[llvm] [AMDGPU] Make globally-addressable-scratch opt-in (PR #189555)
Pierre van Houtryve via llvm-commits
llvm-commits at lists.llvm.org
Thu Jun 4 00:43:32 PDT 2026
https://github.com/Pierre-vh updated https://github.com/llvm/llvm-project/pull/189555
>From a0fe215631b91e2295df6b339544fe4de9da1491 Mon Sep 17 00:00:00 2001
From: pvanhout <pierre.vanhoutryve at amd.com>
Date: Mon, 30 Mar 2026 11:55:36 +0200
Subject: [PATCH 1/3] [AMDGPU] Make globally-addressable-scratch opt-in
This feature is meant to be opt-in for more advanced users, not default-enabled.
It may reduce performance otherwise as we can't assume private AS is thread-local
when it is enabled.
- Add `HasGloballyAddressableScratchSupport` feature to check if a target's scratch
addressing is changed due to support for globally addressable scratch.
- Use `EnableGloballyAddressableScratch` to check whether the user opted into
globally addressable scratch. This affects whether to lower scratch atomics as flat,
and in the future will affect whether NV=1 can be set on scratch accesses.
---
llvm/docs/AMDGPUUsage.rst | 144 +-
llvm/lib/Target/AMDGPU/AMDGPU.td | 17 +-
.../lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp | 8 +-
.../AMDGPU/AMDGPUTargetTransformInfo.cpp | 4 +-
.../AMDGPU/AsmParser/AMDGPUAsmParser.cpp | 6 +-
llvm/lib/Target/AMDGPU/GCNSubtarget.h | 5 +
llvm/lib/Target/AMDGPU/SIISelLowering.cpp | 14 +-
llvm/lib/Target/AMDGPU/SIInstrInfo.cpp | 2 +-
llvm/lib/Target/AMDGPU/SIMemoryLegalizer.cpp | 2 +-
.../AMDGPU/memory-legalizer-private-agent.ll | 8958 +++++++++--------
.../memory-legalizer-private-cluster.ll | 8516 ++++++++--------
.../memory-legalizer-private-lastuse.ll | 6 +-
.../memory-legalizer-private-nontemporal.ll | 6 +-
.../memory-legalizer-private-singlethread.ll | 8141 ++++++++-------
.../AMDGPU/memory-legalizer-private-system.ll | 8522 ++++++++--------
.../memory-legalizer-private-volatile.ll | 6 +-
.../memory-legalizer-private-wavefront.ll | 8141 ++++++++-------
.../memory-legalizer-private-workgroup.ll | 8445 +++++++++-------
.../AMDGPU/expand-atomic-private-gas.ll | 221 +-
19 files changed, 27657 insertions(+), 23507 deletions(-)
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index 2d2419a5479d7..90a792922f59b 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -559,27 +559,21 @@ Every processor supports every OS ABI (see :ref:`amdgpu-os`) with the following
work-item
IDs
- ``gfx1250`` ``amdgcn`` APU - Architected *TBA*
- flat
- scratch .. TODO::
+ ``gfx1250`` ``amdgcn`` APU - globally- - Architected *TBA*
+ addressable- flat
+ scratch scratch .. TODO::
- Packed
work-item Add product
IDs names.
- - Globally
- Accessible
- Scratch
- Workgroup
Clusters
- ``gfx1251`` ``amdgcn`` APU - Architected *TBA*
- flat
- scratch .. TODO::
+ ``gfx1251`` ``amdgcn`` APU - globally- - Architected *TBA*
+ addressable- flat
+ scratch scratch .. TODO::
- Packed
work-item Add product
IDs names.
- - Globally
- Accessible
- Scratch
- Workgroup
Clusters
@@ -771,64 +765,70 @@ For example:
.. table:: AMDGPU Target Features
:name: amdgpu-target-features-table
- =============== ============================ ==================================================
- Target Feature Clang Option to Control Description
+ ============================= ============================ ==================================================
+ Target Feature Clang Option to Control Description
Name
- =============== ============================ ==================================================
- cumode - ``-m[no-]cumode`` Control the wavefront execution mode used
- when generating code for kernels. When disabled
- native WGP wavefront execution mode is used,
- when enabled CU wavefront execution mode is used
- (see :ref:`amdgpu-amdhsa-memory-model`).
-
- sramecc - ``-mcpu`` If specified, generate code that can only be
- - ``--offload-arch`` loaded and executed in a process that has a
- matching setting for SRAMECC.
-
- If not specified for code object V2 to V3, generate
- code that can be loaded and executed in a process
- with SRAMECC enabled.
-
- If not specified for code object V4 or above, generate
- code that can be loaded and executed in a process
- with either setting of SRAMECC.
-
- tgsplit ``-m[no-]tgsplit`` Enable/disable generating code that assumes
- work-groups are launched in threadgroup split mode.
- When enabled the waves of a work-group may be
- launched in different CUs.
-
- wavefrontsize64 - ``-m[no-]wavefrontsize64`` Control the wavefront size used when
- generating code for kernels. When disabled
- native wavefront size 32 is used, when enabled
- wavefront size 64 is used.
-
- xnack - ``-mcpu`` If specified, generate code that can only be
- - ``--offload-arch`` loaded and executed in a process that has a
- matching setting for XNACK replay.
-
- If not specified for code object V2 to V3, generate
- code that can be loaded and executed in a process
- with XNACK replay enabled.
-
- If not specified for code object V4 or above, generate
- code that can be loaded and executed in a process
- with either setting of XNACK replay.
-
- XNACK replay can be used for demand paging and
- page migration. If enabled in the device, then if
- a page fault occurs the code may execute
- incorrectly unless generated with XNACK replay
- enabled, or generated for code object V4 or above without
- specifying XNACK replay. Executing code that was
- generated with XNACK replay enabled, or generated
- for code object V4 or above without specifying XNACK replay,
- on a device that does not have XNACK replay
- enabled will execute correctly but may be less
- performant than code generated for XNACK replay
- disabled.
-
- =============== ============================ ==================================================
+ ============================= ============================ ==================================================
+ cumode - ``-m[no-]cumode`` Control the wavefront execution mode used
+ when generating code for kernels. When disabled
+ native WGP wavefront execution mode is used,
+ when enabled CU wavefront execution mode is used
+ (see :ref:`amdgpu-amdhsa-memory-model`).
+
+ sramecc - ``-mcpu`` If specified, generate code that can only be
+ - ``--offload-arch`` loaded and executed in a process that has a
+ matching setting for SRAMECC.
+
+ If not specified for code object V2 to V3, generate
+ code that can be loaded and executed in a process
+ with SRAMECC enabled.
+
+ If not specified for code object V4 or above, generate
+ code that can be loaded and executed in a process
+ with either setting of SRAMECC.
+
+ tgsplit ``-m[no-]tgsplit`` Enable/disable generating code that assumes
+ work-groups are launched in threadgroup split mode.
+ When enabled the waves of a work-group may be
+ launched in different CUs.
+
+ wavefrontsize64 - ``-m[no-]wavefrontsize64`` Control the wavefront size used when
+ generating code for kernels. When disabled
+ native wavefront size 32 is used, when enabled
+ wavefront size 64 is used.
+
+ xnack - ``-mcpu`` If specified, generate code that can only be
+ - ``--offload-arch`` loaded and executed in a process that has a
+ matching setting for XNACK replay.
+
+ If not specified for code object V2 to V3, generate
+ code that can be loaded and executed in a process
+ with XNACK replay enabled.
+
+ If not specified for code object V4 or above, generate
+ code that can be loaded and executed in a process
+ with either setting of XNACK replay.
+
+ XNACK replay can be used for demand paging and
+ page migration. If enabled in the device, then if
+ a page fault occurs the code may execute
+ incorrectly unless generated with XNACK replay
+ enabled, or generated for code object V4 or above without
+ specifying XNACK replay. Executing code that was
+ generated with XNACK replay enabled, or generated
+ for code object V4 or above without specifying XNACK replay,
+ on a device that does not have XNACK replay
+ enabled will execute correctly but may be less
+ performant than code generated for XNACK replay
+ disabled.
+
+ globally-addressable-scratch - ``--offload-arch`` When enabled, scratch (private) memory can be shared
+ between threads without triggering undefined behavior.
+ Disabled by default as this may incur a performance penalty
+ because the compiler can no longer assume private memory is
+ thread-local when this is enabled.
+
+ ============================= ============================ ==================================================
.. _amdgpu-module-flags:
@@ -1077,7 +1077,7 @@ supported for the ``amdgcn`` target.
access is not supported except by flat and scratch instructions in
GFX9-GFX11.
- On targets without "Globally Accessible Scratch" (introduced in GFX125x), code that
+ On targets without ``globally-addressable-scratch``, or if the feature is disabled, code that
manipulates the stack values in other lanes of a wavefront, such as by
``addrspacecast``-ing stack pointers to generic ones and taking offsets that reach other
lanes or by explicitly constructing the scratch buffer descriptor, triggers undefined
@@ -17424,8 +17424,8 @@ For GFX125x:
This section is currently incomplete as work on the compiler is still ongoing.
The following is a non-exhaustive list of unimplemented/undocumented features:
- non-volatile bit code sequences, globally accessing scratch atomics,
- multicast loads, barriers (including split barriers) and cooperative atomics.
+ non-volatile bit code sequences, multicast loads, barriers (including split barriers)
+ and cooperative atomics.
Scalar operations memory model needs more elaboration as well.
* Vector memory operations are performed as wavefront wide operations, with the
diff --git a/llvm/lib/Target/AMDGPU/AMDGPU.td b/llvm/lib/Target/AMDGPU/AMDGPU.td
index 2d014be12cad7..d662b30d63cec 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPU.td
+++ b/llvm/lib/Target/AMDGPU/AMDGPU.td
@@ -1327,9 +1327,16 @@ defm XF32Insts : AMDGPUSubtargetFeature<"xf32-insts",
"v_mfma_f32_16x16x8_xf32 and v_mfma_f32_32x32x4_xf32"
>;
-defm GloballyAddressableScratch : AMDGPUSubtargetFeature<"globally-addressable-scratch",
- "FLAT instructions can access scratch memory for any thread in any wave",
- /*GenPredicate=*/0
+def FeatureGloballyAddressableScratchSupport : SubtargetFeature<"globally-addressable-scratch-support",
+ "HasGloballyAddressableScratchSupport",
+ "true",
+ "Hardware supports globally-addressable-scratch"
+>;
+
+def FeatureGloballyAddressableScratch : SubtargetFeature<"globally-addressable-scratch",
+ "EnableGloballyAddressableScratch",
+ "true",
+ "FLAT instructions can access scratch memory from any thread in any wave"
>;
// Enable the use of SCRATCH_STORE/LOAD_BLOCK instructions for saving and
@@ -2161,7 +2168,7 @@ def FeatureISAVersion12_50_Common : FeatureSet<
FeatureFlatBufferGlobalAtomicFaddF64Inst,
FeatureMemoryAtomicFAddF32DenormalSupport,
FeatureEmulatedSystemScopeAtomics,
- FeatureGloballyAddressableScratch,
+ FeatureGloballyAddressableScratchSupport,
FeatureKernargPreload,
FeatureVmemPrefInsts,
FeatureLshlAddU64Inst,
@@ -2278,7 +2285,7 @@ def FeatureISAVersion13 : FeatureSet<
FeatureAtomicFMinFMaxF64GlobalInsts,
FeatureAtomicFMinFMaxF64FlatInsts,
FeatureFmaMixBF16Insts,
- FeatureGloballyAddressableScratch,
+ FeatureGloballyAddressableScratchSupport,
FeatureCvtPkF16F32Inst,
FeatureF16BF16ToFP6BF6ConversionScaleInsts,
FeatureIEEEMinimumMaximumInsts,
diff --git a/llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp b/llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp
index 678b444dac077..40f07a90b476a 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp
@@ -2397,7 +2397,7 @@ Register AMDGPULegalizerInfo::getSegmentAperture(
? AMDGPU::SRC_SHARED_BASE
: AMDGPU::SRC_PRIVATE_BASE;
assert((ApertureRegNo != AMDGPU::SRC_PRIVATE_BASE ||
- !ST.hasGloballyAddressableScratch()) &&
+ !ST.hasGloballyAddressableScratchSupport()) &&
"Cannot use src_private_base with globally addressable scratch!");
Register Dst = MRI.createGenericVirtualRegister(S64);
MRI.setRegClass(Dst, &AMDGPU::SReg_64RegClass);
@@ -2522,7 +2522,7 @@ bool AMDGPULegalizerInfo::legalizeAddrSpaceCast(
DestAS == AMDGPUAS::PRIVATE_ADDRESS)) {
auto castFlatToLocalOrPrivate = [&](const DstOp &Dst) -> Register {
if (DestAS == AMDGPUAS::PRIVATE_ADDRESS &&
- ST.hasGloballyAddressableScratch()) {
+ ST.hasGloballyAddressableScratchSupport()) {
// flat -> private with globally addressable scratch: subtract
// src_flat_scratch_base_lo.
const LLT S32 = LLT::scalar(32);
@@ -2573,7 +2573,7 @@ bool AMDGPULegalizerInfo::legalizeAddrSpaceCast(
Register SrcAsInt = B.buildPtrToInt(S32, Src).getReg(0);
if (SrcAS == AMDGPUAS::PRIVATE_ADDRESS &&
- ST.hasGloballyAddressableScratch()) {
+ ST.hasGloballyAddressableScratchSupport()) {
// For wave32: Addr = (TID[4:0] << 52) + FLAT_SCRATCH_BASE + privateAddr
// For wave64: Addr = (TID[5:0] << 51) + FLAT_SCRATCH_BASE + privateAddr
Register AllOnes = B.buildConstant(S32, -1).getReg(0);
@@ -6438,7 +6438,7 @@ bool AMDGPULegalizerInfo::legalizeIsAddrSpace(MachineInstr &MI,
Register Hi32 = Unmerge.getReg(1);
if (AddrSpace == AMDGPUAS::PRIVATE_ADDRESS &&
- ST.hasGloballyAddressableScratch()) {
+ ST.hasGloballyAddressableScratchSupport()) {
Register FlatScratchBaseHi =
B.buildInstr(AMDGPU::S_MOV_B32, {S32},
{Register(AMDGPU::SRC_FLAT_SCRATCH_BASE_HI)})
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.cpp b/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.cpp
index 7631bb2dc6828..bf5e53e13653b 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.cpp
@@ -1148,7 +1148,7 @@ bool GCNTTIImpl::isSourceOfDivergence(const Value *V) const {
unsigned DstAS = Intrinsic->getType()->getPointerAddressSpace();
return SrcAS == AMDGPUAS::PRIVATE_ADDRESS &&
DstAS == AMDGPUAS::FLAT_ADDRESS &&
- ST->hasGloballyAddressableScratch();
+ ST->hasGloballyAddressableScratchSupport();
}
case Intrinsic::amdgcn_workitem_id_y:
case Intrinsic::amdgcn_workitem_id_z: {
@@ -1181,7 +1181,7 @@ bool GCNTTIImpl::isSourceOfDivergence(const Value *V) const {
if (auto *CastI = dyn_cast<AddrSpaceCastInst>(V)) {
return CastI->getSrcAddressSpace() == AMDGPUAS::PRIVATE_ADDRESS &&
CastI->getDestAddressSpace() == AMDGPUAS::FLAT_ADDRESS &&
- ST->hasGloballyAddressableScratch();
+ ST->hasGloballyAddressableScratchSupport();
}
return false;
diff --git a/llvm/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp b/llvm/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp
index 2a9a31e732a6e..57f0ffa621774 100644
--- a/llvm/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp
+++ b/llvm/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp
@@ -1627,8 +1627,8 @@ class AMDGPUAsmParser : public MCTargetAsmParser {
return getFeatureBits()[AMDGPU::FeaturePartialNSAEncoding];
}
- bool hasGloballyAddressableScratch() const {
- return getFeatureBits()[AMDGPU::FeatureGloballyAddressableScratch];
+ bool hasGloballyAddressableScratchSupport() const {
+ return getFeatureBits()[AMDGPU::FeatureGloballyAddressableScratchSupport];
}
unsigned getNSAMaxSize(bool HasSampler = false) const {
@@ -6988,7 +6988,7 @@ bool AMDGPUAsmParser::subtargetHasRegister(const MCRegisterInfo &MRI,
return isGFX9Plus();
case SRC_FLAT_SCRATCH_BASE_LO:
case SRC_FLAT_SCRATCH_BASE_HI:
- return hasGloballyAddressableScratch();
+ return hasGloballyAddressableScratchSupport();
case SRC_POPS_EXITING_WAVE_ID:
return isGFX9Plus() && !isGFX11Plus();
case TBA:
diff --git a/llvm/lib/Target/AMDGPU/GCNSubtarget.h b/llvm/lib/Target/AMDGPU/GCNSubtarget.h
index 082c5c33d6067..b4b69469ad650 100644
--- a/llvm/lib/Target/AMDGPU/GCNSubtarget.h
+++ b/llvm/lib/Target/AMDGPU/GCNSubtarget.h
@@ -1036,6 +1036,11 @@ class GCNSubtarget final : public AMDGPUGenSubtargetInfo,
bool requiresWaitOnWorkgroupReleaseFence() const {
return getGeneration() >= GFX10 || isTgSplitEnabled();
}
+
+ bool isGloballyAddressableScratchEnabled() const {
+ return HasGloballyAddressableScratchSupport &&
+ EnableGloballyAddressableScratch;
+ }
};
class GCNUserSGPRUsageInfo {
diff --git a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
index b393bb904e75b..b0b8a205d63b6 100644
--- a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
+++ b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
@@ -2328,7 +2328,7 @@ bool SITargetLowering::isFreeAddrSpaceCast(unsigned SrcAS,
unsigned DestAS) const {
if (SrcAS == AMDGPUAS::FLAT_ADDRESS) {
if (DestAS == AMDGPUAS::PRIVATE_ADDRESS &&
- Subtarget->hasGloballyAddressableScratch()) {
+ Subtarget->hasGloballyAddressableScratchSupport()) {
// Flat -> private requires subtracting src_flat_scratch_base_lo.
return false;
}
@@ -9164,7 +9164,7 @@ SDValue SITargetLowering::getSegmentAperture(unsigned AS, const SDLoc &DL,
? AMDGPU::SRC_SHARED_BASE
: AMDGPU::SRC_PRIVATE_BASE;
assert((ApertureRegNo != AMDGPU::SRC_PRIVATE_BASE ||
- !Subtarget->hasGloballyAddressableScratch()) &&
+ !Subtarget->hasGloballyAddressableScratchSupport()) &&
"Cannot use src_private_base with globally addressable scratch!");
// Note: this feature (register) is broken. When used as a 32-bit operand,
// it returns a wrong value (all zeroes?). The real value is in the upper 32
@@ -9266,7 +9266,7 @@ SDValue SITargetLowering::lowerADDRSPACECAST(SDValue Op,
SDValue Ptr = DAG.getNode(ISD::TRUNCATE, SL, MVT::i32, Src);
if (DestAS == AMDGPUAS::PRIVATE_ADDRESS &&
- Subtarget->hasGloballyAddressableScratch()) {
+ Subtarget->hasGloballyAddressableScratchSupport()) {
// flat -> private with globally addressable scratch: subtract
// src_flat_scratch_base_lo.
SDValue FlatScratchBaseLo(
@@ -9295,7 +9295,7 @@ SDValue SITargetLowering::lowerADDRSPACECAST(SDValue Op,
SrcAS == AMDGPUAS::PRIVATE_ADDRESS) {
SDValue CvtPtr;
if (SrcAS == AMDGPUAS::PRIVATE_ADDRESS &&
- Subtarget->hasGloballyAddressableScratch()) {
+ Subtarget->hasGloballyAddressableScratchSupport()) {
// For wave32: Addr = (TID[4:0] << 52) + FLAT_SCRATCH_BASE + privateAddr
// For wave64: Addr = (TID[5:0] << 51) + FLAT_SCRATCH_BASE + privateAddr
SDValue AllOnes = DAG.getSignedTargetConstant(-1, SL, MVT::i32);
@@ -11107,7 +11107,7 @@ SDValue SITargetLowering::LowerINTRINSIC_WO_CHAIN(SDValue Op,
? AMDGPUAS::LOCAL_ADDRESS
: AMDGPUAS::PRIVATE_ADDRESS;
if (AS == AMDGPUAS::PRIVATE_ADDRESS &&
- Subtarget->hasGloballyAddressableScratch()) {
+ Subtarget->hasGloballyAddressableScratchSupport()) {
SDValue FlatScratchBaseHi(
DAG.getMachineNode(
AMDGPU::S_MOV_B32, DL, MVT::i32,
@@ -20156,8 +20156,8 @@ static bool flatInstrMayAccessPrivate(const Instruction *I) {
static TargetLowering::AtomicExpansionKind
getPrivateAtomicExpansionKind(const GCNSubtarget &STI) {
- // For GAS, lower to flat atomic.
- return STI.hasGloballyAddressableScratch()
+ // If GAS is enabled, scratch atomics need to be expanded to FLAT.
+ return STI.isGloballyAddressableScratchEnabled()
? TargetLowering::AtomicExpansionKind::CustomExpand
: TargetLowering::AtomicExpansionKind::NotAtomic;
}
diff --git a/llvm/lib/Target/AMDGPU/SIInstrInfo.cpp b/llvm/lib/Target/AMDGPU/SIInstrInfo.cpp
index a544f1380e53d..7dd3bf7587370 100644
--- a/llvm/lib/Target/AMDGPU/SIInstrInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/SIInstrInfo.cpp
@@ -10932,7 +10932,7 @@ SIInstrInfo::getGenericValueUniformity(const MachineInstr &MI) const {
unsigned SrcAS = SrcTy.getAddressSpace();
return SrcAS == AMDGPUAS::PRIVATE_ADDRESS &&
DstAS == AMDGPUAS::FLAT_ADDRESS &&
- ST.hasGloballyAddressableScratch()
+ ST.hasGloballyAddressableScratchSupport()
? ValueUniformity::NeverUniform
: ValueUniformity::Default;
};
diff --git a/llvm/lib/Target/AMDGPU/SIMemoryLegalizer.cpp b/llvm/lib/Target/AMDGPU/SIMemoryLegalizer.cpp
index 3149ebc2f0bf4..bfa90e86fe4f0 100644
--- a/llvm/lib/Target/AMDGPU/SIMemoryLegalizer.cpp
+++ b/llvm/lib/Target/AMDGPU/SIMemoryLegalizer.cpp
@@ -1001,7 +1001,7 @@ bool SICacheControl::enableCPolBits(const MachineBasicBlock::iterator MI,
}
bool SICacheControl::canAffectGlobalAddrSpace(SIAtomicAddrSpace AS) const {
- assert((!ST.hasGloballyAddressableScratch() ||
+ assert((!ST.isGloballyAddressableScratchEnabled() ||
(AS & SIAtomicAddrSpace::GLOBAL) != SIAtomicAddrSpace::NONE ||
(AS & SIAtomicAddrSpace::SCRATCH) == SIAtomicAddrSpace::NONE) &&
"scratch instructions should already be replaced by flat "
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-agent.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-agent.ll
index 1b511c6fca4a7..e53ea71e5cc0b 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-agent.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-agent.ll
@@ -12,7 +12,8 @@
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1100 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX11-CU %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 < %s | FileCheck --check-prefixes=GFX12-WGP %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX12-CU %s
-; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 < %s | FileCheck --check-prefixes=GFX1250 %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 < %s | FileCheck --check-prefixes=GFX1250,GFX1250-NOGAS %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 -mattr=+globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
define amdgpu_kernel void @private_agent_unordered_load(
; GFX6-LABEL: private_agent_unordered_load:
@@ -225,41 +226,47 @@ define amdgpu_kernel void @private_agent_unordered_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_unordered_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_unordered_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_unordered_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("agent") unordered, align 4
@@ -478,41 +485,47 @@ define amdgpu_kernel void @private_agent_monotonic_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_monotonic_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_monotonic_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_monotonic_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("agent") monotonic, align 4
@@ -731,43 +744,49 @@ define amdgpu_kernel void @private_agent_acquire_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acquire_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acquire_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acquire_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("agent") acquire, align 4
@@ -986,45 +1005,51 @@ define amdgpu_kernel void @private_agent_seq_cst_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_seq_cst_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_seq_cst_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_seq_cst_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("agent") seq_cst, align 4
@@ -1227,40 +1252,46 @@ define amdgpu_kernel void @private_agent_unordered_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_unordered_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_unordered_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_unordered_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("agent") unordered, align 4
@@ -1462,40 +1493,46 @@ define amdgpu_kernel void @private_agent_monotonic_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_monotonic_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_monotonic_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_monotonic_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("agent") monotonic, align 4
@@ -1697,45 +1734,51 @@ define amdgpu_kernel void @private_agent_release_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_release_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_release_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_release_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("agent") release, align 4
@@ -1937,45 +1980,51 @@ define amdgpu_kernel void @private_agent_seq_cst_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_seq_cst_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_seq_cst_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_seq_cst_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("agent") seq_cst, align 4
@@ -2210,41 +2259,46 @@ define amdgpu_kernel void @private_agent_monotonic_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_monotonic_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_monotonic_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_monotonic_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent") monotonic
@@ -2479,44 +2533,49 @@ define amdgpu_kernel void @private_agent_acquire_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acquire_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acquire_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acquire_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent") acquire
@@ -2751,46 +2810,51 @@ define amdgpu_kernel void @private_agent_release_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_release_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_release_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_release_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent") release
@@ -3025,49 +3089,54 @@ define amdgpu_kernel void @private_agent_acq_rel_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acq_rel_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acq_rel_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acq_rel_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent") acq_rel
@@ -3302,49 +3371,54 @@ define amdgpu_kernel void @private_agent_seq_cst_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_seq_cst_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_seq_cst_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_seq_cst_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent") seq_cst
@@ -3600,44 +3674,53 @@ define amdgpu_kernel void @private_agent_acquire_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acquire_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acquire_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acquire_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent") acquire
@@ -3894,49 +3977,58 @@ define amdgpu_kernel void @private_agent_acq_rel_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acq_rel_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acq_rel_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acq_rel_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent") acq_rel
@@ -4193,49 +4285,58 @@ define amdgpu_kernel void @private_agent_seq_cst_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_seq_cst_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_seq_cst_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_seq_cst_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent") seq_cst
@@ -4548,49 +4649,56 @@ define amdgpu_kernel void @private_agent_monotonic_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_monotonic_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_monotonic_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_monotonic_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -4903,52 +5011,59 @@ define amdgpu_kernel void @private_agent_acquire_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acquire_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acquire_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acquire_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5261,54 +5376,61 @@ define amdgpu_kernel void @private_agent_release_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_release_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_release_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_release_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5621,57 +5743,64 @@ define amdgpu_kernel void @private_agent_acq_rel_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acq_rel_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acq_rel_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acq_rel_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5984,57 +6113,64 @@ define amdgpu_kernel void @private_agent_seq_cst_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_seq_cst_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_seq_cst_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_seq_cst_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -6347,52 +6483,59 @@ define amdgpu_kernel void @private_agent_monotonic_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_monotonic_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_monotonic_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_monotonic_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -6705,52 +6848,59 @@ define amdgpu_kernel void @private_agent_acquire_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acquire_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acquire_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acquire_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7063,57 +7213,64 @@ define amdgpu_kernel void @private_agent_release_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_release_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_release_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_release_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7426,57 +7583,64 @@ define amdgpu_kernel void @private_agent_acq_rel_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acq_rel_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acq_rel_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acq_rel_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7789,57 +7953,64 @@ define amdgpu_kernel void @private_agent_seq_cst_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_seq_cst_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_seq_cst_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_seq_cst_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8152,57 +8323,64 @@ define amdgpu_kernel void @private_agent_monotonic_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_monotonic_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_monotonic_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_monotonic_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8515,57 +8693,64 @@ define amdgpu_kernel void @private_agent_acquire_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acquire_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acquire_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acquire_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8878,57 +9063,64 @@ define amdgpu_kernel void @private_agent_release_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_release_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_release_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_release_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -9241,57 +9433,64 @@ define amdgpu_kernel void @private_agent_acq_rel_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acq_rel_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acq_rel_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acq_rel_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -9604,57 +9803,64 @@ define amdgpu_kernel void @private_agent_seq_cst_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_seq_cst_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_seq_cst_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_seq_cst_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -9995,50 +10201,59 @@ define amdgpu_kernel void @private_agent_monotonic_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_monotonic_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -10381,52 +10596,61 @@ define amdgpu_kernel void @private_agent_acquire_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acquire_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acquire_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acquire_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -10769,55 +10993,64 @@ define amdgpu_kernel void @private_agent_release_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_release_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_release_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_release_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11160,57 +11393,66 @@ define amdgpu_kernel void @private_agent_acq_rel_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acq_rel_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11553,57 +11795,66 @@ define amdgpu_kernel void @private_agent_seq_cst_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_seq_cst_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11946,52 +12197,61 @@ define amdgpu_kernel void @private_agent_monotonic_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_monotonic_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_monotonic_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_monotonic_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -12334,52 +12594,61 @@ define amdgpu_kernel void @private_agent_acquire_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acquire_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acquire_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acquire_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -12722,57 +12991,66 @@ define amdgpu_kernel void @private_agent_release_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_release_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_release_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_release_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -13115,57 +13393,66 @@ define amdgpu_kernel void @private_agent_acq_rel_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acq_rel_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -13508,57 +13795,66 @@ define amdgpu_kernel void @private_agent_seq_cst_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_seq_cst_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -13901,57 +14197,66 @@ define amdgpu_kernel void @private_agent_monotonic_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_monotonic_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -14294,57 +14599,66 @@ define amdgpu_kernel void @private_agent_acquire_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acquire_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -14687,57 +15001,66 @@ define amdgpu_kernel void @private_agent_release_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_release_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_release_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_release_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -15080,57 +15403,66 @@ define amdgpu_kernel void @private_agent_acq_rel_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acq_rel_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -15473,57 +15805,66 @@ define amdgpu_kernel void @private_agent_seq_cst_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_seq_cst_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -15744,41 +16085,47 @@ define amdgpu_kernel void @private_agent_one_as_unordered_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_unordered_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_unordered_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_unordered_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("agent-one-as") unordered, align 4
@@ -15997,41 +16344,47 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_monotonic_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_monotonic_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_monotonic_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("agent-one-as") monotonic, align 4
@@ -16250,43 +16603,49 @@ define amdgpu_kernel void @private_agent_one_as_acquire_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acquire_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acquire_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acquire_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("agent-one-as") acquire, align 4
@@ -16505,45 +16864,51 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_seq_cst_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_seq_cst_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_seq_cst_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("agent-one-as") seq_cst, align 4
@@ -16746,40 +17111,46 @@ define amdgpu_kernel void @private_agent_one_as_unordered_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_unordered_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_unordered_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_unordered_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("agent-one-as") unordered, align 4
@@ -16981,40 +17352,46 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_monotonic_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_monotonic_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_monotonic_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("agent-one-as") monotonic, align 4
@@ -17216,45 +17593,51 @@ define amdgpu_kernel void @private_agent_one_as_release_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_release_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_release_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_release_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("agent-one-as") release, align 4
@@ -17456,45 +17839,51 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_seq_cst_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_seq_cst_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_seq_cst_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("agent-one-as") seq_cst, align 4
@@ -17729,41 +18118,46 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_monotonic_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_monotonic_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_monotonic_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent-one-as") monotonic
@@ -17998,44 +18392,49 @@ define amdgpu_kernel void @private_agent_one_as_acquire_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acquire_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acquire_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acquire_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent-one-as") acquire
@@ -18270,46 +18669,51 @@ define amdgpu_kernel void @private_agent_one_as_release_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_release_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_release_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_release_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent-one-as") release
@@ -18544,49 +18948,54 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acq_rel_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acq_rel_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acq_rel_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent-one-as") acq_rel
@@ -18821,49 +19230,54 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_seq_cst_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_seq_cst_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_seq_cst_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent-one-as") seq_cst
@@ -19119,44 +19533,53 @@ define amdgpu_kernel void @private_agent_one_as_acquire_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acquire_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acquire_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acquire_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent-one-as") acquire
@@ -19413,49 +19836,58 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acq_rel_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acq_rel_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acq_rel_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent-one-as") acq_rel
@@ -19712,49 +20144,58 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_seq_cst_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_seq_cst_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_seq_cst_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent-one-as") seq_cst
@@ -20067,49 +20508,56 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_monotonic_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_monotonic_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_monotonic_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -20422,52 +20870,59 @@ define amdgpu_kernel void @private_agent_one_as_acquire_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acquire_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acquire_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acquire_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -20780,54 +21235,61 @@ define amdgpu_kernel void @private_agent_one_as_release_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_release_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_release_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_release_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -21140,57 +21602,64 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acq_rel_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acq_rel_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acq_rel_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -21503,57 +21972,64 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_seq_cst_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_seq_cst_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_seq_cst_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -21866,52 +22342,59 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_monotonic_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_monotonic_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_monotonic_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22224,52 +22707,59 @@ define amdgpu_kernel void @private_agent_one_as_acquire_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acquire_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acquire_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acquire_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22582,57 +23072,64 @@ define amdgpu_kernel void @private_agent_one_as_release_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_release_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_release_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_release_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22945,57 +23442,64 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acq_rel_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acq_rel_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acq_rel_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -23308,57 +23812,64 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_seq_cst_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_seq_cst_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_seq_cst_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -23671,57 +24182,64 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_monotonic_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_monotonic_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_monotonic_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24034,57 +24552,64 @@ define amdgpu_kernel void @private_agent_one_as_acquire_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acquire_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acquire_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acquire_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24397,57 +24922,64 @@ define amdgpu_kernel void @private_agent_one_as_release_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_release_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_release_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_release_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24760,57 +25292,64 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acq_rel_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acq_rel_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acq_rel_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -25123,57 +25662,64 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_seq_cst_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_seq_cst_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_seq_cst_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -25514,50 +26060,59 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_monotonic_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -25900,52 +26455,61 @@ define amdgpu_kernel void @private_agent_one_as_acquire_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acquire_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acquire_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acquire_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -26288,57 +26852,66 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acq_rel_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -26681,57 +27254,66 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_seq_cst_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -27074,52 +27656,61 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_monotonic_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_monotonic_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_monotonic_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -27462,52 +28053,61 @@ define amdgpu_kernel void @private_agent_one_as_acquire_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acquire_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acquire_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acquire_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -27850,57 +28450,66 @@ define amdgpu_kernel void @private_agent_one_as_release_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_release_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_release_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_release_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -28243,57 +28852,66 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acq_rel_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -28636,57 +29254,66 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_seq_cst_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -29029,57 +29656,66 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_monotonic_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -29422,57 +30058,66 @@ define amdgpu_kernel void @private_agent_one_as_acquire_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acquire_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -29815,57 +30460,66 @@ define amdgpu_kernel void @private_agent_one_as_release_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_release_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_release_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_release_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30208,57 +30862,66 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acq_rel_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30601,57 +31264,66 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_seq_cst_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30660,3 +31332,5 @@ entry:
store i32 %val0, ptr addrspace(5) %out, align 4
ret void
}
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GFX1250: {{.*}}
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-cluster.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-cluster.ll
index c183a86421726..13e5f3a84e3a2 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-cluster.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-cluster.ll
@@ -12,7 +12,8 @@
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1100 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX11-CU %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 < %s | FileCheck --check-prefixes=GFX12-WGP %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX12-CU %s
-; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 < %s | FileCheck --check-prefixes=GFX1250 %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 < %s | FileCheck --check-prefixes=GFX1250,GFX1250-NOGAS %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 -mattr=+globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
define amdgpu_kernel void @private_cluster_unordered_load(
; GFX6-LABEL: private_cluster_unordered_load:
@@ -225,41 +226,47 @@ define amdgpu_kernel void @private_cluster_unordered_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_unordered_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_unordered_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_unordered_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("cluster") unordered, align 4
@@ -478,41 +485,47 @@ define amdgpu_kernel void @private_cluster_monotonic_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_monotonic_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_monotonic_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_monotonic_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("cluster") monotonic, align 4
@@ -731,42 +744,48 @@ define amdgpu_kernel void @private_cluster_acquire_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acquire_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acquire_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acquire_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("cluster") acquire, align 4
@@ -985,44 +1004,50 @@ define amdgpu_kernel void @private_cluster_seq_cst_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_seq_cst_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_seq_cst_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_seq_cst_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("cluster") seq_cst, align 4
@@ -1225,40 +1250,46 @@ define amdgpu_kernel void @private_cluster_unordered_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_unordered_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_unordered_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_unordered_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("cluster") unordered, align 4
@@ -1460,40 +1491,46 @@ define amdgpu_kernel void @private_cluster_monotonic_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_monotonic_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_monotonic_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_monotonic_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("cluster") monotonic, align 4
@@ -1695,42 +1732,48 @@ define amdgpu_kernel void @private_cluster_release_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_release_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_release_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_release_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("cluster") release, align 4
@@ -1932,42 +1975,48 @@ define amdgpu_kernel void @private_cluster_seq_cst_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_seq_cst_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_seq_cst_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_seq_cst_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("cluster") seq_cst, align 4
@@ -2202,41 +2251,46 @@ define amdgpu_kernel void @private_cluster_monotonic_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_monotonic_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_monotonic_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_monotonic_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster") monotonic
@@ -2471,43 +2525,48 @@ define amdgpu_kernel void @private_cluster_acquire_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acquire_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acquire_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acquire_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster") acquire
@@ -2742,43 +2801,48 @@ define amdgpu_kernel void @private_cluster_release_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_release_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_release_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_release_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster") release
@@ -3013,45 +3077,50 @@ define amdgpu_kernel void @private_cluster_acq_rel_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acq_rel_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acq_rel_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acq_rel_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster") acq_rel
@@ -3286,45 +3355,50 @@ define amdgpu_kernel void @private_cluster_seq_cst_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_seq_cst_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_seq_cst_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_seq_cst_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster") seq_cst
@@ -3580,43 +3654,52 @@ define amdgpu_kernel void @private_cluster_acquire_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acquire_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acquire_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acquire_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster") acquire
@@ -3873,45 +3956,54 @@ define amdgpu_kernel void @private_cluster_acq_rel_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acq_rel_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acq_rel_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acq_rel_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster") acq_rel
@@ -4168,45 +4260,54 @@ define amdgpu_kernel void @private_cluster_seq_cst_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_seq_cst_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_seq_cst_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_seq_cst_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster") seq_cst
@@ -4519,49 +4620,56 @@ define amdgpu_kernel void @private_cluster_monotonic_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_monotonic_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_monotonic_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_monotonic_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -4874,51 +4982,58 @@ define amdgpu_kernel void @private_cluster_acquire_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acquire_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acquire_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acquire_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5231,51 +5346,58 @@ define amdgpu_kernel void @private_cluster_release_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_release_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_release_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_release_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5588,53 +5710,60 @@ define amdgpu_kernel void @private_cluster_acq_rel_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acq_rel_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acq_rel_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acq_rel_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5947,53 +6076,60 @@ define amdgpu_kernel void @private_cluster_seq_cst_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_seq_cst_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_seq_cst_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_seq_cst_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -6306,51 +6442,58 @@ define amdgpu_kernel void @private_cluster_monotonic_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_monotonic_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_monotonic_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_monotonic_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -6663,51 +6806,58 @@ define amdgpu_kernel void @private_cluster_acquire_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acquire_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acquire_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acquire_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7020,53 +7170,60 @@ define amdgpu_kernel void @private_cluster_release_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_release_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_release_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_release_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7379,53 +7536,60 @@ define amdgpu_kernel void @private_cluster_acq_rel_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acq_rel_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acq_rel_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acq_rel_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7738,53 +7902,60 @@ define amdgpu_kernel void @private_cluster_seq_cst_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_seq_cst_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_seq_cst_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_seq_cst_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8097,53 +8268,60 @@ define amdgpu_kernel void @private_cluster_monotonic_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_monotonic_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_monotonic_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_monotonic_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8456,53 +8634,60 @@ define amdgpu_kernel void @private_cluster_acquire_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acquire_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acquire_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acquire_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8815,53 +9000,60 @@ define amdgpu_kernel void @private_cluster_release_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_release_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_release_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_release_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -9174,53 +9366,60 @@ define amdgpu_kernel void @private_cluster_acq_rel_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acq_rel_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acq_rel_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acq_rel_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -9533,53 +9732,60 @@ define amdgpu_kernel void @private_cluster_seq_cst_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_seq_cst_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_seq_cst_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_seq_cst_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -9920,50 +10126,59 @@ define amdgpu_kernel void @private_cluster_monotonic_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_monotonic_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -10306,51 +10521,60 @@ define amdgpu_kernel void @private_cluster_acquire_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acquire_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acquire_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acquire_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -10693,52 +10917,61 @@ define amdgpu_kernel void @private_cluster_release_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_release_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_release_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_release_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11081,53 +11314,62 @@ define amdgpu_kernel void @private_cluster_acq_rel_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acq_rel_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11470,53 +11712,62 @@ define amdgpu_kernel void @private_cluster_seq_cst_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_seq_cst_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11859,51 +12110,60 @@ define amdgpu_kernel void @private_cluster_monotonic_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_monotonic_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_monotonic_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_monotonic_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -12246,51 +12506,60 @@ define amdgpu_kernel void @private_cluster_acquire_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acquire_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acquire_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acquire_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -12633,53 +12902,62 @@ define amdgpu_kernel void @private_cluster_release_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_release_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_release_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_release_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -13022,53 +13300,62 @@ define amdgpu_kernel void @private_cluster_acq_rel_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acq_rel_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -13411,53 +13698,62 @@ define amdgpu_kernel void @private_cluster_seq_cst_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_seq_cst_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -13800,53 +14096,62 @@ define amdgpu_kernel void @private_cluster_monotonic_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_monotonic_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -14189,53 +14494,62 @@ define amdgpu_kernel void @private_cluster_acquire_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acquire_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -14578,53 +14892,62 @@ define amdgpu_kernel void @private_cluster_release_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_release_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_release_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_release_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -14967,53 +15290,62 @@ define amdgpu_kernel void @private_cluster_acq_rel_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acq_rel_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -15356,53 +15688,62 @@ define amdgpu_kernel void @private_cluster_seq_cst_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_seq_cst_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -15623,41 +15964,47 @@ define amdgpu_kernel void @private_cluster_one_as_unordered_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_unordered_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_unordered_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_unordered_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("cluster-one-as") unordered, align 4
@@ -15876,41 +16223,47 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_monotonic_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_monotonic_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_monotonic_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("cluster-one-as") monotonic, align 4
@@ -16129,43 +16482,49 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acquire_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acquire_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acquire_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("cluster-one-as") acquire, align 4
@@ -16384,45 +16743,51 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_seq_cst_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_seq_cst_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_seq_cst_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("cluster-one-as") seq_cst, align 4
@@ -16625,40 +16990,46 @@ define amdgpu_kernel void @private_cluster_one_as_unordered_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_unordered_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_unordered_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_unordered_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("cluster-one-as") unordered, align 4
@@ -16860,40 +17231,46 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_monotonic_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_monotonic_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_monotonic_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("cluster-one-as") monotonic, align 4
@@ -17095,42 +17472,48 @@ define amdgpu_kernel void @private_cluster_one_as_release_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_release_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_release_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_release_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("cluster-one-as") release, align 4
@@ -17332,42 +17715,48 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_seq_cst_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_seq_cst_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_seq_cst_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("cluster-one-as") seq_cst, align 4
@@ -17602,41 +17991,46 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_monotonic_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_monotonic_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_monotonic_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster-one-as") monotonic
@@ -17871,43 +18265,48 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acquire_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acquire_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acquire_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster-one-as") acquire
@@ -18142,43 +18541,48 @@ define amdgpu_kernel void @private_cluster_one_as_release_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_release_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_release_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_release_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster-one-as") release
@@ -18413,45 +18817,50 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acq_rel_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acq_rel_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acq_rel_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster-one-as") acq_rel
@@ -18686,45 +19095,50 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_seq_cst_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_seq_cst_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_seq_cst_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster-one-as") seq_cst
@@ -18980,44 +19394,53 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acquire_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acquire_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acquire_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster-one-as") acquire
@@ -19274,46 +19697,55 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acq_rel_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acq_rel_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acq_rel_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster-one-as") acq_rel
@@ -19570,46 +20002,55 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_seq_cst_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_seq_cst_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_seq_cst_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster-one-as") seq_cst
@@ -19922,49 +20363,56 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_monotonic_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_monotonic_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_monotonic_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -20277,51 +20725,58 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acquire_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acquire_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acquire_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -20634,51 +21089,58 @@ define amdgpu_kernel void @private_cluster_one_as_release_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_release_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_release_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_release_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -20991,53 +21453,60 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acq_rel_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acq_rel_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acq_rel_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -21350,53 +21819,60 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_seq_cst_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_seq_cst_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_seq_cst_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -21709,51 +22185,58 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_monotonic_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_monotonic_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_monotonic_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22066,51 +22549,58 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acquire_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acquire_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acquire_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22423,53 +22913,60 @@ define amdgpu_kernel void @private_cluster_one_as_release_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_release_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_release_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_release_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22782,53 +23279,60 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acq_rel_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acq_rel_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acq_rel_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -23141,53 +23645,60 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_seq_cst_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_seq_cst_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_seq_cst_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -23500,53 +24011,60 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_monotonic_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_monotonic_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_monotonic_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -23859,53 +24377,60 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acquire_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acquire_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acquire_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24218,53 +24743,60 @@ define amdgpu_kernel void @private_cluster_one_as_release_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_release_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_release_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_release_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24577,53 +25109,60 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acq_rel_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acq_rel_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acq_rel_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24936,53 +25475,60 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_seq_cst_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_seq_cst_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_seq_cst_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -25323,50 +25869,59 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_monotonic_ret_cmpxch
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_monotonic_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -25709,52 +26264,61 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acquire_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acquire_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acquire_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -26097,54 +26661,63 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acq_rel_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -26487,54 +27060,63 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_seq_cst_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -26877,52 +27459,61 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_monotonic_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_monotonic_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_monotonic_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -27265,52 +27856,61 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acquire_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acquire_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acquire_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -27653,54 +28253,63 @@ define amdgpu_kernel void @private_cluster_one_as_release_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_release_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_release_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_release_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -28043,54 +28652,63 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acq_rel_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -28433,54 +29051,63 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_seq_cst_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -28823,54 +29450,63 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_monotonic_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -29213,54 +29849,63 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acquire_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -29603,54 +30248,63 @@ define amdgpu_kernel void @private_cluster_one_as_release_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_release_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_release_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_release_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -29993,54 +30647,63 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acq_rel_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30383,54 +31046,63 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_seq_cst_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30439,3 +31111,5 @@ entry:
store i32 %val0, ptr addrspace(5) %out, align 4
ret void
}
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GFX1250: {{.*}}
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-lastuse.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-lastuse.ll
index 07c5c809f0da2..26134c6b14d77 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-lastuse.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-lastuse.ll
@@ -1,7 +1,8 @@
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 < %s | FileCheck --check-prefix=GFX12 %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 -mattr=+cumode < %s | FileCheck --check-prefix=GFX12 %s
-; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 < %s | FileCheck --check-prefixes=GFX1250 %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 < %s | FileCheck --check-prefixes=GFX1250,GFX1250-NOGAS %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 -mattr=+globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
define amdgpu_kernel void @private_last_use_load_0(ptr addrspace(5) %in, ptr addrspace(1) %out) {
; GFX12-LABEL: private_last_use_load_0:
@@ -167,3 +168,6 @@ entry:
!0 = !{i32 1}
declare i32 @llvm.amdgcn.workitem.id.x()
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GFX1250-GAS: {{.*}}
+; GFX1250-NOGAS: {{.*}}
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-nontemporal.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-nontemporal.ll
index 2e0aa7f2fbd0d..183d5d4f79c89 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-nontemporal.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-nontemporal.ll
@@ -12,7 +12,8 @@
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1100 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX11-CU %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 < %s | FileCheck --check-prefixes=GFX12-WGP %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX12-CU %s
-; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 < %s | FileCheck --check-prefixes=GFX1250 %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 < %s | FileCheck --check-prefixes=GFX1250,GFX1250-NOGAS %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 -mattr=+globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
define amdgpu_kernel void @private_nontemporal_load_0(
; GFX6-LABEL: private_nontemporal_load_0:
@@ -1353,3 +1354,6 @@ entry:
!0 = !{i32 1}
declare i32 @llvm.amdgcn.workitem.id.x()
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GFX1250-GAS: {{.*}}
+; GFX1250-NOGAS: {{.*}}
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-singlethread.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-singlethread.ll
index 238ccd2e7baca..f54b1399a11b9 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-singlethread.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-singlethread.ll
@@ -12,7 +12,8 @@
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1100 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX11-CU %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 < %s | FileCheck --check-prefixes=GFX12-WGP %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX12-CU %s
-; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 < %s | FileCheck --check-prefixes=GFX1250 %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 < %s | FileCheck --check-prefixes=GFX1250,GFX1250-NOGAS %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 -mattr=+globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
define amdgpu_kernel void @private_singlethread_unordered_load(
; GFX6-LABEL: private_singlethread_unordered_load:
@@ -225,41 +226,47 @@ define amdgpu_kernel void @private_singlethread_unordered_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_unordered_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_unordered_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_unordered_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("singlethread") unordered, align 4
@@ -478,41 +485,47 @@ define amdgpu_kernel void @private_singlethread_monotonic_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_monotonic_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_monotonic_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_monotonic_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("singlethread") monotonic, align 4
@@ -731,41 +744,47 @@ define amdgpu_kernel void @private_singlethread_acquire_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acquire_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acquire_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acquire_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("singlethread") acquire, align 4
@@ -984,41 +1003,47 @@ define amdgpu_kernel void @private_singlethread_seq_cst_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_seq_cst_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_seq_cst_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_seq_cst_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("singlethread") seq_cst, align 4
@@ -1221,40 +1246,46 @@ define amdgpu_kernel void @private_singlethread_unordered_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_unordered_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_unordered_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_unordered_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("singlethread") unordered, align 4
@@ -1456,40 +1487,46 @@ define amdgpu_kernel void @private_singlethread_monotonic_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_monotonic_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_monotonic_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_monotonic_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("singlethread") monotonic, align 4
@@ -1691,40 +1728,46 @@ define amdgpu_kernel void @private_singlethread_release_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_release_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_release_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_release_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("singlethread") release, align 4
@@ -1926,40 +1969,46 @@ define amdgpu_kernel void @private_singlethread_seq_cst_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_seq_cst_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_seq_cst_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_seq_cst_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("singlethread") seq_cst, align 4
@@ -2194,41 +2243,46 @@ define amdgpu_kernel void @private_singlethread_monotonic_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_monotonic_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_monotonic_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_monotonic_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread") monotonic
@@ -2463,41 +2517,46 @@ define amdgpu_kernel void @private_singlethread_acquire_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acquire_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acquire_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acquire_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread") acquire
@@ -2732,41 +2791,46 @@ define amdgpu_kernel void @private_singlethread_release_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_release_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_release_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_release_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread") release
@@ -3001,41 +3065,46 @@ define amdgpu_kernel void @private_singlethread_acq_rel_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acq_rel_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acq_rel_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acq_rel_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread") acq_rel
@@ -3270,41 +3339,46 @@ define amdgpu_kernel void @private_singlethread_seq_cst_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_seq_cst_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_seq_cst_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_seq_cst_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread") seq_cst
@@ -3560,42 +3634,51 @@ define amdgpu_kernel void @private_singlethread_acquire_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acquire_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acquire_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acquire_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread") acquire
@@ -3852,42 +3935,51 @@ define amdgpu_kernel void @private_singlethread_acq_rel_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acq_rel_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acq_rel_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acq_rel_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread") acq_rel
@@ -4144,42 +4236,51 @@ define amdgpu_kernel void @private_singlethread_seq_cst_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_seq_cst_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_seq_cst_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_seq_cst_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread") seq_cst
@@ -4492,49 +4593,56 @@ define amdgpu_kernel void @private_singlethread_monotonic_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_monotonic_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_monotonic_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_monotonic_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -4847,49 +4955,56 @@ define amdgpu_kernel void @private_singlethread_acquire_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acquire_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acquire_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acquire_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5202,49 +5317,56 @@ define amdgpu_kernel void @private_singlethread_release_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_release_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_release_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_release_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5557,49 +5679,56 @@ define amdgpu_kernel void @private_singlethread_acq_rel_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acq_rel_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acq_rel_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acq_rel_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5912,49 +6041,56 @@ define amdgpu_kernel void @private_singlethread_seq_cst_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_seq_cst_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_seq_cst_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_seq_cst_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -6267,49 +6403,56 @@ define amdgpu_kernel void @private_singlethread_monotonic_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_monotonic_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_monotonic_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_monotonic_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -6622,49 +6765,56 @@ define amdgpu_kernel void @private_singlethread_acquire_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acquire_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acquire_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acquire_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -6977,49 +7127,56 @@ define amdgpu_kernel void @private_singlethread_release_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_release_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_release_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_release_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7332,49 +7489,56 @@ define amdgpu_kernel void @private_singlethread_acq_rel_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acq_rel_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acq_rel_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acq_rel_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7687,49 +7851,56 @@ define amdgpu_kernel void @private_singlethread_seq_cst_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_seq_cst_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_seq_cst_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_seq_cst_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8042,49 +8213,56 @@ define amdgpu_kernel void @private_singlethread_monotonic_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_monotonic_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_monotonic_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_monotonic_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8397,49 +8575,56 @@ define amdgpu_kernel void @private_singlethread_acquire_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acquire_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acquire_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acquire_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8752,49 +8937,56 @@ define amdgpu_kernel void @private_singlethread_release_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_release_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_release_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_release_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -9107,49 +9299,56 @@ define amdgpu_kernel void @private_singlethread_acq_rel_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acq_rel_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acq_rel_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acq_rel_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -9462,49 +9661,56 @@ define amdgpu_kernel void @private_singlethread_seq_cst_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_seq_cst_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_seq_cst_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_seq_cst_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -9845,50 +10051,59 @@ define amdgpu_kernel void @private_singlethread_monotonic_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_monotonic_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -10231,50 +10446,59 @@ define amdgpu_kernel void @private_singlethread_acquire_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acquire_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acquire_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acquire_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -10617,50 +10841,59 @@ define amdgpu_kernel void @private_singlethread_release_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_release_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_release_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_release_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11003,50 +11236,59 @@ define amdgpu_kernel void @private_singlethread_acq_rel_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acq_rel_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11389,50 +11631,59 @@ define amdgpu_kernel void @private_singlethread_seq_cst_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_seq_cst_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11775,50 +12026,59 @@ define amdgpu_kernel void @private_singlethread_monotonic_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_monotonic_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_monotonic_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_monotonic_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -12161,50 +12421,59 @@ define amdgpu_kernel void @private_singlethread_acquire_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acquire_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acquire_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acquire_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -12547,50 +12816,59 @@ define amdgpu_kernel void @private_singlethread_release_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_release_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_release_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_release_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -12933,50 +13211,59 @@ define amdgpu_kernel void @private_singlethread_acq_rel_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acq_rel_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -13319,50 +13606,59 @@ define amdgpu_kernel void @private_singlethread_seq_cst_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_seq_cst_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -13705,50 +14001,59 @@ define amdgpu_kernel void @private_singlethread_monotonic_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_monotonic_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -14091,50 +14396,59 @@ define amdgpu_kernel void @private_singlethread_acquire_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acquire_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -14477,50 +14791,59 @@ define amdgpu_kernel void @private_singlethread_release_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_release_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_release_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_release_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -14863,50 +15186,59 @@ define amdgpu_kernel void @private_singlethread_acq_rel_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acq_rel_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -15249,50 +15581,59 @@ define amdgpu_kernel void @private_singlethread_seq_cst_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_seq_cst_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -15513,41 +15854,47 @@ define amdgpu_kernel void @private_singlethread_one_as_unordered_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_unordered_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_unordered_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_unordered_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("singlethread-one-as") unordered, align 4
@@ -15766,41 +16113,47 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_monotonic_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_monotonic_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_monotonic_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("singlethread-one-as") monotonic, align 4
@@ -16019,41 +16372,47 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acquire_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acquire_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acquire_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("singlethread-one-as") acquire, align 4
@@ -16272,41 +16631,47 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_seq_cst_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_seq_cst_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_seq_cst_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("singlethread-one-as") seq_cst, align 4
@@ -16509,40 +16874,46 @@ define amdgpu_kernel void @private_singlethread_one_as_unordered_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_unordered_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_unordered_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_unordered_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("singlethread-one-as") unordered, align 4
@@ -16744,40 +17115,46 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_monotonic_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_monotonic_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_monotonic_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("singlethread-one-as") monotonic, align 4
@@ -16979,40 +17356,46 @@ define amdgpu_kernel void @private_singlethread_one_as_release_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_release_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_release_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_release_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("singlethread-one-as") release, align 4
@@ -17214,40 +17597,46 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_seq_cst_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_seq_cst_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_seq_cst_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("singlethread-one-as") seq_cst, align 4
@@ -17482,41 +17871,46 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_monotonic_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_monotonic_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_monotonic_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread-one-as") monotonic
@@ -17751,41 +18145,46 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acquire_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acquire_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acquire_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread-one-as") acquire
@@ -18020,41 +18419,46 @@ define amdgpu_kernel void @private_singlethread_one_as_release_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_release_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_release_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_release_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread-one-as") release
@@ -18289,41 +18693,46 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acq_rel_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acq_rel_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acq_rel_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread-one-as") acq_rel
@@ -18558,41 +18967,46 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_seq_cst_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_seq_cst_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_seq_cst_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread-one-as") seq_cst
@@ -18848,42 +19262,51 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acquire_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acquire_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acquire_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread-one-as") acquire
@@ -19140,42 +19563,51 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acq_rel_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acq_rel_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acq_rel_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread-one-as") acq_rel
@@ -19432,42 +19864,51 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_seq_cst_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_seq_cst_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_seq_cst_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread-one-as") seq_cst
@@ -19780,49 +20221,56 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_monotonic_cmpxc
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_monotonic_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_monotonic_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_monotonic_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -20135,49 +20583,56 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_monotonic_cmpxchg
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acquire_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acquire_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acquire_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -20490,49 +20945,56 @@ define amdgpu_kernel void @private_singlethread_one_as_release_monotonic_cmpxchg
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_release_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_release_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_release_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -20845,49 +21307,56 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_monotonic_cmpxchg
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acq_rel_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acq_rel_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acq_rel_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -21200,49 +21669,56 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_monotonic_cmpxchg
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_seq_cst_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_seq_cst_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_seq_cst_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -21555,49 +22031,56 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_acquire_cmpxchg
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_monotonic_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_monotonic_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_monotonic_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -21910,49 +22393,56 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acquire_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acquire_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acquire_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22265,49 +22755,56 @@ define amdgpu_kernel void @private_singlethread_one_as_release_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_release_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_release_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_release_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22620,49 +23117,56 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acq_rel_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acq_rel_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acq_rel_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22975,49 +23479,56 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_seq_cst_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_seq_cst_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_seq_cst_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -23330,49 +23841,56 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_seq_cst_cmpxchg
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_monotonic_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_monotonic_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_monotonic_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -23685,49 +24203,56 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acquire_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acquire_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acquire_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24040,49 +24565,56 @@ define amdgpu_kernel void @private_singlethread_one_as_release_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_release_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_release_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_release_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24395,49 +24927,56 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acq_rel_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acq_rel_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acq_rel_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24750,49 +25289,56 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_seq_cst_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_seq_cst_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_seq_cst_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -25133,50 +25679,59 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_monotonic_ret_c
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_monotonic_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -25519,50 +26074,59 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_monotonic_ret_cmp
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acquire_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acquire_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acquire_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -25905,50 +26469,59 @@ define amdgpu_kernel void @private_singlethread_one_as_release_monotonic_ret_cmp
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_release_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_release_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_release_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -26291,50 +26864,59 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_monotonic_ret_cmp
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acq_rel_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -26677,50 +27259,59 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_monotonic_ret_cmp
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_seq_cst_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -27063,50 +27654,59 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_acquire_ret_cmp
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_monotonic_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_monotonic_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_monotonic_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -27449,50 +28049,59 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_acquire_ret_cmpxc
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acquire_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acquire_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acquire_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -27835,50 +28444,59 @@ define amdgpu_kernel void @private_singlethread_one_as_release_acquire_ret_cmpxc
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_release_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_release_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_release_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -28221,50 +28839,59 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_acquire_ret_cmpxc
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acq_rel_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -28607,50 +29234,59 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_acquire_ret_cmpxc
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_seq_cst_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -28993,50 +29629,59 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_seq_cst_ret_cmp
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_monotonic_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -29379,50 +30024,59 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_seq_cst_ret_cmpxc
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acquire_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -29765,50 +30419,59 @@ define amdgpu_kernel void @private_singlethread_one_as_release_seq_cst_ret_cmpxc
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_release_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_release_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_release_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30151,50 +30814,59 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_seq_cst_ret_cmpxc
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acq_rel_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30537,50 +31209,59 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_seq_cst_ret_cmpxc
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_seq_cst_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30589,3 +31270,5 @@ entry:
store i32 %val0, ptr addrspace(5) %out, align 4
ret void
}
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GFX1250: {{.*}}
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-system.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-system.ll
index 5a5dbffa61148..487e54f55ad0b 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-system.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-system.ll
@@ -12,7 +12,8 @@
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1100 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX11-CU %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 < %s | FileCheck --check-prefixes=GFX12-WGP %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX12-CU %s
-; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 < %s | FileCheck --check-prefixes=GFX1250 %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 < %s | FileCheck --check-prefixes=GFX1250,GFX1250-NOGAS %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 -mattr=+globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
define amdgpu_kernel void @private_system_unordered_load(
; GFX6-LABEL: private_system_unordered_load:
@@ -225,41 +226,47 @@ define amdgpu_kernel void @private_system_unordered_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_unordered_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_unordered_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_unordered_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in unordered, align 4
@@ -478,41 +485,47 @@ define amdgpu_kernel void @private_system_monotonic_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_monotonic_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_monotonic_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_monotonic_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in monotonic, align 4
@@ -731,43 +744,49 @@ define amdgpu_kernel void @private_system_acquire_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_acquire_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_acquire_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_acquire_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in acquire, align 4
@@ -986,45 +1005,51 @@ define amdgpu_kernel void @private_system_seq_cst_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_seq_cst_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_seq_cst_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_seq_cst_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in seq_cst, align 4
@@ -1227,40 +1252,46 @@ define amdgpu_kernel void @private_system_unordered_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_unordered_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_unordered_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_unordered_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out unordered, align 4
@@ -1462,40 +1493,46 @@ define amdgpu_kernel void @private_system_monotonic_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_monotonic_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_monotonic_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_monotonic_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out monotonic, align 4
@@ -1697,45 +1734,51 @@ define amdgpu_kernel void @private_system_release_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_release_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_release_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_release_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out release, align 4
@@ -1937,45 +1980,51 @@ define amdgpu_kernel void @private_system_seq_cst_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_seq_cst_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_seq_cst_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_seq_cst_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out seq_cst, align 4
@@ -2210,41 +2259,46 @@ define amdgpu_kernel void @private_system_monotonic_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_monotonic_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_monotonic_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_monotonic_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in monotonic
@@ -2479,44 +2533,49 @@ define amdgpu_kernel void @private_system_acquire_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_acquire_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_acquire_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_acquire_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in acquire
@@ -2751,46 +2810,51 @@ define amdgpu_kernel void @private_system_release_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_release_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_release_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_release_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in release
@@ -3025,49 +3089,54 @@ define amdgpu_kernel void @private_system_acq_rel_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_acq_rel_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_acq_rel_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_acq_rel_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in acq_rel
@@ -3302,49 +3371,54 @@ define amdgpu_kernel void @private_system_seq_cst_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_seq_cst_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_seq_cst_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_seq_cst_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in seq_cst
@@ -3600,44 +3674,53 @@ define amdgpu_kernel void @private_system_acquire_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_acquire_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_acquire_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_acquire_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in acquire
@@ -3894,49 +3977,58 @@ define amdgpu_kernel void @private_system_acq_rel_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_acq_rel_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_acq_rel_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_acq_rel_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in acq_rel
@@ -4193,49 +4285,58 @@ define amdgpu_kernel void @private_system_seq_cst_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_seq_cst_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_seq_cst_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_seq_cst_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in seq_cst
@@ -4548,49 +4649,56 @@ define amdgpu_kernel void @private_system_monotonic_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_monotonic_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_monotonic_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_monotonic_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -4903,52 +5011,59 @@ define amdgpu_kernel void @private_system_acquire_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_acquire_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_acquire_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_acquire_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5261,54 +5376,61 @@ define amdgpu_kernel void @private_system_release_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_release_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_release_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_release_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5621,57 +5743,64 @@ define amdgpu_kernel void @private_system_acq_rel_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_acq_rel_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_acq_rel_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_acq_rel_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5984,57 +6113,64 @@ define amdgpu_kernel void @private_system_seq_cst_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_seq_cst_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_seq_cst_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_seq_cst_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -6347,52 +6483,59 @@ define amdgpu_kernel void @private_system_monotonic_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_monotonic_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_monotonic_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_monotonic_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -6705,52 +6848,59 @@ define amdgpu_kernel void @private_system_acquire_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_acquire_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_acquire_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_acquire_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7063,57 +7213,64 @@ define amdgpu_kernel void @private_system_release_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_release_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_release_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_release_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7426,57 +7583,64 @@ define amdgpu_kernel void @private_system_acq_rel_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_acq_rel_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_acq_rel_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_acq_rel_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7789,57 +7953,64 @@ define amdgpu_kernel void @private_system_seq_cst_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_seq_cst_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_seq_cst_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_seq_cst_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8152,57 +8323,64 @@ define amdgpu_kernel void @private_system_seq_cst_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_seq_cst_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_seq_cst_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_seq_cst_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8543,50 +8721,59 @@ define amdgpu_kernel void @private_system_monotonic_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_monotonic_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8929,52 +9116,61 @@ define amdgpu_kernel void @private_system_acquire_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_acquire_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_acquire_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_acquire_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -9317,57 +9513,66 @@ define amdgpu_kernel void @private_system_acq_rel_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_acq_rel_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -9710,57 +9915,66 @@ define amdgpu_kernel void @private_system_seq_cst_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_seq_cst_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -10103,52 +10317,61 @@ define amdgpu_kernel void @private_system_monotonic_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_monotonic_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_monotonic_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_monotonic_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -10491,52 +10714,61 @@ define amdgpu_kernel void @private_system_acquire_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_acquire_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_acquire_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_acquire_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -10879,57 +11111,66 @@ define amdgpu_kernel void @private_system_release_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_release_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_release_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_release_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11272,57 +11513,66 @@ define amdgpu_kernel void @private_system_acq_rel_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_acq_rel_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11665,57 +11915,66 @@ define amdgpu_kernel void @private_system_seq_cst_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_seq_cst_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -12058,57 +12317,66 @@ define amdgpu_kernel void @private_system_monotonic_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_monotonic_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -12451,57 +12719,66 @@ define amdgpu_kernel void @private_system_acquire_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_acquire_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -12844,57 +13121,66 @@ define amdgpu_kernel void @private_system_relese_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_relese_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_relese_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_relese_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -13237,57 +13523,66 @@ define amdgpu_kernel void @private_system_acq_rel_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_acq_rel_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -13630,57 +13925,66 @@ define amdgpu_kernel void @private_system_seq_cst_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_seq_cst_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -13901,41 +14205,47 @@ define amdgpu_kernel void @private_system_one_as_unordered_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_unordered_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_unordered_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_unordered_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("one-as") unordered, align 4
@@ -14154,41 +14464,47 @@ define amdgpu_kernel void @private_system_one_as_monotonic_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_monotonic_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_monotonic_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_monotonic_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("one-as") monotonic, align 4
@@ -14407,43 +14723,49 @@ define amdgpu_kernel void @private_system_one_as_acquire_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acquire_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_acquire_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acquire_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("one-as") acquire, align 4
@@ -14662,45 +14984,51 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_seq_cst_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("one-as") seq_cst, align 4
@@ -14903,40 +15231,46 @@ define amdgpu_kernel void @private_system_one_as_unordered_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_unordered_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_unordered_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_unordered_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("one-as") unordered, align 4
@@ -15138,40 +15472,46 @@ define amdgpu_kernel void @private_system_one_as_monotonic_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_monotonic_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_monotonic_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_monotonic_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("one-as") monotonic, align 4
@@ -15373,45 +15713,51 @@ define amdgpu_kernel void @private_system_one_as_release_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_release_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_release_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_release_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("one-as") release, align 4
@@ -15613,45 +15959,51 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_seq_cst_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("one-as") seq_cst, align 4
@@ -15886,41 +16238,46 @@ define amdgpu_kernel void @private_system_one_as_monotonic_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_monotonic_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_monotonic_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_monotonic_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("one-as") monotonic
@@ -16155,44 +16512,49 @@ define amdgpu_kernel void @private_system_one_as_acquire_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acquire_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_acquire_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acquire_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("one-as") acquire
@@ -16427,46 +16789,51 @@ define amdgpu_kernel void @private_system_one_as_release_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_release_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_release_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_release_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("one-as") release
@@ -16701,49 +17068,54 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acq_rel_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_acq_rel_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acq_rel_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("one-as") acq_rel
@@ -16978,49 +17350,54 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_seq_cst_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("one-as") seq_cst
@@ -17276,44 +17653,53 @@ define amdgpu_kernel void @private_system_one_as_acquire_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acquire_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_acquire_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acquire_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("one-as") acquire
@@ -17570,49 +17956,58 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acq_rel_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_acq_rel_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acq_rel_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("one-as") acq_rel
@@ -17869,49 +18264,58 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_seq_cst_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("one-as") seq_cst
@@ -18224,49 +18628,56 @@ define amdgpu_kernel void @private_system_one_as_monotonic_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_monotonic_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_monotonic_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_monotonic_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -18579,52 +18990,59 @@ define amdgpu_kernel void @private_system_one_as_acquire_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acquire_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_acquire_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acquire_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -18937,54 +19355,61 @@ define amdgpu_kernel void @private_system_one_as_release_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_release_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_release_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_release_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -19297,57 +19722,64 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acq_rel_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_acq_rel_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acq_rel_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -19660,57 +20092,64 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_seq_cst_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -20023,52 +20462,59 @@ define amdgpu_kernel void @private_system_one_as_monotonic_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_monotonic_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_monotonic_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_monotonic_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -20381,52 +20827,59 @@ define amdgpu_kernel void @private_system_one_as_acquire_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acquire_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_acquire_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acquire_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -20739,57 +21192,64 @@ define amdgpu_kernel void @private_system_one_as_release_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_release_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_release_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_release_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -21102,57 +21562,64 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acq_rel_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_acq_rel_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acq_rel_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -21465,57 +21932,64 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_seq_cst_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -21828,57 +22302,64 @@ define amdgpu_kernel void @private_system_one_as_monotonic_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_monotonic_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_monotonic_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_monotonic_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22191,57 +22672,64 @@ define amdgpu_kernel void @private_system_one_as_acquire_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acquire_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_acquire_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acquire_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22554,57 +23042,64 @@ define amdgpu_kernel void @private_system_one_as_release_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_release_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_release_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_release_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22917,57 +23412,64 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acq_rel_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_acq_rel_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acq_rel_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -23280,57 +23782,64 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_seq_cst_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -23671,50 +24180,59 @@ define amdgpu_kernel void @private_system_one_as_monotonic_monotonic_ret_cmpxchg
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_monotonic_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24057,52 +24575,61 @@ define amdgpu_kernel void @private_system_one_as_acquire_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acquire_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_acquire_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acquire_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24445,55 +24972,64 @@ define amdgpu_kernel void @private_system_one_as_release_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_release_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_release_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_release_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24836,57 +25372,66 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acq_rel_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -25229,57 +25774,66 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_seq_cst_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -25622,52 +26176,61 @@ define amdgpu_kernel void @private_system_one_as_monotonic_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_monotonic_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_monotonic_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_monotonic_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -26010,52 +26573,61 @@ define amdgpu_kernel void @private_system_one_as_acquire_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acquire_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_acquire_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acquire_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -26398,57 +26970,66 @@ define amdgpu_kernel void @private_system_one_as_release_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_release_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_release_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_release_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -26791,57 +27372,66 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acq_rel_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -27184,57 +27774,66 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_seq_cst_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -27577,57 +28176,66 @@ define amdgpu_kernel void @private_system_one_as_monotonic_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_monotonic_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -27970,57 +28578,66 @@ define amdgpu_kernel void @private_system_one_as_acquire_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acquire_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -28363,57 +28980,66 @@ define amdgpu_kernel void @private_system_one_as_release_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_release_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_release_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_release_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -28756,57 +29382,66 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acq_rel_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -29149,57 +29784,66 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_seq_cst_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -29208,3 +29852,5 @@ entry:
store i32 %val0, ptr addrspace(5) %out, align 4
ret void
}
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GFX1250: {{.*}}
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-volatile.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-volatile.ll
index fd822659c3439..d65875653770f 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-volatile.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-volatile.ll
@@ -8,7 +8,8 @@
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1100 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX11-CU %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 < %s | FileCheck --check-prefixes=GFX12-WGP %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX12-CU %s
-; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 < %s | FileCheck --check-prefixes=GFX1250 %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 < %s | FileCheck --check-prefixes=GFX1250,GFX1250-NOGAS %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 -mattr=+globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
define amdgpu_kernel void @private_volatile_load_0(
; GFX6-LABEL: private_volatile_load_0:
@@ -865,3 +866,6 @@ entry:
}
declare i32 @llvm.amdgcn.workitem.id.x()
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GFX1250-GAS: {{.*}}
+; GFX1250-NOGAS: {{.*}}
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-wavefront.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-wavefront.ll
index c59b6cc15ae9a..5434df9a8e86e 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-wavefront.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-wavefront.ll
@@ -12,7 +12,8 @@
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1100 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX11-CU %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 < %s | FileCheck --check-prefixes=GFX12-WGP %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX12-CU %s
-; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 < %s | FileCheck --check-prefixes=GFX1250 %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 < %s | FileCheck --check-prefixes=GFX1250,GFX1250-NOGAS %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 -mattr=+globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
define amdgpu_kernel void @private_wavefront_unordered_load(
; GFX6-LABEL: private_wavefront_unordered_load:
@@ -225,41 +226,47 @@ define amdgpu_kernel void @private_wavefront_unordered_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_unordered_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_unordered_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_unordered_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("wavefront") unordered, align 4
@@ -478,41 +485,47 @@ define amdgpu_kernel void @private_wavefront_monotonic_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_monotonic_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_monotonic_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_monotonic_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("wavefront") monotonic, align 4
@@ -731,41 +744,47 @@ define amdgpu_kernel void @private_wavefront_acquire_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acquire_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acquire_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acquire_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("wavefront") acquire, align 4
@@ -984,41 +1003,47 @@ define amdgpu_kernel void @private_wavefront_seq_cst_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_seq_cst_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_seq_cst_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_seq_cst_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("wavefront") seq_cst, align 4
@@ -1221,40 +1246,46 @@ define amdgpu_kernel void @private_wavefront_unordered_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_unordered_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_unordered_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_unordered_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("wavefront") unordered, align 4
@@ -1456,40 +1487,46 @@ define amdgpu_kernel void @private_wavefront_monotonic_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_monotonic_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_monotonic_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_monotonic_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("wavefront") monotonic, align 4
@@ -1691,40 +1728,46 @@ define amdgpu_kernel void @private_wavefront_release_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_release_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_release_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_release_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("wavefront") release, align 4
@@ -1926,40 +1969,46 @@ define amdgpu_kernel void @private_wavefront_seq_cst_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_seq_cst_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_seq_cst_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_seq_cst_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("wavefront") seq_cst, align 4
@@ -2194,41 +2243,46 @@ define amdgpu_kernel void @private_wavefront_monotonic_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_monotonic_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_monotonic_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_monotonic_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront") monotonic
@@ -2463,41 +2517,46 @@ define amdgpu_kernel void @private_wavefront_acquire_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acquire_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acquire_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acquire_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront") acquire
@@ -2732,41 +2791,46 @@ define amdgpu_kernel void @private_wavefront_release_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_release_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_release_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_release_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront") release
@@ -3001,41 +3065,46 @@ define amdgpu_kernel void @private_wavefront_acq_rel_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acq_rel_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acq_rel_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acq_rel_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront") acq_rel
@@ -3270,41 +3339,46 @@ define amdgpu_kernel void @private_wavefront_seq_cst_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_seq_cst_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_seq_cst_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_seq_cst_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront") seq_cst
@@ -3560,42 +3634,51 @@ define amdgpu_kernel void @private_wavefront_acquire_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acquire_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acquire_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acquire_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront") acquire
@@ -3852,42 +3935,51 @@ define amdgpu_kernel void @private_wavefront_acq_rel_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acq_rel_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acq_rel_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acq_rel_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront") acq_rel
@@ -4144,42 +4236,51 @@ define amdgpu_kernel void @private_wavefront_seq_cst_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_seq_cst_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_seq_cst_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_seq_cst_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront") seq_cst
@@ -4492,49 +4593,56 @@ define amdgpu_kernel void @private_wavefront_monotonic_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_monotonic_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_monotonic_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_monotonic_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -4847,49 +4955,56 @@ define amdgpu_kernel void @private_wavefront_acquire_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acquire_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acquire_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acquire_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5202,49 +5317,56 @@ define amdgpu_kernel void @private_wavefront_release_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_release_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_release_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_release_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5557,49 +5679,56 @@ define amdgpu_kernel void @private_wavefront_acq_rel_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acq_rel_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acq_rel_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acq_rel_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5912,49 +6041,56 @@ define amdgpu_kernel void @private_wavefront_seq_cst_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_seq_cst_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_seq_cst_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_seq_cst_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -6267,49 +6403,56 @@ define amdgpu_kernel void @private_wavefront_monotonic_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_monotonic_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_monotonic_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_monotonic_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -6622,49 +6765,56 @@ define amdgpu_kernel void @private_wavefront_acquire_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acquire_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acquire_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acquire_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -6977,49 +7127,56 @@ define amdgpu_kernel void @private_wavefront_release_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_release_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_release_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_release_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7332,49 +7489,56 @@ define amdgpu_kernel void @private_wavefront_acq_rel_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acq_rel_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acq_rel_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acq_rel_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7687,49 +7851,56 @@ define amdgpu_kernel void @private_wavefront_seq_cst_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_seq_cst_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_seq_cst_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_seq_cst_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8042,49 +8213,56 @@ define amdgpu_kernel void @private_wavefront_monotonic_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_monotonic_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_monotonic_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_monotonic_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8397,49 +8575,56 @@ define amdgpu_kernel void @private_wavefront_acquire_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acquire_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acquire_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acquire_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8752,49 +8937,56 @@ define amdgpu_kernel void @private_wavefront_release_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_release_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_release_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_release_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -9107,49 +9299,56 @@ define amdgpu_kernel void @private_wavefront_acq_rel_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acq_rel_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acq_rel_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acq_rel_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -9462,49 +9661,56 @@ define amdgpu_kernel void @private_wavefront_seq_cst_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_seq_cst_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_seq_cst_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_seq_cst_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -9845,50 +10051,59 @@ define amdgpu_kernel void @private_wavefront_monotonic_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_monotonic_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -10231,50 +10446,59 @@ define amdgpu_kernel void @private_wavefront_acquire_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acquire_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acquire_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acquire_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -10617,50 +10841,59 @@ define amdgpu_kernel void @private_wavefront_release_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_release_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_release_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_release_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11003,50 +11236,59 @@ define amdgpu_kernel void @private_wavefront_acq_rel_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acq_rel_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11389,50 +11631,59 @@ define amdgpu_kernel void @private_wavefront_seq_cst_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_seq_cst_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11775,50 +12026,59 @@ define amdgpu_kernel void @private_wavefront_monotonic_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_monotonic_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_monotonic_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_monotonic_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -12161,50 +12421,59 @@ define amdgpu_kernel void @private_wavefront_acquire_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acquire_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acquire_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acquire_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -12547,50 +12816,59 @@ define amdgpu_kernel void @private_wavefront_release_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_release_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_release_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_release_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -12933,50 +13211,59 @@ define amdgpu_kernel void @private_wavefront_acq_rel_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acq_rel_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -13319,50 +13606,59 @@ define amdgpu_kernel void @private_wavefront_seq_cst_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_seq_cst_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -13705,50 +14001,59 @@ define amdgpu_kernel void @private_wavefront_monotonic_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_monotonic_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -14091,50 +14396,59 @@ define amdgpu_kernel void @private_wavefront_acquire_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acquire_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -14477,50 +14791,59 @@ define amdgpu_kernel void @private_wavefront_release_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_release_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_release_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_release_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -14863,50 +15186,59 @@ define amdgpu_kernel void @private_wavefront_acq_rel_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acq_rel_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -15249,50 +15581,59 @@ define amdgpu_kernel void @private_wavefront_seq_cst_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_seq_cst_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -15513,41 +15854,47 @@ define amdgpu_kernel void @private_wavefront_one_as_unordered_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_unordered_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_unordered_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_unordered_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("wavefront-one-as") unordered, align 4
@@ -15766,41 +16113,47 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_monotonic_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_monotonic_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_monotonic_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("wavefront-one-as") monotonic, align 4
@@ -16019,41 +16372,47 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acquire_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acquire_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acquire_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("wavefront-one-as") acquire, align 4
@@ -16272,41 +16631,47 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_seq_cst_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_seq_cst_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_seq_cst_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("wavefront-one-as") seq_cst, align 4
@@ -16509,40 +16874,46 @@ define amdgpu_kernel void @private_wavefront_one_as_unordered_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_unordered_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_unordered_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_unordered_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("wavefront-one-as") unordered, align 4
@@ -16744,40 +17115,46 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_monotonic_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_monotonic_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_monotonic_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("wavefront-one-as") monotonic, align 4
@@ -16979,40 +17356,46 @@ define amdgpu_kernel void @private_wavefront_one_as_release_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_release_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_release_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_release_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("wavefront-one-as") release, align 4
@@ -17214,40 +17597,46 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_seq_cst_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_seq_cst_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_seq_cst_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("wavefront-one-as") seq_cst, align 4
@@ -17482,41 +17871,46 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_monotonic_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_monotonic_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_monotonic_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront-one-as") monotonic
@@ -17751,41 +18145,46 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acquire_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acquire_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acquire_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront-one-as") acquire
@@ -18020,41 +18419,46 @@ define amdgpu_kernel void @private_wavefront_one_as_release_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_release_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_release_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_release_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront-one-as") release
@@ -18289,41 +18693,46 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acq_rel_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acq_rel_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acq_rel_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront-one-as") acq_rel
@@ -18558,41 +18967,46 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_seq_cst_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_seq_cst_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_seq_cst_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront-one-as") seq_cst
@@ -18848,42 +19262,51 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acquire_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acquire_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acquire_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront-one-as") acquire
@@ -19140,42 +19563,51 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acq_rel_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acq_rel_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acq_rel_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront-one-as") acq_rel
@@ -19432,42 +19864,51 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_seq_cst_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_seq_cst_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_seq_cst_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront-one-as") seq_cst
@@ -19780,49 +20221,56 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_monotonic_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_monotonic_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_monotonic_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -20135,49 +20583,56 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acquire_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acquire_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acquire_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -20490,49 +20945,56 @@ define amdgpu_kernel void @private_wavefront_one_as_release_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_release_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_release_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_release_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -20845,49 +21307,56 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acq_rel_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acq_rel_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acq_rel_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -21200,49 +21669,56 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_seq_cst_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_seq_cst_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_seq_cst_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -21555,49 +22031,56 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_monotonic_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_monotonic_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_monotonic_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -21910,49 +22393,56 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acquire_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acquire_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acquire_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22265,49 +22755,56 @@ define amdgpu_kernel void @private_wavefront_one_as_release_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_release_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_release_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_release_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22620,49 +23117,56 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acq_rel_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acq_rel_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acq_rel_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22975,49 +23479,56 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_seq_cst_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_seq_cst_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_seq_cst_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -23330,49 +23841,56 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_monotonic_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_monotonic_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_monotonic_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -23685,49 +24203,56 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acquire_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acquire_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acquire_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24040,49 +24565,56 @@ define amdgpu_kernel void @private_wavefront_one_as_release_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_release_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_release_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_release_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24395,49 +24927,56 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acq_rel_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acq_rel_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acq_rel_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24750,49 +25289,56 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_seq_cst_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_seq_cst_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_seq_cst_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -25133,50 +25679,59 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_monotonic_ret_cmpx
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_monotonic_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -25519,50 +26074,59 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_monotonic_ret_cmpxch
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acquire_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acquire_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acquire_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -25905,50 +26469,59 @@ define amdgpu_kernel void @private_wavefront_one_as_release_monotonic_ret_cmpxch
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_release_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_release_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_release_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -26291,50 +26864,59 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_monotonic_ret_cmpxch
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acq_rel_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -26677,50 +27259,59 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_monotonic_ret_cmpxch
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_seq_cst_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -27063,50 +27654,59 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_acquire_ret_cmpxch
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_monotonic_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_monotonic_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_monotonic_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -27449,50 +28049,59 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acquire_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acquire_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acquire_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -27835,50 +28444,59 @@ define amdgpu_kernel void @private_wavefront_one_as_release_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_release_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_release_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_release_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -28221,50 +28839,59 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acq_rel_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -28607,50 +29234,59 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_seq_cst_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -28993,50 +29629,59 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_seq_cst_ret_cmpxch
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_monotonic_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -29379,50 +30024,59 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acquire_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -29765,50 +30419,59 @@ define amdgpu_kernel void @private_wavefront_one_as_release_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_release_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_release_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_release_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30151,50 +30814,59 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acq_rel_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30537,50 +31209,59 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_seq_cst_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30589,3 +31270,5 @@ entry:
store i32 %val0, ptr addrspace(5) %out, align 4
ret void
}
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GFX1250: {{.*}}
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-workgroup.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-workgroup.ll
index d50c8a875a208..994539720e878 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-workgroup.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-workgroup.ll
@@ -12,7 +12,8 @@
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1100 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX11-CU %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 < %s | FileCheck --check-prefixes=GFX12-WGP %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX12-CU %s
-; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 < %s | FileCheck --check-prefixes=GFX1250 %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 < %s | FileCheck --check-prefixes=GFX1250,GFX1250-NOGAS %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 -mattr=+globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
define amdgpu_kernel void @private_workgroup_unordered_load(
; GFX6-LABEL: private_workgroup_unordered_load:
@@ -225,41 +226,47 @@ define amdgpu_kernel void @private_workgroup_unordered_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_unordered_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_unordered_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_unordered_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("workgroup") unordered, align 4
@@ -478,41 +485,47 @@ define amdgpu_kernel void @private_workgroup_monotonic_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_monotonic_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_monotonic_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_monotonic_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("workgroup") monotonic, align 4
@@ -731,41 +744,47 @@ define amdgpu_kernel void @private_workgroup_acquire_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acquire_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acquire_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acquire_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("workgroup") acquire, align 4
@@ -984,43 +1003,49 @@ define amdgpu_kernel void @private_workgroup_seq_cst_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_seq_cst_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_seq_cst_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_seq_cst_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("workgroup") seq_cst, align 4
@@ -1223,40 +1248,46 @@ define amdgpu_kernel void @private_workgroup_unordered_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_unordered_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_unordered_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_unordered_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("workgroup") unordered, align 4
@@ -1458,40 +1489,46 @@ define amdgpu_kernel void @private_workgroup_monotonic_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_monotonic_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_monotonic_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_monotonic_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("workgroup") monotonic, align 4
@@ -1693,42 +1730,48 @@ define amdgpu_kernel void @private_workgroup_release_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_release_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_release_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_release_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("workgroup") release, align 4
@@ -1930,42 +1973,48 @@ define amdgpu_kernel void @private_workgroup_seq_cst_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_seq_cst_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_seq_cst_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_seq_cst_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("workgroup") seq_cst, align 4
@@ -2200,41 +2249,46 @@ define amdgpu_kernel void @private_workgroup_monotonic_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_monotonic_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_monotonic_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_monotonic_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup") monotonic
@@ -2469,42 +2523,47 @@ define amdgpu_kernel void @private_workgroup_acquire_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acquire_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acquire_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acquire_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup") acquire
@@ -2739,43 +2798,48 @@ define amdgpu_kernel void @private_workgroup_release_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_release_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_release_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_release_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup") release
@@ -3010,44 +3074,49 @@ define amdgpu_kernel void @private_workgroup_acq_rel_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acq_rel_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acq_rel_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acq_rel_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup") acq_rel
@@ -3282,44 +3351,49 @@ define amdgpu_kernel void @private_workgroup_seq_cst_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_seq_cst_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_seq_cst_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_seq_cst_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup") seq_cst
@@ -3575,42 +3649,51 @@ define amdgpu_kernel void @private_workgroup_acquire_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acquire_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acquire_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acquire_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup") acquire
@@ -3867,44 +3950,53 @@ define amdgpu_kernel void @private_workgroup_acq_rel_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acq_rel_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acq_rel_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acq_rel_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup") acq_rel
@@ -4161,44 +4253,53 @@ define amdgpu_kernel void @private_workgroup_seq_cst_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_seq_cst_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_seq_cst_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_seq_cst_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup") seq_cst
@@ -4511,49 +4612,56 @@ define amdgpu_kernel void @private_workgroup_monotonic_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_monotonic_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_monotonic_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_monotonic_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -4866,50 +4974,57 @@ define amdgpu_kernel void @private_workgroup_acquire_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acquire_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acquire_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acquire_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5222,51 +5337,58 @@ define amdgpu_kernel void @private_workgroup_release_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_release_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_release_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_release_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5579,52 +5701,59 @@ define amdgpu_kernel void @private_workgroup_acq_rel_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acq_rel_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acq_rel_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acq_rel_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5937,52 +6066,59 @@ define amdgpu_kernel void @private_workgroup_seq_cst_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_seq_cst_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_seq_cst_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_seq_cst_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -6295,50 +6431,57 @@ define amdgpu_kernel void @private_workgroup_monotonic_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_monotonic_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_monotonic_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_monotonic_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -6651,50 +6794,57 @@ define amdgpu_kernel void @private_workgroup_acquire_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acquire_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acquire_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acquire_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7007,52 +7157,59 @@ define amdgpu_kernel void @private_workgroup_release_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_release_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_release_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_release_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7365,52 +7522,59 @@ define amdgpu_kernel void @private_workgroup_acq_rel_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acq_rel_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acq_rel_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acq_rel_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7723,52 +7887,59 @@ define amdgpu_kernel void @private_workgroup_seq_cst_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_seq_cst_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_seq_cst_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_seq_cst_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8081,52 +8252,59 @@ define amdgpu_kernel void @private_workgroup_monotonic_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_monotonic_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_monotonic_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_monotonic_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8439,52 +8617,59 @@ define amdgpu_kernel void @private_workgroup_acquire_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acquire_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acquire_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acquire_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8797,52 +8982,59 @@ define amdgpu_kernel void @private_workgroup_release_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_release_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_release_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_release_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -9155,52 +9347,59 @@ define amdgpu_kernel void @private_workgroup_acq_rel_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acq_rel_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acq_rel_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acq_rel_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -9513,52 +9712,59 @@ define amdgpu_kernel void @private_workgroup_seq_cst_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_seq_cst_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_seq_cst_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_seq_cst_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -9899,50 +10105,59 @@ define amdgpu_kernel void @private_workgroup_monotonic_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_monotonic_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -10285,50 +10500,59 @@ define amdgpu_kernel void @private_workgroup_acquire_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acquire_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acquire_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acquire_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -10671,52 +10895,61 @@ define amdgpu_kernel void @private_workgroup_release_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_release_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_release_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_release_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11059,52 +11292,61 @@ define amdgpu_kernel void @private_workgroup_acq_rel_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acq_rel_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11447,52 +11689,61 @@ define amdgpu_kernel void @private_workgroup_seq_cst_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_seq_cst_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11835,50 +12086,59 @@ define amdgpu_kernel void @private_workgroup_monotonic_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_monotonic_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_monotonic_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_monotonic_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -12221,50 +12481,59 @@ define amdgpu_kernel void @private_workgroup_acquire_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acquire_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acquire_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acquire_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -12607,52 +12876,61 @@ define amdgpu_kernel void @private_workgroup_release_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_release_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_release_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_release_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -12995,52 +13273,61 @@ define amdgpu_kernel void @private_workgroup_acq_rel_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acq_rel_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -13383,52 +13670,61 @@ define amdgpu_kernel void @private_workgroup_seq_cst_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_seq_cst_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -13771,52 +14067,61 @@ define amdgpu_kernel void @private_workgroup_monotonic_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_monotonic_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -14159,52 +14464,61 @@ define amdgpu_kernel void @private_workgroup_acquire_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acquire_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -14547,52 +14861,61 @@ define amdgpu_kernel void @private_workgroup_release_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_release_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_release_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_release_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -14935,52 +15258,61 @@ define amdgpu_kernel void @private_workgroup_acq_rel_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acq_rel_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -15323,52 +15655,61 @@ define amdgpu_kernel void @private_workgroup_seq_cst_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_seq_cst_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -15589,41 +15930,47 @@ define amdgpu_kernel void @private_workgroup_one_as_unordered_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_unordered_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_unordered_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_unordered_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("workgroup-one-as") unordered, align 4
@@ -15842,41 +16189,47 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_monotonic_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_monotonic_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_monotonic_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("workgroup-one-as") monotonic, align 4
@@ -16095,41 +16448,47 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acquire_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acquire_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acquire_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("workgroup-one-as") acquire, align 4
@@ -16348,43 +16707,49 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_seq_cst_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_seq_cst_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_seq_cst_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("workgroup-one-as") seq_cst, align 4
@@ -16587,40 +16952,46 @@ define amdgpu_kernel void @private_workgroup_one_as_unordered_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_unordered_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_unordered_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_unordered_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("workgroup-one-as") unordered, align 4
@@ -16822,40 +17193,46 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_monotonic_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_monotonic_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_monotonic_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("workgroup-one-as") monotonic, align 4
@@ -17057,42 +17434,48 @@ define amdgpu_kernel void @private_workgroup_one_as_release_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_release_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_release_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_release_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("workgroup-one-as") release, align 4
@@ -17294,42 +17677,48 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_seq_cst_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_seq_cst_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_seq_cst_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("workgroup-one-as") seq_cst, align 4
@@ -17564,41 +17953,46 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_monotonic_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_monotonic_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_monotonic_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup-one-as") monotonic
@@ -17833,42 +18227,47 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acquire_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acquire_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acquire_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup-one-as") acquire
@@ -18103,43 +18502,48 @@ define amdgpu_kernel void @private_workgroup_one_as_release_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_release_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_release_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_release_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup-one-as") release
@@ -18374,44 +18778,49 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acq_rel_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acq_rel_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acq_rel_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup-one-as") acq_rel
@@ -18646,44 +19055,49 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_seq_cst_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_seq_cst_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_seq_cst_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
+; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup-one-as") seq_cst
@@ -18939,42 +19353,51 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acquire_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acquire_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acquire_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup-one-as") acquire
@@ -19231,44 +19654,53 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acq_rel_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acq_rel_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acq_rel_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup-one-as") acq_rel
@@ -19525,44 +19957,53 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_seq_cst_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_seq_cst_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_seq_cst_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup-one-as") seq_cst
@@ -19875,49 +20316,56 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_monotonic_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_monotonic_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_monotonic_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -20230,50 +20678,57 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acquire_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acquire_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acquire_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -20586,51 +21041,58 @@ define amdgpu_kernel void @private_workgroup_one_as_release_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_release_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_release_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_release_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -20943,52 +21405,59 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acq_rel_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acq_rel_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acq_rel_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -21301,52 +21770,59 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_seq_cst_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_seq_cst_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_seq_cst_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -21659,50 +22135,57 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_monotonic_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_monotonic_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_monotonic_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22015,50 +22498,57 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acquire_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acquire_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acquire_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22371,52 +22861,59 @@ define amdgpu_kernel void @private_workgroup_one_as_release_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_release_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_release_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_release_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22729,52 +23226,59 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acq_rel_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acq_rel_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acq_rel_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -23087,52 +23591,59 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_seq_cst_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_seq_cst_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_seq_cst_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -23445,52 +23956,59 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_monotonic_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_monotonic_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_monotonic_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -23803,52 +24321,59 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acquire_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acquire_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acquire_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24161,52 +24686,59 @@ define amdgpu_kernel void @private_workgroup_one_as_release_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_release_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_release_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_release_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24519,52 +25051,59 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acq_rel_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acq_rel_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acq_rel_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24877,52 +25416,59 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_seq_cst_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_seq_cst_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_seq_cst_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -25263,50 +25809,59 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_monotonic_ret_cmpx
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_monotonic_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -25649,50 +26204,59 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_monotonic_ret_cmpxch
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acquire_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acquire_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acquire_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -26035,52 +26599,61 @@ define amdgpu_kernel void @private_workgroup_one_as_release_monotonic_ret_cmpxch
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_release_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_release_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_release_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -26423,52 +26996,61 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_monotonic_ret_cmpxch
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acq_rel_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -26811,52 +27393,61 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_monotonic_ret_cmpxch
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_seq_cst_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -27199,50 +27790,59 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_acquire_ret_cmpxch
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_monotonic_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_monotonic_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_monotonic_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -27585,50 +28185,59 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acquire_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acquire_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acquire_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -27971,52 +28580,61 @@ define amdgpu_kernel void @private_workgroup_one_as_release_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_release_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_release_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_release_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -28359,52 +28977,61 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acq_rel_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -28747,52 +29374,61 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_seq_cst_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -29135,52 +29771,61 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_seq_cst_ret_cmpxch
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_monotonic_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -29523,52 +30168,61 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acquire_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -29911,52 +30565,61 @@ define amdgpu_kernel void @private_workgroup_one_as_release_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_release_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_release_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_release_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30299,52 +30962,61 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acq_rel_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30687,52 +31359,61 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_seq_cst_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30741,3 +31422,5 @@ entry:
store i32 %val0, ptr addrspace(5) %out, align 4
ret void
}
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GFX1250: {{.*}}
diff --git a/llvm/test/Transforms/AtomicExpand/AMDGPU/expand-atomic-private-gas.ll b/llvm/test/Transforms/AtomicExpand/AMDGPU/expand-atomic-private-gas.ll
index 2cb805ba873a8..00c29b5875217 100644
--- a/llvm/test/Transforms/AtomicExpand/AMDGPU/expand-atomic-private-gas.ll
+++ b/llvm/test/Transforms/AtomicExpand/AMDGPU/expand-atomic-private-gas.ll
@@ -1,6 +1,7 @@
; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 5
; RUN: opt -S -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1200 -passes='require<libcall-lowering-info>,atomic-expand' %s | FileCheck -check-prefixes=GFX1200 %s
-; RUN: opt -S -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1250 -passes='require<libcall-lowering-info>,atomic-expand' %s | FileCheck -check-prefixes=GFX1250 %s
+; RUN: opt -S -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1250 -passes='require<libcall-lowering-info>,atomic-expand' %s | FileCheck -check-prefixes=GFX1250,GFX1250-NOGAS %s
+; RUN: opt -S -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1250 -mattr=+globally-addressable-scratch -passes='require<libcall-lowering-info>,atomic-expand' %s | FileCheck -check-prefixes=GFX1250,GFX1250-GAS %s
define void @system_atomic_store_unordered_float(ptr addrspace(5) %addr, float %val) {
; GFX1200-LABEL: define void @system_atomic_store_unordered_float(
@@ -8,11 +9,16 @@ define void @system_atomic_store_unordered_float(ptr addrspace(5) %addr, float %
; GFX1200-NEXT: store float [[VAL]], ptr addrspace(5) [[ADDR]], align 4
; GFX1200-NEXT: ret void
;
-; GFX1250-LABEL: define void @system_atomic_store_unordered_float(
-; GFX1250-SAME: ptr addrspace(5) [[ADDR:%.*]], float [[VAL:%.*]]) #[[ATTR0:[0-9]+]] {
-; GFX1250-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
-; GFX1250-NEXT: store atomic float [[VAL]], ptr [[SCRATCH_ASCAST]] unordered, align 4
-; GFX1250-NEXT: ret void
+; GFX1250-NOGAS-LABEL: define void @system_atomic_store_unordered_float(
+; GFX1250-NOGAS-SAME: ptr addrspace(5) [[ADDR:%.*]], float [[VAL:%.*]]) #[[ATTR0:[0-9]+]] {
+; GFX1250-NOGAS-NEXT: store float [[VAL]], ptr addrspace(5) [[ADDR]], align 4
+; GFX1250-NOGAS-NEXT: ret void
+;
+; GFX1250-GAS-LABEL: define void @system_atomic_store_unordered_float(
+; GFX1250-GAS-SAME: ptr addrspace(5) [[ADDR:%.*]], float [[VAL:%.*]]) #[[ATTR0:[0-9]+]] {
+; GFX1250-GAS-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
+; GFX1250-GAS-NEXT: store atomic float [[VAL]], ptr [[SCRATCH_ASCAST]] unordered, align 4
+; GFX1250-GAS-NEXT: ret void
;
store atomic float %val, ptr addrspace(5) %addr unordered, align 4
ret void
@@ -24,11 +30,16 @@ define void @system_atomic_store_unordered_i32(ptr addrspace(5) %addr, i32 %val)
; GFX1200-NEXT: store i32 [[VAL]], ptr addrspace(5) [[ADDR]], align 4
; GFX1200-NEXT: ret void
;
-; GFX1250-LABEL: define void @system_atomic_store_unordered_i32(
-; GFX1250-SAME: ptr addrspace(5) [[ADDR:%.*]], i32 [[VAL:%.*]]) #[[ATTR0]] {
-; GFX1250-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
-; GFX1250-NEXT: store atomic i32 [[VAL]], ptr [[SCRATCH_ASCAST]] unordered, align 4
-; GFX1250-NEXT: ret void
+; GFX1250-NOGAS-LABEL: define void @system_atomic_store_unordered_i32(
+; GFX1250-NOGAS-SAME: ptr addrspace(5) [[ADDR:%.*]], i32 [[VAL:%.*]]) #[[ATTR0]] {
+; GFX1250-NOGAS-NEXT: store i32 [[VAL]], ptr addrspace(5) [[ADDR]], align 4
+; GFX1250-NOGAS-NEXT: ret void
+;
+; GFX1250-GAS-LABEL: define void @system_atomic_store_unordered_i32(
+; GFX1250-GAS-SAME: ptr addrspace(5) [[ADDR:%.*]], i32 [[VAL:%.*]]) #[[ATTR0]] {
+; GFX1250-GAS-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
+; GFX1250-GAS-NEXT: store atomic i32 [[VAL]], ptr [[SCRATCH_ASCAST]] unordered, align 4
+; GFX1250-GAS-NEXT: ret void
;
store atomic i32 %val, ptr addrspace(5) %addr unordered, align 4
ret void
@@ -40,11 +51,16 @@ define void @system_atomic_store_release_i32(ptr addrspace(5) %addr, i32 %val) {
; GFX1200-NEXT: store i32 [[VAL]], ptr addrspace(5) [[ADDR]], align 4
; GFX1200-NEXT: ret void
;
-; GFX1250-LABEL: define void @system_atomic_store_release_i32(
-; GFX1250-SAME: ptr addrspace(5) [[ADDR:%.*]], i32 [[VAL:%.*]]) #[[ATTR0]] {
-; GFX1250-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
-; GFX1250-NEXT: store atomic i32 [[VAL]], ptr [[SCRATCH_ASCAST]] release, align 4
-; GFX1250-NEXT: ret void
+; GFX1250-NOGAS-LABEL: define void @system_atomic_store_release_i32(
+; GFX1250-NOGAS-SAME: ptr addrspace(5) [[ADDR:%.*]], i32 [[VAL:%.*]]) #[[ATTR0]] {
+; GFX1250-NOGAS-NEXT: store i32 [[VAL]], ptr addrspace(5) [[ADDR]], align 4
+; GFX1250-NOGAS-NEXT: ret void
+;
+; GFX1250-GAS-LABEL: define void @system_atomic_store_release_i32(
+; GFX1250-GAS-SAME: ptr addrspace(5) [[ADDR:%.*]], i32 [[VAL:%.*]]) #[[ATTR0]] {
+; GFX1250-GAS-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
+; GFX1250-GAS-NEXT: store atomic i32 [[VAL]], ptr [[SCRATCH_ASCAST]] release, align 4
+; GFX1250-GAS-NEXT: ret void
;
store atomic i32 %val, ptr addrspace(5) %addr release, align 4
ret void
@@ -56,11 +72,16 @@ define void @workgroup_atomic_store_release_i32(ptr addrspace(5) %addr, i32 %val
; GFX1200-NEXT: store i32 [[VAL]], ptr addrspace(5) [[ADDR]], align 4
; GFX1200-NEXT: ret void
;
-; GFX1250-LABEL: define void @workgroup_atomic_store_release_i32(
-; GFX1250-SAME: ptr addrspace(5) [[ADDR:%.*]], i32 [[VAL:%.*]]) #[[ATTR0]] {
-; GFX1250-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
-; GFX1250-NEXT: store atomic i32 [[VAL]], ptr [[SCRATCH_ASCAST]] syncscope("workgroup") release, align 4
-; GFX1250-NEXT: ret void
+; GFX1250-NOGAS-LABEL: define void @workgroup_atomic_store_release_i32(
+; GFX1250-NOGAS-SAME: ptr addrspace(5) [[ADDR:%.*]], i32 [[VAL:%.*]]) #[[ATTR0]] {
+; GFX1250-NOGAS-NEXT: store i32 [[VAL]], ptr addrspace(5) [[ADDR]], align 4
+; GFX1250-NOGAS-NEXT: ret void
+;
+; GFX1250-GAS-LABEL: define void @workgroup_atomic_store_release_i32(
+; GFX1250-GAS-SAME: ptr addrspace(5) [[ADDR:%.*]], i32 [[VAL:%.*]]) #[[ATTR0]] {
+; GFX1250-GAS-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
+; GFX1250-GAS-NEXT: store atomic i32 [[VAL]], ptr [[SCRATCH_ASCAST]] syncscope("workgroup") release, align 4
+; GFX1250-GAS-NEXT: ret void
;
store atomic i32 %val, ptr addrspace(5) %addr syncscope("workgroup") release, align 4
ret void
@@ -72,11 +93,16 @@ define float @system_atomic_load_unordered_float(ptr addrspace(5) %addr) {
; GFX1200-NEXT: [[VAL:%.*]] = load float, ptr addrspace(5) [[ADDR]], align 4, !invariant.load [[META0:![0-9]+]], !nontemporal [[META1:![0-9]+]]
; GFX1200-NEXT: ret float [[VAL]]
;
-; GFX1250-LABEL: define float @system_atomic_load_unordered_float(
-; GFX1250-SAME: ptr addrspace(5) [[ADDR:%.*]]) #[[ATTR0]] {
-; GFX1250-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
-; GFX1250-NEXT: [[VAL:%.*]] = load atomic float, ptr [[SCRATCH_ASCAST]] unordered, align 4, !invariant.load [[META0:![0-9]+]], !nontemporal [[META1:![0-9]+]]
-; GFX1250-NEXT: ret float [[VAL]]
+; GFX1250-NOGAS-LABEL: define float @system_atomic_load_unordered_float(
+; GFX1250-NOGAS-SAME: ptr addrspace(5) [[ADDR:%.*]]) #[[ATTR0]] {
+; GFX1250-NOGAS-NEXT: [[VAL:%.*]] = load float, ptr addrspace(5) [[ADDR]], align 4, !invariant.load [[META0:![0-9]+]], !nontemporal [[META1:![0-9]+]]
+; GFX1250-NOGAS-NEXT: ret float [[VAL]]
+;
+; GFX1250-GAS-LABEL: define float @system_atomic_load_unordered_float(
+; GFX1250-GAS-SAME: ptr addrspace(5) [[ADDR:%.*]]) #[[ATTR0]] {
+; GFX1250-GAS-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
+; GFX1250-GAS-NEXT: [[VAL:%.*]] = load atomic float, ptr [[SCRATCH_ASCAST]] unordered, align 4, !invariant.load [[META0:![0-9]+]], !nontemporal [[META1:![0-9]+]]
+; GFX1250-GAS-NEXT: ret float [[VAL]]
;
%val = load atomic float, ptr addrspace(5) %addr unordered, align 4, !invariant.load !1, !nontemporal !0
ret float %val
@@ -88,11 +114,16 @@ define i32 @system_atomic_load_unordered_i32(ptr addrspace(5) %addr) {
; GFX1200-NEXT: [[VAL:%.*]] = load i32, ptr addrspace(5) [[ADDR]], align 4
; GFX1200-NEXT: ret i32 [[VAL]]
;
-; GFX1250-LABEL: define i32 @system_atomic_load_unordered_i32(
-; GFX1250-SAME: ptr addrspace(5) [[ADDR:%.*]]) #[[ATTR0]] {
-; GFX1250-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
-; GFX1250-NEXT: [[VAL:%.*]] = load atomic i32, ptr [[SCRATCH_ASCAST]] unordered, align 4
-; GFX1250-NEXT: ret i32 [[VAL]]
+; GFX1250-NOGAS-LABEL: define i32 @system_atomic_load_unordered_i32(
+; GFX1250-NOGAS-SAME: ptr addrspace(5) [[ADDR:%.*]]) #[[ATTR0]] {
+; GFX1250-NOGAS-NEXT: [[VAL:%.*]] = load i32, ptr addrspace(5) [[ADDR]], align 4
+; GFX1250-NOGAS-NEXT: ret i32 [[VAL]]
+;
+; GFX1250-GAS-LABEL: define i32 @system_atomic_load_unordered_i32(
+; GFX1250-GAS-SAME: ptr addrspace(5) [[ADDR:%.*]]) #[[ATTR0]] {
+; GFX1250-GAS-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
+; GFX1250-GAS-NEXT: [[VAL:%.*]] = load atomic i32, ptr [[SCRATCH_ASCAST]] unordered, align 4
+; GFX1250-GAS-NEXT: ret i32 [[VAL]]
;
%val = load atomic i32, ptr addrspace(5) %addr unordered, align 4
ret i32 %val
@@ -104,11 +135,16 @@ define i32 @system_atomic_load_acquire_i32(ptr addrspace(5) %addr) {
; GFX1200-NEXT: [[VAL:%.*]] = load i32, ptr addrspace(5) [[ADDR]], align 4
; GFX1200-NEXT: ret i32 [[VAL]]
;
-; GFX1250-LABEL: define i32 @system_atomic_load_acquire_i32(
-; GFX1250-SAME: ptr addrspace(5) [[ADDR:%.*]]) #[[ATTR0]] {
-; GFX1250-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
-; GFX1250-NEXT: [[VAL:%.*]] = load atomic i32, ptr [[SCRATCH_ASCAST]] acquire, align 4
-; GFX1250-NEXT: ret i32 [[VAL]]
+; GFX1250-NOGAS-LABEL: define i32 @system_atomic_load_acquire_i32(
+; GFX1250-NOGAS-SAME: ptr addrspace(5) [[ADDR:%.*]]) #[[ATTR0]] {
+; GFX1250-NOGAS-NEXT: [[VAL:%.*]] = load i32, ptr addrspace(5) [[ADDR]], align 4
+; GFX1250-NOGAS-NEXT: ret i32 [[VAL]]
+;
+; GFX1250-GAS-LABEL: define i32 @system_atomic_load_acquire_i32(
+; GFX1250-GAS-SAME: ptr addrspace(5) [[ADDR:%.*]]) #[[ATTR0]] {
+; GFX1250-GAS-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
+; GFX1250-GAS-NEXT: [[VAL:%.*]] = load atomic i32, ptr [[SCRATCH_ASCAST]] acquire, align 4
+; GFX1250-GAS-NEXT: ret i32 [[VAL]]
;
%val = load atomic i32, ptr addrspace(5) %addr acquire, align 4
ret i32 %val
@@ -120,11 +156,16 @@ define i32 @workgroup_atomic_load_acquire_i32(ptr addrspace(5) %addr) {
; GFX1200-NEXT: [[VAL:%.*]] = load i32, ptr addrspace(5) [[ADDR]], align 4
; GFX1200-NEXT: ret i32 [[VAL]]
;
-; GFX1250-LABEL: define i32 @workgroup_atomic_load_acquire_i32(
-; GFX1250-SAME: ptr addrspace(5) [[ADDR:%.*]]) #[[ATTR0]] {
-; GFX1250-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
-; GFX1250-NEXT: [[VAL:%.*]] = load atomic i32, ptr [[SCRATCH_ASCAST]] syncscope("workgroup") acquire, align 4
-; GFX1250-NEXT: ret i32 [[VAL]]
+; GFX1250-NOGAS-LABEL: define i32 @workgroup_atomic_load_acquire_i32(
+; GFX1250-NOGAS-SAME: ptr addrspace(5) [[ADDR:%.*]]) #[[ATTR0]] {
+; GFX1250-NOGAS-NEXT: [[VAL:%.*]] = load i32, ptr addrspace(5) [[ADDR]], align 4
+; GFX1250-NOGAS-NEXT: ret i32 [[VAL]]
+;
+; GFX1250-GAS-LABEL: define i32 @workgroup_atomic_load_acquire_i32(
+; GFX1250-GAS-SAME: ptr addrspace(5) [[ADDR:%.*]]) #[[ATTR0]] {
+; GFX1250-GAS-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
+; GFX1250-GAS-NEXT: [[VAL:%.*]] = load atomic i32, ptr [[SCRATCH_ASCAST]] syncscope("workgroup") acquire, align 4
+; GFX1250-GAS-NEXT: ret i32 [[VAL]]
;
%val = load atomic i32, ptr addrspace(5) %addr syncscope("workgroup") acquire, align 4
ret i32 %val
@@ -142,12 +183,23 @@ define i32 @system_atomic_cmpxchg_acq_rel_acquire_i32(ptr addrspace(5) %addr, i3
; GFX1200-NEXT: [[RES:%.*]] = extractvalue { i32, i1 } [[TMP5]], 0
; GFX1200-NEXT: ret i32 [[RES]]
;
-; GFX1250-LABEL: define i32 @system_atomic_cmpxchg_acq_rel_acquire_i32(
-; GFX1250-SAME: ptr addrspace(5) [[ADDR:%.*]], i32 [[OLD:%.*]], i32 [[IN:%.*]]) #[[ATTR0]] {
-; GFX1250-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
-; GFX1250-NEXT: [[VAL:%.*]] = cmpxchg volatile ptr [[SCRATCH_ASCAST]], i32 [[OLD]], i32 [[IN]] acq_rel acquire, align 4, !nontemporal [[META1]]
-; GFX1250-NEXT: [[RES:%.*]] = extractvalue { i32, i1 } [[VAL]], 0
-; GFX1250-NEXT: ret i32 [[RES]]
+; GFX1250-NOGAS-LABEL: define i32 @system_atomic_cmpxchg_acq_rel_acquire_i32(
+; GFX1250-NOGAS-SAME: ptr addrspace(5) [[ADDR:%.*]], i32 [[OLD:%.*]], i32 [[IN:%.*]]) #[[ATTR0]] {
+; GFX1250-NOGAS-NEXT: [[TMP1:%.*]] = load i32, ptr addrspace(5) [[ADDR]], align 4
+; GFX1250-NOGAS-NEXT: [[TMP2:%.*]] = icmp eq i32 [[TMP1]], [[OLD]]
+; GFX1250-NOGAS-NEXT: [[TMP3:%.*]] = select i1 [[TMP2]], i32 [[IN]], i32 [[TMP1]]
+; GFX1250-NOGAS-NEXT: store i32 [[TMP3]], ptr addrspace(5) [[ADDR]], align 4
+; GFX1250-NOGAS-NEXT: [[TMP4:%.*]] = insertvalue { i32, i1 } poison, i32 [[TMP1]], 0
+; GFX1250-NOGAS-NEXT: [[TMP5:%.*]] = insertvalue { i32, i1 } [[TMP4]], i1 [[TMP2]], 1
+; GFX1250-NOGAS-NEXT: [[RES:%.*]] = extractvalue { i32, i1 } [[TMP5]], 0
+; GFX1250-NOGAS-NEXT: ret i32 [[RES]]
+;
+; GFX1250-GAS-LABEL: define i32 @system_atomic_cmpxchg_acq_rel_acquire_i32(
+; GFX1250-GAS-SAME: ptr addrspace(5) [[ADDR:%.*]], i32 [[OLD:%.*]], i32 [[IN:%.*]]) #[[ATTR0]] {
+; GFX1250-GAS-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
+; GFX1250-GAS-NEXT: [[VAL:%.*]] = cmpxchg volatile ptr [[SCRATCH_ASCAST]], i32 [[OLD]], i32 [[IN]] acq_rel acquire, align 4, !nontemporal [[META1]]
+; GFX1250-GAS-NEXT: [[RES:%.*]] = extractvalue { i32, i1 } [[VAL]], 0
+; GFX1250-GAS-NEXT: ret i32 [[RES]]
;
%val = cmpxchg volatile ptr addrspace(5) %addr, i32 %old, i32 %in acq_rel acquire, !nontemporal !0
%res = extractvalue { i32, i1 } %val, 0
@@ -161,11 +213,17 @@ define i32 @system_atomicrmw_xchg_acq_rel_i32(ptr addrspace(5) %addr, i32 %in) {
; GFX1200-NEXT: store i32 [[IN]], ptr addrspace(5) [[ADDR]], align 4
; GFX1200-NEXT: ret i32 [[TMP1]]
;
-; GFX1250-LABEL: define i32 @system_atomicrmw_xchg_acq_rel_i32(
-; GFX1250-SAME: ptr addrspace(5) [[ADDR:%.*]], i32 [[IN:%.*]]) #[[ATTR0]] {
-; GFX1250-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
-; GFX1250-NEXT: [[VAL:%.*]] = atomicrmw volatile xchg ptr [[SCRATCH_ASCAST]], i32 [[IN]] acq_rel, align 4
-; GFX1250-NEXT: ret i32 [[VAL]]
+; GFX1250-NOGAS-LABEL: define i32 @system_atomicrmw_xchg_acq_rel_i32(
+; GFX1250-NOGAS-SAME: ptr addrspace(5) [[ADDR:%.*]], i32 [[IN:%.*]]) #[[ATTR0]] {
+; GFX1250-NOGAS-NEXT: [[TMP1:%.*]] = load i32, ptr addrspace(5) [[ADDR]], align 4
+; GFX1250-NOGAS-NEXT: store i32 [[IN]], ptr addrspace(5) [[ADDR]], align 4
+; GFX1250-NOGAS-NEXT: ret i32 [[TMP1]]
+;
+; GFX1250-GAS-LABEL: define i32 @system_atomicrmw_xchg_acq_rel_i32(
+; GFX1250-GAS-SAME: ptr addrspace(5) [[ADDR:%.*]], i32 [[IN:%.*]]) #[[ATTR0]] {
+; GFX1250-GAS-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
+; GFX1250-GAS-NEXT: [[VAL:%.*]] = atomicrmw volatile xchg ptr [[SCRATCH_ASCAST]], i32 [[IN]] acq_rel, align 4
+; GFX1250-GAS-NEXT: ret i32 [[VAL]]
;
%val = atomicrmw volatile xchg ptr addrspace(5) %addr, i32 %in acq_rel
ret i32 %val
@@ -178,11 +236,17 @@ define i16 @system_atomicrmw_xchg_acq_rel_i16(ptr addrspace(5) %addr, i16 %in) {
; GFX1200-NEXT: store i16 [[IN]], ptr addrspace(5) [[ADDR]], align 2
; GFX1200-NEXT: ret i16 [[TMP1]]
;
-; GFX1250-LABEL: define i16 @system_atomicrmw_xchg_acq_rel_i16(
-; GFX1250-SAME: ptr addrspace(5) [[ADDR:%.*]], i16 [[IN:%.*]]) #[[ATTR0]] {
-; GFX1250-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
-; GFX1250-NEXT: [[VAL:%.*]] = atomicrmw volatile xchg ptr [[SCRATCH_ASCAST]], i16 [[IN]] acq_rel, align 2
-; GFX1250-NEXT: ret i16 [[VAL]]
+; GFX1250-NOGAS-LABEL: define i16 @system_atomicrmw_xchg_acq_rel_i16(
+; GFX1250-NOGAS-SAME: ptr addrspace(5) [[ADDR:%.*]], i16 [[IN:%.*]]) #[[ATTR0]] {
+; GFX1250-NOGAS-NEXT: [[TMP1:%.*]] = load i16, ptr addrspace(5) [[ADDR]], align 2
+; GFX1250-NOGAS-NEXT: store i16 [[IN]], ptr addrspace(5) [[ADDR]], align 2
+; GFX1250-NOGAS-NEXT: ret i16 [[TMP1]]
+;
+; GFX1250-GAS-LABEL: define i16 @system_atomicrmw_xchg_acq_rel_i16(
+; GFX1250-GAS-SAME: ptr addrspace(5) [[ADDR:%.*]], i16 [[IN:%.*]]) #[[ATTR0]] {
+; GFX1250-GAS-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
+; GFX1250-GAS-NEXT: [[VAL:%.*]] = atomicrmw volatile xchg ptr [[SCRATCH_ASCAST]], i16 [[IN]] acq_rel, align 2
+; GFX1250-GAS-NEXT: ret i16 [[VAL]]
;
%val = atomicrmw volatile xchg ptr addrspace(5) %addr, i16 %in acq_rel
ret i16 %val
@@ -196,11 +260,18 @@ define half @system_atomicrmw_fmax_acq_rel_half(ptr addrspace(5) %addr, half %in
; GFX1200-NEXT: store half [[TMP2]], ptr addrspace(5) [[ADDR]], align 2
; GFX1200-NEXT: ret half [[TMP1]]
;
-; GFX1250-LABEL: define half @system_atomicrmw_fmax_acq_rel_half(
-; GFX1250-SAME: ptr addrspace(5) [[ADDR:%.*]], half [[IN:%.*]]) #[[ATTR0]] {
-; GFX1250-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
-; GFX1250-NEXT: [[VAL:%.*]] = atomicrmw volatile fmax ptr [[SCRATCH_ASCAST]], half [[IN]] acq_rel, align 2
-; GFX1250-NEXT: ret half [[VAL]]
+; GFX1250-NOGAS-LABEL: define half @system_atomicrmw_fmax_acq_rel_half(
+; GFX1250-NOGAS-SAME: ptr addrspace(5) [[ADDR:%.*]], half [[IN:%.*]]) #[[ATTR0]] {
+; GFX1250-NOGAS-NEXT: [[TMP1:%.*]] = load half, ptr addrspace(5) [[ADDR]], align 2
+; GFX1250-NOGAS-NEXT: [[TMP2:%.*]] = call half @llvm.maxnum.f16(half [[TMP1]], half [[IN]])
+; GFX1250-NOGAS-NEXT: store half [[TMP2]], ptr addrspace(5) [[ADDR]], align 2
+; GFX1250-NOGAS-NEXT: ret half [[TMP1]]
+;
+; GFX1250-GAS-LABEL: define half @system_atomicrmw_fmax_acq_rel_half(
+; GFX1250-GAS-SAME: ptr addrspace(5) [[ADDR:%.*]], half [[IN:%.*]]) #[[ATTR0]] {
+; GFX1250-GAS-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
+; GFX1250-GAS-NEXT: [[VAL:%.*]] = atomicrmw volatile fmax ptr [[SCRATCH_ASCAST]], half [[IN]] acq_rel, align 2
+; GFX1250-GAS-NEXT: ret half [[VAL]]
;
%val = atomicrmw volatile fmax ptr addrspace(5) %addr, half %in acq_rel
ret half %val
@@ -214,11 +285,18 @@ define float @system_atomicrmw_fminimum_acq_rel_float(ptr addrspace(5) %addr, fl
; GFX1200-NEXT: store float [[TMP2]], ptr addrspace(5) [[ADDR]], align 4
; GFX1200-NEXT: ret float [[TMP1]]
;
-; GFX1250-LABEL: define float @system_atomicrmw_fminimum_acq_rel_float(
-; GFX1250-SAME: ptr addrspace(5) [[ADDR:%.*]], float [[IN:%.*]]) #[[ATTR0]] {
-; GFX1250-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
-; GFX1250-NEXT: [[VAL:%.*]] = atomicrmw volatile fminimum ptr [[SCRATCH_ASCAST]], float [[IN]] acq_rel, align 4, !nontemporal [[META1]]
-; GFX1250-NEXT: ret float [[VAL]]
+; GFX1250-NOGAS-LABEL: define float @system_atomicrmw_fminimum_acq_rel_float(
+; GFX1250-NOGAS-SAME: ptr addrspace(5) [[ADDR:%.*]], float [[IN:%.*]]) #[[ATTR0]] {
+; GFX1250-NOGAS-NEXT: [[TMP1:%.*]] = load float, ptr addrspace(5) [[ADDR]], align 4
+; GFX1250-NOGAS-NEXT: [[TMP2:%.*]] = call float @llvm.minimum.f32(float [[TMP1]], float [[IN]])
+; GFX1250-NOGAS-NEXT: store float [[TMP2]], ptr addrspace(5) [[ADDR]], align 4
+; GFX1250-NOGAS-NEXT: ret float [[TMP1]]
+;
+; GFX1250-GAS-LABEL: define float @system_atomicrmw_fminimum_acq_rel_float(
+; GFX1250-GAS-SAME: ptr addrspace(5) [[ADDR:%.*]], float [[IN:%.*]]) #[[ATTR0]] {
+; GFX1250-GAS-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
+; GFX1250-GAS-NEXT: [[VAL:%.*]] = atomicrmw volatile fminimum ptr [[SCRATCH_ASCAST]], float [[IN]] acq_rel, align 4, !nontemporal [[META1]]
+; GFX1250-GAS-NEXT: ret float [[VAL]]
;
%val = atomicrmw volatile fminimum ptr addrspace(5) %addr, float %in acq_rel, !nontemporal !0
ret float %val
@@ -230,6 +308,11 @@ define float @system_atomicrmw_fminimum_acq_rel_float(ptr addrspace(5) %addr, fl
; GFX1200: [[META0]] = !{i32 1}
; GFX1200: [[META1]] = !{}
;.
-; GFX1250: [[META0]] = !{i32 1}
-; GFX1250: [[META1]] = !{}
+; GFX1250-NOGAS: [[META0]] = !{i32 1}
+; GFX1250-NOGAS: [[META1]] = !{}
+;.
+; GFX1250-GAS: [[META0]] = !{i32 1}
+; GFX1250-GAS: [[META1]] = !{}
;.
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GFX1250: {{.*}}
>From 1d88b835190600ce37f9266318e23991fbfa57e5 Mon Sep 17 00:00:00 2001
From: pvanhout <pierre.vanhoutryve at amd.com>
Date: Wed, 1 Apr 2026 13:26:44 +0200
Subject: [PATCH 2/3] Comments
---
llvm/docs/AMDGPUUsage.rst | 158 ++++++++++--------
llvm/lib/Target/AMDGPU/AMDGPU.td | 8 +-
llvm/lib/Target/AMDGPU/GCNSubtarget.cpp | 9 +
llvm/lib/Target/AMDGPU/GCNSubtarget.h | 5 +-
.../AMDGPU/memory-legalizer-private-agent.ll | 2 +-
.../memory-legalizer-private-cluster.ll | 2 +-
.../memory-legalizer-private-lastuse.ll | 2 +-
.../memory-legalizer-private-nontemporal.ll | 2 +-
.../memory-legalizer-private-singlethread.ll | 2 +-
.../AMDGPU/memory-legalizer-private-system.ll | 2 +-
.../memory-legalizer-private-volatile.ll | 2 +-
.../memory-legalizer-private-wavefront.ll | 2 +-
.../memory-legalizer-private-workgroup.ll | 2 +-
.../AMDGPU/expand-atomic-private-gas.ll | 2 +-
14 files changed, 105 insertions(+), 95 deletions(-)
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index 90a792922f59b..2a350edcb786f 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -559,21 +559,29 @@ Every processor supports every OS ABI (see :ref:`amdgpu-os`) with the following
work-item
IDs
- ``gfx1250`` ``amdgcn`` APU - globally- - Architected *TBA*
- addressable- flat
- scratch scratch .. TODO::
+ ``gfx1250`` ``amdgcn`` APU - Architected *TBA*
+ flat
+ scratch .. TODO::
- Packed
work-item Add product
IDs names.
+ - Globally
+ Addressable
+ Scratch
+ :sup:`1`
- Workgroup
Clusters
- ``gfx1251`` ``amdgcn`` APU - globally- - Architected *TBA*
- addressable- flat
- scratch scratch .. TODO::
+ ``gfx1251`` ``amdgcn`` APU - Architected *TBA*
+ flat
+ scratch .. TODO::
- Packed
work-item Add product
IDs names.
+ - Globally
+ Addressable
+ Scratch
+ :sup:`1`
- Workgroup
Clusters
@@ -588,6 +596,9 @@ Every processor supports every OS ABI (see :ref:`amdgpu-os`) with the following
=========== =============== ============ ===== ================= =============== =============== ======================
+:sup:`1`: Globally addressable scratch (GAS) is not enabled by default, but can be enabled using ``-amdgpu-globally-addressable-scratch``.
+There is currently no front-end support for GAS, and thus back-end support is limited.
+
Generic processors allow execution of a single code object on any of the processors that
it supports. Such code objects may not perform as well as those for the non-generic processors.
@@ -765,70 +776,64 @@ For example:
.. table:: AMDGPU Target Features
:name: amdgpu-target-features-table
- ============================= ============================ ==================================================
- Target Feature Clang Option to Control Description
+ =============== ============================ ==================================================
+ Target Feature Clang Option to Control Description
Name
- ============================= ============================ ==================================================
- cumode - ``-m[no-]cumode`` Control the wavefront execution mode used
- when generating code for kernels. When disabled
- native WGP wavefront execution mode is used,
- when enabled CU wavefront execution mode is used
- (see :ref:`amdgpu-amdhsa-memory-model`).
-
- sramecc - ``-mcpu`` If specified, generate code that can only be
- - ``--offload-arch`` loaded and executed in a process that has a
- matching setting for SRAMECC.
-
- If not specified for code object V2 to V3, generate
- code that can be loaded and executed in a process
- with SRAMECC enabled.
-
- If not specified for code object V4 or above, generate
- code that can be loaded and executed in a process
- with either setting of SRAMECC.
-
- tgsplit ``-m[no-]tgsplit`` Enable/disable generating code that assumes
- work-groups are launched in threadgroup split mode.
- When enabled the waves of a work-group may be
- launched in different CUs.
-
- wavefrontsize64 - ``-m[no-]wavefrontsize64`` Control the wavefront size used when
- generating code for kernels. When disabled
- native wavefront size 32 is used, when enabled
- wavefront size 64 is used.
-
- xnack - ``-mcpu`` If specified, generate code that can only be
- - ``--offload-arch`` loaded and executed in a process that has a
- matching setting for XNACK replay.
-
- If not specified for code object V2 to V3, generate
- code that can be loaded and executed in a process
- with XNACK replay enabled.
-
- If not specified for code object V4 or above, generate
- code that can be loaded and executed in a process
- with either setting of XNACK replay.
-
- XNACK replay can be used for demand paging and
- page migration. If enabled in the device, then if
- a page fault occurs the code may execute
- incorrectly unless generated with XNACK replay
- enabled, or generated for code object V4 or above without
- specifying XNACK replay. Executing code that was
- generated with XNACK replay enabled, or generated
- for code object V4 or above without specifying XNACK replay,
- on a device that does not have XNACK replay
- enabled will execute correctly but may be less
- performant than code generated for XNACK replay
- disabled.
-
- globally-addressable-scratch - ``--offload-arch`` When enabled, scratch (private) memory can be shared
- between threads without triggering undefined behavior.
- Disabled by default as this may incur a performance penalty
- because the compiler can no longer assume private memory is
- thread-local when this is enabled.
-
- ============================= ============================ ==================================================
+ =============== ============================ ==================================================
+ cumode - ``-m[no-]cumode`` Control the wavefront execution mode used
+ when generating code for kernels. When disabled
+ native WGP wavefront execution mode is used,
+ when enabled CU wavefront execution mode is used
+ (see :ref:`amdgpu-amdhsa-memory-model`).
+
+ sramecc - ``-mcpu`` If specified, generate code that can only be
+ - ``--offload-arch`` loaded and executed in a process that has a
+ matching setting for SRAMECC.
+
+ If not specified for code object V2 to V3, generate
+ code that can be loaded and executed in a process
+ with SRAMECC enabled.
+
+ If not specified for code object V4 or above, generate
+ code that can be loaded and executed in a process
+ with either setting of SRAMECC.
+
+ tgsplit ``-m[no-]tgsplit`` Enable/disable generating code that assumes
+ work-groups are launched in threadgroup split mode.
+ When enabled the waves of a work-group may be
+ launched in different CUs.
+
+ wavefrontsize64 - ``-m[no-]wavefrontsize64`` Control the wavefront size used when
+ generating code for kernels. When disabled
+ native wavefront size 32 is used, when enabled
+ wavefront size 64 is used.
+
+ xnack - ``-mcpu`` If specified, generate code that can only be
+ - ``--offload-arch`` loaded and executed in a process that has a
+ matching setting for XNACK replay.
+
+ If not specified for code object V2 to V3, generate
+ code that can be loaded and executed in a process
+ with XNACK replay enabled.
+
+ If not specified for code object V4 or above, generate
+ code that can be loaded and executed in a process
+ with either setting of XNACK replay.
+
+ XNACK replay can be used for demand paging and
+ page migration. If enabled in the device, then if
+ a page fault occurs the code may execute
+ incorrectly unless generated with XNACK replay
+ enabled, or generated for code object V4 or above without
+ specifying XNACK replay. Executing code that was
+ generated with XNACK replay enabled, or generated
+ for code object V4 or above without specifying XNACK replay,
+ on a device that does not have XNACK replay
+ enabled will execute correctly but may be less
+ performant than code generated for XNACK replay
+ disabled.
+
+ =============== ============================ ==================================================
.. _amdgpu-module-flags:
@@ -1077,8 +1082,8 @@ supported for the ``amdgcn`` target.
access is not supported except by flat and scratch instructions in
GFX9-GFX11.
- On targets without ``globally-addressable-scratch``, or if the feature is disabled, code that
- manipulates the stack values in other lanes of a wavefront, such as by
+ If globally addressable scratch is not supported by the target, or is not enabled,
+ code that manipulates the stack values in other lanes of a wavefront, such as by
``addrspacecast``-ing stack pointers to generic ones and taking offsets that reach other
lanes or by explicitly constructing the scratch buffer descriptor, triggers undefined
behavior when it modifies the scratch values of other lanes. The compiler may assume
@@ -7186,9 +7191,14 @@ explicitly track the completion of these instructions before using their
side-effects.
Private address space uses ``buffer_load/store`` using the scratch V#
-(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX11). Since only a single thread
-is accessing the memory, atomic memory orderings are not meaningful, and all
-accesses are treated as non-atomic.
+(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX11).
+
+* On targets where globally addressable scratch is not supported, or not enabled,
+ Only a single thread can access private memory. Thus, atomic memory orderings
+ are not meaningful, and all private accesses are treated as non-atomic.
+* On targets where globally addressable scratch is supported **and** enabled, private
+ atomic accesses are converted into atomics in the generic (``flat``) address space.
+ All properties of the atomic (atomic ordering, volatility, alignment, etc.) are preserved.
Constant address space uses ``buffer/global_load`` instructions (or equivalent
scalar memory instructions). Since the constant address space contents do not
diff --git a/llvm/lib/Target/AMDGPU/AMDGPU.td b/llvm/lib/Target/AMDGPU/AMDGPU.td
index d662b30d63cec..a6fcb3ab23b80 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPU.td
+++ b/llvm/lib/Target/AMDGPU/AMDGPU.td
@@ -1330,13 +1330,7 @@ defm XF32Insts : AMDGPUSubtargetFeature<"xf32-insts",
def FeatureGloballyAddressableScratchSupport : SubtargetFeature<"globally-addressable-scratch-support",
"HasGloballyAddressableScratchSupport",
"true",
- "Hardware supports globally-addressable-scratch"
->;
-
-def FeatureGloballyAddressableScratch : SubtargetFeature<"globally-addressable-scratch",
- "EnableGloballyAddressableScratch",
- "true",
- "FLAT instructions can access scratch memory from any thread in any wave"
+ "Hardware supports Globally Addressable Scratch"
>;
// Enable the use of SCRATCH_STORE/LOAD_BLOCK instructions for saving and
diff --git a/llvm/lib/Target/AMDGPU/GCNSubtarget.cpp b/llvm/lib/Target/AMDGPU/GCNSubtarget.cpp
index 622a28312fa21..287a586409e00 100644
--- a/llvm/lib/Target/AMDGPU/GCNSubtarget.cpp
+++ b/llvm/lib/Target/AMDGPU/GCNSubtarget.cpp
@@ -52,6 +52,10 @@ static cl::opt<unsigned>
cl::desc("Number of addresses from which to enable MIMG NSA."),
cl::init(2), cl::Hidden);
+static cl::opt<bool> EnableGloballyAddressableScratch(
+ "amdgpu-globally-addressable-scratch",
+ cl::desc("Enable Globally Addressable Scratch"), cl::init(false));
+
GCNSubtarget::~GCNSubtarget() = default;
GCNSubtarget &GCNSubtarget::initializeSubtargetDependencies(const Triple &TT,
@@ -847,6 +851,11 @@ unsigned GCNSubtarget::getNSAThreshold(const MachineFunction &MF) const {
return NSAThreshold;
}
+bool GCNSubtarget::isGloballyAddressableScratchEnabled() const {
+ return HasGloballyAddressableScratchSupport &&
+ EnableGloballyAddressableScratch;
+}
+
GCNUserSGPRUsageInfo::GCNUserSGPRUsageInfo(const Function &F,
const GCNSubtarget &ST)
: ST(ST) {
diff --git a/llvm/lib/Target/AMDGPU/GCNSubtarget.h b/llvm/lib/Target/AMDGPU/GCNSubtarget.h
index b4b69469ad650..8b56afc93ad9d 100644
--- a/llvm/lib/Target/AMDGPU/GCNSubtarget.h
+++ b/llvm/lib/Target/AMDGPU/GCNSubtarget.h
@@ -1037,10 +1037,7 @@ class GCNSubtarget final : public AMDGPUGenSubtargetInfo,
return getGeneration() >= GFX10 || isTgSplitEnabled();
}
- bool isGloballyAddressableScratchEnabled() const {
- return HasGloballyAddressableScratchSupport &&
- EnableGloballyAddressableScratch;
- }
+ bool isGloballyAddressableScratchEnabled() const;
};
class GCNUserSGPRUsageInfo {
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-agent.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-agent.ll
index e53ea71e5cc0b..fb6e82f864a5a 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-agent.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-agent.ll
@@ -13,7 +13,7 @@
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 < %s | FileCheck --check-prefixes=GFX12-WGP %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX12-CU %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 < %s | FileCheck --check-prefixes=GFX1250,GFX1250-NOGAS %s
-; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 -mattr=+globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 -amdgpu-globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
define amdgpu_kernel void @private_agent_unordered_load(
; GFX6-LABEL: private_agent_unordered_load:
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-cluster.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-cluster.ll
index 13e5f3a84e3a2..1bff0e551a8c8 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-cluster.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-cluster.ll
@@ -13,7 +13,7 @@
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 < %s | FileCheck --check-prefixes=GFX12-WGP %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX12-CU %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 < %s | FileCheck --check-prefixes=GFX1250,GFX1250-NOGAS %s
-; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 -mattr=+globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 -amdgpu-globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
define amdgpu_kernel void @private_cluster_unordered_load(
; GFX6-LABEL: private_cluster_unordered_load:
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-lastuse.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-lastuse.ll
index 26134c6b14d77..95bb2d2441243 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-lastuse.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-lastuse.ll
@@ -2,7 +2,7 @@
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 < %s | FileCheck --check-prefix=GFX12 %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 -mattr=+cumode < %s | FileCheck --check-prefix=GFX12 %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 < %s | FileCheck --check-prefixes=GFX1250,GFX1250-NOGAS %s
-; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 -mattr=+globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 -amdgpu-globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
define amdgpu_kernel void @private_last_use_load_0(ptr addrspace(5) %in, ptr addrspace(1) %out) {
; GFX12-LABEL: private_last_use_load_0:
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-nontemporal.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-nontemporal.ll
index 183d5d4f79c89..255cbaa0c403c 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-nontemporal.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-nontemporal.ll
@@ -13,7 +13,7 @@
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 < %s | FileCheck --check-prefixes=GFX12-WGP %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX12-CU %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 < %s | FileCheck --check-prefixes=GFX1250,GFX1250-NOGAS %s
-; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 -mattr=+globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 -amdgpu-globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
define amdgpu_kernel void @private_nontemporal_load_0(
; GFX6-LABEL: private_nontemporal_load_0:
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-singlethread.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-singlethread.ll
index f54b1399a11b9..6b90662e26475 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-singlethread.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-singlethread.ll
@@ -13,7 +13,7 @@
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 < %s | FileCheck --check-prefixes=GFX12-WGP %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX12-CU %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 < %s | FileCheck --check-prefixes=GFX1250,GFX1250-NOGAS %s
-; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 -mattr=+globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 -amdgpu-globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
define amdgpu_kernel void @private_singlethread_unordered_load(
; GFX6-LABEL: private_singlethread_unordered_load:
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-system.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-system.ll
index 487e54f55ad0b..08a2e7b287eec 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-system.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-system.ll
@@ -13,7 +13,7 @@
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 < %s | FileCheck --check-prefixes=GFX12-WGP %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX12-CU %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 < %s | FileCheck --check-prefixes=GFX1250,GFX1250-NOGAS %s
-; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 -mattr=+globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 -amdgpu-globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
define amdgpu_kernel void @private_system_unordered_load(
; GFX6-LABEL: private_system_unordered_load:
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-volatile.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-volatile.ll
index d65875653770f..7d533b35f7bcd 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-volatile.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-volatile.ll
@@ -9,7 +9,7 @@
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 < %s | FileCheck --check-prefixes=GFX12-WGP %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX12-CU %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 < %s | FileCheck --check-prefixes=GFX1250,GFX1250-NOGAS %s
-; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 -mattr=+globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 -amdgpu-globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
define amdgpu_kernel void @private_volatile_load_0(
; GFX6-LABEL: private_volatile_load_0:
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-wavefront.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-wavefront.ll
index 5434df9a8e86e..91f559e66bfee 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-wavefront.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-wavefront.ll
@@ -13,7 +13,7 @@
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 < %s | FileCheck --check-prefixes=GFX12-WGP %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX12-CU %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 < %s | FileCheck --check-prefixes=GFX1250,GFX1250-NOGAS %s
-; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 -mattr=+globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 -amdgpu-globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
define amdgpu_kernel void @private_wavefront_unordered_load(
; GFX6-LABEL: private_wavefront_unordered_load:
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-workgroup.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-workgroup.ll
index 994539720e878..5f1845fdd024f 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-workgroup.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-workgroup.ll
@@ -13,7 +13,7 @@
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 < %s | FileCheck --check-prefixes=GFX12-WGP %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1200 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX12-CU %s
; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 < %s | FileCheck --check-prefixes=GFX1250,GFX1250-NOGAS %s
-; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 -mattr=+globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -O0 -mcpu=gfx1250 -amdgpu-globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
define amdgpu_kernel void @private_workgroup_unordered_load(
; GFX6-LABEL: private_workgroup_unordered_load:
diff --git a/llvm/test/Transforms/AtomicExpand/AMDGPU/expand-atomic-private-gas.ll b/llvm/test/Transforms/AtomicExpand/AMDGPU/expand-atomic-private-gas.ll
index 00c29b5875217..6e9d13121ab94 100644
--- a/llvm/test/Transforms/AtomicExpand/AMDGPU/expand-atomic-private-gas.ll
+++ b/llvm/test/Transforms/AtomicExpand/AMDGPU/expand-atomic-private-gas.ll
@@ -1,7 +1,7 @@
; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 5
; RUN: opt -S -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1200 -passes='require<libcall-lowering-info>,atomic-expand' %s | FileCheck -check-prefixes=GFX1200 %s
; RUN: opt -S -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1250 -passes='require<libcall-lowering-info>,atomic-expand' %s | FileCheck -check-prefixes=GFX1250,GFX1250-NOGAS %s
-; RUN: opt -S -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1250 -mattr=+globally-addressable-scratch -passes='require<libcall-lowering-info>,atomic-expand' %s | FileCheck -check-prefixes=GFX1250,GFX1250-GAS %s
+; RUN: opt -S -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1250 -amdgpu-globally-addressable-scratch -passes='require<libcall-lowering-info>,atomic-expand' %s | FileCheck -check-prefixes=GFX1250,GFX1250-GAS %s
define void @system_atomic_store_unordered_float(ptr addrspace(5) %addr, float %val) {
; GFX1200-LABEL: define void @system_atomic_store_unordered_float(
>From 6c33fec8b27cb728b622fc7be9cd6bc477781796 Mon Sep 17 00:00:00 2001
From: pvanhout <pierre.vanhoutryve at amd.com>
Date: Mon, 20 Apr 2026 10:02:43 +0200
Subject: [PATCH 3/3] Rebase
---
.../AMDGPU/memory-legalizer-private-agent.ll | 2850 ++++++++++------
.../memory-legalizer-private-cluster.ll | 2850 ++++++++++------
.../memory-legalizer-private-singlethread.ll | 2882 +++++++++++------
.../AMDGPU/memory-legalizer-private-system.ll | 2722 +++++++++++-----
.../memory-legalizer-private-wavefront.ll | 2882 +++++++++++------
.../memory-legalizer-private-workgroup.ll | 2882 +++++++++++------
6 files changed, 11622 insertions(+), 5446 deletions(-)
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-agent.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-agent.ll
index fb6e82f864a5a..7b29528177fdf 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-agent.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-agent.ll
@@ -229,34 +229,42 @@ define amdgpu_kernel void @private_agent_unordered_load(
; GFX1250-NOGAS-LABEL: private_agent_unordered_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_agent_unordered_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -265,6 +273,7 @@ define amdgpu_kernel void @private_agent_unordered_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -488,34 +497,42 @@ define amdgpu_kernel void @private_agent_monotonic_load(
; GFX1250-NOGAS-LABEL: private_agent_monotonic_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_agent_monotonic_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -524,6 +541,7 @@ define amdgpu_kernel void @private_agent_monotonic_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_DEV
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -747,34 +765,42 @@ define amdgpu_kernel void @private_agent_acquire_load(
; GFX1250-NOGAS-LABEL: private_agent_acquire_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_agent_acquire_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -785,6 +811,7 @@ define amdgpu_kernel void @private_agent_acquire_load(
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -1008,34 +1035,42 @@ define amdgpu_kernel void @private_agent_seq_cst_load(
; GFX1250-NOGAS-LABEL: private_agent_seq_cst_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_agent_seq_cst_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -1048,6 +1083,7 @@ define amdgpu_kernel void @private_agent_seq_cst_load(
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -1255,10 +1291,14 @@ define amdgpu_kernel void @private_agent_unordered_store(
; GFX1250-NOGAS-LABEL: private_agent_unordered_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -1266,22 +1306,26 @@ define amdgpu_kernel void @private_agent_unordered_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -1496,10 +1540,14 @@ define amdgpu_kernel void @private_agent_monotonic_store(
; GFX1250-NOGAS-LABEL: private_agent_monotonic_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -1507,22 +1555,26 @@ define amdgpu_kernel void @private_agent_monotonic_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -1737,10 +1789,14 @@ define amdgpu_kernel void @private_agent_release_store(
; GFX1250-NOGAS-LABEL: private_agent_release_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -1748,22 +1804,26 @@ define amdgpu_kernel void @private_agent_release_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -1983,10 +2043,14 @@ define amdgpu_kernel void @private_agent_seq_cst_store(
; GFX1250-NOGAS-LABEL: private_agent_seq_cst_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -1994,22 +2058,26 @@ define amdgpu_kernel void @private_agent_seq_cst_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -2263,8 +2331,14 @@ define amdgpu_kernel void @private_agent_monotonic_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -2272,29 +2346,34 @@ define amdgpu_kernel void @private_agent_monotonic_atomicrmw(
; GFX1250-GAS-LABEL: private_agent_monotonic_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
@@ -2537,8 +2616,14 @@ define amdgpu_kernel void @private_agent_acquire_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -2546,29 +2631,34 @@ define amdgpu_kernel void @private_agent_acquire_atomicrmw(
; GFX1250-GAS-LABEL: private_agent_acquire_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
@@ -2814,8 +2904,14 @@ define amdgpu_kernel void @private_agent_release_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -2823,29 +2919,34 @@ define amdgpu_kernel void @private_agent_release_atomicrmw(
; GFX1250-GAS-LABEL: private_agent_release_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
@@ -3093,8 +3194,14 @@ define amdgpu_kernel void @private_agent_acq_rel_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -3102,29 +3209,34 @@ define amdgpu_kernel void @private_agent_acq_rel_atomicrmw(
; GFX1250-GAS-LABEL: private_agent_acq_rel_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
@@ -3375,8 +3487,14 @@ define amdgpu_kernel void @private_agent_seq_cst_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -3384,29 +3502,34 @@ define amdgpu_kernel void @private_agent_seq_cst_atomicrmw(
; GFX1250-GAS-LABEL: private_agent_seq_cst_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
@@ -3678,6 +3801,10 @@ define amdgpu_kernel void @private_agent_acquire_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -3691,22 +3818,26 @@ define amdgpu_kernel void @private_agent_acquire_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -3981,6 +4112,10 @@ define amdgpu_kernel void @private_agent_acq_rel_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -3994,22 +4129,26 @@ define amdgpu_kernel void @private_agent_acq_rel_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -4289,6 +4428,10 @@ define amdgpu_kernel void @private_agent_seq_cst_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -4302,22 +4445,26 @@ define amdgpu_kernel void @private_agent_seq_cst_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -4653,6 +4800,12 @@ define amdgpu_kernel void @private_agent_monotonic_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -4666,26 +4819,32 @@ define amdgpu_kernel void @private_agent_monotonic_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_agent_monotonic_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -4693,6 +4852,7 @@ define amdgpu_kernel void @private_agent_monotonic_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -5015,6 +5175,12 @@ define amdgpu_kernel void @private_agent_acquire_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -5028,26 +5194,32 @@ define amdgpu_kernel void @private_agent_acquire_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_agent_acquire_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -5055,6 +5227,7 @@ define amdgpu_kernel void @private_agent_acquire_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -5380,6 +5553,12 @@ define amdgpu_kernel void @private_agent_release_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -5393,26 +5572,32 @@ define amdgpu_kernel void @private_agent_release_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_agent_release_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -5420,6 +5605,7 @@ define amdgpu_kernel void @private_agent_release_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -5747,6 +5933,12 @@ define amdgpu_kernel void @private_agent_acq_rel_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -5760,26 +5952,32 @@ define amdgpu_kernel void @private_agent_acq_rel_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_agent_acq_rel_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -5787,6 +5985,7 @@ define amdgpu_kernel void @private_agent_acq_rel_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -6117,6 +6316,12 @@ define amdgpu_kernel void @private_agent_seq_cst_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -6130,26 +6335,32 @@ define amdgpu_kernel void @private_agent_seq_cst_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_agent_seq_cst_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -6157,6 +6368,7 @@ define amdgpu_kernel void @private_agent_seq_cst_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -6487,6 +6699,12 @@ define amdgpu_kernel void @private_agent_monotonic_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -6500,26 +6718,32 @@ define amdgpu_kernel void @private_agent_monotonic_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_agent_monotonic_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -6527,6 +6751,7 @@ define amdgpu_kernel void @private_agent_monotonic_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -6852,6 +7077,12 @@ define amdgpu_kernel void @private_agent_acquire_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -6865,26 +7096,32 @@ define amdgpu_kernel void @private_agent_acquire_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_agent_acquire_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -6892,6 +7129,7 @@ define amdgpu_kernel void @private_agent_acquire_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -7217,6 +7455,12 @@ define amdgpu_kernel void @private_agent_release_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -7230,26 +7474,32 @@ define amdgpu_kernel void @private_agent_release_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_agent_release_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -7257,6 +7507,7 @@ define amdgpu_kernel void @private_agent_release_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -7587,6 +7838,12 @@ define amdgpu_kernel void @private_agent_acq_rel_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -7600,26 +7857,32 @@ define amdgpu_kernel void @private_agent_acq_rel_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_agent_acq_rel_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -7627,6 +7890,7 @@ define amdgpu_kernel void @private_agent_acq_rel_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -7957,6 +8221,12 @@ define amdgpu_kernel void @private_agent_seq_cst_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -7970,26 +8240,32 @@ define amdgpu_kernel void @private_agent_seq_cst_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_agent_seq_cst_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -7997,6 +8273,7 @@ define amdgpu_kernel void @private_agent_seq_cst_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -8327,6 +8604,12 @@ define amdgpu_kernel void @private_agent_monotonic_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -8340,26 +8623,32 @@ define amdgpu_kernel void @private_agent_monotonic_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_agent_monotonic_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -8367,6 +8656,7 @@ define amdgpu_kernel void @private_agent_monotonic_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -8697,6 +8987,12 @@ define amdgpu_kernel void @private_agent_acquire_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -8710,26 +9006,32 @@ define amdgpu_kernel void @private_agent_acquire_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_agent_acquire_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -8737,6 +9039,7 @@ define amdgpu_kernel void @private_agent_acquire_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -9067,6 +9370,12 @@ define amdgpu_kernel void @private_agent_release_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -9080,26 +9389,32 @@ define amdgpu_kernel void @private_agent_release_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_agent_release_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -9107,6 +9422,7 @@ define amdgpu_kernel void @private_agent_release_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -9437,6 +9753,12 @@ define amdgpu_kernel void @private_agent_acq_rel_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -9450,26 +9772,32 @@ define amdgpu_kernel void @private_agent_acq_rel_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_agent_acq_rel_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -9477,6 +9805,7 @@ define amdgpu_kernel void @private_agent_acq_rel_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -9807,6 +10136,12 @@ define amdgpu_kernel void @private_agent_seq_cst_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -9820,26 +10155,32 @@ define amdgpu_kernel void @private_agent_seq_cst_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_agent_seq_cst_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -9847,6 +10188,7 @@ define amdgpu_kernel void @private_agent_seq_cst_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -10205,6 +10547,12 @@ define amdgpu_kernel void @private_agent_monotonic_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -10220,25 +10568,31 @@ define amdgpu_kernel void @private_agent_monotonic_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -10600,6 +10954,12 @@ define amdgpu_kernel void @private_agent_acquire_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -10615,25 +10975,31 @@ define amdgpu_kernel void @private_agent_acquire_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -10997,6 +11363,12 @@ define amdgpu_kernel void @private_agent_release_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -11012,25 +11384,31 @@ define amdgpu_kernel void @private_agent_release_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -11397,6 +11775,12 @@ define amdgpu_kernel void @private_agent_acq_rel_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -11412,25 +11796,31 @@ define amdgpu_kernel void @private_agent_acq_rel_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -11799,6 +12189,12 @@ define amdgpu_kernel void @private_agent_seq_cst_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -11814,25 +12210,31 @@ define amdgpu_kernel void @private_agent_seq_cst_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -12201,6 +12603,12 @@ define amdgpu_kernel void @private_agent_monotonic_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -12216,25 +12624,31 @@ define amdgpu_kernel void @private_agent_monotonic_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -12598,6 +13012,12 @@ define amdgpu_kernel void @private_agent_acquire_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -12613,25 +13033,31 @@ define amdgpu_kernel void @private_agent_acquire_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -12995,6 +13421,12 @@ define amdgpu_kernel void @private_agent_release_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -13010,25 +13442,31 @@ define amdgpu_kernel void @private_agent_release_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -13397,6 +13835,12 @@ define amdgpu_kernel void @private_agent_acq_rel_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -13412,25 +13856,31 @@ define amdgpu_kernel void @private_agent_acq_rel_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -13799,6 +14249,12 @@ define amdgpu_kernel void @private_agent_seq_cst_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -13814,25 +14270,31 @@ define amdgpu_kernel void @private_agent_seq_cst_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -14201,6 +14663,12 @@ define amdgpu_kernel void @private_agent_monotonic_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -14216,25 +14684,31 @@ define amdgpu_kernel void @private_agent_monotonic_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -14603,6 +15077,12 @@ define amdgpu_kernel void @private_agent_acquire_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -14618,25 +15098,31 @@ define amdgpu_kernel void @private_agent_acquire_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -15005,6 +15491,12 @@ define amdgpu_kernel void @private_agent_release_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -15020,25 +15512,31 @@ define amdgpu_kernel void @private_agent_release_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -15407,6 +15905,12 @@ define amdgpu_kernel void @private_agent_acq_rel_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -15422,25 +15926,31 @@ define amdgpu_kernel void @private_agent_acq_rel_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -15809,6 +16319,12 @@ define amdgpu_kernel void @private_agent_seq_cst_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -15824,25 +16340,31 @@ define amdgpu_kernel void @private_agent_seq_cst_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -16088,34 +16610,42 @@ define amdgpu_kernel void @private_agent_one_as_unordered_load(
; GFX1250-NOGAS-LABEL: private_agent_one_as_unordered_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_agent_one_as_unordered_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -16124,6 +16654,7 @@ define amdgpu_kernel void @private_agent_one_as_unordered_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -16347,34 +16878,42 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_load(
; GFX1250-NOGAS-LABEL: private_agent_one_as_monotonic_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_agent_one_as_monotonic_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -16383,6 +16922,7 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_DEV
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -16606,34 +17146,42 @@ define amdgpu_kernel void @private_agent_one_as_acquire_load(
; GFX1250-NOGAS-LABEL: private_agent_one_as_acquire_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_agent_one_as_acquire_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -16644,6 +17192,7 @@ define amdgpu_kernel void @private_agent_one_as_acquire_load(
; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -16867,34 +17416,42 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_load(
; GFX1250-NOGAS-LABEL: private_agent_one_as_seq_cst_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_agent_one_as_seq_cst_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -16907,6 +17464,7 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_load(
; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -17114,10 +17672,14 @@ define amdgpu_kernel void @private_agent_one_as_unordered_store(
; GFX1250-NOGAS-LABEL: private_agent_one_as_unordered_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -17125,22 +17687,26 @@ define amdgpu_kernel void @private_agent_one_as_unordered_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -17355,10 +17921,14 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_store(
; GFX1250-NOGAS-LABEL: private_agent_one_as_monotonic_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -17366,22 +17936,26 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -17596,10 +18170,14 @@ define amdgpu_kernel void @private_agent_one_as_release_store(
; GFX1250-NOGAS-LABEL: private_agent_one_as_release_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -17607,22 +18185,26 @@ define amdgpu_kernel void @private_agent_one_as_release_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -17842,10 +18424,14 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_store(
; GFX1250-NOGAS-LABEL: private_agent_one_as_seq_cst_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -17853,22 +18439,26 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -18122,8 +18712,14 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -18131,29 +18727,34 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_atomicrmw(
; GFX1250-GAS-LABEL: private_agent_one_as_monotonic_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
@@ -18396,8 +18997,14 @@ define amdgpu_kernel void @private_agent_one_as_acquire_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -18405,29 +19012,34 @@ define amdgpu_kernel void @private_agent_one_as_acquire_atomicrmw(
; GFX1250-GAS-LABEL: private_agent_one_as_acquire_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
@@ -18673,8 +19285,14 @@ define amdgpu_kernel void @private_agent_one_as_release_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -18682,29 +19300,34 @@ define amdgpu_kernel void @private_agent_one_as_release_atomicrmw(
; GFX1250-GAS-LABEL: private_agent_one_as_release_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
@@ -18952,8 +19575,14 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -18961,29 +19590,34 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_atomicrmw(
; GFX1250-GAS-LABEL: private_agent_one_as_acq_rel_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
@@ -19234,8 +19868,14 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -19243,29 +19883,34 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_atomicrmw(
; GFX1250-GAS-LABEL: private_agent_one_as_seq_cst_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
@@ -19537,6 +20182,10 @@ define amdgpu_kernel void @private_agent_one_as_acquire_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -19550,22 +20199,26 @@ define amdgpu_kernel void @private_agent_one_as_acquire_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -19840,6 +20493,10 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -19853,22 +20510,26 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -20148,6 +20809,10 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -20161,22 +20826,26 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -20512,6 +21181,12 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -20525,26 +21200,32 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_agent_one_as_monotonic_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -20552,6 +21233,7 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -20874,6 +21556,12 @@ define amdgpu_kernel void @private_agent_one_as_acquire_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -20887,26 +21575,32 @@ define amdgpu_kernel void @private_agent_one_as_acquire_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_agent_one_as_acquire_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -20914,6 +21608,7 @@ define amdgpu_kernel void @private_agent_one_as_acquire_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -21239,6 +21934,12 @@ define amdgpu_kernel void @private_agent_one_as_release_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -21252,26 +21953,32 @@ define amdgpu_kernel void @private_agent_one_as_release_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_agent_one_as_release_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -21279,6 +21986,7 @@ define amdgpu_kernel void @private_agent_one_as_release_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -21606,6 +22314,12 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -21619,26 +22333,32 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_agent_one_as_acq_rel_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -21646,6 +22366,7 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -21976,6 +22697,12 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -21989,26 +22716,32 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_agent_one_as_seq_cst_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -22016,6 +22749,7 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -22346,6 +23080,12 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -22359,26 +23099,32 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_agent_one_as_monotonic_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -22386,6 +23132,7 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -22711,6 +23458,12 @@ define amdgpu_kernel void @private_agent_one_as_acquire_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -22724,26 +23477,32 @@ define amdgpu_kernel void @private_agent_one_as_acquire_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_agent_one_as_acquire_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -22751,6 +23510,7 @@ define amdgpu_kernel void @private_agent_one_as_acquire_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -23076,6 +23836,12 @@ define amdgpu_kernel void @private_agent_one_as_release_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -23089,26 +23855,32 @@ define amdgpu_kernel void @private_agent_one_as_release_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_agent_one_as_release_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -23116,6 +23888,7 @@ define amdgpu_kernel void @private_agent_one_as_release_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -23446,6 +24219,12 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -23459,26 +24238,32 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_agent_one_as_acq_rel_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -23486,6 +24271,7 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -23816,6 +24602,12 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -23829,26 +24621,32 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_agent_one_as_seq_cst_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -23856,6 +24654,7 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -24186,6 +24985,12 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -24199,26 +25004,32 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_agent_one_as_monotonic_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -24226,6 +25037,7 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -24556,6 +25368,12 @@ define amdgpu_kernel void @private_agent_one_as_acquire_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -24569,26 +25387,32 @@ define amdgpu_kernel void @private_agent_one_as_acquire_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_agent_one_as_acquire_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -24596,6 +25420,7 @@ define amdgpu_kernel void @private_agent_one_as_acquire_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -24926,6 +25751,12 @@ define amdgpu_kernel void @private_agent_one_as_release_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -24939,26 +25770,32 @@ define amdgpu_kernel void @private_agent_one_as_release_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_agent_one_as_release_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -24966,6 +25803,7 @@ define amdgpu_kernel void @private_agent_one_as_release_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -25296,6 +26134,12 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -25309,26 +26153,32 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_agent_one_as_acq_rel_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -25336,6 +26186,7 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -25666,6 +26517,12 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -25679,26 +26536,32 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_agent_one_as_seq_cst_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -25706,6 +26569,7 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -26064,6 +26928,12 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -26079,25 +26949,31 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -26459,6 +27335,12 @@ define amdgpu_kernel void @private_agent_one_as_acquire_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -26474,25 +27356,31 @@ define amdgpu_kernel void @private_agent_one_as_acquire_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -26856,6 +27744,12 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -26871,25 +27765,31 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -27258,6 +28158,12 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -27273,25 +28179,31 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -27660,6 +28572,12 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -27675,25 +28593,31 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -28057,6 +28981,12 @@ define amdgpu_kernel void @private_agent_one_as_acquire_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -28072,25 +29002,31 @@ define amdgpu_kernel void @private_agent_one_as_acquire_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -28454,6 +29390,12 @@ define amdgpu_kernel void @private_agent_one_as_release_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -28469,25 +29411,31 @@ define amdgpu_kernel void @private_agent_one_as_release_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -28856,6 +29804,12 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -28871,25 +29825,31 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -29258,6 +30218,12 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -29273,25 +30239,31 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -29660,6 +30632,12 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -29675,25 +30653,31 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -30062,6 +31046,12 @@ define amdgpu_kernel void @private_agent_one_as_acquire_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -30077,25 +31067,31 @@ define amdgpu_kernel void @private_agent_one_as_acquire_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -30464,6 +31460,12 @@ define amdgpu_kernel void @private_agent_one_as_release_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -30479,25 +31481,31 @@ define amdgpu_kernel void @private_agent_one_as_release_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -30866,6 +31874,12 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -30881,25 +31895,31 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -31268,6 +32288,12 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -31283,25 +32309,31 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-cluster.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-cluster.ll
index 1bff0e551a8c8..1fb4fd30ca4b7 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-cluster.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-cluster.ll
@@ -229,34 +229,42 @@ define amdgpu_kernel void @private_cluster_unordered_load(
; GFX1250-NOGAS-LABEL: private_cluster_unordered_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_cluster_unordered_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -265,6 +273,7 @@ define amdgpu_kernel void @private_cluster_unordered_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -488,34 +497,42 @@ define amdgpu_kernel void @private_cluster_monotonic_load(
; GFX1250-NOGAS-LABEL: private_cluster_monotonic_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_cluster_monotonic_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -524,6 +541,7 @@ define amdgpu_kernel void @private_cluster_monotonic_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -747,34 +765,42 @@ define amdgpu_kernel void @private_cluster_acquire_load(
; GFX1250-NOGAS-LABEL: private_cluster_acquire_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_cluster_acquire_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -784,6 +810,7 @@ define amdgpu_kernel void @private_cluster_acquire_load(
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -1007,34 +1034,42 @@ define amdgpu_kernel void @private_cluster_seq_cst_load(
; GFX1250-NOGAS-LABEL: private_cluster_seq_cst_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_cluster_seq_cst_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -1046,6 +1081,7 @@ define amdgpu_kernel void @private_cluster_seq_cst_load(
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -1253,10 +1289,14 @@ define amdgpu_kernel void @private_cluster_unordered_store(
; GFX1250-NOGAS-LABEL: private_cluster_unordered_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -1264,22 +1304,26 @@ define amdgpu_kernel void @private_cluster_unordered_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -1494,10 +1538,14 @@ define amdgpu_kernel void @private_cluster_monotonic_store(
; GFX1250-NOGAS-LABEL: private_cluster_monotonic_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -1505,22 +1553,26 @@ define amdgpu_kernel void @private_cluster_monotonic_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -1735,10 +1787,14 @@ define amdgpu_kernel void @private_cluster_release_store(
; GFX1250-NOGAS-LABEL: private_cluster_release_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -1746,22 +1802,26 @@ define amdgpu_kernel void @private_cluster_release_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -1978,10 +2038,14 @@ define amdgpu_kernel void @private_cluster_seq_cst_store(
; GFX1250-NOGAS-LABEL: private_cluster_seq_cst_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -1989,22 +2053,26 @@ define amdgpu_kernel void @private_cluster_seq_cst_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -2255,8 +2323,14 @@ define amdgpu_kernel void @private_cluster_monotonic_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -2264,29 +2338,34 @@ define amdgpu_kernel void @private_cluster_monotonic_atomicrmw(
; GFX1250-GAS-LABEL: private_cluster_monotonic_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
@@ -2529,8 +2608,14 @@ define amdgpu_kernel void @private_cluster_acquire_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -2538,29 +2623,34 @@ define amdgpu_kernel void @private_cluster_acquire_atomicrmw(
; GFX1250-GAS-LABEL: private_cluster_acquire_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
@@ -2805,8 +2895,14 @@ define amdgpu_kernel void @private_cluster_release_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -2814,29 +2910,34 @@ define amdgpu_kernel void @private_cluster_release_atomicrmw(
; GFX1250-GAS-LABEL: private_cluster_release_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
@@ -3081,8 +3182,14 @@ define amdgpu_kernel void @private_cluster_acq_rel_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -3090,29 +3197,34 @@ define amdgpu_kernel void @private_cluster_acq_rel_atomicrmw(
; GFX1250-GAS-LABEL: private_cluster_acq_rel_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
@@ -3359,8 +3471,14 @@ define amdgpu_kernel void @private_cluster_seq_cst_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -3368,29 +3486,34 @@ define amdgpu_kernel void @private_cluster_seq_cst_atomicrmw(
; GFX1250-GAS-LABEL: private_cluster_seq_cst_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
@@ -3658,6 +3781,10 @@ define amdgpu_kernel void @private_cluster_acquire_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -3671,22 +3798,26 @@ define amdgpu_kernel void @private_cluster_acquire_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -3960,6 +4091,10 @@ define amdgpu_kernel void @private_cluster_acq_rel_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -3973,22 +4108,26 @@ define amdgpu_kernel void @private_cluster_acq_rel_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -4264,6 +4403,10 @@ define amdgpu_kernel void @private_cluster_seq_cst_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -4277,22 +4420,26 @@ define amdgpu_kernel void @private_cluster_seq_cst_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -4624,6 +4771,12 @@ define amdgpu_kernel void @private_cluster_monotonic_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -4637,26 +4790,32 @@ define amdgpu_kernel void @private_cluster_monotonic_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_cluster_monotonic_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -4664,6 +4823,7 @@ define amdgpu_kernel void @private_cluster_monotonic_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -4986,6 +5146,12 @@ define amdgpu_kernel void @private_cluster_acquire_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -4999,26 +5165,32 @@ define amdgpu_kernel void @private_cluster_acquire_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_cluster_acquire_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -5026,6 +5198,7 @@ define amdgpu_kernel void @private_cluster_acquire_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -5350,6 +5523,12 @@ define amdgpu_kernel void @private_cluster_release_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -5363,26 +5542,32 @@ define amdgpu_kernel void @private_cluster_release_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_cluster_release_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -5390,6 +5575,7 @@ define amdgpu_kernel void @private_cluster_release_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -5714,6 +5900,12 @@ define amdgpu_kernel void @private_cluster_acq_rel_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -5727,26 +5919,32 @@ define amdgpu_kernel void @private_cluster_acq_rel_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_cluster_acq_rel_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -5754,6 +5952,7 @@ define amdgpu_kernel void @private_cluster_acq_rel_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -6080,6 +6279,12 @@ define amdgpu_kernel void @private_cluster_seq_cst_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -6093,26 +6298,32 @@ define amdgpu_kernel void @private_cluster_seq_cst_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_cluster_seq_cst_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -6120,6 +6331,7 @@ define amdgpu_kernel void @private_cluster_seq_cst_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -6446,6 +6658,12 @@ define amdgpu_kernel void @private_cluster_monotonic_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -6459,26 +6677,32 @@ define amdgpu_kernel void @private_cluster_monotonic_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_cluster_monotonic_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -6486,6 +6710,7 @@ define amdgpu_kernel void @private_cluster_monotonic_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -6810,6 +7035,12 @@ define amdgpu_kernel void @private_cluster_acquire_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -6823,26 +7054,32 @@ define amdgpu_kernel void @private_cluster_acquire_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_cluster_acquire_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -6850,6 +7087,7 @@ define amdgpu_kernel void @private_cluster_acquire_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -7174,6 +7412,12 @@ define amdgpu_kernel void @private_cluster_release_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -7187,26 +7431,32 @@ define amdgpu_kernel void @private_cluster_release_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_cluster_release_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -7214,6 +7464,7 @@ define amdgpu_kernel void @private_cluster_release_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -7540,6 +7791,12 @@ define amdgpu_kernel void @private_cluster_acq_rel_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -7553,26 +7810,32 @@ define amdgpu_kernel void @private_cluster_acq_rel_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_cluster_acq_rel_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -7580,6 +7843,7 @@ define amdgpu_kernel void @private_cluster_acq_rel_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -7906,6 +8170,12 @@ define amdgpu_kernel void @private_cluster_seq_cst_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -7919,26 +8189,32 @@ define amdgpu_kernel void @private_cluster_seq_cst_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_cluster_seq_cst_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -7946,6 +8222,7 @@ define amdgpu_kernel void @private_cluster_seq_cst_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -8272,6 +8549,12 @@ define amdgpu_kernel void @private_cluster_monotonic_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -8285,26 +8568,32 @@ define amdgpu_kernel void @private_cluster_monotonic_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_cluster_monotonic_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -8312,6 +8601,7 @@ define amdgpu_kernel void @private_cluster_monotonic_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -8638,6 +8928,12 @@ define amdgpu_kernel void @private_cluster_acquire_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -8651,26 +8947,32 @@ define amdgpu_kernel void @private_cluster_acquire_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_cluster_acquire_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -8678,6 +8980,7 @@ define amdgpu_kernel void @private_cluster_acquire_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -9004,6 +9307,12 @@ define amdgpu_kernel void @private_cluster_release_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -9017,26 +9326,32 @@ define amdgpu_kernel void @private_cluster_release_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_cluster_release_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -9044,6 +9359,7 @@ define amdgpu_kernel void @private_cluster_release_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -9370,6 +9686,12 @@ define amdgpu_kernel void @private_cluster_acq_rel_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -9383,26 +9705,32 @@ define amdgpu_kernel void @private_cluster_acq_rel_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_cluster_acq_rel_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -9410,6 +9738,7 @@ define amdgpu_kernel void @private_cluster_acq_rel_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -9736,6 +10065,12 @@ define amdgpu_kernel void @private_cluster_seq_cst_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -9749,26 +10084,32 @@ define amdgpu_kernel void @private_cluster_seq_cst_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_cluster_seq_cst_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -9776,6 +10117,7 @@ define amdgpu_kernel void @private_cluster_seq_cst_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -10130,6 +10472,12 @@ define amdgpu_kernel void @private_cluster_monotonic_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -10145,25 +10493,31 @@ define amdgpu_kernel void @private_cluster_monotonic_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -10525,6 +10879,12 @@ define amdgpu_kernel void @private_cluster_acquire_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -10540,25 +10900,31 @@ define amdgpu_kernel void @private_cluster_acquire_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -10921,6 +11287,12 @@ define amdgpu_kernel void @private_cluster_release_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -10936,25 +11308,31 @@ define amdgpu_kernel void @private_cluster_release_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -11318,6 +11696,12 @@ define amdgpu_kernel void @private_cluster_acq_rel_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -11333,25 +11717,31 @@ define amdgpu_kernel void @private_cluster_acq_rel_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -11716,6 +12106,12 @@ define amdgpu_kernel void @private_cluster_seq_cst_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -11731,25 +12127,31 @@ define amdgpu_kernel void @private_cluster_seq_cst_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -12114,6 +12516,12 @@ define amdgpu_kernel void @private_cluster_monotonic_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -12129,25 +12537,31 @@ define amdgpu_kernel void @private_cluster_monotonic_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -12510,6 +12924,12 @@ define amdgpu_kernel void @private_cluster_acquire_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -12525,25 +12945,31 @@ define amdgpu_kernel void @private_cluster_acquire_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -12906,6 +13332,12 @@ define amdgpu_kernel void @private_cluster_release_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -12921,25 +13353,31 @@ define amdgpu_kernel void @private_cluster_release_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -13304,6 +13742,12 @@ define amdgpu_kernel void @private_cluster_acq_rel_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -13319,25 +13763,31 @@ define amdgpu_kernel void @private_cluster_acq_rel_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -13702,6 +14152,12 @@ define amdgpu_kernel void @private_cluster_seq_cst_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -13717,25 +14173,31 @@ define amdgpu_kernel void @private_cluster_seq_cst_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -14100,6 +14562,12 @@ define amdgpu_kernel void @private_cluster_monotonic_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -14115,25 +14583,31 @@ define amdgpu_kernel void @private_cluster_monotonic_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -14498,6 +14972,12 @@ define amdgpu_kernel void @private_cluster_acquire_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -14513,25 +14993,31 @@ define amdgpu_kernel void @private_cluster_acquire_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -14896,6 +15382,12 @@ define amdgpu_kernel void @private_cluster_release_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -14911,25 +15403,31 @@ define amdgpu_kernel void @private_cluster_release_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -15294,6 +15792,12 @@ define amdgpu_kernel void @private_cluster_acq_rel_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -15309,25 +15813,31 @@ define amdgpu_kernel void @private_cluster_acq_rel_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -15692,6 +16202,12 @@ define amdgpu_kernel void @private_cluster_seq_cst_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -15707,25 +16223,31 @@ define amdgpu_kernel void @private_cluster_seq_cst_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -15967,34 +16489,42 @@ define amdgpu_kernel void @private_cluster_one_as_unordered_load(
; GFX1250-NOGAS-LABEL: private_cluster_one_as_unordered_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_cluster_one_as_unordered_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -16003,6 +16533,7 @@ define amdgpu_kernel void @private_cluster_one_as_unordered_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -16226,34 +16757,42 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_load(
; GFX1250-NOGAS-LABEL: private_cluster_one_as_monotonic_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_cluster_one_as_monotonic_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -16262,6 +16801,7 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -16485,34 +17025,42 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_load(
; GFX1250-NOGAS-LABEL: private_cluster_one_as_acquire_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_cluster_one_as_acquire_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -16523,6 +17071,7 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_load(
; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -16746,34 +17295,42 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_load(
; GFX1250-NOGAS-LABEL: private_cluster_one_as_seq_cst_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_cluster_one_as_seq_cst_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -16786,6 +17343,7 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_load(
; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -16993,10 +17551,14 @@ define amdgpu_kernel void @private_cluster_one_as_unordered_store(
; GFX1250-NOGAS-LABEL: private_cluster_one_as_unordered_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -17004,22 +17566,26 @@ define amdgpu_kernel void @private_cluster_one_as_unordered_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -17234,10 +17800,14 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_store(
; GFX1250-NOGAS-LABEL: private_cluster_one_as_monotonic_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -17245,22 +17815,26 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -17475,10 +18049,14 @@ define amdgpu_kernel void @private_cluster_one_as_release_store(
; GFX1250-NOGAS-LABEL: private_cluster_one_as_release_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -17486,22 +18064,26 @@ define amdgpu_kernel void @private_cluster_one_as_release_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -17718,10 +18300,14 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_store(
; GFX1250-NOGAS-LABEL: private_cluster_one_as_seq_cst_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -17729,22 +18315,26 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -17995,8 +18585,14 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -18004,29 +18600,34 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_atomicrmw(
; GFX1250-GAS-LABEL: private_cluster_one_as_monotonic_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
@@ -18269,8 +18870,14 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -18278,29 +18885,34 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_atomicrmw(
; GFX1250-GAS-LABEL: private_cluster_one_as_acquire_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
@@ -18545,8 +19157,14 @@ define amdgpu_kernel void @private_cluster_one_as_release_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -18554,29 +19172,34 @@ define amdgpu_kernel void @private_cluster_one_as_release_atomicrmw(
; GFX1250-GAS-LABEL: private_cluster_one_as_release_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
@@ -18821,8 +19444,14 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -18830,29 +19459,34 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_atomicrmw(
; GFX1250-GAS-LABEL: private_cluster_one_as_acq_rel_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
@@ -19099,8 +19733,14 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -19108,29 +19748,34 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_atomicrmw(
; GFX1250-GAS-LABEL: private_cluster_one_as_seq_cst_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
@@ -19398,6 +20043,10 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -19411,22 +20060,26 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -19701,6 +20354,10 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -19714,22 +20371,26 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -20006,6 +20667,10 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -20019,22 +20684,26 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -20367,6 +21036,12 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -20380,26 +21055,32 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_cluster_one_as_monotonic_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -20407,6 +21088,7 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -20729,6 +21411,12 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -20742,26 +21430,32 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_cluster_one_as_acquire_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -20769,6 +21463,7 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -21093,6 +21788,12 @@ define amdgpu_kernel void @private_cluster_one_as_release_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -21106,26 +21807,32 @@ define amdgpu_kernel void @private_cluster_one_as_release_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_cluster_one_as_release_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -21133,6 +21840,7 @@ define amdgpu_kernel void @private_cluster_one_as_release_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -21457,6 +22165,12 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -21470,26 +22184,32 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_cluster_one_as_acq_rel_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -21497,6 +22217,7 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -21823,6 +22544,12 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -21836,26 +22563,32 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_cluster_one_as_seq_cst_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -21863,6 +22596,7 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -22189,6 +22923,12 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -22202,26 +22942,32 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_cluster_one_as_monotonic_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -22229,6 +22975,7 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -22553,6 +23300,12 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -22566,26 +23319,32 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_cluster_one_as_acquire_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -22593,6 +23352,7 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -22917,6 +23677,12 @@ define amdgpu_kernel void @private_cluster_one_as_release_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -22930,26 +23696,32 @@ define amdgpu_kernel void @private_cluster_one_as_release_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_cluster_one_as_release_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -22957,6 +23729,7 @@ define amdgpu_kernel void @private_cluster_one_as_release_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -23283,6 +24056,12 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -23296,26 +24075,32 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_cluster_one_as_acq_rel_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -23323,6 +24108,7 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -23649,6 +24435,12 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -23662,26 +24454,32 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_cluster_one_as_seq_cst_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -23689,6 +24487,7 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -24015,6 +24814,12 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -24028,26 +24833,32 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_cluster_one_as_monotonic_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -24055,6 +24866,7 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -24381,6 +25193,12 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -24394,26 +25212,32 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_cluster_one_as_acquire_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -24421,6 +25245,7 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -24747,6 +25572,12 @@ define amdgpu_kernel void @private_cluster_one_as_release_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -24760,26 +25591,32 @@ define amdgpu_kernel void @private_cluster_one_as_release_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_cluster_one_as_release_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -24787,6 +25624,7 @@ define amdgpu_kernel void @private_cluster_one_as_release_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -25113,6 +25951,12 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -25126,26 +25970,32 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_cluster_one_as_acq_rel_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -25153,6 +26003,7 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -25479,6 +26330,12 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -25492,26 +26349,32 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_cluster_one_as_seq_cst_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -25519,6 +26382,7 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -25873,6 +26737,12 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_monotonic_ret_cmpxch
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -25888,25 +26758,31 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_monotonic_ret_cmpxch
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -26268,6 +27144,12 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -26283,25 +27165,31 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -26665,6 +27553,12 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -26680,25 +27574,31 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -27064,6 +27964,12 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -27079,25 +27985,31 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -27463,6 +28375,12 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -27478,25 +28396,31 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -27860,6 +28784,12 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -27875,25 +28805,31 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -28257,6 +29193,12 @@ define amdgpu_kernel void @private_cluster_one_as_release_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -28272,25 +29214,31 @@ define amdgpu_kernel void @private_cluster_one_as_release_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -28656,6 +29604,12 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -28671,25 +29625,31 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -29055,6 +30015,12 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -29070,25 +30036,31 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -29454,6 +30426,12 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -29469,25 +30447,31 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -29853,6 +30837,12 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -29868,25 +30858,31 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -30252,6 +31248,12 @@ define amdgpu_kernel void @private_cluster_one_as_release_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -30267,25 +31269,31 @@ define amdgpu_kernel void @private_cluster_one_as_release_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -30651,6 +31659,12 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -30666,25 +31680,31 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -31050,6 +32070,12 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -31065,25 +32091,31 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-singlethread.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-singlethread.ll
index 6b90662e26475..378c2daae0ded 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-singlethread.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-singlethread.ll
@@ -229,34 +229,42 @@ define amdgpu_kernel void @private_singlethread_unordered_load(
; GFX1250-NOGAS-LABEL: private_singlethread_unordered_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_singlethread_unordered_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -265,6 +273,7 @@ define amdgpu_kernel void @private_singlethread_unordered_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -488,34 +497,42 @@ define amdgpu_kernel void @private_singlethread_monotonic_load(
; GFX1250-NOGAS-LABEL: private_singlethread_monotonic_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_singlethread_monotonic_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -524,6 +541,7 @@ define amdgpu_kernel void @private_singlethread_monotonic_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -747,34 +765,42 @@ define amdgpu_kernel void @private_singlethread_acquire_load(
; GFX1250-NOGAS-LABEL: private_singlethread_acquire_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_singlethread_acquire_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -783,6 +809,7 @@ define amdgpu_kernel void @private_singlethread_acquire_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -1006,34 +1033,42 @@ define amdgpu_kernel void @private_singlethread_seq_cst_load(
; GFX1250-NOGAS-LABEL: private_singlethread_seq_cst_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_singlethread_seq_cst_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -1042,6 +1077,7 @@ define amdgpu_kernel void @private_singlethread_seq_cst_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -1249,10 +1285,14 @@ define amdgpu_kernel void @private_singlethread_unordered_store(
; GFX1250-NOGAS-LABEL: private_singlethread_unordered_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -1260,22 +1300,26 @@ define amdgpu_kernel void @private_singlethread_unordered_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -1490,10 +1534,14 @@ define amdgpu_kernel void @private_singlethread_monotonic_store(
; GFX1250-NOGAS-LABEL: private_singlethread_monotonic_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -1501,22 +1549,26 @@ define amdgpu_kernel void @private_singlethread_monotonic_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -1731,10 +1783,14 @@ define amdgpu_kernel void @private_singlethread_release_store(
; GFX1250-NOGAS-LABEL: private_singlethread_release_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -1742,22 +1798,26 @@ define amdgpu_kernel void @private_singlethread_release_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -1972,10 +2032,14 @@ define amdgpu_kernel void @private_singlethread_seq_cst_store(
; GFX1250-NOGAS-LABEL: private_singlethread_seq_cst_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -1983,22 +2047,26 @@ define amdgpu_kernel void @private_singlethread_seq_cst_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -2247,8 +2315,14 @@ define amdgpu_kernel void @private_singlethread_monotonic_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -2256,29 +2330,34 @@ define amdgpu_kernel void @private_singlethread_monotonic_atomicrmw(
; GFX1250-GAS-LABEL: private_singlethread_monotonic_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
@@ -2521,8 +2600,14 @@ define amdgpu_kernel void @private_singlethread_acquire_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -2530,29 +2615,34 @@ define amdgpu_kernel void @private_singlethread_acquire_atomicrmw(
; GFX1250-GAS-LABEL: private_singlethread_acquire_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
@@ -2795,8 +2885,14 @@ define amdgpu_kernel void @private_singlethread_release_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -2804,29 +2900,34 @@ define amdgpu_kernel void @private_singlethread_release_atomicrmw(
; GFX1250-GAS-LABEL: private_singlethread_release_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
@@ -3069,8 +3170,14 @@ define amdgpu_kernel void @private_singlethread_acq_rel_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -3078,29 +3185,34 @@ define amdgpu_kernel void @private_singlethread_acq_rel_atomicrmw(
; GFX1250-GAS-LABEL: private_singlethread_acq_rel_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
@@ -3343,8 +3455,14 @@ define amdgpu_kernel void @private_singlethread_seq_cst_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -3352,29 +3470,34 @@ define amdgpu_kernel void @private_singlethread_seq_cst_atomicrmw(
; GFX1250-GAS-LABEL: private_singlethread_seq_cst_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
@@ -3638,6 +3761,10 @@ define amdgpu_kernel void @private_singlethread_acquire_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -3651,22 +3778,26 @@ define amdgpu_kernel void @private_singlethread_acquire_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -3939,6 +4070,10 @@ define amdgpu_kernel void @private_singlethread_acq_rel_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -3952,22 +4087,26 @@ define amdgpu_kernel void @private_singlethread_acq_rel_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -4240,6 +4379,10 @@ define amdgpu_kernel void @private_singlethread_seq_cst_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -4253,22 +4396,26 @@ define amdgpu_kernel void @private_singlethread_seq_cst_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -4597,6 +4744,12 @@ define amdgpu_kernel void @private_singlethread_monotonic_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -4610,26 +4763,32 @@ define amdgpu_kernel void @private_singlethread_monotonic_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_singlethread_monotonic_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -4637,6 +4796,7 @@ define amdgpu_kernel void @private_singlethread_monotonic_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -4959,6 +5119,12 @@ define amdgpu_kernel void @private_singlethread_acquire_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -4972,26 +5138,32 @@ define amdgpu_kernel void @private_singlethread_acquire_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_singlethread_acquire_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -4999,6 +5171,7 @@ define amdgpu_kernel void @private_singlethread_acquire_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -5321,6 +5494,12 @@ define amdgpu_kernel void @private_singlethread_release_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -5334,26 +5513,32 @@ define amdgpu_kernel void @private_singlethread_release_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_singlethread_release_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -5361,6 +5546,7 @@ define amdgpu_kernel void @private_singlethread_release_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -5683,6 +5869,12 @@ define amdgpu_kernel void @private_singlethread_acq_rel_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -5696,26 +5888,32 @@ define amdgpu_kernel void @private_singlethread_acq_rel_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_singlethread_acq_rel_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -5723,6 +5921,7 @@ define amdgpu_kernel void @private_singlethread_acq_rel_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -6045,6 +6244,12 @@ define amdgpu_kernel void @private_singlethread_seq_cst_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -6058,26 +6263,32 @@ define amdgpu_kernel void @private_singlethread_seq_cst_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_singlethread_seq_cst_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -6085,6 +6296,7 @@ define amdgpu_kernel void @private_singlethread_seq_cst_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -6407,6 +6619,12 @@ define amdgpu_kernel void @private_singlethread_monotonic_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -6420,26 +6638,32 @@ define amdgpu_kernel void @private_singlethread_monotonic_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_singlethread_monotonic_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -6447,6 +6671,7 @@ define amdgpu_kernel void @private_singlethread_monotonic_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -6769,6 +6994,12 @@ define amdgpu_kernel void @private_singlethread_acquire_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -6782,26 +7013,32 @@ define amdgpu_kernel void @private_singlethread_acquire_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_singlethread_acquire_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -6809,6 +7046,7 @@ define amdgpu_kernel void @private_singlethread_acquire_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -7131,6 +7369,12 @@ define amdgpu_kernel void @private_singlethread_release_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -7144,26 +7388,32 @@ define amdgpu_kernel void @private_singlethread_release_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_singlethread_release_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -7171,6 +7421,7 @@ define amdgpu_kernel void @private_singlethread_release_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -7493,6 +7744,12 @@ define amdgpu_kernel void @private_singlethread_acq_rel_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -7506,26 +7763,32 @@ define amdgpu_kernel void @private_singlethread_acq_rel_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_singlethread_acq_rel_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -7533,6 +7796,7 @@ define amdgpu_kernel void @private_singlethread_acq_rel_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -7855,6 +8119,12 @@ define amdgpu_kernel void @private_singlethread_seq_cst_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -7868,26 +8138,32 @@ define amdgpu_kernel void @private_singlethread_seq_cst_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_singlethread_seq_cst_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -7895,6 +8171,7 @@ define amdgpu_kernel void @private_singlethread_seq_cst_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -8217,6 +8494,12 @@ define amdgpu_kernel void @private_singlethread_monotonic_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -8230,26 +8513,32 @@ define amdgpu_kernel void @private_singlethread_monotonic_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_singlethread_monotonic_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -8257,6 +8546,7 @@ define amdgpu_kernel void @private_singlethread_monotonic_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -8579,6 +8869,12 @@ define amdgpu_kernel void @private_singlethread_acquire_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -8592,26 +8888,32 @@ define amdgpu_kernel void @private_singlethread_acquire_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_singlethread_acquire_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -8619,6 +8921,7 @@ define amdgpu_kernel void @private_singlethread_acquire_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -8941,6 +9244,12 @@ define amdgpu_kernel void @private_singlethread_release_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -8954,26 +9263,32 @@ define amdgpu_kernel void @private_singlethread_release_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_singlethread_release_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -8981,6 +9296,7 @@ define amdgpu_kernel void @private_singlethread_release_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -9303,6 +9619,12 @@ define amdgpu_kernel void @private_singlethread_acq_rel_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -9316,26 +9638,32 @@ define amdgpu_kernel void @private_singlethread_acq_rel_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_singlethread_acq_rel_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -9343,6 +9671,7 @@ define amdgpu_kernel void @private_singlethread_acq_rel_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -9665,6 +9994,12 @@ define amdgpu_kernel void @private_singlethread_seq_cst_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -9678,26 +10013,32 @@ define amdgpu_kernel void @private_singlethread_seq_cst_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_singlethread_seq_cst_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -9705,6 +10046,7 @@ define amdgpu_kernel void @private_singlethread_seq_cst_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -10055,6 +10397,12 @@ define amdgpu_kernel void @private_singlethread_monotonic_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -10070,25 +10418,31 @@ define amdgpu_kernel void @private_singlethread_monotonic_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -10450,6 +10804,12 @@ define amdgpu_kernel void @private_singlethread_acquire_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -10465,25 +10825,31 @@ define amdgpu_kernel void @private_singlethread_acquire_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -10845,6 +11211,12 @@ define amdgpu_kernel void @private_singlethread_release_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -10860,25 +11232,31 @@ define amdgpu_kernel void @private_singlethread_release_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -11240,6 +11618,12 @@ define amdgpu_kernel void @private_singlethread_acq_rel_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -11255,25 +11639,31 @@ define amdgpu_kernel void @private_singlethread_acq_rel_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -11635,6 +12025,12 @@ define amdgpu_kernel void @private_singlethread_seq_cst_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -11650,25 +12046,31 @@ define amdgpu_kernel void @private_singlethread_seq_cst_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -12030,6 +12432,12 @@ define amdgpu_kernel void @private_singlethread_monotonic_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -12045,25 +12453,31 @@ define amdgpu_kernel void @private_singlethread_monotonic_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -12425,6 +12839,12 @@ define amdgpu_kernel void @private_singlethread_acquire_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -12440,25 +12860,31 @@ define amdgpu_kernel void @private_singlethread_acquire_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -12820,6 +13246,12 @@ define amdgpu_kernel void @private_singlethread_release_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -12835,25 +13267,31 @@ define amdgpu_kernel void @private_singlethread_release_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -13215,6 +13653,12 @@ define amdgpu_kernel void @private_singlethread_acq_rel_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -13230,25 +13674,31 @@ define amdgpu_kernel void @private_singlethread_acq_rel_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -13610,6 +14060,12 @@ define amdgpu_kernel void @private_singlethread_seq_cst_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -13625,25 +14081,31 @@ define amdgpu_kernel void @private_singlethread_seq_cst_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -14005,6 +14467,12 @@ define amdgpu_kernel void @private_singlethread_monotonic_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -14020,25 +14488,31 @@ define amdgpu_kernel void @private_singlethread_monotonic_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -14400,6 +14874,12 @@ define amdgpu_kernel void @private_singlethread_acquire_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -14415,25 +14895,31 @@ define amdgpu_kernel void @private_singlethread_acquire_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -14795,6 +15281,12 @@ define amdgpu_kernel void @private_singlethread_release_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -14810,25 +15302,31 @@ define amdgpu_kernel void @private_singlethread_release_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -15190,6 +15688,12 @@ define amdgpu_kernel void @private_singlethread_acq_rel_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -15205,25 +15709,31 @@ define amdgpu_kernel void @private_singlethread_acq_rel_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -15585,6 +16095,12 @@ define amdgpu_kernel void @private_singlethread_seq_cst_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -15600,25 +16116,31 @@ define amdgpu_kernel void @private_singlethread_seq_cst_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -15857,34 +16379,42 @@ define amdgpu_kernel void @private_singlethread_one_as_unordered_load(
; GFX1250-NOGAS-LABEL: private_singlethread_one_as_unordered_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_singlethread_one_as_unordered_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -15893,6 +16423,7 @@ define amdgpu_kernel void @private_singlethread_one_as_unordered_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -16116,34 +16647,42 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_load(
; GFX1250-NOGAS-LABEL: private_singlethread_one_as_monotonic_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_singlethread_one_as_monotonic_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -16152,6 +16691,7 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -16375,34 +16915,42 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_load(
; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acquire_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_singlethread_one_as_acquire_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -16411,6 +16959,7 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -16634,34 +17183,42 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_load(
; GFX1250-NOGAS-LABEL: private_singlethread_one_as_seq_cst_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_singlethread_one_as_seq_cst_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -16670,6 +17227,7 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -16877,10 +17435,14 @@ define amdgpu_kernel void @private_singlethread_one_as_unordered_store(
; GFX1250-NOGAS-LABEL: private_singlethread_one_as_unordered_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -16888,22 +17450,26 @@ define amdgpu_kernel void @private_singlethread_one_as_unordered_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -17118,10 +17684,14 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_store(
; GFX1250-NOGAS-LABEL: private_singlethread_one_as_monotonic_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -17129,22 +17699,26 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -17359,10 +17933,14 @@ define amdgpu_kernel void @private_singlethread_one_as_release_store(
; GFX1250-NOGAS-LABEL: private_singlethread_one_as_release_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -17370,22 +17948,26 @@ define amdgpu_kernel void @private_singlethread_one_as_release_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -17600,10 +18182,14 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_store(
; GFX1250-NOGAS-LABEL: private_singlethread_one_as_seq_cst_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -17611,22 +18197,26 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -17875,8 +18465,14 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -17884,29 +18480,34 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_atomicrmw(
; GFX1250-GAS-LABEL: private_singlethread_one_as_monotonic_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
@@ -18149,8 +18750,14 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -18158,29 +18765,34 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_atomicrmw(
; GFX1250-GAS-LABEL: private_singlethread_one_as_acquire_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
@@ -18423,8 +19035,14 @@ define amdgpu_kernel void @private_singlethread_one_as_release_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -18432,29 +19050,34 @@ define amdgpu_kernel void @private_singlethread_one_as_release_atomicrmw(
; GFX1250-GAS-LABEL: private_singlethread_one_as_release_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
@@ -18697,8 +19320,14 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -18706,29 +19335,34 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_atomicrmw(
; GFX1250-GAS-LABEL: private_singlethread_one_as_acq_rel_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
@@ -18971,8 +19605,14 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -18980,29 +19620,34 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_atomicrmw(
; GFX1250-GAS-LABEL: private_singlethread_one_as_seq_cst_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
@@ -19266,6 +19911,10 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -19279,22 +19928,26 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -19567,6 +20220,10 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -19580,22 +20237,26 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -19868,6 +20529,10 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -19881,22 +20546,26 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -20225,6 +20894,12 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_monotonic_cmpxc
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -20238,26 +20913,32 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_monotonic_cmpxc
; GFX1250-GAS-LABEL: private_singlethread_one_as_monotonic_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -20265,6 +20946,7 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_monotonic_cmpxc
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -20587,6 +21269,12 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_monotonic_cmpxchg
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -20600,26 +21288,32 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_monotonic_cmpxchg
; GFX1250-GAS-LABEL: private_singlethread_one_as_acquire_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -20627,6 +21321,7 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_monotonic_cmpxchg
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -20949,6 +21644,12 @@ define amdgpu_kernel void @private_singlethread_one_as_release_monotonic_cmpxchg
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -20962,26 +21663,32 @@ define amdgpu_kernel void @private_singlethread_one_as_release_monotonic_cmpxchg
; GFX1250-GAS-LABEL: private_singlethread_one_as_release_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -20989,6 +21696,7 @@ define amdgpu_kernel void @private_singlethread_one_as_release_monotonic_cmpxchg
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -21311,6 +22019,12 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_monotonic_cmpxchg
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -21324,26 +22038,32 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_monotonic_cmpxchg
; GFX1250-GAS-LABEL: private_singlethread_one_as_acq_rel_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -21351,6 +22071,7 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_monotonic_cmpxchg
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -21673,6 +22394,12 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_monotonic_cmpxchg
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -21686,26 +22413,32 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_monotonic_cmpxchg
; GFX1250-GAS-LABEL: private_singlethread_one_as_seq_cst_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -21713,6 +22446,7 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_monotonic_cmpxchg
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -22035,6 +22769,12 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_acquire_cmpxchg
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -22048,26 +22788,32 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_acquire_cmpxchg
; GFX1250-GAS-LABEL: private_singlethread_one_as_monotonic_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -22075,6 +22821,7 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_acquire_cmpxchg
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -22397,6 +23144,12 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -22410,26 +23163,32 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_singlethread_one_as_acquire_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -22437,6 +23196,7 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -22759,6 +23519,12 @@ define amdgpu_kernel void @private_singlethread_one_as_release_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -22772,26 +23538,32 @@ define amdgpu_kernel void @private_singlethread_one_as_release_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_singlethread_one_as_release_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -22799,6 +23571,7 @@ define amdgpu_kernel void @private_singlethread_one_as_release_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -23121,6 +23894,12 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -23134,26 +23913,32 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_singlethread_one_as_acq_rel_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -23161,6 +23946,7 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -23483,6 +24269,12 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -23496,26 +24288,32 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_singlethread_one_as_seq_cst_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -23523,6 +24321,7 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -23845,6 +24644,12 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_seq_cst_cmpxchg
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -23858,26 +24663,32 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_seq_cst_cmpxchg
; GFX1250-GAS-LABEL: private_singlethread_one_as_monotonic_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -23885,6 +24696,7 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_seq_cst_cmpxchg
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -24207,6 +25019,12 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -24220,26 +25038,32 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_singlethread_one_as_acquire_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -24247,6 +25071,7 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -24569,6 +25394,12 @@ define amdgpu_kernel void @private_singlethread_one_as_release_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -24582,26 +25413,32 @@ define amdgpu_kernel void @private_singlethread_one_as_release_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_singlethread_one_as_release_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -24609,6 +25446,7 @@ define amdgpu_kernel void @private_singlethread_one_as_release_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -24931,6 +25769,12 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -24944,26 +25788,32 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_singlethread_one_as_acq_rel_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -24971,6 +25821,7 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -25293,6 +26144,12 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -25306,26 +26163,32 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_singlethread_one_as_seq_cst_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -25333,6 +26196,7 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -25683,6 +26547,12 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_monotonic_ret_c
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -25698,25 +26568,31 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_monotonic_ret_c
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -26078,6 +26954,12 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_monotonic_ret_cmp
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -26093,25 +26975,31 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_monotonic_ret_cmp
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -26473,6 +27361,12 @@ define amdgpu_kernel void @private_singlethread_one_as_release_monotonic_ret_cmp
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -26488,25 +27382,31 @@ define amdgpu_kernel void @private_singlethread_one_as_release_monotonic_ret_cmp
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -26868,6 +27768,12 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_monotonic_ret_cmp
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -26883,25 +27789,31 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_monotonic_ret_cmp
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -27263,6 +28175,12 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_monotonic_ret_cmp
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -27278,25 +28196,31 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_monotonic_ret_cmp
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -27658,6 +28582,12 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_acquire_ret_cmp
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -27673,25 +28603,31 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_acquire_ret_cmp
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -28053,6 +28989,12 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_acquire_ret_cmpxc
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -28068,25 +29010,31 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_acquire_ret_cmpxc
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -28448,6 +29396,12 @@ define amdgpu_kernel void @private_singlethread_one_as_release_acquire_ret_cmpxc
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -28463,25 +29417,31 @@ define amdgpu_kernel void @private_singlethread_one_as_release_acquire_ret_cmpxc
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -28843,6 +29803,12 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_acquire_ret_cmpxc
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -28858,25 +29824,31 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_acquire_ret_cmpxc
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -29238,6 +30210,12 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_acquire_ret_cmpxc
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -29253,25 +30231,31 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_acquire_ret_cmpxc
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -29633,6 +30617,12 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_seq_cst_ret_cmp
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -29648,25 +30638,31 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_seq_cst_ret_cmp
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -30028,6 +31024,12 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_seq_cst_ret_cmpxc
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -30043,25 +31045,31 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_seq_cst_ret_cmpxc
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -30423,6 +31431,12 @@ define amdgpu_kernel void @private_singlethread_one_as_release_seq_cst_ret_cmpxc
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -30438,25 +31452,31 @@ define amdgpu_kernel void @private_singlethread_one_as_release_seq_cst_ret_cmpxc
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -30818,6 +31838,12 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_seq_cst_ret_cmpxc
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -30833,25 +31859,31 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_seq_cst_ret_cmpxc
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -31213,6 +32245,12 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_seq_cst_ret_cmpxc
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -31228,25 +32266,31 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_seq_cst_ret_cmpxc
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-system.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-system.ll
index 08a2e7b287eec..21b4667ac69ea 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-system.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-system.ll
@@ -229,34 +229,42 @@ define amdgpu_kernel void @private_system_unordered_load(
; GFX1250-NOGAS-LABEL: private_system_unordered_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_system_unordered_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -265,6 +273,7 @@ define amdgpu_kernel void @private_system_unordered_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -488,34 +497,42 @@ define amdgpu_kernel void @private_system_monotonic_load(
; GFX1250-NOGAS-LABEL: private_system_monotonic_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_system_monotonic_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -524,6 +541,7 @@ define amdgpu_kernel void @private_system_monotonic_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SYS
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -747,34 +765,42 @@ define amdgpu_kernel void @private_system_acquire_load(
; GFX1250-NOGAS-LABEL: private_system_acquire_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_system_acquire_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -785,6 +811,7 @@ define amdgpu_kernel void @private_system_acquire_load(
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -1008,34 +1035,42 @@ define amdgpu_kernel void @private_system_seq_cst_load(
; GFX1250-NOGAS-LABEL: private_system_seq_cst_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_system_seq_cst_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -1048,6 +1083,7 @@ define amdgpu_kernel void @private_system_seq_cst_load(
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -1255,10 +1291,14 @@ define amdgpu_kernel void @private_system_unordered_store(
; GFX1250-NOGAS-LABEL: private_system_unordered_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -1266,22 +1306,26 @@ define amdgpu_kernel void @private_system_unordered_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -1496,10 +1540,14 @@ define amdgpu_kernel void @private_system_monotonic_store(
; GFX1250-NOGAS-LABEL: private_system_monotonic_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -1507,22 +1555,26 @@ define amdgpu_kernel void @private_system_monotonic_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -1737,10 +1789,14 @@ define amdgpu_kernel void @private_system_release_store(
; GFX1250-NOGAS-LABEL: private_system_release_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -1748,22 +1804,26 @@ define amdgpu_kernel void @private_system_release_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -1983,10 +2043,14 @@ define amdgpu_kernel void @private_system_seq_cst_store(
; GFX1250-NOGAS-LABEL: private_system_seq_cst_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -1994,22 +2058,26 @@ define amdgpu_kernel void @private_system_seq_cst_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -2263,8 +2331,14 @@ define amdgpu_kernel void @private_system_monotonic_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -2272,29 +2346,34 @@ define amdgpu_kernel void @private_system_monotonic_atomicrmw(
; GFX1250-GAS-LABEL: private_system_monotonic_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
@@ -2537,8 +2616,14 @@ define amdgpu_kernel void @private_system_acquire_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -2546,29 +2631,34 @@ define amdgpu_kernel void @private_system_acquire_atomicrmw(
; GFX1250-GAS-LABEL: private_system_acquire_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
@@ -2814,8 +2904,14 @@ define amdgpu_kernel void @private_system_release_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -2823,29 +2919,34 @@ define amdgpu_kernel void @private_system_release_atomicrmw(
; GFX1250-GAS-LABEL: private_system_release_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
@@ -3093,8 +3194,14 @@ define amdgpu_kernel void @private_system_acq_rel_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -3102,29 +3209,34 @@ define amdgpu_kernel void @private_system_acq_rel_atomicrmw(
; GFX1250-GAS-LABEL: private_system_acq_rel_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
@@ -3375,8 +3487,14 @@ define amdgpu_kernel void @private_system_seq_cst_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -3384,29 +3502,34 @@ define amdgpu_kernel void @private_system_seq_cst_atomicrmw(
; GFX1250-GAS-LABEL: private_system_seq_cst_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
@@ -3678,6 +3801,10 @@ define amdgpu_kernel void @private_system_acquire_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -3691,22 +3818,26 @@ define amdgpu_kernel void @private_system_acquire_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -3981,6 +4112,10 @@ define amdgpu_kernel void @private_system_acq_rel_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -3994,22 +4129,26 @@ define amdgpu_kernel void @private_system_acq_rel_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -4289,6 +4428,10 @@ define amdgpu_kernel void @private_system_seq_cst_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -4302,22 +4445,26 @@ define amdgpu_kernel void @private_system_seq_cst_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -4653,6 +4800,12 @@ define amdgpu_kernel void @private_system_monotonic_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -4666,26 +4819,32 @@ define amdgpu_kernel void @private_system_monotonic_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_system_monotonic_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -4693,6 +4852,7 @@ define amdgpu_kernel void @private_system_monotonic_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -5015,6 +5175,12 @@ define amdgpu_kernel void @private_system_acquire_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -5028,26 +5194,32 @@ define amdgpu_kernel void @private_system_acquire_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_system_acquire_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -5055,6 +5227,7 @@ define amdgpu_kernel void @private_system_acquire_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -5380,6 +5553,12 @@ define amdgpu_kernel void @private_system_release_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -5393,26 +5572,32 @@ define amdgpu_kernel void @private_system_release_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_system_release_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -5420,6 +5605,7 @@ define amdgpu_kernel void @private_system_release_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -5747,6 +5933,12 @@ define amdgpu_kernel void @private_system_acq_rel_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -5760,26 +5952,32 @@ define amdgpu_kernel void @private_system_acq_rel_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_system_acq_rel_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -5787,6 +5985,7 @@ define amdgpu_kernel void @private_system_acq_rel_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -6117,6 +6316,12 @@ define amdgpu_kernel void @private_system_seq_cst_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -6130,26 +6335,32 @@ define amdgpu_kernel void @private_system_seq_cst_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_system_seq_cst_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -6157,6 +6368,7 @@ define amdgpu_kernel void @private_system_seq_cst_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -6487,6 +6699,12 @@ define amdgpu_kernel void @private_system_monotonic_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -6500,26 +6718,32 @@ define amdgpu_kernel void @private_system_monotonic_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_system_monotonic_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -6527,6 +6751,7 @@ define amdgpu_kernel void @private_system_monotonic_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -6852,6 +7077,12 @@ define amdgpu_kernel void @private_system_acquire_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -6865,26 +7096,32 @@ define amdgpu_kernel void @private_system_acquire_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_system_acquire_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -6892,6 +7129,7 @@ define amdgpu_kernel void @private_system_acquire_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -7217,6 +7455,12 @@ define amdgpu_kernel void @private_system_release_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -7230,26 +7474,32 @@ define amdgpu_kernel void @private_system_release_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_system_release_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -7257,6 +7507,7 @@ define amdgpu_kernel void @private_system_release_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -7587,6 +7838,12 @@ define amdgpu_kernel void @private_system_acq_rel_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -7600,26 +7857,32 @@ define amdgpu_kernel void @private_system_acq_rel_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_system_acq_rel_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -7627,6 +7890,7 @@ define amdgpu_kernel void @private_system_acq_rel_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -7957,6 +8221,12 @@ define amdgpu_kernel void @private_system_seq_cst_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -7970,26 +8240,32 @@ define amdgpu_kernel void @private_system_seq_cst_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_system_seq_cst_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -7997,6 +8273,7 @@ define amdgpu_kernel void @private_system_seq_cst_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -8327,6 +8604,12 @@ define amdgpu_kernel void @private_system_seq_cst_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -8340,26 +8623,32 @@ define amdgpu_kernel void @private_system_seq_cst_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_system_seq_cst_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -8367,6 +8656,7 @@ define amdgpu_kernel void @private_system_seq_cst_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -8725,6 +9015,12 @@ define amdgpu_kernel void @private_system_monotonic_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -8740,25 +9036,31 @@ define amdgpu_kernel void @private_system_monotonic_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -9120,6 +9422,12 @@ define amdgpu_kernel void @private_system_acquire_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -9135,25 +9443,31 @@ define amdgpu_kernel void @private_system_acquire_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -9517,6 +9831,12 @@ define amdgpu_kernel void @private_system_acq_rel_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -9532,25 +9852,31 @@ define amdgpu_kernel void @private_system_acq_rel_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -9919,6 +10245,12 @@ define amdgpu_kernel void @private_system_seq_cst_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -9934,25 +10266,31 @@ define amdgpu_kernel void @private_system_seq_cst_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -10321,6 +10659,12 @@ define amdgpu_kernel void @private_system_monotonic_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -10336,25 +10680,31 @@ define amdgpu_kernel void @private_system_monotonic_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -10718,6 +11068,12 @@ define amdgpu_kernel void @private_system_acquire_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -10733,25 +11089,31 @@ define amdgpu_kernel void @private_system_acquire_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -11115,6 +11477,12 @@ define amdgpu_kernel void @private_system_release_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -11130,25 +11498,31 @@ define amdgpu_kernel void @private_system_release_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -11517,6 +11891,12 @@ define amdgpu_kernel void @private_system_acq_rel_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -11532,25 +11912,31 @@ define amdgpu_kernel void @private_system_acq_rel_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -11919,6 +12305,12 @@ define amdgpu_kernel void @private_system_seq_cst_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -11934,25 +12326,31 @@ define amdgpu_kernel void @private_system_seq_cst_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -12321,6 +12719,12 @@ define amdgpu_kernel void @private_system_monotonic_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -12336,25 +12740,31 @@ define amdgpu_kernel void @private_system_monotonic_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -12723,6 +13133,12 @@ define amdgpu_kernel void @private_system_acquire_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -12738,25 +13154,31 @@ define amdgpu_kernel void @private_system_acquire_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -13125,6 +13547,12 @@ define amdgpu_kernel void @private_system_relese_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -13140,25 +13568,31 @@ define amdgpu_kernel void @private_system_relese_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -13527,6 +13961,12 @@ define amdgpu_kernel void @private_system_acq_rel_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -13542,25 +13982,31 @@ define amdgpu_kernel void @private_system_acq_rel_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -13929,6 +14375,12 @@ define amdgpu_kernel void @private_system_seq_cst_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -13944,25 +14396,31 @@ define amdgpu_kernel void @private_system_seq_cst_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -14208,34 +14666,42 @@ define amdgpu_kernel void @private_system_one_as_unordered_load(
; GFX1250-NOGAS-LABEL: private_system_one_as_unordered_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_system_one_as_unordered_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -14244,6 +14710,7 @@ define amdgpu_kernel void @private_system_one_as_unordered_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -14467,34 +14934,42 @@ define amdgpu_kernel void @private_system_one_as_monotonic_load(
; GFX1250-NOGAS-LABEL: private_system_one_as_monotonic_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_system_one_as_monotonic_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -14503,6 +14978,7 @@ define amdgpu_kernel void @private_system_one_as_monotonic_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SYS
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -14726,34 +15202,42 @@ define amdgpu_kernel void @private_system_one_as_acquire_load(
; GFX1250-NOGAS-LABEL: private_system_one_as_acquire_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_system_one_as_acquire_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -14764,6 +15248,7 @@ define amdgpu_kernel void @private_system_one_as_acquire_load(
; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -14987,34 +15472,42 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_load(
; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -15027,6 +15520,7 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_load(
; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -15234,10 +15728,14 @@ define amdgpu_kernel void @private_system_one_as_unordered_store(
; GFX1250-NOGAS-LABEL: private_system_one_as_unordered_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -15245,22 +15743,26 @@ define amdgpu_kernel void @private_system_one_as_unordered_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -15475,10 +15977,14 @@ define amdgpu_kernel void @private_system_one_as_monotonic_store(
; GFX1250-NOGAS-LABEL: private_system_one_as_monotonic_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -15486,22 +15992,26 @@ define amdgpu_kernel void @private_system_one_as_monotonic_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -15716,10 +16226,14 @@ define amdgpu_kernel void @private_system_one_as_release_store(
; GFX1250-NOGAS-LABEL: private_system_one_as_release_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -15727,22 +16241,26 @@ define amdgpu_kernel void @private_system_one_as_release_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -15962,10 +16480,14 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_store(
; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -15973,22 +16495,26 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -16242,8 +16768,14 @@ define amdgpu_kernel void @private_system_one_as_monotonic_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -16251,29 +16783,34 @@ define amdgpu_kernel void @private_system_one_as_monotonic_atomicrmw(
; GFX1250-GAS-LABEL: private_system_one_as_monotonic_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
@@ -16516,8 +17053,14 @@ define amdgpu_kernel void @private_system_one_as_acquire_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -16525,29 +17068,34 @@ define amdgpu_kernel void @private_system_one_as_acquire_atomicrmw(
; GFX1250-GAS-LABEL: private_system_one_as_acquire_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
@@ -16793,8 +17341,14 @@ define amdgpu_kernel void @private_system_one_as_release_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -16802,29 +17356,34 @@ define amdgpu_kernel void @private_system_one_as_release_atomicrmw(
; GFX1250-GAS-LABEL: private_system_one_as_release_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
@@ -17072,8 +17631,14 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -17081,29 +17646,34 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_atomicrmw(
; GFX1250-GAS-LABEL: private_system_one_as_acq_rel_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
@@ -17354,8 +17924,14 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -17363,29 +17939,34 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_atomicrmw(
; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
@@ -17657,6 +18238,10 @@ define amdgpu_kernel void @private_system_one_as_acquire_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -17670,22 +18255,26 @@ define amdgpu_kernel void @private_system_one_as_acquire_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -17960,6 +18549,10 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -17973,22 +18566,26 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -18268,6 +18865,10 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -18281,22 +18882,26 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -18632,6 +19237,12 @@ define amdgpu_kernel void @private_system_one_as_monotonic_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -18645,26 +19256,32 @@ define amdgpu_kernel void @private_system_one_as_monotonic_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_monotonic_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -18672,6 +19289,7 @@ define amdgpu_kernel void @private_system_one_as_monotonic_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -18994,6 +19612,12 @@ define amdgpu_kernel void @private_system_one_as_acquire_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -19007,26 +19631,32 @@ define amdgpu_kernel void @private_system_one_as_acquire_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_acquire_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -19034,6 +19664,7 @@ define amdgpu_kernel void @private_system_one_as_acquire_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -19359,6 +19990,12 @@ define amdgpu_kernel void @private_system_one_as_release_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -19372,26 +20009,32 @@ define amdgpu_kernel void @private_system_one_as_release_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_release_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -19399,6 +20042,7 @@ define amdgpu_kernel void @private_system_one_as_release_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -19726,6 +20370,12 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -19739,26 +20389,32 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_acq_rel_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -19766,6 +20422,7 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -20096,6 +20753,12 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -20109,26 +20772,32 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -20136,6 +20805,7 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -20466,6 +21136,12 @@ define amdgpu_kernel void @private_system_one_as_monotonic_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -20479,26 +21155,32 @@ define amdgpu_kernel void @private_system_one_as_monotonic_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_monotonic_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -20506,6 +21188,7 @@ define amdgpu_kernel void @private_system_one_as_monotonic_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -20831,6 +21514,12 @@ define amdgpu_kernel void @private_system_one_as_acquire_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -20844,26 +21533,32 @@ define amdgpu_kernel void @private_system_one_as_acquire_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_acquire_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -20871,6 +21566,7 @@ define amdgpu_kernel void @private_system_one_as_acquire_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -21196,6 +21892,12 @@ define amdgpu_kernel void @private_system_one_as_release_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -21209,26 +21911,32 @@ define amdgpu_kernel void @private_system_one_as_release_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_release_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -21236,6 +21944,7 @@ define amdgpu_kernel void @private_system_one_as_release_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -21566,6 +22275,12 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -21579,26 +22294,32 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_acq_rel_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -21606,6 +22327,7 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -21936,6 +22658,12 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -21949,26 +22677,32 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -21976,6 +22710,7 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -22306,6 +23041,12 @@ define amdgpu_kernel void @private_system_one_as_monotonic_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -22319,26 +23060,32 @@ define amdgpu_kernel void @private_system_one_as_monotonic_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_monotonic_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -22346,6 +23093,7 @@ define amdgpu_kernel void @private_system_one_as_monotonic_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -22676,6 +23424,12 @@ define amdgpu_kernel void @private_system_one_as_acquire_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -22689,26 +23443,32 @@ define amdgpu_kernel void @private_system_one_as_acquire_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_acquire_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -22716,6 +23476,7 @@ define amdgpu_kernel void @private_system_one_as_acquire_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -23046,6 +23807,12 @@ define amdgpu_kernel void @private_system_one_as_release_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -23059,26 +23826,32 @@ define amdgpu_kernel void @private_system_one_as_release_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_release_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -23086,6 +23859,7 @@ define amdgpu_kernel void @private_system_one_as_release_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -23416,6 +24190,12 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -23429,26 +24209,32 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_acq_rel_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -23456,6 +24242,7 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -23786,6 +24573,12 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -23799,26 +24592,32 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -23826,6 +24625,7 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -24184,6 +24984,12 @@ define amdgpu_kernel void @private_system_one_as_monotonic_monotonic_ret_cmpxchg
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -24199,25 +25005,31 @@ define amdgpu_kernel void @private_system_one_as_monotonic_monotonic_ret_cmpxchg
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -24579,6 +25391,12 @@ define amdgpu_kernel void @private_system_one_as_acquire_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -24594,25 +25412,31 @@ define amdgpu_kernel void @private_system_one_as_acquire_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -24976,6 +25800,12 @@ define amdgpu_kernel void @private_system_one_as_release_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -24991,25 +25821,31 @@ define amdgpu_kernel void @private_system_one_as_release_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -25376,6 +26212,12 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -25391,25 +26233,31 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -25778,6 +26626,12 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -25793,25 +26647,31 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -26180,6 +27040,12 @@ define amdgpu_kernel void @private_system_one_as_monotonic_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -26195,25 +27061,31 @@ define amdgpu_kernel void @private_system_one_as_monotonic_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -26577,6 +27449,12 @@ define amdgpu_kernel void @private_system_one_as_acquire_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -26592,25 +27470,31 @@ define amdgpu_kernel void @private_system_one_as_acquire_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -26974,6 +27858,12 @@ define amdgpu_kernel void @private_system_one_as_release_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -26989,25 +27879,31 @@ define amdgpu_kernel void @private_system_one_as_release_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -27376,6 +28272,12 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -27391,25 +28293,31 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -27778,6 +28686,12 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -27793,25 +28707,31 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -28180,6 +29100,12 @@ define amdgpu_kernel void @private_system_one_as_monotonic_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -28195,25 +29121,31 @@ define amdgpu_kernel void @private_system_one_as_monotonic_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -28582,6 +29514,12 @@ define amdgpu_kernel void @private_system_one_as_acquire_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -28597,25 +29535,31 @@ define amdgpu_kernel void @private_system_one_as_acquire_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -28984,6 +29928,12 @@ define amdgpu_kernel void @private_system_one_as_release_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -28999,25 +29949,31 @@ define amdgpu_kernel void @private_system_one_as_release_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -29386,6 +30342,12 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -29401,25 +30363,31 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -29788,6 +30756,12 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -29803,25 +30777,31 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-wavefront.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-wavefront.ll
index 91f559e66bfee..9c8d226ffd723 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-wavefront.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-wavefront.ll
@@ -229,34 +229,42 @@ define amdgpu_kernel void @private_wavefront_unordered_load(
; GFX1250-NOGAS-LABEL: private_wavefront_unordered_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_wavefront_unordered_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -265,6 +273,7 @@ define amdgpu_kernel void @private_wavefront_unordered_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -488,34 +497,42 @@ define amdgpu_kernel void @private_wavefront_monotonic_load(
; GFX1250-NOGAS-LABEL: private_wavefront_monotonic_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_wavefront_monotonic_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -524,6 +541,7 @@ define amdgpu_kernel void @private_wavefront_monotonic_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -747,34 +765,42 @@ define amdgpu_kernel void @private_wavefront_acquire_load(
; GFX1250-NOGAS-LABEL: private_wavefront_acquire_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_wavefront_acquire_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -783,6 +809,7 @@ define amdgpu_kernel void @private_wavefront_acquire_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -1006,34 +1033,42 @@ define amdgpu_kernel void @private_wavefront_seq_cst_load(
; GFX1250-NOGAS-LABEL: private_wavefront_seq_cst_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_wavefront_seq_cst_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -1042,6 +1077,7 @@ define amdgpu_kernel void @private_wavefront_seq_cst_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -1249,10 +1285,14 @@ define amdgpu_kernel void @private_wavefront_unordered_store(
; GFX1250-NOGAS-LABEL: private_wavefront_unordered_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -1260,22 +1300,26 @@ define amdgpu_kernel void @private_wavefront_unordered_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -1490,10 +1534,14 @@ define amdgpu_kernel void @private_wavefront_monotonic_store(
; GFX1250-NOGAS-LABEL: private_wavefront_monotonic_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -1501,22 +1549,26 @@ define amdgpu_kernel void @private_wavefront_monotonic_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -1731,10 +1783,14 @@ define amdgpu_kernel void @private_wavefront_release_store(
; GFX1250-NOGAS-LABEL: private_wavefront_release_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -1742,22 +1798,26 @@ define amdgpu_kernel void @private_wavefront_release_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -1972,10 +2032,14 @@ define amdgpu_kernel void @private_wavefront_seq_cst_store(
; GFX1250-NOGAS-LABEL: private_wavefront_seq_cst_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -1983,22 +2047,26 @@ define amdgpu_kernel void @private_wavefront_seq_cst_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -2247,8 +2315,14 @@ define amdgpu_kernel void @private_wavefront_monotonic_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -2256,29 +2330,34 @@ define amdgpu_kernel void @private_wavefront_monotonic_atomicrmw(
; GFX1250-GAS-LABEL: private_wavefront_monotonic_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
@@ -2521,8 +2600,14 @@ define amdgpu_kernel void @private_wavefront_acquire_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -2530,29 +2615,34 @@ define amdgpu_kernel void @private_wavefront_acquire_atomicrmw(
; GFX1250-GAS-LABEL: private_wavefront_acquire_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
@@ -2795,8 +2885,14 @@ define amdgpu_kernel void @private_wavefront_release_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -2804,29 +2900,34 @@ define amdgpu_kernel void @private_wavefront_release_atomicrmw(
; GFX1250-GAS-LABEL: private_wavefront_release_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
@@ -3069,8 +3170,14 @@ define amdgpu_kernel void @private_wavefront_acq_rel_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -3078,29 +3185,34 @@ define amdgpu_kernel void @private_wavefront_acq_rel_atomicrmw(
; GFX1250-GAS-LABEL: private_wavefront_acq_rel_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
@@ -3343,8 +3455,14 @@ define amdgpu_kernel void @private_wavefront_seq_cst_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -3352,29 +3470,34 @@ define amdgpu_kernel void @private_wavefront_seq_cst_atomicrmw(
; GFX1250-GAS-LABEL: private_wavefront_seq_cst_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
@@ -3638,6 +3761,10 @@ define amdgpu_kernel void @private_wavefront_acquire_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -3651,22 +3778,26 @@ define amdgpu_kernel void @private_wavefront_acquire_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -3939,6 +4070,10 @@ define amdgpu_kernel void @private_wavefront_acq_rel_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -3952,22 +4087,26 @@ define amdgpu_kernel void @private_wavefront_acq_rel_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -4240,6 +4379,10 @@ define amdgpu_kernel void @private_wavefront_seq_cst_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -4253,22 +4396,26 @@ define amdgpu_kernel void @private_wavefront_seq_cst_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -4597,6 +4744,12 @@ define amdgpu_kernel void @private_wavefront_monotonic_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -4610,26 +4763,32 @@ define amdgpu_kernel void @private_wavefront_monotonic_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_wavefront_monotonic_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -4637,6 +4796,7 @@ define amdgpu_kernel void @private_wavefront_monotonic_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -4959,6 +5119,12 @@ define amdgpu_kernel void @private_wavefront_acquire_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -4972,26 +5138,32 @@ define amdgpu_kernel void @private_wavefront_acquire_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_wavefront_acquire_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -4999,6 +5171,7 @@ define amdgpu_kernel void @private_wavefront_acquire_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -5321,6 +5494,12 @@ define amdgpu_kernel void @private_wavefront_release_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -5334,26 +5513,32 @@ define amdgpu_kernel void @private_wavefront_release_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_wavefront_release_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -5361,6 +5546,7 @@ define amdgpu_kernel void @private_wavefront_release_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -5683,6 +5869,12 @@ define amdgpu_kernel void @private_wavefront_acq_rel_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -5696,26 +5888,32 @@ define amdgpu_kernel void @private_wavefront_acq_rel_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_wavefront_acq_rel_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -5723,6 +5921,7 @@ define amdgpu_kernel void @private_wavefront_acq_rel_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -6045,6 +6244,12 @@ define amdgpu_kernel void @private_wavefront_seq_cst_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -6058,26 +6263,32 @@ define amdgpu_kernel void @private_wavefront_seq_cst_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_wavefront_seq_cst_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -6085,6 +6296,7 @@ define amdgpu_kernel void @private_wavefront_seq_cst_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -6407,6 +6619,12 @@ define amdgpu_kernel void @private_wavefront_monotonic_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -6420,26 +6638,32 @@ define amdgpu_kernel void @private_wavefront_monotonic_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_wavefront_monotonic_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -6447,6 +6671,7 @@ define amdgpu_kernel void @private_wavefront_monotonic_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -6769,6 +6994,12 @@ define amdgpu_kernel void @private_wavefront_acquire_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -6782,26 +7013,32 @@ define amdgpu_kernel void @private_wavefront_acquire_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_wavefront_acquire_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -6809,6 +7046,7 @@ define amdgpu_kernel void @private_wavefront_acquire_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -7131,6 +7369,12 @@ define amdgpu_kernel void @private_wavefront_release_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -7144,26 +7388,32 @@ define amdgpu_kernel void @private_wavefront_release_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_wavefront_release_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -7171,6 +7421,7 @@ define amdgpu_kernel void @private_wavefront_release_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -7493,6 +7744,12 @@ define amdgpu_kernel void @private_wavefront_acq_rel_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -7506,26 +7763,32 @@ define amdgpu_kernel void @private_wavefront_acq_rel_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_wavefront_acq_rel_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -7533,6 +7796,7 @@ define amdgpu_kernel void @private_wavefront_acq_rel_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -7855,6 +8119,12 @@ define amdgpu_kernel void @private_wavefront_seq_cst_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -7868,26 +8138,32 @@ define amdgpu_kernel void @private_wavefront_seq_cst_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_wavefront_seq_cst_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -7895,6 +8171,7 @@ define amdgpu_kernel void @private_wavefront_seq_cst_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -8217,6 +8494,12 @@ define amdgpu_kernel void @private_wavefront_monotonic_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -8230,26 +8513,32 @@ define amdgpu_kernel void @private_wavefront_monotonic_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_wavefront_monotonic_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -8257,6 +8546,7 @@ define amdgpu_kernel void @private_wavefront_monotonic_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -8579,6 +8869,12 @@ define amdgpu_kernel void @private_wavefront_acquire_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -8592,26 +8888,32 @@ define amdgpu_kernel void @private_wavefront_acquire_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_wavefront_acquire_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -8619,6 +8921,7 @@ define amdgpu_kernel void @private_wavefront_acquire_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -8941,6 +9244,12 @@ define amdgpu_kernel void @private_wavefront_release_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -8954,26 +9263,32 @@ define amdgpu_kernel void @private_wavefront_release_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_wavefront_release_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -8981,6 +9296,7 @@ define amdgpu_kernel void @private_wavefront_release_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -9303,6 +9619,12 @@ define amdgpu_kernel void @private_wavefront_acq_rel_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -9316,26 +9638,32 @@ define amdgpu_kernel void @private_wavefront_acq_rel_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_wavefront_acq_rel_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -9343,6 +9671,7 @@ define amdgpu_kernel void @private_wavefront_acq_rel_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -9665,6 +9994,12 @@ define amdgpu_kernel void @private_wavefront_seq_cst_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -9678,26 +10013,32 @@ define amdgpu_kernel void @private_wavefront_seq_cst_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_wavefront_seq_cst_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -9705,6 +10046,7 @@ define amdgpu_kernel void @private_wavefront_seq_cst_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -10055,6 +10397,12 @@ define amdgpu_kernel void @private_wavefront_monotonic_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -10070,25 +10418,31 @@ define amdgpu_kernel void @private_wavefront_monotonic_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -10450,6 +10804,12 @@ define amdgpu_kernel void @private_wavefront_acquire_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -10465,25 +10825,31 @@ define amdgpu_kernel void @private_wavefront_acquire_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -10845,6 +11211,12 @@ define amdgpu_kernel void @private_wavefront_release_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -10860,25 +11232,31 @@ define amdgpu_kernel void @private_wavefront_release_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -11240,6 +11618,12 @@ define amdgpu_kernel void @private_wavefront_acq_rel_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -11255,25 +11639,31 @@ define amdgpu_kernel void @private_wavefront_acq_rel_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -11635,6 +12025,12 @@ define amdgpu_kernel void @private_wavefront_seq_cst_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -11650,25 +12046,31 @@ define amdgpu_kernel void @private_wavefront_seq_cst_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -12030,6 +12432,12 @@ define amdgpu_kernel void @private_wavefront_monotonic_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -12045,25 +12453,31 @@ define amdgpu_kernel void @private_wavefront_monotonic_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -12425,6 +12839,12 @@ define amdgpu_kernel void @private_wavefront_acquire_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -12440,25 +12860,31 @@ define amdgpu_kernel void @private_wavefront_acquire_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -12820,6 +13246,12 @@ define amdgpu_kernel void @private_wavefront_release_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -12835,25 +13267,31 @@ define amdgpu_kernel void @private_wavefront_release_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -13215,6 +13653,12 @@ define amdgpu_kernel void @private_wavefront_acq_rel_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -13230,25 +13674,31 @@ define amdgpu_kernel void @private_wavefront_acq_rel_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -13610,6 +14060,12 @@ define amdgpu_kernel void @private_wavefront_seq_cst_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -13625,25 +14081,31 @@ define amdgpu_kernel void @private_wavefront_seq_cst_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -14005,6 +14467,12 @@ define amdgpu_kernel void @private_wavefront_monotonic_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -14020,25 +14488,31 @@ define amdgpu_kernel void @private_wavefront_monotonic_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -14400,6 +14874,12 @@ define amdgpu_kernel void @private_wavefront_acquire_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -14415,25 +14895,31 @@ define amdgpu_kernel void @private_wavefront_acquire_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -14795,6 +15281,12 @@ define amdgpu_kernel void @private_wavefront_release_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -14810,25 +15302,31 @@ define amdgpu_kernel void @private_wavefront_release_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -15190,6 +15688,12 @@ define amdgpu_kernel void @private_wavefront_acq_rel_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -15205,25 +15709,31 @@ define amdgpu_kernel void @private_wavefront_acq_rel_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -15585,6 +16095,12 @@ define amdgpu_kernel void @private_wavefront_seq_cst_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -15600,25 +16116,31 @@ define amdgpu_kernel void @private_wavefront_seq_cst_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -15857,34 +16379,42 @@ define amdgpu_kernel void @private_wavefront_one_as_unordered_load(
; GFX1250-NOGAS-LABEL: private_wavefront_one_as_unordered_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_wavefront_one_as_unordered_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -15893,6 +16423,7 @@ define amdgpu_kernel void @private_wavefront_one_as_unordered_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -16116,34 +16647,42 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_load(
; GFX1250-NOGAS-LABEL: private_wavefront_one_as_monotonic_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_wavefront_one_as_monotonic_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -16152,6 +16691,7 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -16375,34 +16915,42 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_load(
; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acquire_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_wavefront_one_as_acquire_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -16411,6 +16959,7 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -16634,34 +17183,42 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_load(
; GFX1250-NOGAS-LABEL: private_wavefront_one_as_seq_cst_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_wavefront_one_as_seq_cst_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -16670,6 +17227,7 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -16877,10 +17435,14 @@ define amdgpu_kernel void @private_wavefront_one_as_unordered_store(
; GFX1250-NOGAS-LABEL: private_wavefront_one_as_unordered_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -16888,22 +17450,26 @@ define amdgpu_kernel void @private_wavefront_one_as_unordered_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -17118,10 +17684,14 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_store(
; GFX1250-NOGAS-LABEL: private_wavefront_one_as_monotonic_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -17129,22 +17699,26 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -17359,10 +17933,14 @@ define amdgpu_kernel void @private_wavefront_one_as_release_store(
; GFX1250-NOGAS-LABEL: private_wavefront_one_as_release_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -17370,22 +17948,26 @@ define amdgpu_kernel void @private_wavefront_one_as_release_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -17600,10 +18182,14 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_store(
; GFX1250-NOGAS-LABEL: private_wavefront_one_as_seq_cst_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -17611,22 +18197,26 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -17875,8 +18465,14 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -17884,29 +18480,34 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_atomicrmw(
; GFX1250-GAS-LABEL: private_wavefront_one_as_monotonic_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
@@ -18149,8 +18750,14 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -18158,29 +18765,34 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_atomicrmw(
; GFX1250-GAS-LABEL: private_wavefront_one_as_acquire_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
@@ -18423,8 +19035,14 @@ define amdgpu_kernel void @private_wavefront_one_as_release_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -18432,29 +19050,34 @@ define amdgpu_kernel void @private_wavefront_one_as_release_atomicrmw(
; GFX1250-GAS-LABEL: private_wavefront_one_as_release_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
@@ -18697,8 +19320,14 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -18706,29 +19335,34 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_atomicrmw(
; GFX1250-GAS-LABEL: private_wavefront_one_as_acq_rel_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
@@ -18971,8 +19605,14 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -18980,29 +19620,34 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_atomicrmw(
; GFX1250-GAS-LABEL: private_wavefront_one_as_seq_cst_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
@@ -19266,6 +19911,10 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -19279,22 +19928,26 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -19567,6 +20220,10 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -19580,22 +20237,26 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -19868,6 +20529,10 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -19881,22 +20546,26 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -20225,6 +20894,12 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -20238,26 +20913,32 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_wavefront_one_as_monotonic_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -20265,6 +20946,7 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -20587,6 +21269,12 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -20600,26 +21288,32 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_wavefront_one_as_acquire_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -20627,6 +21321,7 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -20949,6 +21644,12 @@ define amdgpu_kernel void @private_wavefront_one_as_release_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -20962,26 +21663,32 @@ define amdgpu_kernel void @private_wavefront_one_as_release_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_wavefront_one_as_release_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -20989,6 +21696,7 @@ define amdgpu_kernel void @private_wavefront_one_as_release_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -21311,6 +22019,12 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -21324,26 +22038,32 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_wavefront_one_as_acq_rel_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -21351,6 +22071,7 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -21673,6 +22394,12 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -21686,26 +22413,32 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_wavefront_one_as_seq_cst_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -21713,6 +22446,7 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -22035,6 +22769,12 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -22048,26 +22788,32 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_wavefront_one_as_monotonic_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -22075,6 +22821,7 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -22397,6 +23144,12 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -22410,26 +23163,32 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_wavefront_one_as_acquire_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -22437,6 +23196,7 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -22759,6 +23519,12 @@ define amdgpu_kernel void @private_wavefront_one_as_release_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -22772,26 +23538,32 @@ define amdgpu_kernel void @private_wavefront_one_as_release_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_wavefront_one_as_release_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -22799,6 +23571,7 @@ define amdgpu_kernel void @private_wavefront_one_as_release_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -23121,6 +23894,12 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -23134,26 +23913,32 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_wavefront_one_as_acq_rel_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -23161,6 +23946,7 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -23483,6 +24269,12 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -23496,26 +24288,32 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_wavefront_one_as_seq_cst_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -23523,6 +24321,7 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -23845,6 +24644,12 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -23858,26 +24663,32 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_wavefront_one_as_monotonic_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -23885,6 +24696,7 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -24207,6 +25019,12 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -24220,26 +25038,32 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_wavefront_one_as_acquire_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -24247,6 +25071,7 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -24569,6 +25394,12 @@ define amdgpu_kernel void @private_wavefront_one_as_release_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -24582,26 +25413,32 @@ define amdgpu_kernel void @private_wavefront_one_as_release_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_wavefront_one_as_release_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -24609,6 +25446,7 @@ define amdgpu_kernel void @private_wavefront_one_as_release_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -24931,6 +25769,12 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -24944,26 +25788,32 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_wavefront_one_as_acq_rel_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -24971,6 +25821,7 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -25293,6 +26144,12 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -25306,26 +26163,32 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_wavefront_one_as_seq_cst_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -25333,6 +26196,7 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -25683,6 +26547,12 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_monotonic_ret_cmpx
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -25698,25 +26568,31 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_monotonic_ret_cmpx
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -26078,6 +26954,12 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_monotonic_ret_cmpxch
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -26093,25 +26975,31 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_monotonic_ret_cmpxch
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -26473,6 +27361,12 @@ define amdgpu_kernel void @private_wavefront_one_as_release_monotonic_ret_cmpxch
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -26488,25 +27382,31 @@ define amdgpu_kernel void @private_wavefront_one_as_release_monotonic_ret_cmpxch
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -26868,6 +27768,12 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_monotonic_ret_cmpxch
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -26883,25 +27789,31 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_monotonic_ret_cmpxch
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -27263,6 +28175,12 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_monotonic_ret_cmpxch
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -27278,25 +28196,31 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_monotonic_ret_cmpxch
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -27658,6 +28582,12 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_acquire_ret_cmpxch
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -27673,25 +28603,31 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_acquire_ret_cmpxch
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -28053,6 +28989,12 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -28068,25 +29010,31 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -28448,6 +29396,12 @@ define amdgpu_kernel void @private_wavefront_one_as_release_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -28463,25 +29417,31 @@ define amdgpu_kernel void @private_wavefront_one_as_release_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -28843,6 +29803,12 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -28858,25 +29824,31 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -29238,6 +30210,12 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -29253,25 +30231,31 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -29633,6 +30617,12 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_seq_cst_ret_cmpxch
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -29648,25 +30638,31 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_seq_cst_ret_cmpxch
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -30028,6 +31024,12 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -30043,25 +31045,31 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -30423,6 +31431,12 @@ define amdgpu_kernel void @private_wavefront_one_as_release_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -30438,25 +31452,31 @@ define amdgpu_kernel void @private_wavefront_one_as_release_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -30818,6 +31838,12 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -30833,25 +31859,31 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -31213,6 +32245,12 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -31228,25 +32266,31 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-workgroup.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-workgroup.ll
index 5f1845fdd024f..b0e6dd7ef8499 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-workgroup.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-workgroup.ll
@@ -229,34 +229,42 @@ define amdgpu_kernel void @private_workgroup_unordered_load(
; GFX1250-NOGAS-LABEL: private_workgroup_unordered_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_workgroup_unordered_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -265,6 +273,7 @@ define amdgpu_kernel void @private_workgroup_unordered_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -488,34 +497,42 @@ define amdgpu_kernel void @private_workgroup_monotonic_load(
; GFX1250-NOGAS-LABEL: private_workgroup_monotonic_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_workgroup_monotonic_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -524,6 +541,7 @@ define amdgpu_kernel void @private_workgroup_monotonic_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -747,34 +765,42 @@ define amdgpu_kernel void @private_workgroup_acquire_load(
; GFX1250-NOGAS-LABEL: private_workgroup_acquire_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_workgroup_acquire_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -783,6 +809,7 @@ define amdgpu_kernel void @private_workgroup_acquire_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -1006,34 +1033,42 @@ define amdgpu_kernel void @private_workgroup_seq_cst_load(
; GFX1250-NOGAS-LABEL: private_workgroup_seq_cst_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_workgroup_seq_cst_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -1044,6 +1079,7 @@ define amdgpu_kernel void @private_workgroup_seq_cst_load(
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -1251,10 +1287,14 @@ define amdgpu_kernel void @private_workgroup_unordered_store(
; GFX1250-NOGAS-LABEL: private_workgroup_unordered_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -1262,22 +1302,26 @@ define amdgpu_kernel void @private_workgroup_unordered_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -1492,10 +1536,14 @@ define amdgpu_kernel void @private_workgroup_monotonic_store(
; GFX1250-NOGAS-LABEL: private_workgroup_monotonic_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -1503,22 +1551,26 @@ define amdgpu_kernel void @private_workgroup_monotonic_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -1733,10 +1785,14 @@ define amdgpu_kernel void @private_workgroup_release_store(
; GFX1250-NOGAS-LABEL: private_workgroup_release_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -1744,22 +1800,26 @@ define amdgpu_kernel void @private_workgroup_release_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -1976,10 +2036,14 @@ define amdgpu_kernel void @private_workgroup_seq_cst_store(
; GFX1250-NOGAS-LABEL: private_workgroup_seq_cst_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -1987,22 +2051,26 @@ define amdgpu_kernel void @private_workgroup_seq_cst_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -2253,8 +2321,14 @@ define amdgpu_kernel void @private_workgroup_monotonic_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -2262,29 +2336,34 @@ define amdgpu_kernel void @private_workgroup_monotonic_atomicrmw(
; GFX1250-GAS-LABEL: private_workgroup_monotonic_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
@@ -2527,8 +2606,14 @@ define amdgpu_kernel void @private_workgroup_acquire_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -2536,29 +2621,34 @@ define amdgpu_kernel void @private_workgroup_acquire_atomicrmw(
; GFX1250-GAS-LABEL: private_workgroup_acquire_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
@@ -2802,8 +2892,14 @@ define amdgpu_kernel void @private_workgroup_release_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -2811,29 +2907,34 @@ define amdgpu_kernel void @private_workgroup_release_atomicrmw(
; GFX1250-GAS-LABEL: private_workgroup_release_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
@@ -3078,8 +3179,14 @@ define amdgpu_kernel void @private_workgroup_acq_rel_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -3087,29 +3194,34 @@ define amdgpu_kernel void @private_workgroup_acq_rel_atomicrmw(
; GFX1250-GAS-LABEL: private_workgroup_acq_rel_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
@@ -3355,8 +3467,14 @@ define amdgpu_kernel void @private_workgroup_seq_cst_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -3364,29 +3482,34 @@ define amdgpu_kernel void @private_workgroup_seq_cst_atomicrmw(
; GFX1250-GAS-LABEL: private_workgroup_seq_cst_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
@@ -3653,6 +3776,10 @@ define amdgpu_kernel void @private_workgroup_acquire_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -3666,22 +3793,26 @@ define amdgpu_kernel void @private_workgroup_acquire_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -3954,6 +4085,10 @@ define amdgpu_kernel void @private_workgroup_acq_rel_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -3967,22 +4102,26 @@ define amdgpu_kernel void @private_workgroup_acq_rel_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -4257,6 +4396,10 @@ define amdgpu_kernel void @private_workgroup_seq_cst_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -4270,22 +4413,26 @@ define amdgpu_kernel void @private_workgroup_seq_cst_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -4616,6 +4763,12 @@ define amdgpu_kernel void @private_workgroup_monotonic_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -4629,26 +4782,32 @@ define amdgpu_kernel void @private_workgroup_monotonic_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_workgroup_monotonic_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -4656,6 +4815,7 @@ define amdgpu_kernel void @private_workgroup_monotonic_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -4978,6 +5138,12 @@ define amdgpu_kernel void @private_workgroup_acquire_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -4991,26 +5157,32 @@ define amdgpu_kernel void @private_workgroup_acquire_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_workgroup_acquire_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -5018,6 +5190,7 @@ define amdgpu_kernel void @private_workgroup_acquire_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -5341,6 +5514,12 @@ define amdgpu_kernel void @private_workgroup_release_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -5354,26 +5533,32 @@ define amdgpu_kernel void @private_workgroup_release_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_workgroup_release_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -5381,6 +5566,7 @@ define amdgpu_kernel void @private_workgroup_release_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -5705,6 +5891,12 @@ define amdgpu_kernel void @private_workgroup_acq_rel_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -5718,26 +5910,32 @@ define amdgpu_kernel void @private_workgroup_acq_rel_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_workgroup_acq_rel_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -5745,6 +5943,7 @@ define amdgpu_kernel void @private_workgroup_acq_rel_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -6070,6 +6269,12 @@ define amdgpu_kernel void @private_workgroup_seq_cst_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -6083,26 +6288,32 @@ define amdgpu_kernel void @private_workgroup_seq_cst_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_workgroup_seq_cst_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -6110,6 +6321,7 @@ define amdgpu_kernel void @private_workgroup_seq_cst_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -6435,6 +6647,12 @@ define amdgpu_kernel void @private_workgroup_monotonic_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -6448,26 +6666,32 @@ define amdgpu_kernel void @private_workgroup_monotonic_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_workgroup_monotonic_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -6475,6 +6699,7 @@ define amdgpu_kernel void @private_workgroup_monotonic_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -6798,6 +7023,12 @@ define amdgpu_kernel void @private_workgroup_acquire_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -6811,26 +7042,32 @@ define amdgpu_kernel void @private_workgroup_acquire_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_workgroup_acquire_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -6838,6 +7075,7 @@ define amdgpu_kernel void @private_workgroup_acquire_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -7161,6 +7399,12 @@ define amdgpu_kernel void @private_workgroup_release_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -7174,26 +7418,32 @@ define amdgpu_kernel void @private_workgroup_release_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_workgroup_release_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -7201,6 +7451,7 @@ define amdgpu_kernel void @private_workgroup_release_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -7526,6 +7777,12 @@ define amdgpu_kernel void @private_workgroup_acq_rel_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -7539,26 +7796,32 @@ define amdgpu_kernel void @private_workgroup_acq_rel_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_workgroup_acq_rel_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -7566,6 +7829,7 @@ define amdgpu_kernel void @private_workgroup_acq_rel_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -7891,6 +8155,12 @@ define amdgpu_kernel void @private_workgroup_seq_cst_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -7904,26 +8174,32 @@ define amdgpu_kernel void @private_workgroup_seq_cst_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_workgroup_seq_cst_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -7931,6 +8207,7 @@ define amdgpu_kernel void @private_workgroup_seq_cst_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -8256,6 +8533,12 @@ define amdgpu_kernel void @private_workgroup_monotonic_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -8269,26 +8552,32 @@ define amdgpu_kernel void @private_workgroup_monotonic_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_workgroup_monotonic_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -8296,6 +8585,7 @@ define amdgpu_kernel void @private_workgroup_monotonic_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -8621,6 +8911,12 @@ define amdgpu_kernel void @private_workgroup_acquire_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -8634,26 +8930,32 @@ define amdgpu_kernel void @private_workgroup_acquire_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_workgroup_acquire_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -8661,6 +8963,7 @@ define amdgpu_kernel void @private_workgroup_acquire_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -8986,6 +9289,12 @@ define amdgpu_kernel void @private_workgroup_release_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -8999,26 +9308,32 @@ define amdgpu_kernel void @private_workgroup_release_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_workgroup_release_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -9026,6 +9341,7 @@ define amdgpu_kernel void @private_workgroup_release_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -9351,6 +9667,12 @@ define amdgpu_kernel void @private_workgroup_acq_rel_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -9364,26 +9686,32 @@ define amdgpu_kernel void @private_workgroup_acq_rel_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_workgroup_acq_rel_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -9391,6 +9719,7 @@ define amdgpu_kernel void @private_workgroup_acq_rel_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -9716,6 +10045,12 @@ define amdgpu_kernel void @private_workgroup_seq_cst_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -9729,26 +10064,32 @@ define amdgpu_kernel void @private_workgroup_seq_cst_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_workgroup_seq_cst_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -9756,6 +10097,7 @@ define amdgpu_kernel void @private_workgroup_seq_cst_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -10109,6 +10451,12 @@ define amdgpu_kernel void @private_workgroup_monotonic_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -10124,25 +10472,31 @@ define amdgpu_kernel void @private_workgroup_monotonic_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -10504,6 +10858,12 @@ define amdgpu_kernel void @private_workgroup_acquire_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -10519,25 +10879,31 @@ define amdgpu_kernel void @private_workgroup_acquire_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -10899,6 +11265,12 @@ define amdgpu_kernel void @private_workgroup_release_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -10914,25 +11286,31 @@ define amdgpu_kernel void @private_workgroup_release_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -11296,6 +11674,12 @@ define amdgpu_kernel void @private_workgroup_acq_rel_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -11311,25 +11695,31 @@ define amdgpu_kernel void @private_workgroup_acq_rel_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -11693,6 +12083,12 @@ define amdgpu_kernel void @private_workgroup_seq_cst_monotonic_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -11708,25 +12104,31 @@ define amdgpu_kernel void @private_workgroup_seq_cst_monotonic_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -12090,6 +12492,12 @@ define amdgpu_kernel void @private_workgroup_monotonic_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -12105,25 +12513,31 @@ define amdgpu_kernel void @private_workgroup_monotonic_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -12485,6 +12899,12 @@ define amdgpu_kernel void @private_workgroup_acquire_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -12500,25 +12920,31 @@ define amdgpu_kernel void @private_workgroup_acquire_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -12880,6 +13306,12 @@ define amdgpu_kernel void @private_workgroup_release_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -12895,25 +13327,31 @@ define amdgpu_kernel void @private_workgroup_release_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -13277,6 +13715,12 @@ define amdgpu_kernel void @private_workgroup_acq_rel_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -13292,25 +13736,31 @@ define amdgpu_kernel void @private_workgroup_acq_rel_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -13674,6 +14124,12 @@ define amdgpu_kernel void @private_workgroup_seq_cst_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -13689,25 +14145,31 @@ define amdgpu_kernel void @private_workgroup_seq_cst_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -14071,6 +14533,12 @@ define amdgpu_kernel void @private_workgroup_monotonic_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -14086,25 +14554,31 @@ define amdgpu_kernel void @private_workgroup_monotonic_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -14468,6 +14942,12 @@ define amdgpu_kernel void @private_workgroup_acquire_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -14483,25 +14963,31 @@ define amdgpu_kernel void @private_workgroup_acquire_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -14865,6 +15351,12 @@ define amdgpu_kernel void @private_workgroup_release_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -14880,25 +15372,31 @@ define amdgpu_kernel void @private_workgroup_release_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -15262,6 +15760,12 @@ define amdgpu_kernel void @private_workgroup_acq_rel_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -15277,25 +15781,31 @@ define amdgpu_kernel void @private_workgroup_acq_rel_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -15659,6 +16169,12 @@ define amdgpu_kernel void @private_workgroup_seq_cst_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -15674,25 +16190,31 @@ define amdgpu_kernel void @private_workgroup_seq_cst_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -15933,34 +16455,42 @@ define amdgpu_kernel void @private_workgroup_one_as_unordered_load(
; GFX1250-NOGAS-LABEL: private_workgroup_one_as_unordered_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_workgroup_one_as_unordered_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -15969,6 +16499,7 @@ define amdgpu_kernel void @private_workgroup_one_as_unordered_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -16192,34 +16723,42 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_load(
; GFX1250-NOGAS-LABEL: private_workgroup_one_as_monotonic_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_workgroup_one_as_monotonic_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -16228,6 +16767,7 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -16451,34 +16991,42 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_load(
; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acquire_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_workgroup_one_as_acquire_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -16487,6 +17035,7 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_load(
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -16710,34 +17259,42 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_load(
; GFX1250-NOGAS-LABEL: private_workgroup_one_as_seq_cst_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
; GFX1250-GAS-LABEL: private_workgroup_one_as_seq_cst_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -16748,6 +17305,7 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_load(
; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) {
@@ -16955,10 +17513,14 @@ define amdgpu_kernel void @private_workgroup_one_as_unordered_store(
; GFX1250-NOGAS-LABEL: private_workgroup_one_as_unordered_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -16966,22 +17528,26 @@ define amdgpu_kernel void @private_workgroup_one_as_unordered_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -17196,10 +17762,14 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_store(
; GFX1250-NOGAS-LABEL: private_workgroup_one_as_monotonic_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -17207,22 +17777,26 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -17437,10 +18011,14 @@ define amdgpu_kernel void @private_workgroup_one_as_release_store(
; GFX1250-NOGAS-LABEL: private_workgroup_one_as_release_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -17448,22 +18026,26 @@ define amdgpu_kernel void @private_workgroup_one_as_release_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -17680,10 +18262,14 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_store(
; GFX1250-NOGAS-LABEL: private_workgroup_one_as_seq_cst_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
;
@@ -17691,22 +18277,26 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_store(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
@@ -17957,8 +18547,14 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -17966,29 +18562,34 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_atomicrmw(
; GFX1250-GAS-LABEL: private_workgroup_one_as_monotonic_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
@@ -18231,8 +18832,14 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -18240,29 +18847,34 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_atomicrmw(
; GFX1250-GAS-LABEL: private_workgroup_one_as_acquire_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
@@ -18506,8 +19118,14 @@ define amdgpu_kernel void @private_workgroup_one_as_release_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -18515,29 +19133,34 @@ define amdgpu_kernel void @private_workgroup_one_as_release_atomicrmw(
; GFX1250-GAS-LABEL: private_workgroup_one_as_release_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
@@ -18782,8 +19405,14 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -18791,29 +19420,34 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_atomicrmw(
; GFX1250-GAS-LABEL: private_workgroup_one_as_acq_rel_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
@@ -19059,8 +19693,14 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
; GFX1250-NOGAS-NEXT: s_endpgm
@@ -19068,29 +19708,34 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_atomicrmw(
; GFX1250-GAS-LABEL: private_workgroup_one_as_seq_cst_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s1
-; GFX1250-GAS-NEXT: s_mov_b32 s1, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s1, v0
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
@@ -19357,6 +20002,10 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -19370,22 +20019,26 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -19658,6 +20311,10 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -19671,22 +20328,26 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -19961,6 +20622,10 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_ret_atomicrmw(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
@@ -19974,22 +20639,26 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_ret_atomicrmw(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[2:3], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[2:3]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s3
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -20320,6 +20989,12 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -20333,26 +21008,32 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_workgroup_one_as_monotonic_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -20360,6 +21041,7 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -20682,6 +21364,12 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -20695,26 +21383,32 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_workgroup_one_as_acquire_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -20722,6 +21416,7 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -21045,6 +21740,12 @@ define amdgpu_kernel void @private_workgroup_one_as_release_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -21058,26 +21759,32 @@ define amdgpu_kernel void @private_workgroup_one_as_release_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_workgroup_one_as_release_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -21085,6 +21792,7 @@ define amdgpu_kernel void @private_workgroup_one_as_release_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -21409,6 +22117,12 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -21422,26 +22136,32 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_workgroup_one_as_acq_rel_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -21449,6 +22169,7 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -21774,6 +22495,12 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_monotonic_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -21787,26 +22514,32 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_workgroup_one_as_seq_cst_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -21814,6 +22547,7 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_monotonic_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -22139,6 +22873,12 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -22152,26 +22892,32 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_workgroup_one_as_monotonic_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -22179,6 +22925,7 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -22502,6 +23249,12 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -22515,26 +23268,32 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_workgroup_one_as_acquire_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -22542,6 +23301,7 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -22865,6 +23625,12 @@ define amdgpu_kernel void @private_workgroup_one_as_release_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -22878,26 +23644,32 @@ define amdgpu_kernel void @private_workgroup_one_as_release_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_workgroup_one_as_release_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -22905,6 +23677,7 @@ define amdgpu_kernel void @private_workgroup_one_as_release_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -23230,6 +24003,12 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -23243,26 +24022,32 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_workgroup_one_as_acq_rel_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -23270,6 +24055,7 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -23595,6 +24381,12 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_acquire_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -23608,26 +24400,32 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_workgroup_one_as_seq_cst_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -23635,6 +24433,7 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_acquire_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -23960,6 +24759,12 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -23973,26 +24778,32 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_workgroup_one_as_monotonic_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -24000,6 +24811,7 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -24325,6 +25137,12 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -24338,26 +25156,32 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_workgroup_one_as_acquire_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -24365,6 +25189,7 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -24690,6 +25515,12 @@ define amdgpu_kernel void @private_workgroup_one_as_release_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -24703,26 +25534,32 @@ define amdgpu_kernel void @private_workgroup_one_as_release_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_workgroup_one_as_release_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -24730,6 +25567,7 @@ define amdgpu_kernel void @private_workgroup_one_as_release_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -25055,6 +25893,12 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -25068,26 +25912,32 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_workgroup_one_as_acq_rel_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -25095,6 +25945,7 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -25420,6 +26271,12 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_seq_cst_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -25433,26 +26290,32 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_workgroup_one_as_seq_cst_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
-; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s3, s2, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
-; GFX1250-GAS-NEXT: s_mov_b32 s6, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s6
-; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
@@ -25460,6 +26323,7 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_seq_cst_cmpxchg(
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
@@ -25813,6 +26677,12 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_monotonic_ret_cmpx
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -25828,25 +26698,31 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_monotonic_ret_cmpx
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -26208,6 +27084,12 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_monotonic_ret_cmpxch
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -26223,25 +27105,31 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_monotonic_ret_cmpxch
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -26603,6 +27491,12 @@ define amdgpu_kernel void @private_workgroup_one_as_release_monotonic_ret_cmpxch
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -26618,25 +27512,31 @@ define amdgpu_kernel void @private_workgroup_one_as_release_monotonic_ret_cmpxch
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -27000,6 +27900,12 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_monotonic_ret_cmpxch
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -27015,25 +27921,31 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_monotonic_ret_cmpxch
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -27397,6 +28309,12 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_monotonic_ret_cmpxch
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -27412,25 +28330,31 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_monotonic_ret_cmpxch
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -27794,6 +28718,12 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_acquire_ret_cmpxch
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -27809,25 +28739,31 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_acquire_ret_cmpxch
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -28189,6 +29125,12 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -28204,25 +29146,31 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -28584,6 +29532,12 @@ define amdgpu_kernel void @private_workgroup_one_as_release_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -28599,25 +29553,31 @@ define amdgpu_kernel void @private_workgroup_one_as_release_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -28981,6 +29941,12 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -28996,25 +29962,31 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -29378,6 +30350,12 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_acquire_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -29393,25 +30371,31 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_acquire_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -29775,6 +30759,12 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_seq_cst_ret_cmpxch
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -29790,25 +30780,31 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_seq_cst_ret_cmpxch
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -30172,6 +31168,12 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -30187,25 +31189,31 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -30569,6 +31577,12 @@ define amdgpu_kernel void @private_workgroup_one_as_release_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -30584,25 +31598,31 @@ define amdgpu_kernel void @private_workgroup_one_as_release_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -30966,6 +31986,12 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -30981,25 +32007,31 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
@@ -31363,6 +32395,12 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
@@ -31378,25 +32416,31 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_seq_cst_ret_cmpxchg(
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GAS-NEXT: s_add_co_i32 s4, s0, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
-; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
-; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s4
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], src_flat_scratch_base_lo
-; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[6:7]
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
-; GFX1250-GAS-NEXT: s_mov_b32 s5, -1
-; GFX1250-GAS-NEXT: s_cmp_lg_u32 s4, s5
-; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
More information about the llvm-commits
mailing list