[llvm-branch-commits] [llvm] [AMDGPU] Add synthetic apertures and use them for barriers (PR #204127)
Pierre van Houtryve via llvm-branch-commits
llvm-branch-commits at lists.llvm.org
Tue Jun 16 04:48:41 PDT 2026
https://github.com/Pierre-vh created https://github.com/llvm/llvm-project/pull/204127
Define what a synthetic aperture is, and adjust the barrier AS
to use this new system. This makes the barrier AS even safer to
use as now we can use all 32 bits of it without ever risking
hitting a valid address of any kind (LDS or outside LDS).
>From b244676654dc4e3ac852e02e1409c9d80a3e7f6d Mon Sep 17 00:00:00 2001
From: pvanhout <pierre.vanhoutryve at amd.com>
Date: Tue, 16 Jun 2026 10:21:06 +0200
Subject: [PATCH] [AMDGPU] Add synthetic apertures and use them for barriers
Define what a synthetic aperture is, and adjust the barrier AS
to use this new system. This makes the barrier AS even safer to
use as now we can use all 32 bits of it without ever risking
hitting a valid address of any kind (LDS or outside LDS).
---
llvm/docs/AMDGPUUsage.rst | 49 +++++++-
llvm/include/llvm/Support/AMDGPUAddrSpace.h | 5 -
.../lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp | 44 ++++---
llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.h | 4 +
llvm/lib/Target/AMDGPU/AMDGPUMemoryUtils.cpp | 9 ++
llvm/lib/Target/AMDGPU/AMDGPUMemoryUtils.h | 4 +
llvm/lib/Target/AMDGPU/SIDefines.h | 12 ++
llvm/lib/Target/AMDGPU/SIISelLowering.cpp | 36 +++---
llvm/lib/Target/AMDGPU/SIISelLowering.h | 2 +
.../CodeGen/AMDGPU/addrspacecast-barrier.ll | 107 +++++++++---------
10 files changed, 169 insertions(+), 103 deletions(-)
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index e925b18a0e77f..4a2857989240c 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -1015,6 +1015,9 @@ supported for the ``amdgcn`` target.
A global address space address has the same value when used as a flat address
so no conversion is needed.
+ See also :ref:`synthetic apertures<amdgpu-synthetic-apertures>` which exist
+ in the generic address space.
+
**Global and Constant**
The global and constant address spaces both use global virtual addresses,
which are the same virtual address space used by the CPU. However, some
@@ -1195,7 +1198,9 @@ supported for the ``amdgcn`` target.
**Barrier**
This address space represents barrier IDs (introduced in GFX12) as addresses.
- It does not map directly to any addressable memory, thus pointers into this address space:
+ It does not map directly to any addressable memory and is implemented using
+ :ref:`synthetic apertures<amdgpu-synthetic-apertures>`, thus pointers into
+ this address space:
* Never alias with any other pointers outside this address space.
* Cannot be dereferenced.
@@ -1206,10 +1211,6 @@ supported for the ``amdgcn`` target.
have a value corresponding to a valid barrier ID on the target.
Otherwise, the behavior is undefined
- These pointers do not have a corresponding hardware aperture but safe round-tripping
- through the generic address space is still possible. Attempting to dereference a
- generic pointer derived from a barrier pointer is undefined behavior.
-
**Streamout Registers**
Dedicated registers used by the GS NGG Streamout Instructions. The register
file is modelled as a memory in a distinct address space because it is indexed
@@ -1217,6 +1218,44 @@ supported for the ``amdgcn`` target.
accesses affect LGKMcnt. This is an internal address space used only by the
compiler. Do not use this address space for IR pointers.
+.. _amdgpu-synthetic-apertures:
+
+Synthetic Apertures
+~~~~~~~~~~~~~~~~~~~
+
+*Synthetic apertures* are defined that enable safe roundtrips of pointers
+from special address spaces through the generic address space. Attempting to
+dereference generic pointers obtained in this way (using e.g. `load` or
+`store`) has undefined behavior. The following synthetic apertures are defined:
+
+.. table:: AMDGPU Synthetic Apertures
+ :name: amdgpu-synthetic-apertures-table
+ :widths: 40 20 40
+
+ ============ ====== ===============================================================
+ Name Number Corresponding :ref:`Address Space<amdgpu-address-spaces-table>`
+ ============ ====== ===============================================================
+ BARRIER 1 Barrier
+ ============ ====== ===============================================================
+
+Note that the address size of an address spaces implemented via synthetic apertures
+can only be 32 bits wide or less. The full width of the source pointer is usable and
+preserved when converting it from/to the generic address space.
+
+Converting a pointer to generic (64 bits) using synthetic apertures is done as follows:
+
+ * The value of the source pointer (32 bits) becomes the lower 32 bits of the generic pointer.
+ * The upper 32 bits are a bitwise ``OR`` of:
+
+ * The upper 32 bits of the LDS segment aperture.
+ * The synthetic aperture number.
+
+The conversion back to the original address space can simply be done by discarding the
+upper 32 bits of the generic pointer.
+
+As the LDS aperture is defined by its 16 most significant bits, we can theoretically
+support up to ``2 << 16`` synthetic apertures safely.
+
.. _amdgpu-memory-scopes:
Memory Scopes
diff --git a/llvm/include/llvm/Support/AMDGPUAddrSpace.h b/llvm/include/llvm/Support/AMDGPUAddrSpace.h
index d72ba0a1415c0..1824c1bcc2e89 100644
--- a/llvm/include/llvm/Support/AMDGPUAddrSpace.h
+++ b/llvm/include/llvm/Support/AMDGPUAddrSpace.h
@@ -91,11 +91,6 @@ enum : unsigned {
// Some places use this if the address space can't be determined.
UNKNOWN_ADDRESS_SPACE = ~0u,
};
-
-/// The BARRIER AS does not have an aperture in HW, so when converting
-/// BARRIER addresses from/to generic, we represent them as LDS addresses
-/// offset by a large amount so they can never alias with real LDS memory.
-static constexpr unsigned BarrierAddrLDSOffset = 0x802000u;
} // end namespace AMDGPUAS
namespace AMDGPU {
diff --git a/llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp b/llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp
index 9c40cbcc7439a..5838ec33a63d0 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp
@@ -2408,10 +2408,27 @@ bool AMDGPULegalizerInfo::legalizeCustom(
llvm_unreachable("expected switch to return");
}
-Register AMDGPULegalizerInfo::getSegmentAperture(
- unsigned AS,
- MachineRegisterInfo &MRI,
- MachineIRBuilder &B) const {
+Register AMDGPULegalizerInfo::getSegmentAperture(unsigned AS,
+ MachineRegisterInfo &MRI,
+ MachineIRBuilder &B) const {
+ unsigned BaseAS = AS;
+ unsigned SANum = AMDGPU::tryGetSyntheticApertureNumber(AS);
+ if (SANum != AMDGPU::SyntheticAperture::None)
+ BaseAS = AMDGPUAS::LOCAL_ADDRESS;
+
+ Register Aperture = getBaseSegmentAperture(BaseAS, MRI, B);
+
+ if (SANum != AMDGPU::SyntheticAperture::None) {
+ const LLT S32 = LLT::scalar(32);
+ auto Tag = B.buildConstant(S32, SANum);
+ return B.buildOr(S32, Aperture, Tag).getReg(0);
+ }
+
+ return Aperture;
+}
+
+Register AMDGPULegalizerInfo::getBaseSegmentAperture(
+ unsigned AS, MachineRegisterInfo &MRI, MachineIRBuilder &B) const {
MachineFunction &MF = B.getMF();
const GCNSubtarget &ST = MF.getSubtarget<GCNSubtarget>();
const LLT S32 = LLT::scalar(32);
@@ -2566,17 +2583,6 @@ bool AMDGPULegalizerInfo::legalizeAddrSpaceCast(
return B.buildIntToPtr(Dst, Sub).getReg(0);
}
- if (DestAS == AMDGPUAS::BARRIER) {
- // flat -> barrier: extract the low 32 bits, then sub the barrier AS
- // offset.
- Register LoBits = B.buildExtract(S32, Src, 0).getReg(0);
- Register Sub =
- B.buildSub(S32, LoBits,
- B.buildConstant(S32, AMDGPUAS::BarrierAddrLDSOffset))
- .getReg(0);
- return B.buildIntToPtr(Dst, Sub).getReg(0);
- }
-
return B.buildExtract(Dst, Src, 0).getReg(0);
};
@@ -2647,14 +2653,6 @@ bool AMDGPULegalizerInfo::legalizeAddrSpaceCast(
if (!ApertureReg.isValid())
return false;
- if (SrcAS == AMDGPUAS::BARRIER) {
- // barrier -> flat: add the barrier AS offset
- SrcAsInt =
- B.buildAdd(S32, SrcAsInt,
- B.buildConstant(S32, AMDGPUAS::BarrierAddrLDSOffset))
- .getReg(0);
- }
-
// TODO: Should we allow mismatched types but matching sizes in merges to
// avoid the ptrtoint?
return B.buildMergeLikeInstr(Dst, {SrcAsInt, ApertureReg}).getReg(0);
diff --git a/llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.h b/llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.h
index 53cd6d786ee2f..93ff4195250bb 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.h
+++ b/llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.h
@@ -266,6 +266,10 @@ class AMDGPULegalizerInfo final : public LegalizerInfo {
bool legalizeIntrinsic(LegalizerHelper &Helper,
MachineInstr &MI) const override;
+
+private:
+ Register getBaseSegmentAperture(unsigned AS, MachineRegisterInfo &MRI,
+ MachineIRBuilder &B) const;
};
} // End llvm namespace.
#endif
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUMemoryUtils.cpp b/llvm/lib/Target/AMDGPU/AMDGPUMemoryUtils.cpp
index d3b815008f4cd..e58c0e861b2ff 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUMemoryUtils.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUMemoryUtils.cpp
@@ -32,6 +32,15 @@ Align getAlign(const DataLayout &DL, const GlobalVariable *GV) {
GV->getValueType());
}
+unsigned tryGetSyntheticApertureNumber(unsigned AS) {
+ switch (AS) {
+ case AMDGPUAS::BARRIER:
+ return SyntheticAperture::BARRIER;
+ default:
+ return SyntheticAperture::None;
+ }
+}
+
void copyMetadataForWidenedLoad(LoadInst &Dest, const LoadInst &Source) {
SmallVector<std::pair<unsigned, MDNode *>, 8> MD;
Source.getAllMetadata(MD);
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUMemoryUtils.h b/llvm/lib/Target/AMDGPU/AMDGPUMemoryUtils.h
index 4e164d08549dd..f81b677051561 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUMemoryUtils.h
+++ b/llvm/lib/Target/AMDGPU/AMDGPUMemoryUtils.h
@@ -35,6 +35,10 @@ using VariableFunctionMap = DenseMap<GlobalVariable *, DenseSet<Function *>>;
Align getAlign(const DataLayout &DL, const GlobalVariable *GV);
+// Get the synthetic aperture number for the given address space, or None (0)
+// if the address space does not have one.
+unsigned tryGetSyntheticApertureNumber(unsigned AS);
+
// Copy metadata onto a load widened to read a superset of Source's bytes. Only
// value-independent metadata is copied; metadata describing the loaded value
// (!range, !noundef, !nofpclass, !tbaa, ...) is dropped.
diff --git a/llvm/lib/Target/AMDGPU/SIDefines.h b/llvm/lib/Target/AMDGPU/SIDefines.h
index 509c21b1e7788..a1819652df515 100644
--- a/llvm/lib/Target/AMDGPU/SIDefines.h
+++ b/llvm/lib/Target/AMDGPU/SIDefines.h
@@ -1137,6 +1137,18 @@ enum Type {
};
} // namespace Barrier
+
+namespace SyntheticAperture {
+/// Synthetic aperture numbers.
+///
+/// NOTE: This is also documented in AMDGPUUsage.
+enum SyntheticAperture {
+ None = 0,
+
+ BARRIER = 1,
+};
+} // namespace SyntheticAperture
+
} // namespace AMDGPU
// clang-format off
diff --git a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
index 27aa19c330a2a..901ea2bd1d07a 100644
--- a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
+++ b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
@@ -15,6 +15,7 @@
#include "AMDGPU.h"
#include "AMDGPUInstrInfo.h"
#include "AMDGPULaneMaskUtils.h"
+#include "AMDGPUMemoryUtils.h"
#include "AMDGPUSelectionDAGInfo.h"
#include "AMDGPUTargetMachine.h"
#include "GCNSubtarget.h"
@@ -9196,6 +9197,23 @@ SDValue SITargetLowering::LowerINLINEASM(SDValue Op, SelectionDAG &DAG) const {
SDValue SITargetLowering::getSegmentAperture(unsigned AS, const SDLoc &DL,
SelectionDAG &DAG) const {
+ unsigned BaseAS = AS;
+ unsigned SANum = AMDGPU::tryGetSyntheticApertureNumber(AS);
+ if (SANum != AMDGPU::SyntheticAperture::None)
+ BaseAS = AMDGPUAS::LOCAL_ADDRESS;
+
+ SDValue Aperture = getBaseSegmentAperture(BaseAS, DL, DAG);
+
+ if (SANum != AMDGPU::SyntheticAperture::None) {
+ SDValue Tag = DAG.getConstant(SANum, DL, MVT::i32);
+ return DAG.getNode(ISD::OR, DL, MVT::i32, Aperture, Tag);
+ }
+
+ return Aperture;
+}
+
+SDValue SITargetLowering::getBaseSegmentAperture(unsigned AS, const SDLoc &DL,
+ SelectionDAG &DAG) const {
const bool IsLDS = (AS == AMDGPUAS::LOCAL_ADDRESS || AS == AMDGPUAS::BARRIER);
if (Subtarget->hasApertureRegs()) {
@@ -9312,11 +9330,6 @@ SDValue SITargetLowering::lowerADDRSPACECAST(SDValue Op,
DAG.getRegister(AMDGPU::SRC_FLAT_SCRATCH_BASE_LO, MVT::i32)),
0);
Ptr = DAG.getNode(ISD::SUB, SL, MVT::i32, Ptr, FlatScratchBaseLo);
- } else if (DestAS == AMDGPUAS::BARRIER) {
- // flat -> barrier: sub the barrier AS offset.
- Ptr = DAG.getNode(
- ISD::SUB, SL, MVT::i32, Ptr,
- DAG.getConstant(AMDGPUAS::BarrierAddrLDSOffset, SL, MVT::i32));
}
if (IsNonNull || isKnownNonNull(Op, DAG, TM, SrcAS))
@@ -9367,18 +9380,7 @@ SDValue SITargetLowering::lowerADDRSPACECAST(SDValue Op,
} else {
SDValue Aperture = getSegmentAperture(SrcAS, SL, DAG);
- if (SrcAS == AMDGPUAS::BARRIER) {
- // barrier -> flat: add the barrier AS offset.
- SDValue SrcOffset = DAG.getNode(
- ISD::ADD, SL, MVT::i32, Src,
- DAG.getConstant(AMDGPUAS::BarrierAddrLDSOffset, SL, MVT::i32));
- CvtPtr = DAG.getNode(ISD::BUILD_VECTOR, SL, MVT::v2i32, SrcOffset,
- Aperture);
- } else {
- CvtPtr =
- DAG.getNode(ISD::BUILD_VECTOR, SL, MVT::v2i32, Src, Aperture);
- }
-
+ CvtPtr = DAG.getNode(ISD::BUILD_VECTOR, SL, MVT::v2i32, Src, Aperture);
CvtPtr = DAG.getNode(ISD::BITCAST, SL, MVT::i64, CvtPtr);
}
diff --git a/llvm/lib/Target/AMDGPU/SIISelLowering.h b/llvm/lib/Target/AMDGPU/SIISelLowering.h
index 95ff5bba7cfff..475199a809e7a 100644
--- a/llvm/lib/Target/AMDGPU/SIISelLowering.h
+++ b/llvm/lib/Target/AMDGPU/SIISelLowering.h
@@ -173,6 +173,8 @@ class SITargetLowering final : public AMDGPUTargetLowering {
SDValue lowerXMULO(SDValue Op, SelectionDAG &DAG) const;
SDValue lowerXMUL_LOHI(SDValue Op, SelectionDAG &DAG) const;
+ SDValue getBaseSegmentAperture(unsigned AS, const SDLoc &DL,
+ SelectionDAG &DAG) const;
SDValue getSegmentAperture(unsigned AS, const SDLoc &DL,
SelectionDAG &DAG) const;
diff --git a/llvm/test/CodeGen/AMDGPU/addrspacecast-barrier.ll b/llvm/test/CodeGen/AMDGPU/addrspacecast-barrier.ll
index 34f9a51a0455c..8ae387015debd 100644
--- a/llvm/test/CodeGen/AMDGPU/addrspacecast-barrier.ll
+++ b/llvm/test/CodeGen/AMDGPU/addrspacecast-barrier.ll
@@ -19,11 +19,11 @@ define amdgpu_kernel void @barrier_to_generic(ptr addrspace(15) %bar, ptr %out)
; GFX942-SDAG-NEXT: s_mov_b64 s[0:1], src_shared_base
; GFX942-SDAG-NEXT: s_load_dword s0, s[4:5], 0x0
; GFX942-SDAG-NEXT: s_load_dwordx2 s[2:3], s[4:5], 0x8
+; GFX942-SDAG-NEXT: s_or_b32 s1, s1, 1
; GFX942-SDAG-NEXT: s_waitcnt lgkmcnt(0)
-; GFX942-SDAG-NEXT: s_add_i32 s4, s0, 0x802000
; GFX942-SDAG-NEXT: s_cmp_lg_u32 s0, 0
-; GFX942-SDAG-NEXT: s_cselect_b32 s0, s4, 0
; GFX942-SDAG-NEXT: s_cselect_b32 s1, s1, 0
+; GFX942-SDAG-NEXT: s_cselect_b32 s0, s0, 0
; GFX942-SDAG-NEXT: v_mov_b32_e32 v2, s0
; GFX942-SDAG-NEXT: v_mov_b32_e32 v3, s1
; GFX942-SDAG-NEXT: v_mov_b64_e32 v[0:1], s[2:3]
@@ -32,12 +32,12 @@ define amdgpu_kernel void @barrier_to_generic(ptr addrspace(15) %bar, ptr %out)
;
; GFX942-GISEL-LABEL: barrier_to_generic:
; GFX942-GISEL: ; %bb.0:
-; GFX942-GISEL-NEXT: s_load_dword s6, s[4:5], 0x0
-; GFX942-GISEL-NEXT: s_load_dwordx2 s[2:3], s[4:5], 0x8
; GFX942-GISEL-NEXT: s_mov_b64 s[0:1], src_shared_base
+; GFX942-GISEL-NEXT: s_load_dword s0, s[4:5], 0x0
+; GFX942-GISEL-NEXT: s_load_dwordx2 s[2:3], s[4:5], 0x8
+; GFX942-GISEL-NEXT: s_or_b32 s1, s1, 1
; GFX942-GISEL-NEXT: s_waitcnt lgkmcnt(0)
-; GFX942-GISEL-NEXT: s_add_u32 s0, s6, 0x802000
-; GFX942-GISEL-NEXT: s_cmp_lg_u32 s6, 0
+; GFX942-GISEL-NEXT: s_cmp_lg_u32 s0, 0
; GFX942-GISEL-NEXT: s_cselect_b64 s[0:1], s[0:1], 0
; GFX942-GISEL-NEXT: v_mov_b64_e32 v[0:1], s[0:1]
; GFX942-GISEL-NEXT: v_mov_b64_e32 v[2:3], s[2:3]
@@ -54,11 +54,11 @@ define amdgpu_kernel void @barrier_to_generic(ptr addrspace(15) %bar, ptr %out)
; GFX1030-SDAG-NEXT: s_clause 0x1
; GFX1030-SDAG-NEXT: s_load_dword s0, s[8:9], 0x0
; GFX1030-SDAG-NEXT: s_load_dwordx2 s[2:3], s[8:9], 0x8
+; GFX1030-SDAG-NEXT: s_or_b32 s1, s1, 1
; GFX1030-SDAG-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1030-SDAG-NEXT: s_add_i32 s4, s0, 0x802000
; GFX1030-SDAG-NEXT: s_cmp_lg_u32 s0, 0
; GFX1030-SDAG-NEXT: v_mov_b32_e32 v2, s2
-; GFX1030-SDAG-NEXT: s_cselect_b32 s0, s4, 0
+; GFX1030-SDAG-NEXT: s_cselect_b32 s0, s0, 0
; GFX1030-SDAG-NEXT: s_cselect_b32 s1, s1, 0
; GFX1030-SDAG-NEXT: v_mov_b32_e32 v0, s0
; GFX1030-SDAG-NEXT: v_mov_b32_e32 v1, s1
@@ -72,13 +72,13 @@ define amdgpu_kernel void @barrier_to_generic(ptr addrspace(15) %bar, ptr %out)
; GFX1030-GISEL-NEXT: s_addc_u32 s13, s13, 0
; GFX1030-GISEL-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12
; GFX1030-GISEL-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13
+; GFX1030-GISEL-NEXT: s_mov_b64 s[0:1], src_shared_base
; GFX1030-GISEL-NEXT: s_clause 0x1
-; GFX1030-GISEL-NEXT: s_load_dword s4, s[8:9], 0x0
+; GFX1030-GISEL-NEXT: s_load_dword s0, s[8:9], 0x0
; GFX1030-GISEL-NEXT: s_load_dwordx2 s[2:3], s[8:9], 0x8
-; GFX1030-GISEL-NEXT: s_mov_b64 s[0:1], src_shared_base
+; GFX1030-GISEL-NEXT: s_or_b32 s1, s1, 1
; GFX1030-GISEL-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1030-GISEL-NEXT: s_add_u32 s0, s4, 0x802000
-; GFX1030-GISEL-NEXT: s_cmp_lg_u32 s4, 0
+; GFX1030-GISEL-NEXT: s_cmp_lg_u32 s0, 0
; GFX1030-GISEL-NEXT: v_mov_b32_e32 v2, s2
; GFX1030-GISEL-NEXT: s_cselect_b64 s[0:1], s[0:1], 0
; GFX1030-GISEL-NEXT: v_mov_b32_e32 v3, s3
@@ -93,11 +93,11 @@ define amdgpu_kernel void @barrier_to_generic(ptr addrspace(15) %bar, ptr %out)
; GFX1200-SDAG-NEXT: s_clause 0x1
; GFX1200-SDAG-NEXT: s_load_b32 s0, s[4:5], 0x0
; GFX1200-SDAG-NEXT: s_load_b64 s[2:3], s[4:5], 0x8
+; GFX1200-SDAG-NEXT: s_or_b32 s1, s1, 1
; GFX1200-SDAG-NEXT: s_wait_kmcnt 0x0
-; GFX1200-SDAG-NEXT: s_add_co_i32 s4, s0, 0x802000
; GFX1200-SDAG-NEXT: s_cmp_lg_u32 s0, 0
; GFX1200-SDAG-NEXT: v_dual_mov_b32 v2, s2 :: v_dual_mov_b32 v3, s3
-; GFX1200-SDAG-NEXT: s_cselect_b32 s0, s4, 0
+; GFX1200-SDAG-NEXT: s_cselect_b32 s0, s0, 0
; GFX1200-SDAG-NEXT: s_cselect_b32 s1, s1, 0
; GFX1200-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
; GFX1200-SDAG-NEXT: v_dual_mov_b32 v0, s0 :: v_dual_mov_b32 v1, s1
@@ -106,13 +106,13 @@ define amdgpu_kernel void @barrier_to_generic(ptr addrspace(15) %bar, ptr %out)
;
; GFX1200-GISEL-LABEL: barrier_to_generic:
; GFX1200-GISEL: ; %bb.0:
+; GFX1200-GISEL-NEXT: s_mov_b64 s[0:1], src_shared_base
; GFX1200-GISEL-NEXT: s_clause 0x1
-; GFX1200-GISEL-NEXT: s_load_b32 s6, s[4:5], 0x0
+; GFX1200-GISEL-NEXT: s_load_b32 s0, s[4:5], 0x0
; GFX1200-GISEL-NEXT: s_load_b64 s[2:3], s[4:5], 0x8
-; GFX1200-GISEL-NEXT: s_mov_b64 s[0:1], src_shared_base
+; GFX1200-GISEL-NEXT: s_or_b32 s1, s1, 1
; GFX1200-GISEL-NEXT: s_wait_kmcnt 0x0
-; GFX1200-GISEL-NEXT: s_add_co_u32 s0, s6, 0x802000
-; GFX1200-GISEL-NEXT: s_cmp_lg_u32 s6, 0
+; GFX1200-GISEL-NEXT: s_cmp_lg_u32 s0, 0
; GFX1200-GISEL-NEXT: v_dual_mov_b32 v2, s2 :: v_dual_mov_b32 v3, s3
; GFX1200-GISEL-NEXT: s_cselect_b64 s[0:1], s[0:1], 0
; GFX1200-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -127,10 +127,10 @@ define amdgpu_kernel void @barrier_to_generic(ptr addrspace(15) %bar, ptr %out)
; GFX1250-SDAG-NEXT: s_clause 0x1
; GFX1250-SDAG-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-SDAG-NEXT: s_load_b64 s[2:3], s[4:5], 0x8 nv
+; GFX1250-SDAG-NEXT: s_or_b32 s1, s1, 1
; GFX1250-SDAG-NEXT: s_wait_kmcnt 0x0
-; GFX1250-SDAG-NEXT: s_add_co_i32 s4, s0, 0x802000
; GFX1250-SDAG-NEXT: s_cmp_lg_u32 s0, 0
-; GFX1250-SDAG-NEXT: s_cselect_b32 s0, s4, 0
+; GFX1250-SDAG-NEXT: s_cselect_b32 s0, s0, 0
; GFX1250-SDAG-NEXT: s_cselect_b32 s1, s1, 0
; GFX1250-SDAG-NEXT: v_dual_mov_b32 v2, 0 :: v_dual_mov_b32 v0, s0
; GFX1250-SDAG-NEXT: v_mov_b32_e32 v1, s1
@@ -140,14 +140,14 @@ define amdgpu_kernel void @barrier_to_generic(ptr addrspace(15) %bar, ptr %out)
; GFX1250-GISEL-LABEL: barrier_to_generic:
; GFX1250-GISEL: ; %bb.0:
; GFX1250-GISEL-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GISEL-NEXT: s_mov_b64 s[0:1], src_shared_base
; GFX1250-GISEL-NEXT: s_clause 0x1
-; GFX1250-GISEL-NEXT: s_load_b32 s6, s[4:5], 0x0 nv
+; GFX1250-GISEL-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GISEL-NEXT: s_load_b64 s[2:3], s[4:5], 0x8 nv
-; GFX1250-GISEL-NEXT: s_mov_b64 s[0:1], src_shared_base
+; GFX1250-GISEL-NEXT: s_or_b32 s1, s1, 1
; GFX1250-GISEL-NEXT: v_mov_b32_e32 v2, 0
; GFX1250-GISEL-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GISEL-NEXT: s_add_co_u32 s0, s6, 0x802000
-; GFX1250-GISEL-NEXT: s_cmp_lg_u32 s6, 0
+; GFX1250-GISEL-NEXT: s_cmp_lg_u32 s0, 0
; GFX1250-GISEL-NEXT: s_cselect_b64 s[0:1], s[0:1], 0
; GFX1250-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
; GFX1250-GISEL-NEXT: v_mov_b64_e32 v[0:1], s[0:1]
@@ -163,18 +163,20 @@ define amdgpu_kernel void @barrier_gv_to_generic(ptr %out) {
; GFX942-SDAG: ; %bb.0:
; GFX942-SDAG-NEXT: s_load_dwordx2 s[2:3], s[4:5], 0x0
; GFX942-SDAG-NEXT: s_mov_b64 s[0:1], src_shared_base
-; GFX942-SDAG-NEXT: v_mov_b32_e32 v0, 0x802001
-; GFX942-SDAG-NEXT: v_mov_b32_e32 v1, s1
+; GFX942-SDAG-NEXT: s_or_b32 s0, s1, 1
+; GFX942-SDAG-NEXT: v_mov_b32_e32 v2, 1
+; GFX942-SDAG-NEXT: v_mov_b32_e32 v3, s0
; GFX942-SDAG-NEXT: s_waitcnt lgkmcnt(0)
-; GFX942-SDAG-NEXT: v_mov_b64_e32 v[2:3], s[2:3]
-; GFX942-SDAG-NEXT: flat_store_dwordx2 v[2:3], v[0:1]
+; GFX942-SDAG-NEXT: v_mov_b64_e32 v[0:1], s[2:3]
+; GFX942-SDAG-NEXT: flat_store_dwordx2 v[0:1], v[2:3]
; GFX942-SDAG-NEXT: s_endpgm
;
; GFX942-GISEL-LABEL: barrier_gv_to_generic:
; GFX942-GISEL: ; %bb.0:
; GFX942-GISEL-NEXT: s_load_dwordx2 s[2:3], s[4:5], 0x0
; GFX942-GISEL-NEXT: s_mov_b64 s[0:1], src_shared_base
-; GFX942-GISEL-NEXT: s_mov_b32 s0, 0x802001
+; GFX942-GISEL-NEXT: s_mov_b32 s0, 1
+; GFX942-GISEL-NEXT: s_or_b32 s1, s1, 1
; GFX942-GISEL-NEXT: v_mov_b64_e32 v[0:1], s[0:1]
; GFX942-GISEL-NEXT: s_waitcnt lgkmcnt(0)
; GFX942-GISEL-NEXT: v_mov_b64_e32 v[2:3], s[2:3]
@@ -189,8 +191,9 @@ define amdgpu_kernel void @barrier_gv_to_generic(ptr %out) {
; GFX1030-SDAG-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13
; GFX1030-SDAG-NEXT: s_load_dwordx2 s[2:3], s[8:9], 0x0
; GFX1030-SDAG-NEXT: s_mov_b64 s[0:1], src_shared_base
-; GFX1030-SDAG-NEXT: v_mov_b32_e32 v0, 0x802001
-; GFX1030-SDAG-NEXT: v_mov_b32_e32 v1, s1
+; GFX1030-SDAG-NEXT: v_mov_b32_e32 v0, 1
+; GFX1030-SDAG-NEXT: s_or_b32 s0, s1, 1
+; GFX1030-SDAG-NEXT: v_mov_b32_e32 v1, s0
; GFX1030-SDAG-NEXT: s_waitcnt lgkmcnt(0)
; GFX1030-SDAG-NEXT: v_mov_b32_e32 v2, s2
; GFX1030-SDAG-NEXT: v_mov_b32_e32 v3, s3
@@ -205,7 +208,8 @@ define amdgpu_kernel void @barrier_gv_to_generic(ptr %out) {
; GFX1030-GISEL-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13
; GFX1030-GISEL-NEXT: s_load_dwordx2 s[2:3], s[8:9], 0x0
; GFX1030-GISEL-NEXT: s_mov_b64 s[0:1], src_shared_base
-; GFX1030-GISEL-NEXT: s_mov_b32 s0, 0x802001
+; GFX1030-GISEL-NEXT: s_mov_b32 s0, 1
+; GFX1030-GISEL-NEXT: s_or_b32 s1, s1, 1
; GFX1030-GISEL-NEXT: v_mov_b32_e32 v0, s0
; GFX1030-GISEL-NEXT: v_mov_b32_e32 v1, s1
; GFX1030-GISEL-NEXT: s_waitcnt lgkmcnt(0)
@@ -218,8 +222,9 @@ define amdgpu_kernel void @barrier_gv_to_generic(ptr %out) {
; GFX1200-SDAG: ; %bb.0:
; GFX1200-SDAG-NEXT: s_load_b64 s[2:3], s[4:5], 0x0
; GFX1200-SDAG-NEXT: s_mov_b64 s[0:1], src_shared_base
-; GFX1200-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
-; GFX1200-SDAG-NEXT: v_dual_mov_b32 v0, 0x802001 :: v_dual_mov_b32 v1, s1
+; GFX1200-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX1200-SDAG-NEXT: s_or_b32 s0, s1, 1
+; GFX1200-SDAG-NEXT: v_dual_mov_b32 v0, 1 :: v_dual_mov_b32 v1, s0
; GFX1200-SDAG-NEXT: s_wait_kmcnt 0x0
; GFX1200-SDAG-NEXT: v_dual_mov_b32 v2, s2 :: v_dual_mov_b32 v3, s3
; GFX1200-SDAG-NEXT: flat_store_b64 v[2:3], v[0:1]
@@ -229,7 +234,8 @@ define amdgpu_kernel void @barrier_gv_to_generic(ptr %out) {
; GFX1200-GISEL: ; %bb.0:
; GFX1200-GISEL-NEXT: s_load_b64 s[2:3], s[4:5], 0x0
; GFX1200-GISEL-NEXT: s_mov_b64 s[0:1], src_shared_base
-; GFX1200-GISEL-NEXT: s_mov_b32 s0, 0x802001
+; GFX1200-GISEL-NEXT: s_mov_b32 s0, 1
+; GFX1200-GISEL-NEXT: s_or_b32 s1, s1, 1
; GFX1200-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
; GFX1200-GISEL-NEXT: v_dual_mov_b32 v0, s0 :: v_dual_mov_b32 v1, s1
; GFX1200-GISEL-NEXT: s_wait_kmcnt 0x0
@@ -242,9 +248,10 @@ define amdgpu_kernel void @barrier_gv_to_generic(ptr %out) {
; GFX1250-SDAG-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-SDAG-NEXT: s_load_b64 s[2:3], s[4:5], 0x0 nv
; GFX1250-SDAG-NEXT: s_mov_b64 s[0:1], src_shared_base
+; GFX1250-SDAG-NEXT: v_dual_mov_b32 v2, 0 :: v_dual_mov_b32 v0, 1
+; GFX1250-SDAG-NEXT: s_or_b32 s0, s1, 1
; GFX1250-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
-; GFX1250-SDAG-NEXT: v_dual_mov_b32 v2, 0 :: v_dual_mov_b32 v1, s1
-; GFX1250-SDAG-NEXT: v_mov_b32_e32 v0, 0x802001
+; GFX1250-SDAG-NEXT: v_mov_b32_e32 v1, s0
; GFX1250-SDAG-NEXT: s_wait_kmcnt 0x0
; GFX1250-SDAG-NEXT: flat_store_b64 v2, v[0:1], s[2:3]
; GFX1250-SDAG-NEXT: s_endpgm
@@ -254,7 +261,8 @@ define amdgpu_kernel void @barrier_gv_to_generic(ptr %out) {
; GFX1250-GISEL-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-GISEL-NEXT: s_load_b64 s[2:3], s[4:5], 0x0 nv
; GFX1250-GISEL-NEXT: s_mov_b64 s[0:1], src_shared_base
-; GFX1250-GISEL-NEXT: s_mov_b32 s0, 0x802001
+; GFX1250-GISEL-NEXT: s_mov_b32 s0, 1
+; GFX1250-GISEL-NEXT: s_or_b32 s1, s1, 1
; GFX1250-GISEL-NEXT: v_mov_b32_e32 v2, 0
; GFX1250-GISEL-NEXT: v_mov_b64_e32 v[0:1], s[0:1]
; GFX1250-GISEL-NEXT: s_wait_kmcnt 0x0
@@ -344,10 +352,9 @@ define amdgpu_kernel void @generic_to_barrier(ptr %generic, ptr %out) {
; GFX942-SDAG: ; %bb.0:
; GFX942-SDAG-NEXT: s_load_dwordx4 s[0:3], s[4:5], 0x0
; GFX942-SDAG-NEXT: s_waitcnt lgkmcnt(0)
-; GFX942-SDAG-NEXT: v_mov_b32_e32 v0, s2
-; GFX942-SDAG-NEXT: s_add_i32 s2, s0, 0xff7fe000
; GFX942-SDAG-NEXT: s_cmp_lg_u64 s[0:1], 0
-; GFX942-SDAG-NEXT: s_cselect_b32 s0, s2, 0
+; GFX942-SDAG-NEXT: s_cselect_b32 s0, s0, 0
+; GFX942-SDAG-NEXT: v_mov_b32_e32 v0, s2
; GFX942-SDAG-NEXT: v_mov_b32_e32 v1, s3
; GFX942-SDAG-NEXT: v_mov_b32_e32 v2, s0
; GFX942-SDAG-NEXT: flat_store_dword v[0:1], v2
@@ -357,9 +364,8 @@ define amdgpu_kernel void @generic_to_barrier(ptr %generic, ptr %out) {
; GFX942-GISEL: ; %bb.0:
; GFX942-GISEL-NEXT: s_load_dwordx4 s[0:3], s[4:5], 0x0
; GFX942-GISEL-NEXT: s_waitcnt lgkmcnt(0)
-; GFX942-GISEL-NEXT: s_add_i32 s4, s0, 0xff7fe000
; GFX942-GISEL-NEXT: s_cmp_lg_u64 s[0:1], 0
-; GFX942-GISEL-NEXT: s_cselect_b32 s0, s4, 0
+; GFX942-GISEL-NEXT: s_cselect_b32 s0, s0, 0
; GFX942-GISEL-NEXT: v_mov_b32_e32 v2, s0
; GFX942-GISEL-NEXT: v_mov_b64_e32 v[0:1], s[2:3]
; GFX942-GISEL-NEXT: flat_store_dword v[0:1], v2
@@ -373,10 +379,9 @@ define amdgpu_kernel void @generic_to_barrier(ptr %generic, ptr %out) {
; GFX1030-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13
; GFX1030-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0
; GFX1030-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1030-NEXT: s_add_i32 s4, s0, 0xff7fe000
; GFX1030-NEXT: s_cmp_lg_u64 s[0:1], 0
; GFX1030-NEXT: v_mov_b32_e32 v0, s2
-; GFX1030-NEXT: s_cselect_b32 s0, s4, 0
+; GFX1030-NEXT: s_cselect_b32 s0, s0, 0
; GFX1030-NEXT: v_mov_b32_e32 v1, s3
; GFX1030-NEXT: v_mov_b32_e32 v2, s0
; GFX1030-NEXT: flat_store_dword v[0:1], v2
@@ -386,10 +391,9 @@ define amdgpu_kernel void @generic_to_barrier(ptr %generic, ptr %out) {
; GFX1200-SDAG: ; %bb.0:
; GFX1200-SDAG-NEXT: s_load_b128 s[0:3], s[4:5], 0x0
; GFX1200-SDAG-NEXT: s_wait_kmcnt 0x0
-; GFX1200-SDAG-NEXT: s_add_co_i32 s4, s0, 0xff7fe000
; GFX1200-SDAG-NEXT: s_cmp_lg_u64 s[0:1], 0
; GFX1200-SDAG-NEXT: v_dual_mov_b32 v0, s2 :: v_dual_mov_b32 v1, s3
-; GFX1200-SDAG-NEXT: s_cselect_b32 s0, s4, 0
+; GFX1200-SDAG-NEXT: s_cselect_b32 s0, s0, 0
; GFX1200-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
; GFX1200-SDAG-NEXT: v_mov_b32_e32 v2, s0
; GFX1200-SDAG-NEXT: flat_store_b32 v[0:1], v2
@@ -399,10 +403,9 @@ define amdgpu_kernel void @generic_to_barrier(ptr %generic, ptr %out) {
; GFX1200-GISEL: ; %bb.0:
; GFX1200-GISEL-NEXT: s_load_b128 s[0:3], s[4:5], 0x0
; GFX1200-GISEL-NEXT: s_wait_kmcnt 0x0
-; GFX1200-GISEL-NEXT: s_add_co_i32 s4, s0, 0xff7fe000
; GFX1200-GISEL-NEXT: s_cmp_lg_u64 s[0:1], 0
; GFX1200-GISEL-NEXT: v_mov_b32_e32 v0, s2
-; GFX1200-GISEL-NEXT: s_cselect_b32 s0, s4, 0
+; GFX1200-GISEL-NEXT: s_cselect_b32 s0, s0, 0
; GFX1200-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
; GFX1200-GISEL-NEXT: v_dual_mov_b32 v1, s3 :: v_dual_mov_b32 v2, s0
; GFX1200-GISEL-NEXT: flat_store_b32 v[0:1], v2
@@ -413,9 +416,8 @@ define amdgpu_kernel void @generic_to_barrier(ptr %generic, ptr %out) {
; GFX1250-SDAG-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
; GFX1250-SDAG-NEXT: s_load_b128 s[0:3], s[4:5], 0x0 nv
; GFX1250-SDAG-NEXT: s_wait_kmcnt 0x0
-; GFX1250-SDAG-NEXT: s_add_co_i32 s4, s0, 0xff7fe000
; GFX1250-SDAG-NEXT: s_cmp_lg_u64 s[0:1], 0
-; GFX1250-SDAG-NEXT: s_cselect_b32 s0, s4, 0
+; GFX1250-SDAG-NEXT: s_cselect_b32 s0, s0, 0
; GFX1250-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
; GFX1250-SDAG-NEXT: v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s0
; GFX1250-SDAG-NEXT: flat_store_b32 v0, v1, s[2:3]
@@ -427,9 +429,8 @@ define amdgpu_kernel void @generic_to_barrier(ptr %generic, ptr %out) {
; GFX1250-GISEL-NEXT: s_load_b128 s[0:3], s[4:5], 0x0 nv
; GFX1250-GISEL-NEXT: v_mov_b32_e32 v1, 0
; GFX1250-GISEL-NEXT: s_wait_kmcnt 0x0
-; GFX1250-GISEL-NEXT: s_add_co_i32 s4, s0, 0xff7fe000
; GFX1250-GISEL-NEXT: s_cmp_lg_u64 s[0:1], 0
-; GFX1250-GISEL-NEXT: s_cselect_b32 s0, s4, 0
+; GFX1250-GISEL-NEXT: s_cselect_b32 s0, s0, 0
; GFX1250-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
; GFX1250-GISEL-NEXT: v_mov_b32_e32 v0, s0
; GFX1250-GISEL-NEXT: flat_store_b32 v1, v0, s[2:3]
More information about the llvm-branch-commits
mailing list