[clang] [llvm] [AMDGPU] Lower loads and stores for address space 13 (PR #209541)
Gheorghe-Teodor Bercea via cfe-commits
cfe-commits at lists.llvm.org
Tue Jul 14 23:49:16 PDT 2026
https://github.com/doru1004 updated https://github.com/llvm/llvm-project/pull/209541
>From 6c920f0248f56a5b34a1d1f8fb9c540841208809 Mon Sep 17 00:00:00 2001
From: Gheorghe-Teodor Bercea <dobercea at amd.com>
Date: Thu, 9 Jul 2026 15:46:01 -0500
Subject: [PATCH 1/2] Introduce address space 13 for VGPR as memory
---
clang/test/CodeGen/target-data.c | 4 +-
clang/test/CodeGenOpenCL/amdgpu-env-amdgcn.cl | 2 +-
llvm/docs/AMDGPUUsage.rst | 19 ++++-
llvm/include/llvm/Support/AMDGPUAddrSpace.h | 5 ++
llvm/lib/IR/AutoUpgrade.cpp | 4 +
llvm/lib/IR/VerifierAMDGPU.cpp | 12 ++-
llvm/lib/Target/AMDGPU/AMDGPUInstructions.td | 1 +
llvm/lib/TargetParser/TargetDataLayout.cpp | 3 +-
llvm/test/Verifier/AMDGPU/alloca.ll | 75 ++++++++++++-------
.../Bitcode/DataLayoutUpgradeTest.cpp | 38 +++++-----
10 files changed, 110 insertions(+), 53 deletions(-)
diff --git a/clang/test/CodeGen/target-data.c b/clang/test/CodeGen/target-data.c
index a5e0b814c7042..f03aaba8b53dd 100644
--- a/clang/test/CodeGen/target-data.c
+++ b/clang/test/CodeGen/target-data.c
@@ -160,12 +160,12 @@
// RUN: %clang_cc1 -triple amdgcn-unknown -target-cpu hawaii -o - -emit-llvm %s \
// RUN: | FileCheck %s -check-prefix=R600SI
-// R600SI: target datalayout = "e-m:e-p:64:64-p1:64:64-p2:32:32-p3:32:32-p4:64:64-p5:32:32-p6:32:32-p7:160:256:256:32-p8:128:128:128:48-p9:192:256:256:32-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-v2048:2048-n32:64-S32-A5-G1-ni:7:8:9"
+// R600SI: target datalayout = "e-m:e-p:64:64-p1:64:64-p2:32:32-p3:32:32-p4:64:64-p5:32:32-p6:32:32-p7:160:256:256:32-p8:128:128:128:48-p9:192:256:256:32-p13:32:32-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-v2048:2048-n32:64-S32-A5-G1-ni:7:8:9"
// Test default -target-cpu
// RUN: %clang_cc1 -triple amdgcn-unknown -o - -emit-llvm %s \
// RUN: | FileCheck %s -check-prefix=R600SIDefault
-// R600SIDefault: target datalayout = "e-m:e-p:64:64-p1:64:64-p2:32:32-p3:32:32-p4:64:64-p5:32:32-p6:32:32-p7:160:256:256:32-p8:128:128:128:48-p9:192:256:256:32-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-v2048:2048-n32:64-S32-A5-G1-ni:7:8:9"
+// R600SIDefault: target datalayout = "e-m:e-p:64:64-p1:64:64-p2:32:32-p3:32:32-p4:64:64-p5:32:32-p6:32:32-p7:160:256:256:32-p8:128:128:128:48-p9:192:256:256:32-p13:32:32-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-v2048:2048-n32:64-S32-A5-G1-ni:7:8:9"
// RUN: %clang_cc1 -triple arm64-unknown -o - -emit-llvm %s | \
// RUN: FileCheck %s -check-prefix=AARCH64
diff --git a/clang/test/CodeGenOpenCL/amdgpu-env-amdgcn.cl b/clang/test/CodeGenOpenCL/amdgpu-env-amdgcn.cl
index 72ce72644b8ea..f120db1aaf6cd 100644
--- a/clang/test/CodeGenOpenCL/amdgpu-env-amdgcn.cl
+++ b/clang/test/CodeGenOpenCL/amdgpu-env-amdgcn.cl
@@ -1,5 +1,5 @@
// RUN: %clang_cc1 %s -O0 -triple amdgcn -emit-llvm -o - | FileCheck %s
// RUN: %clang_cc1 %s -O0 -triple amdgcn---opencl -emit-llvm -o - | FileCheck %s
-// CHECK: target datalayout = "e-m:e-p:64:64-p1:64:64-p2:32:32-p3:32:32-p4:64:64-p5:32:32-p6:32:32-p7:160:256:256:32-p8:128:128:128:48-p9:192:256:256:32-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-v2048:2048-n32:64-S32-A5-G1-ni:7:8:9"
+// CHECK: target datalayout = "e-m:e-p:64:64-p1:64:64-p2:32:32-p3:32:32-p4:64:64-p5:32:32-p6:32:32-p7:160:256:256:32-p8:128:128:128:48-p9:192:256:256:32-p13:32:32-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-v2048:2048-n32:64-S32-A5-G1-ni:7:8:9"
void foo(void) {}
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index 741f1f766eedd..e2464066554ed 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -997,7 +997,7 @@ supported for the ``amdgcn`` target.
*reserved for future use* 10
*reserved for future use* 11
*reserved for downstream use (LLPC)* 12
- *reserved for future use* 13
+ VGPR 13 N/A VGPR 32 0xFFFFFFFF
*reserved for future use* 14
*reserved for future use* 16
Streamout Registers 128 N/A GS_REGS
@@ -1212,6 +1212,23 @@ supported for the ``amdgcn`` target.
a buffer strided pointer, this means that the base pointer is ``align(4)``, that
the offset is a multiple of 4 bytes, and that the stride is a multiple of 4.
+**VGPR**
+ The VGPR address space presents a memory view of the wave's vector registers.
+ The 32-bit address is a byte address into the thread's view of vector
+ registers. For example, loading 4 bytes from address ``12`` reads the contents
+ of ``v3``. Storing 8 bytes to address ``32`` overwrites the contents of
+ ``v[8:9]``.
+
+ Use of this address space by frontends is strongly discouraged. It has unusual
+ and subtle lifetime rules due to the potential for interaction with normal
+ register allocation. It exists primarily for internal purposes of the backend,
+ such as promoting ``alloca`` instructions from the private address space into
+ VGPRs.
+
+ In particular, memory in this address space that was allocated by an
+ ``alloca`` is not visible while in a called function. Attempting to dereference
+ a pointer to such memory in a called function is undefined behavior.
+
**Streamout Registers**
Dedicated registers used by the GS NGG Streamout Instructions. The register
file is modelled as a memory in a distinct address space because it is indexed
diff --git a/llvm/include/llvm/Support/AMDGPUAddrSpace.h b/llvm/include/llvm/Support/AMDGPUAddrSpace.h
index 01b1510524d0f..b1af8047ce580 100644
--- a/llvm/include/llvm/Support/AMDGPUAddrSpace.h
+++ b/llvm/include/llvm/Support/AMDGPUAddrSpace.h
@@ -47,6 +47,11 @@ enum : unsigned {
BUFFER_STRIDED_POINTER = 9, ///< Address space for 192-bit fat buffer
///< pointers with an additional index.
+ VGPR = 13, ///< Address space for VGPRs. The 32-bit address is a byte offset
+ ///< into the wave's view of its vector registers. Note this shares
+ ///< its numeric value with CONSTANT_BUFFER_5, which is only used
+ ///< by the (graphics) R600 path.
+
RESERVED_ADDRESS_SPACE_16 = 16, ///< Reserved for downstream use.
/// Internal address spaces. Can be freely renumbered.
diff --git a/llvm/lib/IR/AutoUpgrade.cpp b/llvm/lib/IR/AutoUpgrade.cpp
index 229c7757db3dc..e903e561386c2 100644
--- a/llvm/lib/IR/AutoUpgrade.cpp
+++ b/llvm/lib/IR/AutoUpgrade.cpp
@@ -6958,6 +6958,10 @@ std::string llvm::UpgradeDataLayoutString(StringRef DL, StringRef TT) {
Res.replace(Res.find(OldP8), OldP8.size(), "-p8:128:128:128:48-");
if (!DL.contains("-p9") && !DL.starts_with("p9"))
Res.append("-p9:192:256:256:32");
+ // Add sizing for the VGPR address space (32-bit byte offset into the
+ // wave's vector registers).
+ if (!DL.contains("-p13") && !DL.starts_with("p13"))
+ Res.append("-p13:32:32");
}
// Upgrade the ELF mangling mode.
diff --git a/llvm/lib/IR/VerifierAMDGPU.cpp b/llvm/lib/IR/VerifierAMDGPU.cpp
index 04cb214ef2520..a9cbdc20ace35 100644
--- a/llvm/lib/IR/VerifierAMDGPU.cpp
+++ b/llvm/lib/IR/VerifierAMDGPU.cpp
@@ -122,8 +122,16 @@ void llvm::verifyAMDGPUAlloca(VerifierSupport &VS, const AllocaInst &AI) {
if (!VS.TT.isAMDGPU())
return;
- if (AI.getAddressSpace() != AMDGPUAS::PRIVATE_ADDRESS)
- VS.CheckFailed("alloca on amdgpu must be in addrspace(5)", &AI);
+ if (AI.getAddressSpace() != AMDGPUAS::PRIVATE_ADDRESS &&
+ AI.getAddressSpace() != AMDGPUAS::VGPR)
+ VS.CheckFailed("alloca on amdgpu must be in addrspace(5) or addrspace(13)",
+ &AI);
+
+ // Only static allocas can live in VGPRs; a dynamically sized one has no
+ // register-file representation. (Other address spaces are already rejected
+ // above, so this only adds the more specific diagnostic for addrspace(13).)
+ if (!AI.isStaticAlloca() && AI.getAddressSpace() == AMDGPUAS::VGPR)
+ VS.CheckFailed("dynamic alloca on amdgpu must be in addrspace(5)", &AI);
}
bool llvm::isAMDGPUCallBrIntrinsic(Intrinsic::ID ID) {
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUInstructions.td b/llvm/lib/Target/AMDGPU/AMDGPUInstructions.td
index 2387afe1cd362..aa06e49c68e03 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUInstructions.td
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInstructions.td
@@ -19,6 +19,7 @@ def AddrSpaces {
int Constant = 4;
int Private = 5;
int Constant32Bit = 6;
+ int VGPR = 13;
}
diff --git a/llvm/lib/TargetParser/TargetDataLayout.cpp b/llvm/lib/TargetParser/TargetDataLayout.cpp
index cab9d7792c787..976297ae62922 100644
--- a/llvm/lib/TargetParser/TargetDataLayout.cpp
+++ b/llvm/lib/TargetParser/TargetDataLayout.cpp
@@ -274,7 +274,8 @@ static std::string computeAMDDataLayout(const Triple &TT) {
// space 8) which cannot be non-trivilally accessed by LLVM memory operations
// like getelementptr.
return "e-m:e-p:64:64-p1:64:64-p2:32:32-p3:32:32-p4:64:64-p5:32:32-p6:32:32"
- "-p7:160:256:256:32-p8:128:128:128:48-p9:192:256:256:32-i64:64-"
+ "-p7:160:256:256:32-p8:128:128:128:48-p9:192:256:256:32-p13:32:32-"
+ "i64:64-"
"v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-"
"v1024:1024-v2048:2048-n32:64-S32-A5-G1-ni:7:8:9";
}
diff --git a/llvm/test/Verifier/AMDGPU/alloca.ll b/llvm/test/Verifier/AMDGPU/alloca.ll
index f31d6228d7936..24b422a37bd7d 100644
--- a/llvm/test/Verifier/AMDGPU/alloca.ll
+++ b/llvm/test/Verifier/AMDGPU/alloca.ll
@@ -2,23 +2,25 @@
target triple = "amdgcn-amd-amdhsa"
-; CHECK: alloca on amdgpu must be in addrspace(5)
+; A static alloca is allowed in addrspace(5) (private) and addrspace(13) (VGPR);
+; any other address space is rejected.
+; CHECK: alloca on amdgpu must be in addrspace(5) or addrspace(13)
; CHECK-NEXT: %alloca.0 = alloca i32, align 4
-; CHECK-NEXT: alloca on amdgpu must be in addrspace(5)
+; CHECK-NEXT: alloca on amdgpu must be in addrspace(5) or addrspace(13)
; CHECK-NEXT: %alloca.1 = alloca i32, align 4, addrspace(1)
-; CHECK-NEXT: alloca on amdgpu must be in addrspace(5)
+; CHECK-NEXT: alloca on amdgpu must be in addrspace(5) or addrspace(13)
; CHECK-NEXT: %alloca.2 = alloca i32, align 4, addrspace(2)
-; CHECK-NEXT: alloca on amdgpu must be in addrspace(5)
+; CHECK-NEXT: alloca on amdgpu must be in addrspace(5) or addrspace(13)
; CHECK-NEXT: %alloca.3 = alloca i32, align 4, addrspace(3)
-; CHECK-NEXT: alloca on amdgpu must be in addrspace(5)
+; CHECK-NEXT: alloca on amdgpu must be in addrspace(5) or addrspace(13)
; CHECK-NEXT: %alloca.4 = alloca i32, align 4, addrspace(4)
-; CHECK-NEXT: alloca on amdgpu must be in addrspace(5)
+; CHECK-NEXT: alloca on amdgpu must be in addrspace(5) or addrspace(13)
; CHECK-NEXT: %alloca.6 = alloca i32, align 4, addrspace(6)
-; CHECK-NEXT: alloca on amdgpu must be in addrspace(5)
+; CHECK-NEXT: alloca on amdgpu must be in addrspace(5) or addrspace(13)
; CHECK-NEXT: %alloca.7 = alloca i32, align 4, addrspace(7)
-; CHECK-NEXT: alloca on amdgpu must be in addrspace(5)
+; CHECK-NEXT: alloca on amdgpu must be in addrspace(5) or addrspace(13)
; CHECK-NEXT: %alloca.8 = alloca i32, align 4, addrspace(8)
-; CHECK-NEXT: alloca on amdgpu must be in addrspace(5)
+; CHECK-NEXT: alloca on amdgpu must be in addrspace(5) or addrspace(13)
; CHECK-NEXT: %alloca.9 = alloca i32, align 4, addrspace(9)
define void @static_alloca() {
entry:
@@ -32,27 +34,32 @@ entry:
%alloca.7 = alloca i32, align 4, addrspace(7)
%alloca.8 = alloca i32, align 4, addrspace(8)
%alloca.9 = alloca i32, align 4, addrspace(9)
+ %alloca.13 = alloca i32, align 4, addrspace(13)
ret void
}
-; CHECK: alloca on amdgpu must be in addrspace(5)
+; A dynamically sized alloca is only allowed in addrspace(5); addrspace(13) is
+; rejected because it has no register-file representation.
+; CHECK: alloca on amdgpu must be in addrspace(5) or addrspace(13)
; CHECK-NEXT: %alloca.0 = alloca i32, i32 %n, align 4
-; CHECK-NEXT: alloca on amdgpu must be in addrspace(5)
+; CHECK-NEXT: alloca on amdgpu must be in addrspace(5) or addrspace(13)
; CHECK-NEXT: %alloca.1 = alloca i32, i32 %n, align 4, addrspace(1)
-; CHECK-NEXT: alloca on amdgpu must be in addrspace(5)
+; CHECK-NEXT: alloca on amdgpu must be in addrspace(5) or addrspace(13)
; CHECK-NEXT: %alloca.2 = alloca i32, i32 %n, align 4, addrspace(2)
-; CHECK-NEXT: alloca on amdgpu must be in addrspace(5)
+; CHECK-NEXT: alloca on amdgpu must be in addrspace(5) or addrspace(13)
; CHECK-NEXT: %alloca.3 = alloca i32, i32 %n, align 4, addrspace(3)
-; CHECK-NEXT: alloca on amdgpu must be in addrspace(5)
+; CHECK-NEXT: alloca on amdgpu must be in addrspace(5) or addrspace(13)
; CHECK-NEXT: %alloca.4 = alloca i32, i32 %n, align 4, addrspace(4)
-; CHECK-NEXT: alloca on amdgpu must be in addrspace(5)
+; CHECK-NEXT: alloca on amdgpu must be in addrspace(5) or addrspace(13)
; CHECK-NEXT: %alloca.6 = alloca i32, i32 %n, align 4, addrspace(6)
-; CHECK-NEXT: alloca on amdgpu must be in addrspace(5)
+; CHECK-NEXT: alloca on amdgpu must be in addrspace(5) or addrspace(13)
; CHECK-NEXT: %alloca.7 = alloca i32, i32 %n, align 4, addrspace(7)
-; CHECK-NEXT: alloca on amdgpu must be in addrspace(5)
+; CHECK-NEXT: alloca on amdgpu must be in addrspace(5) or addrspace(13)
; CHECK-NEXT: %alloca.8 = alloca i32, i32 %n, align 4, addrspace(8)
-; CHECK-NEXT: alloca on amdgpu must be in addrspace(5)
+; CHECK-NEXT: alloca on amdgpu must be in addrspace(5) or addrspace(13)
; CHECK-NEXT: %alloca.9 = alloca i32, i32 %n, align 4, addrspace(9)
+; CHECK-NEXT: dynamic alloca on amdgpu must be in addrspace(5)
+; CHECK-NEXT: %alloca.13 = alloca i32, i32 %n, align 4, addrspace(13)
define void @dynamic_alloca_i32(i32 %n) {
entry:
%alloca.0 = alloca i32, i32 %n, align 4
@@ -65,26 +72,27 @@ entry:
%alloca.7 = alloca i32, i32 %n, align 4, addrspace(7)
%alloca.8 = alloca i32, i32 %n, align 4, addrspace(8)
%alloca.9 = alloca i32, i32 %n, align 4, addrspace(9)
+ %alloca.13 = alloca i32, i32 %n, align 4, addrspace(13)
ret void
}
-; CHECK: alloca on amdgpu must be in addrspace(5)
+; CHECK: alloca on amdgpu must be in addrspace(5) or addrspace(13)
; CHECK-NEXT: %alloca.0 = alloca i32, i64 %n, align 4
-; CHECK-NEXT: alloca on amdgpu must be in addrspace(5)
+; CHECK-NEXT: alloca on amdgpu must be in addrspace(5) or addrspace(13)
; CHECK-NEXT: %alloca.1 = alloca i32, i64 %n, align 4, addrspace(1)
-; CHECK-NEXT: alloca on amdgpu must be in addrspace(5)
+; CHECK-NEXT: alloca on amdgpu must be in addrspace(5) or addrspace(13)
; CHECK-NEXT: %alloca.2 = alloca i32, i64 %n, align 4, addrspace(2)
-; CHECK-NEXT: alloca on amdgpu must be in addrspace(5)
+; CHECK-NEXT: alloca on amdgpu must be in addrspace(5) or addrspace(13)
; CHECK-NEXT: %alloca.3 = alloca i32, i64 %n, align 4, addrspace(3)
-; CHECK-NEXT: alloca on amdgpu must be in addrspace(5)
+; CHECK-NEXT: alloca on amdgpu must be in addrspace(5) or addrspace(13)
; CHECK-NEXT: %alloca.4 = alloca i32, i64 %n, align 4, addrspace(4)
-; CHECK-NEXT: alloca on amdgpu must be in addrspace(5)
+; CHECK-NEXT: alloca on amdgpu must be in addrspace(5) or addrspace(13)
; CHECK-NEXT: %alloca.6 = alloca i32, i64 %n, align 4, addrspace(6)
-; CHECK-NEXT: alloca on amdgpu must be in addrspace(5)
+; CHECK-NEXT: alloca on amdgpu must be in addrspace(5) or addrspace(13)
; CHECK-NEXT: %alloca.7 = alloca i32, i64 %n, align 4, addrspace(7)
-; CHECK-NEXT: alloca on amdgpu must be in addrspace(5)
+; CHECK-NEXT: alloca on amdgpu must be in addrspace(5) or addrspace(13)
; CHECK-NEXT: %alloca.8 = alloca i32, i64 %n, align 4, addrspace(8)
-; CHECK-NEXT: alloca on amdgpu must be in addrspace(5)
+; CHECK-NEXT: alloca on amdgpu must be in addrspace(5) or addrspace(13)
; CHECK-NEXT: %alloca.9 = alloca i32, i64 %n, align 4, addrspace(9)
define void @dynamic_alloca_i64(i64 %n) {
entry:
@@ -100,3 +108,16 @@ entry:
%alloca.9 = alloca i32, i64 %n, align 4, addrspace(9)
ret void
}
+
+; A static alloca that is not in the entry block is treated as dynamic, so
+; addrspace(13) is rejected there too.
+; CHECK: dynamic alloca on amdgpu must be in addrspace(5)
+; CHECK-NEXT: %alloca.13 = alloca i32, align 4, addrspace(13)
+define void @nonentry_alloca() {
+entry:
+ br label %nonentry
+
+nonentry:
+ %alloca.13 = alloca i32, align 4, addrspace(13)
+ ret void
+}
diff --git a/llvm/unittests/Bitcode/DataLayoutUpgradeTest.cpp b/llvm/unittests/Bitcode/DataLayoutUpgradeTest.cpp
index a082adbf6565e..0ec3c753c10f1 100644
--- a/llvm/unittests/Bitcode/DataLayoutUpgradeTest.cpp
+++ b/llvm/unittests/Bitcode/DataLayoutUpgradeTest.cpp
@@ -43,14 +43,14 @@ TEST(DataLayoutUpgradeTest, ValidDataLayoutUpgrade) {
// and that ANDGCN adds p7 and p8 as well.
EXPECT_EQ(UpgradeDataLayoutString("e-p:64:64", "amdgcn"),
"m:e-e-p:64:64-G1-ni:7:8:9-p7:160:256:256:32-p8:128:128:128:48-p9:"
- "192:256:256:32");
+ "192:256:256:32-p13:32:32");
EXPECT_EQ(UpgradeDataLayoutString("e-p:64:64-G1", "amdgcn"),
"m:e-e-p:64:64-G1-ni:7:8:9-p7:160:256:256:32-p8:128:128:128:48-p9:"
- "192:256:256:32");
+ "192:256:256:32-p13:32:32");
// Check that the old AMDGCN p8:128:128 definition is upgraded
EXPECT_EQ(UpgradeDataLayoutString("e-p:64:64-p8:128:128-G1", "amdgcn"),
"m:e-e-p:64:64-p8:128:128:128:48-G1-ni:7:8:9-p7:160:256:256:32-p9:"
- "192:256:256:32");
+ "192:256:256:32-p13:32:32");
// but that r600 does not.
EXPECT_EQ(UpgradeDataLayoutString("e-p:32:32-G1", "r600"),
"m:e-e-p:32:32-G1");
@@ -66,7 +66,7 @@ TEST(DataLayoutUpgradeTest, ValidDataLayoutUpgrade) {
"m:e-e-p:64:64-p1:64:64-p2:32:32-p3:32:32-p4:64:64-p5:32:32-p6:32:32-i64:"
"64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:"
"1024-v2048:2048-n32:64-S32-A5-G1-ni:7:8:9-p7:160:256:256:32-p8:128:128:"
- "128:48-p9:192:256:256:32");
+ "128:48-p9:192:256:256:32-p13:32:32");
// Check that SystemZ adds -S64 if needed.
EXPECT_EQ(UpgradeDataLayoutString(
@@ -158,24 +158,24 @@ TEST(DataLayoutUpgradeTest, NoDataLayoutUpgrade) {
EXPECT_EQ(UpgradeDataLayoutString("G2", "r600"), "m:e-G2");
EXPECT_EQ(UpgradeDataLayoutString("e-p:64:64-G2", "amdgcn"),
"m:e-e-p:64:64-G2-ni:7:8:9-p7:160:256:256:32-p8:128:128:128:48-p9:"
- "192:256:256:32");
+ "192:256:256:32-p13:32:32");
EXPECT_EQ(UpgradeDataLayoutString("G2-e-p:64:64", "amdgcn"),
"m:e-G2-e-p:64:64-ni:7:8:9-p7:160:256:256:32-p8:128:128:128:48-p9:"
- "192:256:256:32");
+ "192:256:256:32-p13:32:32");
EXPECT_EQ(UpgradeDataLayoutString("e-p:64:64-G0", "amdgcn"),
"m:e-e-p:64:64-G0-ni:7:8:9-p7:160:256:256:32-p8:128:128:128:48-p9:"
- "192:256:256:32");
+ "192:256:256:32-p13:32:32");
// Check that AMDGCN targets don't add already declared address space 7.
- EXPECT_EQ(
- UpgradeDataLayoutString("e-p:64:64-p7:64:64", "amdgcn"),
- "m:e-e-p:64:64-p7:64:64-G1-ni:7:8:9-p8:128:128:128:48-p9:192:256:256:32");
- EXPECT_EQ(
- UpgradeDataLayoutString("p7:64:64-G2-e-p:64:64", "amdgcn"),
- "m:e-p7:64:64-G2-e-p:64:64-ni:7:8:9-p8:128:128:128:48-p9:192:256:256:32");
- EXPECT_EQ(
- UpgradeDataLayoutString("e-p:64:64-p7:64:64-G1", "amdgcn"),
- "m:e-e-p:64:64-p7:64:64-G1-ni:7:8:9-p8:128:128:128:48-p9:192:256:256:32");
+ EXPECT_EQ(UpgradeDataLayoutString("e-p:64:64-p7:64:64", "amdgcn"),
+ "m:e-e-p:64:64-p7:64:64-G1-ni:7:8:9-p8:128:128:128:48-p9:192:256:"
+ "256:32-p13:32:32");
+ EXPECT_EQ(UpgradeDataLayoutString("p7:64:64-G2-e-p:64:64", "amdgcn"),
+ "m:e-p7:64:64-G2-e-p:64:64-ni:7:8:9-p8:128:128:128:48-p9:192:256:"
+ "256:32-p13:32:32");
+ EXPECT_EQ(UpgradeDataLayoutString("e-p:64:64-p7:64:64-G1", "amdgcn"),
+ "m:e-e-p:64:64-p7:64:64-G1-ni:7:8:9-p8:128:128:128:48-p9:192:256:"
+ "256:32-p13:32:32");
// Check that SPIR & SPIRV targets don't add -G1 if there is already a -G
// flag.
@@ -216,9 +216,9 @@ TEST(DataLayoutUpgradeTest, EmptyDataLayout) {
// Check that AMDGPU targets add G1 if it's not present.
EXPECT_EQ(UpgradeDataLayoutString("", "r600"), "m:e-G1");
- EXPECT_EQ(
- UpgradeDataLayoutString("", "amdgcn"),
- "m:e-G1-ni:7:8:9-p7:160:256:256:32-p8:128:128:128:48-p9:192:256:256:32");
+ EXPECT_EQ(UpgradeDataLayoutString("", "amdgcn"),
+ "m:e-G1-ni:7:8:9-p7:160:256:256:32-p8:128:128:128:48-p9:192:256:"
+ "256:32-p13:32:32");
// Check that SPIR & SPIRV targets add G1 if it's not present.
EXPECT_EQ(UpgradeDataLayoutString("", "spir"), "G1");
>From f4904da67970fa6eef7255934f98764807483766 Mon Sep 17 00:00:00 2001
From: Gheorghe-Teodor Bercea <dobercea at amd.com>
Date: Tue, 14 Jul 2026 00:06:53 -0500
Subject: [PATCH 2/2] Lower loads and stores for address space 13
---
llvm/lib/Target/AMDGPU/AMDGPU.h | 9 +
.../lib/Target/AMDGPU/AMDGPUAssignIdxToM0.cpp | 108 ++++
llvm/lib/Target/AMDGPU/AMDGPUGISel.td | 3 +
.../AMDGPU/AMDGPUInstructionSelector.cpp | 27 +
.../Target/AMDGPU/AMDGPUInstructionSelector.h | 1 +
.../lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp | 96 ++++
.../Target/AMDGPU/AMDGPULowerVGPREncoding.cpp | 100 +++-
.../lib/Target/AMDGPU/AMDGPUMachineInstrs.cpp | 55 +++
llvm/lib/Target/AMDGPU/AMDGPUMachineInstrs.h | 71 +++
llvm/lib/Target/AMDGPU/AMDGPUPassRegistry.def | 1 +
.../AMDGPU/AMDGPURegBankLegalizeRules.cpp | 7 +
.../Target/AMDGPU/AMDGPURegisterBankInfo.cpp | 15 +
.../Target/AMDGPU/AMDGPUSearchableTables.td | 17 +
.../lib/Target/AMDGPU/AMDGPUTargetMachine.cpp | 9 +
.../AMDGPU/AMDGPUTargetTransformInfo.cpp | 7 +-
llvm/lib/Target/AMDGPU/CMakeLists.txt | 2 +
llvm/lib/Target/AMDGPU/SIISelLowering.cpp | 117 +++++
llvm/lib/Target/AMDGPU/SIISelLowering.h | 3 +
llvm/lib/Target/AMDGPU/SIInstrInfo.cpp | 74 ++-
llvm/lib/Target/AMDGPU/SIInstrInfo.td | 11 +
llvm/lib/Target/AMDGPU/SIInstructions.td | 100 ++++
.../Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp | 11 +
llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h | 13 +
.../AMDGPU/AddressSpaceVGPR/as-vgpr-basic.ll | 384 +++++++++++++++
.../AMDGPU/AddressSpaceVGPR/as-vgpr-copy.ll | 461 ++++++++++++++++++
.../AddressSpaceVGPR/as-vgpr-divergent.ll | 130 +++++
.../AddressSpaceVGPR/as-vgpr-inttoptr.ll | 48 ++
.../AddressSpaceVGPR/as-vgpr-unsupported.ll | 31 ++
llvm/test/CodeGen/AMDGPU/llc-pipeline-npm.ll | 3 +
llvm/test/CodeGen/AMDGPU/llc-pipeline.ll | 5 +
30 files changed, 1895 insertions(+), 24 deletions(-)
create mode 100644 llvm/lib/Target/AMDGPU/AMDGPUAssignIdxToM0.cpp
create mode 100644 llvm/lib/Target/AMDGPU/AMDGPUMachineInstrs.cpp
create mode 100644 llvm/lib/Target/AMDGPU/AMDGPUMachineInstrs.h
create mode 100644 llvm/test/CodeGen/AMDGPU/AddressSpaceVGPR/as-vgpr-basic.ll
create mode 100644 llvm/test/CodeGen/AMDGPU/AddressSpaceVGPR/as-vgpr-copy.ll
create mode 100644 llvm/test/CodeGen/AMDGPU/AddressSpaceVGPR/as-vgpr-divergent.ll
create mode 100644 llvm/test/CodeGen/AMDGPU/AddressSpaceVGPR/as-vgpr-inttoptr.ll
create mode 100644 llvm/test/CodeGen/AMDGPU/AddressSpaceVGPR/as-vgpr-unsupported.ll
diff --git a/llvm/lib/Target/AMDGPU/AMDGPU.h b/llvm/lib/Target/AMDGPU/AMDGPU.h
index e4367811f1ed4..6c0c8f07a4b51 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPU.h
+++ b/llvm/lib/Target/AMDGPU/AMDGPU.h
@@ -223,6 +223,9 @@ extern char &AMDGPURegBankLegalizeID;
void initializeAMDGPUMarkLastScratchLoadLegacyPass(PassRegistry &);
extern char &AMDGPUMarkLastScratchLoadID;
+void initializeAMDGPUAssignIdxToM0LegacyPass(PassRegistry &);
+extern char &AMDGPUAssignIdxToM0ID;
+
void initializeSILowerSGPRSpillsLegacyPass(PassRegistry &);
extern char &SILowerSGPRSpillsLegacyID;
@@ -437,6 +440,12 @@ class AMDGPUMarkLastScratchLoadPass
MachineFunctionAnalysisManager &AM);
};
+class AMDGPUAssignIdxToM0Pass : public PassInfoMixin<AMDGPUAssignIdxToM0Pass> {
+public:
+ PreservedAnalyses run(MachineFunction &MF,
+ MachineFunctionAnalysisManager &MFAM);
+};
+
class SIInsertWaitcntsPass
: public RequiredPassInfoMixin<SIInsertWaitcntsPass> {
public:
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAssignIdxToM0.cpp b/llvm/lib/Target/AMDGPU/AMDGPUAssignIdxToM0.cpp
new file mode 100644
index 0000000000000..8c5de37eb46e5
--- /dev/null
+++ b/llvm/lib/Target/AMDGPU/AMDGPUAssignIdxToM0.cpp
@@ -0,0 +1,108 @@
+//===- AMDGPUAssignIdxToM0.cpp - Copy VGPR-memory indices to M0 ----------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+/// \file
+/// Copy the register index of a VGPR "as memory" (address space 13)
+/// V_LOAD_IDX / V_STORE_IDX pseudo into M0, which V_MOVREL[SD] reads when the
+/// pseudo is lowered (see AMDGPULowerVGPREncoding). This runs before register
+/// allocation so the copy to M0 is inserted while the index is still virtual.
+//
+//===----------------------------------------------------------------------===//
+
+#include "AMDGPU.h"
+#include "AMDGPUMachineInstrs.h"
+#include "GCNSubtarget.h"
+#include "SIInstrInfo.h"
+#include "llvm/CodeGen/MachineFunctionPass.h"
+#include "llvm/CodeGen/MachineInstrBuilder.h"
+#include "llvm/CodeGen/MachinePassManager.h"
+#include "llvm/InitializePasses.h"
+
+using namespace llvm;
+
+#define DEBUG_TYPE "amdgpu-assign-idx-to-m0"
+
+static bool assignIdxToM0(MachineFunction &MF) {
+ const GCNSubtarget &ST = MF.getSubtarget<GCNSubtarget>();
+ if (!ST.hasMovrel())
+ return false;
+
+ const SIInstrInfo *TII = ST.getInstrInfo();
+
+ bool Changed = false;
+ for (MachineBasicBlock &MBB : MF) {
+ for (MachineInstr &MI : MBB) {
+ auto *LdSt = dyn_cast<AMDGPUMI::VLoadStoreIdxInst>(&MI);
+ if (!LdSt)
+ continue;
+
+ MachineOperand &IdxOp = LdSt->getIdxOp();
+ if (!IdxOp.isReg())
+ continue;
+
+ assert(!MI.isBundled());
+
+ // Remove the implicit-def $m0 that instruction selection added (to pin a
+ // divergent access inside its waterfall loop); M0 is written for real
+ // below.
+ int DefIdx = MI.findRegisterDefOperandIdx(AMDGPU::M0, /*TRI=*/nullptr);
+ assert(DefIdx >= 0);
+ MI.removeOperand(DefIdx);
+
+ // Add a copy from the index register to M0 and rewrite MI to read M0.
+ BuildMI(MBB, &MI, MI.getDebugLoc(), TII->get(AMDGPU::COPY), AMDGPU::M0)
+ .add(IdxOp);
+ IdxOp.setReg(AMDGPU::M0);
+ IdxOp.setIsKill();
+ Changed = true;
+ }
+ }
+
+ return Changed;
+}
+
+namespace {
+
+class AMDGPUAssignIdxToM0Legacy : public MachineFunctionPass {
+public:
+ static char ID;
+
+ AMDGPUAssignIdxToM0Legacy() : MachineFunctionPass(ID) {}
+
+ bool runOnMachineFunction(MachineFunction &MF) override {
+ if (skipFunction(MF.getFunction()))
+ return false;
+ return assignIdxToM0(MF);
+ }
+
+ void getAnalysisUsage(AnalysisUsage &AU) const override {
+ AU.setPreservesCFG();
+ MachineFunctionPass::getAnalysisUsage(AU);
+ }
+
+ StringRef getPassName() const override { return "AMDGPU Assign Idx To M0"; }
+};
+
+} // end anonymous namespace
+
+PreservedAnalyses
+AMDGPUAssignIdxToM0Pass::run(MachineFunction &MF,
+ MachineFunctionAnalysisManager &MFAM) {
+ if (!assignIdxToM0(MF))
+ return PreservedAnalyses::all();
+ auto PA = getMachineFunctionPassPreservedAnalyses();
+ PA.preserveSet<CFGAnalyses>();
+ return PA;
+}
+
+char AMDGPUAssignIdxToM0Legacy::ID = 0;
+
+char &llvm::AMDGPUAssignIdxToM0ID = AMDGPUAssignIdxToM0Legacy::ID;
+
+INITIALIZE_PASS(AMDGPUAssignIdxToM0Legacy, DEBUG_TYPE,
+ "AMDGPU Assign Idx To M0", false, false)
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUGISel.td b/llvm/lib/Target/AMDGPU/AMDGPUGISel.td
index a88322df79036..38a692a45fa29 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUGISel.td
+++ b/llvm/lib/Target/AMDGPU/AMDGPUGISel.td
@@ -282,6 +282,9 @@ def : GINodeEquiv<G_AMDGPU_FMINIMUM3, AMDGPUfminimum3>;
def : GINodeEquiv<G_AMDGPU_CLAMP, AMDGPUclamp>;
+def : GINodeEquiv<G_AMDGPU_REG_LOAD, SIreg_load>;
+def : GINodeEquiv<G_AMDGPU_REG_STORE, SIreg_store>;
+
def : GINodeEquiv<G_AMDGPU_ATOMIC_CMPXCHG, AMDGPUatomic_cmp_swap>;
def : GINodeEquiv<G_AMDGPU_BUFFER_LOAD, SIbuffer_load>;
def : GINodeEquiv<G_AMDGPU_BUFFER_LOAD_USHORT, SIbuffer_load_ushort>;
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp b/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp
index c85d7124f62bb..974af5bd30b76 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp
@@ -15,6 +15,7 @@
#include "AMDGPU.h"
#include "AMDGPUGlobalISelUtils.h"
#include "AMDGPUInstrInfo.h"
+#include "AMDGPUMachineInstrs.h"
#include "AMDGPURegisterBankInfo.h"
#include "AMDGPUTargetMachine.h"
#include "SIMachineFunctionInfo.h"
@@ -4550,6 +4551,29 @@ bool AMDGPUInstructionSelector::selectStackRestore(MachineInstr &MI) const {
return true;
}
+bool AMDGPUInstructionSelector::selectRegLoadStore(MachineInstr &I) const {
+ // Remember where the selected machine instruction will land.
+ MachineBasicBlock::iterator II = std::next(I.getIterator());
+
+ if (!selectImpl(I, *CoverageInfo))
+ return false;
+
+ // On a movrel subtarget the selected V_LOAD_IDX / V_STORE_IDX expands to an
+ // M0-relative move (see AMDGPUAssignIdxToM0 and AMDGPULowerVGPREncoding). Add
+ // an implicit-def of $m0: it records that the eventual move clobbers M0, and
+ // - because an instruction defining a physical register is not hoisted/sunk -
+ // keeps a divergent access pinned inside its waterfall loop.
+ // AMDGPUAssignIdxToM0 removes it when it writes M0 for real.
+ if (!Subtarget->hasMovrel())
+ return true;
+
+ auto *LdStIdx = cast<AMDGPUMI::VLoadStoreIdxInst>(&*std::prev(II));
+ if (LdStIdx->getIdxOp().isReg())
+ LdStIdx->addOperand(MachineOperand::CreateReg(AMDGPU::M0, /*isDef=*/true,
+ /*isImp=*/true));
+ return true;
+}
+
bool AMDGPUInstructionSelector::select(MachineInstr &I) {
if (!I.isPreISelOpcode()) {
@@ -4685,6 +4709,9 @@ bool AMDGPUInstructionSelector::select(MachineInstr &I) {
case AMDGPU::G_AMDGPU_BVH_INTERSECT_RAY:
case AMDGPU::G_AMDGPU_BVH8_INTERSECT_RAY:
return selectBVHIntersectRayIntrinsic(I);
+ case AMDGPU::G_AMDGPU_REG_LOAD:
+ case AMDGPU::G_AMDGPU_REG_STORE:
+ return selectRegLoadStore(I);
case AMDGPU::G_SBFX:
case AMDGPU::G_UBFX:
return selectG_SBFX_UBFX(I);
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.h b/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.h
index 19028ef58a166..bff0a6cc90e3e 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.h
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.h
@@ -91,6 +91,7 @@ class AMDGPUInstructionSelector final : public InstructionSelector {
bool selectCOPY_VCC_SCC(MachineInstr &I) const;
bool selectReadAnyLane(MachineInstr &I) const;
bool selectPHI(MachineInstr &I) const;
+ bool selectRegLoadStore(MachineInstr &I) const;
bool selectG_TRUNC(MachineInstr &I) const;
bool selectG_SZA_EXT(MachineInstr &I) const;
bool selectG_FPEXT(MachineInstr &I) const;
diff --git a/llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp b/llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp
index c57ef9392a4ca..9e7934454b6bb 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp
@@ -16,6 +16,7 @@
#include "AMDGPU.h"
#include "AMDGPUGlobalISelUtils.h"
#include "AMDGPUInstrInfo.h"
+#include "AMDGPUMachineInstrs.h"
#include "AMDGPUMemoryUtils.h"
#include "AMDGPUTargetMachine.h"
#include "MCTargetDesc/AMDGPUMCTargetDesc.h"
@@ -428,6 +429,8 @@ static unsigned maxSizeForAddrSpace(const GCNSubtarget &ST, unsigned AS,
// register bank/uniformity and if the memory is invariant or not written in a
// kernel.
return IsLoad ? 512 : 128;
+ case AMDGPUAS::VGPR:
+ return 1024;
default:
// FIXME: Flat addresses may contextually need to be split to 32-bit parts
// if they may alias scratch depending on the subtarget. This needs to be
@@ -544,10 +547,23 @@ static bool loadStoreBitcastWorkaround(const LLT Ty) {
static bool isLoadStoreLegal(const GCNSubtarget &ST, const LegalityQuery &Query) {
const LLT Ty = Query.Types[0];
+ // VGPR ("as memory") accesses are never plain-legal; they are custom-lowered
+ // to G_AMDGPU_REG_LOAD/STORE.
+ if (Query.Types[1].getAddressSpace() == AMDGPUAS::VGPR)
+ return false;
return isRegisterType(ST, Ty) && isLoadStoreSizeLegal(ST, Query) &&
!hasBufferRsrcWorkaround(Ty) && !loadStoreBitcastWorkaround(Ty);
}
+// Whether the VGPR ("as memory") load/store lowering handles a MemSize-bit
+// memory access producing/consuming a ValSize-bit value. Only whole-dword
+// accesses (those with a matching V_LOAD_IDX/V_STORE_IDX pseudo) are supported
+// for now; sub-dword (8/16-bit) support lands later.
+static bool isVGPRLoadStoreSizeSupported(unsigned MemSize, unsigned ValSize) {
+ return MemSize == ValSize &&
+ AMDGPUMI::VLoadIdxInst::tryGetOpcodeForBitWidth(MemSize) != -1;
+}
+
/// Return true if a load or store of the type should be lowered with a bitcast
/// to a different type.
static bool shouldBitcastLoadStoreType(const GCNSubtarget &ST, const LLT Ty,
@@ -1655,6 +1671,14 @@ AMDGPULegalizerInfo::AMDGPULegalizerInfo(const GCNSubtarget &ST_,
// inserting addrspacecasts.
Actions.customIf(typeIs(1, Constant32Ptr));
+ // VGPR ("as memory") accesses are custom-lowered to the legal
+ // G_AMDGPU_REG_LOAD/STORE target instructions. Always take the custom path
+ // so an unsupported (e.g. sub-dword) access is diagnosed cleanly rather
+ // than failing to legalize.
+ Actions.customIf([=](const LegalityQuery &Query) -> bool {
+ return Query.Types[1].getAddressSpace() == AMDGPUAS::VGPR;
+ });
+
// Turn any illegal element vectors into something easier to deal
// with. These will ultimately produce 32-bit scalar shifts to extract the
// parts anyway.
@@ -3441,6 +3465,71 @@ static LLT widenToNextPowerOf2(LLT Ty) {
return LLT::scalar(PowerOf2Ceil(Ty.getSizeInBits()));
}
+/// Lower a whole-dword G_LOAD / G_STORE on AMDGPUAS::VGPR into a legal
+/// G_AMDGPU_REG_LOAD / G_AMDGPU_REG_STORE indexed by the pointer's dword offset
+/// (pointer >> 2). Parallels the SelectionDAG LowerLoadStoreVGPR.
+static bool lowerLoadStoreVGPR(LegalizerHelper &Helper, MachineInstr &MI) {
+ MachineIRBuilder &B = Helper.MIRBuilder;
+ MachineRegisterInfo &MRI = *B.getMRI();
+ MachineMemOperand &MMO = **MI.memoperands_begin();
+
+ const bool IsStore = MI.getOpcode() == AMDGPU::G_STORE;
+ Register ValReg = MI.getOperand(0).getReg();
+ Register PtrReg = MI.getOperand(1).getReg();
+
+ const LLT ValTy = MRI.getType(ValReg);
+ const unsigned ValSize = ValTy.getSizeInBits();
+ const LLT S32 = LLT::scalar(32);
+
+ // Only whole-dword, non-extending/non-truncating accesses are implemented.
+ // Reject anything else with a diagnostic instead of failing to legalize
+ // (sub-dword support lands in a later change).
+ if (!isVGPRLoadStoreSizeSupported(MMO.getMemoryType().getSizeInBits(),
+ ValSize)) {
+ const Function &F = B.getMF().getFunction();
+ F.getContext().diagnose(DiagnosticInfoUnsupported(
+ F,
+ "unsupported access of VGPR 'as memory' address space (13); only "
+ "whole-dword loads and stores are implemented",
+ MI.getDebugLoc()));
+ if (!IsStore)
+ B.buildUndef(ValReg);
+ MI.eraseFromParent();
+ return true;
+ }
+
+ const auto PtrAsInt = B.buildPtrToInt(S32, PtrReg);
+ auto Two = B.buildConstant(S32, 2);
+ const auto Index = B.buildLShr(S32, PtrAsInt, Two);
+
+ // Normalize the value to i32 / <N x i32> so a selection pattern always
+ // exists (e.g. for v4i8).
+ LLT RegTy = ValTy;
+ if (ValTy.getScalarSizeInBits() != 32) {
+ unsigned NumDwords = ValSize / 32;
+ RegTy = NumDwords == 1 ? S32 : LLT::fixed_vector(NumDwords, 32);
+ }
+
+ if (IsStore) {
+ Register Value = ValReg;
+ if (RegTy != ValTy)
+ Value = B.buildBitcast(RegTy, Value).getReg(0);
+ B.buildInstr(AMDGPU::G_AMDGPU_REG_STORE, {}, {Value, Index.getReg(0)})
+ .addMemOperand(&MMO);
+ } else if (RegTy == ValTy) {
+ B.buildInstr(AMDGPU::G_AMDGPU_REG_LOAD, {ValReg}, {Index.getReg(0)})
+ .addMemOperand(&MMO);
+ } else {
+ const auto Result =
+ B.buildInstr(AMDGPU::G_AMDGPU_REG_LOAD, {RegTy}, {Index.getReg(0)})
+ .addMemOperand(&MMO);
+ B.buildBitcast(ValReg, Result);
+ }
+
+ MI.eraseFromParent();
+ return true;
+}
+
bool AMDGPULegalizerInfo::legalizeLoad(LegalizerHelper &Helper,
MachineInstr &MI) const {
MachineIRBuilder &B = Helper.MIRBuilder;
@@ -3451,6 +3540,9 @@ bool AMDGPULegalizerInfo::legalizeLoad(LegalizerHelper &Helper,
LLT PtrTy = MRI.getType(PtrReg);
unsigned AddrSpace = PtrTy.getAddressSpace();
+ if (AddrSpace == AMDGPUAS::VGPR && MI.getOpcode() == AMDGPU::G_LOAD)
+ return lowerLoadStoreVGPR(Helper, MI);
+
if (AddrSpace == AMDGPUAS::CONSTANT_ADDRESS_32BIT) {
LLT ConstPtr = LLT::pointer(AMDGPUAS::CONSTANT_ADDRESS, 64);
auto Cast = B.buildAddrSpaceCast(ConstPtr, PtrReg);
@@ -3539,6 +3631,10 @@ bool AMDGPULegalizerInfo::legalizeStore(LegalizerHelper &Helper,
Register DataReg = MI.getOperand(0).getReg();
LLT DataTy = MRI.getType(DataReg);
+ if (MRI.getType(MI.getOperand(1).getReg()).getAddressSpace() ==
+ AMDGPUAS::VGPR)
+ return lowerLoadStoreVGPR(Helper, MI);
+
if (hasBufferRsrcWorkaround(DataTy)) {
Observer.changingInstr(MI);
castBufferRsrcArgToV4I32(MI, B, 0);
diff --git a/llvm/lib/Target/AMDGPU/AMDGPULowerVGPREncoding.cpp b/llvm/lib/Target/AMDGPU/AMDGPULowerVGPREncoding.cpp
index 54fc33a4a47cd..49bb4c513a709 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPULowerVGPREncoding.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPULowerVGPREncoding.cpp
@@ -42,9 +42,11 @@
#include "AMDGPULowerVGPREncoding.h"
#include "AMDGPU.h"
+#include "AMDGPUMachineInstrs.h"
#include "GCNSubtarget.h"
#include "SIDefines.h"
#include "SIInstrInfo.h"
+#include "SIMachineFunctionInfo.h"
#include "llvm/ADT/bit.h"
#include "llvm/Support/Debug.h"
#include "llvm/Support/MathExtras.h"
@@ -131,6 +133,7 @@ class AMDGPULowerVGPREncoding {
bool run(MachineFunction &MF);
private:
+ const GCNSubtarget *ST;
const SIInstrInfo *TII;
const SIRegisterInfo *TRI;
@@ -176,6 +179,13 @@ class AMDGPULowerVGPREncoding {
/// Handle single \p MI. \return true if changed.
bool runOnMachineInstr(MachineInstr &MI);
+ /// Lower a VGPR "as memory" (address space 13) indexed load/store pseudo
+ /// (V_LOAD_IDX_B<N> / V_STORE_IDX_B<N>) into a sequence of M0-relative moves
+ /// (v_movrels_b32 for loads, v_movreld_b32 for stores) over the wave's vector
+ /// registers. M0 must already hold the dword index (see AMDGPUAssignIdxToM0).
+ /// This replaces the pseudo, which is erased.
+ void lowerLoadStoreIdx(MachineInstr &MI);
+
/// Compute the mode for a single \p MI given \p Ops operands
/// bit mapping. Optionally takes second array \p Ops2 for VOPD.
/// If provided and an operand from \p Ops is not a VGPR, then \p Ops2
@@ -378,6 +388,63 @@ bool AMDGPULowerVGPREncoding::runOnMachineInstr(MachineInstr &MI) {
return false;
}
+void AMDGPULowerVGPREncoding::lowerLoadStoreIdx(MachineInstr &MI) {
+ auto &LdSt = cast<AMDGPUMI::VLoadStoreIdxInst>(MI);
+ MachineBasicBlock &BB = *MI.getParent();
+ const DebugLoc &DL = MI.getDebugLoc();
+ const bool IsStore = LdSt.mayStore();
+
+ // $data is operand 0 of both the load (def) and store (use) pseudos; M0
+ // already holds the dword index (see AMDGPUAssignIdxToM0).
+ Register Data = LdSt.getDataOp().getReg();
+ unsigned Offset = LdSt.getOffsetOp().getImm();
+ unsigned NumDwords = LdSt.getBitWidth() / 32;
+
+ // A statically out-of-range dword offset would fold into a base VGPR outside
+ // the addressable register file - an out-of-bounds access of the VGPR
+ // "as memory" (address space 13) region. When the whole file is addressable
+ // the index is allowed to wrap; otherwise it must stay in range.
+#ifndef NDEBUG
+ unsigned NumAddressableVGPRs = ST->getAddressableNumVGPRs(
+ MI.getMF()->getInfo<SIMachineFunctionInfo>()->getDynamicVGPRBlockSize());
+ bool AllowOffsetWrap =
+ NumAddressableVGPRs == AMDGPU::IsaInfo::getTotalNumVGPRs(*ST);
+ assert((AllowOffsetWrap || Offset + NumDwords <= NumAddressableVGPRs) &&
+ "out of bounds VGPR 'as memory' (address space 13) access");
+#endif
+
+ unsigned Opcode =
+ IsStore ? AMDGPU::V_MOVRELD_B32_e32 : AMDGPU::V_MOVRELS_B32_e32;
+
+ // The dword index is (M0 + $offset). Fold $offset into the base register so
+ // each dword i reads/writes VGPR($offset + i) relative to M0.
+ for (unsigned i = 0; i < NumDwords; ++i) {
+ Register Base = AMDGPU::VGPR0 + Offset + i;
+ Register Sub = Data;
+ if (NumDwords != 1)
+ Sub = TRI->getSubReg(Data, TRI->getSubRegFromChannel(i));
+
+ MachineInstr *Mov;
+ if (IsStore)
+ Mov = BuildMI(BB, MI, DL, TII->get(Opcode))
+ .addReg(Base, RegState::Undef)
+ .addReg(Sub)
+ .getInstr();
+ else
+ Mov = BuildMI(BB, MI, DL, TII->get(Opcode), Sub)
+ .addReg(Base, RegState::Undef)
+ .getInstr();
+
+ // On subtargets with more than 256 addressable VGPRs the referenced
+ // register may need high address bits; reuse the S_SET_VGPR_MSB machinery
+ // to encode them. This is a no-op on movrel-only (<=256 VGPR) subtargets.
+ if (ST->has1024AddressableVGPRs())
+ runOnMachineInstr(*Mov);
+ }
+
+ MI.eraseFromParent();
+}
+
MachineBasicBlock::instr_iterator
AMDGPULowerVGPREncoding::handleClause(MachineBasicBlock::instr_iterator I) {
if (!ClauseRemaining)
@@ -551,12 +618,14 @@ bool AMDGPULowerVGPREncoding::handleSetregMode(MachineInstr &MI) {
}
bool AMDGPULowerVGPREncoding::run(MachineFunction &MF) {
- const GCNSubtarget &ST = MF.getSubtarget<GCNSubtarget>();
- if (!ST.has1024AddressableVGPRs())
- return false;
+ ST = &MF.getSubtarget<GCNSubtarget>();
+ TII = ST->getInstrInfo();
+ TRI = ST->getRegisterInfo();
- TII = ST.getInstrInfo();
- TRI = ST.getRegisterInfo();
+ // The S_SET_VGPR_MSB encoding is only required on subtargets with more than
+ // 256 addressable VGPRs (gfx1250). On movrel-only subtargets the pass still
+ // runs, but only to lower the VGPR "as memory" indexed load/store pseudos.
+ const bool LowerVGPRMSBs = ST->has1024AddressableVGPRs();
LLVM_DEBUG(dbgs() << "*** AMDGPULowerVGPREncoding on " << MF.getName()
<< " ***\n");
@@ -573,6 +642,19 @@ bool AMDGPULowerVGPREncoding::run(MachineFunction &MF) {
<< ":\n");
for (auto &MI : llvm::make_early_inc_range(MBB.instrs())) {
+ // Lower VGPR "as memory" indexed load/store pseudos on any subtarget that
+ // reaches this pass (movrel-only or gfx1250). This replaces the pseudo.
+ if (isa<AMDGPUMI::VLoadStoreIdxInst>(&MI)) {
+ lowerLoadStoreIdx(MI);
+ Changed = true;
+ continue;
+ }
+
+ // The remaining work only inserts VGPR MSB encoding, which is unnecessary
+ // on movrel-only subtargets.
+ if (!LowerVGPRMSBs)
+ continue;
+
if (MI.isMetaInstruction())
continue;
@@ -605,7 +687,7 @@ bool AMDGPULowerVGPREncoding::run(MachineFunction &MF) {
}
if (MI.getOpcode() == AMDGPU::S_SETREG_IMM32_B32 &&
- ST.hasSetregVGPRMSBFixup()) {
+ ST->hasSetregVGPRMSBFixup()) {
Changed |= handleSetregMode(MI);
continue;
}
@@ -629,8 +711,10 @@ bool AMDGPULowerVGPREncoding::run(MachineFunction &MF) {
}
// Reset the mode if we are falling through.
- LLVM_DEBUG(dbgs() << " end of BB, resetting mode\n");
- resetMode(MBB.instr_end());
+ if (LowerVGPRMSBs) {
+ LLVM_DEBUG(dbgs() << " end of BB, resetting mode\n");
+ resetMode(MBB.instr_end());
+ }
}
return Changed;
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUMachineInstrs.cpp b/llvm/lib/Target/AMDGPU/AMDGPUMachineInstrs.cpp
new file mode 100644
index 0000000000000..697c63e2079d1
--- /dev/null
+++ b/llvm/lib/Target/AMDGPU/AMDGPUMachineInstrs.cpp
@@ -0,0 +1,55 @@
+//===-- AMDGPUMachineInstrs.cpp -*- C++ -*---------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+/// Convenience wrappers and helpers for AMDGPU-specific machine instructions.
+//
+//===----------------------------------------------------------------------===//
+
+#include "AMDGPUMachineInstrs.h"
+#include "Utils/AMDGPUBaseInfo.h"
+#include "llvm/CodeGen/MachineInstr.h"
+#include "llvm/Support/ErrorHandling.h"
+
+using namespace llvm;
+using namespace AMDGPUMI;
+
+unsigned VLoadStoreIdxInst::getBitWidth() const {
+ const AMDGPU::VLdStIdxOpcodeInfo *Info =
+ AMDGPU::getVLdStIdxOpcodeInfoByOpcode(getOpcode());
+ if (!Info)
+ llvm_unreachable("unsupported V_LOAD/STORE_IDX opcode");
+ return Info->BitWidth;
+}
+
+int VLoadIdxInst::tryGetOpcodeForBitWidth(unsigned Bits) {
+ const AMDGPU::VLdStIdxOpcodeInfo *Info =
+ AMDGPU::getVLdStIdxOpcodeInfoByKey(Bits, /*IsStore=*/false);
+ if (!Info)
+ return -1;
+ return Info->Opcode;
+}
+
+unsigned VLoadIdxInst::getOpcodeForBitWidth(unsigned Bits) {
+ int Opcode = tryGetOpcodeForBitWidth(Bits);
+ assert(Opcode != -1);
+ return Opcode;
+}
+
+int VStoreIdxInst::tryGetOpcodeForBitWidth(unsigned Bits) {
+ const AMDGPU::VLdStIdxOpcodeInfo *Info =
+ AMDGPU::getVLdStIdxOpcodeInfoByKey(Bits, /*IsStore=*/true);
+ if (!Info)
+ return -1;
+ return Info->Opcode;
+}
+
+unsigned VStoreIdxInst::getOpcodeForBitWidth(unsigned Bits) {
+ int Opcode = tryGetOpcodeForBitWidth(Bits);
+ assert(Opcode != -1);
+ return Opcode;
+}
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUMachineInstrs.h b/llvm/lib/Target/AMDGPU/AMDGPUMachineInstrs.h
new file mode 100644
index 0000000000000..24e24aab53152
--- /dev/null
+++ b/llvm/lib/Target/AMDGPU/AMDGPUMachineInstrs.h
@@ -0,0 +1,71 @@
+//===-- AMDGPUMachineInstrs.h -*- C++ -*-----------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+/// Convenience wrappers and helpers for AMDGPU-specific machine instructions.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIB_TARGET_AMDGPU_AMDGPUMACHINEINSTRS_H
+#define LLVM_LIB_TARGET_AMDGPU_AMDGPUMACHINEINSTRS_H
+
+#include "SIInstrInfo.h"
+#include "Utils/AMDGPUBaseInfo.h"
+#include "llvm/CodeGen/MachineInstr.h"
+
+namespace llvm {
+namespace AMDGPUMI {
+
+// Wrapper for the whole-dword VGPR "as memory" (address space 13) indexed
+// load/store pseudos (V_LOAD_IDX_B<N> / V_STORE_IDX_B<N>). Operand layout:
+// load: (outs data), (ins idx, offset)
+// store: (outs), (ins data, idx, offset)
+// so data/idx/offset are always operands 0/1/2.
+class VLoadStoreIdxInst : public MachineInstr {
+public:
+ MachineOperand &getDataOp() { return getOperand(0); }
+ MachineOperand &getIdxOp() { return getOperand(1); }
+ MachineOperand &getOffsetOp() { return getOperand(2); }
+ const MachineOperand &getDataOp() const { return getOperand(0); }
+ const MachineOperand &getIdxOp() const { return getOperand(1); }
+ const MachineOperand &getOffsetOp() const { return getOperand(2); }
+
+ unsigned getBitWidth() const;
+
+ static bool classof(const MachineInstr *MI) {
+ return AMDGPU::getVLdStIdxOpcodeInfoByOpcode(MI->getOpcode()) != nullptr;
+ }
+};
+
+class VLoadIdxInst : public VLoadStoreIdxInst {
+public:
+ static int tryGetOpcodeForBitWidth(unsigned Bits);
+ static unsigned getOpcodeForBitWidth(unsigned Bits);
+
+ static bool classof(const MachineInstr *MI) {
+ const AMDGPU::VLdStIdxOpcodeInfo *Info =
+ AMDGPU::getVLdStIdxOpcodeInfoByOpcode(MI->getOpcode());
+ return Info && !Info->IsStore;
+ }
+};
+
+class VStoreIdxInst : public VLoadStoreIdxInst {
+public:
+ static int tryGetOpcodeForBitWidth(unsigned Bits);
+ static unsigned getOpcodeForBitWidth(unsigned Bits);
+
+ static bool classof(const MachineInstr *MI) {
+ const AMDGPU::VLdStIdxOpcodeInfo *Info =
+ AMDGPU::getVLdStIdxOpcodeInfoByOpcode(MI->getOpcode());
+ return Info && Info->IsStore;
+ }
+};
+
+} // end namespace AMDGPUMI
+} // end namespace llvm
+
+#endif // LLVM_LIB_TARGET_AMDGPU_AMDGPUMACHINEINSTRS_H
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUPassRegistry.def b/llvm/lib/Target/AMDGPU/AMDGPUPassRegistry.def
index d052f3c73920c..d0255025298d7 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUPassRegistry.def
+++ b/llvm/lib/Target/AMDGPU/AMDGPUPassRegistry.def
@@ -113,6 +113,7 @@ MACHINE_FUNCTION_ANALYSIS("amdgpu-next-use-analysis", AMDGPUNextUseAnalysisPass(
#ifndef MACHINE_FUNCTION_PASS
#define MACHINE_FUNCTION_PASS(NAME, CREATE_PASS)
#endif
+MACHINE_FUNCTION_PASS("amdgpu-assign-idx-to-m0", AMDGPUAssignIdxToM0Pass())
MACHINE_FUNCTION_PASS("amdgpu-insert-delay-alu", AMDGPUInsertDelayAluPass())
MACHINE_FUNCTION_PASS("amdgpu-isel", AMDGPUISelDAGToDAGPass(*this))
MACHINE_FUNCTION_PASS("amdgpu-lower-vgpr-encoding", AMDGPULowerVGPREncodingPass())
diff --git a/llvm/lib/Target/AMDGPU/AMDGPURegBankLegalizeRules.cpp b/llvm/lib/Target/AMDGPU/AMDGPURegBankLegalizeRules.cpp
index 7f7225ad6d311..6c426c6da11e8 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPURegBankLegalizeRules.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPURegBankLegalizeRules.cpp
@@ -1380,6 +1380,13 @@ RegBankLegalizeRules::RegBankLegalizeRules(const GCNSubtarget &_ST,
{{VgprV2S16},
{VgprV2S16, SgprV4S32_WF, Vgpr32, Vgpr32, Sgpr32_WF}}});
+ // VGPR ("as memory") indexed load/store: the data is a VGPR value of any
+ // register class; the dword index is made uniform (waterfall) as an SGPR.
+ addRulesForGOpcs({G_AMDGPU_REG_LOAD}).Any({{BRC}, {{VgprBRC}, {Sgpr32_WF}}});
+
+ addRulesForGOpcs({G_AMDGPU_REG_STORE})
+ .Any({{BRC}, {{}, {VgprBRC, Sgpr32_WF}}});
+
addRulesForGOpcs({G_PTR_ADD})
.Any({{UniPtr32}, {{SgprPtr32}, {SgprPtr32, Sgpr32}}})
.Any({{DivPtr32}, {{VgprPtr32}, {VgprPtr32, Vgpr32}}})
diff --git a/llvm/lib/Target/AMDGPU/AMDGPURegisterBankInfo.cpp b/llvm/lib/Target/AMDGPU/AMDGPURegisterBankInfo.cpp
index 200234c23c886..5bdf0768179ec 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPURegisterBankInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPURegisterBankInfo.cpp
@@ -3087,6 +3087,14 @@ void AMDGPURegisterBankInfo::applyMappingImpl(
return;
}
+ case AMDGPU::G_AMDGPU_REG_LOAD:
+ case AMDGPU::G_AMDGPU_REG_STORE: {
+ // The dword index (operand 1) must be uniform; a divergent index needs a
+ // waterfall loop.
+ applyDefaultMapping(OpdMapper);
+ executeInWaterfallLoop(B, MI, {1});
+ return;
+ }
case AMDGPU::G_AMDGPU_BUFFER_LOAD:
case AMDGPU::G_AMDGPU_BUFFER_LOAD_USHORT:
case AMDGPU::G_AMDGPU_BUFFER_LOAD_SSHORT:
@@ -4486,6 +4494,13 @@ AMDGPURegisterBankInfo::getInstrMapping(const MachineInstr &MI) const {
}
break;
}
+ case AMDGPU::G_AMDGPU_REG_LOAD:
+ case AMDGPU::G_AMDGPU_REG_STORE: {
+ // data/result is a VGPR value; the dword index is uniform (SGPR).
+ OpdsMapping[0] = getVGPROpMapping(MI.getOperand(0).getReg(), MRI, *TRI);
+ OpdsMapping[1] = getSGPROpMapping(MI.getOperand(1).getReg(), MRI, *TRI);
+ break;
+ }
case AMDGPU::G_AMDGPU_BUFFER_LOAD:
case AMDGPU::G_AMDGPU_BUFFER_LOAD_UBYTE:
case AMDGPU::G_AMDGPU_BUFFER_LOAD_SBYTE:
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUSearchableTables.td b/llvm/lib/Target/AMDGPU/AMDGPUSearchableTables.td
index 2f7f72bd58f0e..abb38d86468e6 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUSearchableTables.td
+++ b/llvm/lib/Target/AMDGPU/AMDGPUSearchableTables.td
@@ -431,3 +431,20 @@ def AMDGPUImageDMaskIntrinsicTable : GenericTable {
let PrimaryKeyName = "getAMDGPUImageDMaskIntrinsic";
let PrimaryKeyEarlyOut = 1;
}
+
+//===----------------------------------------------------------------------===//
+// V_LOAD/STORE_IDX opcode mapping table.
+//===----------------------------------------------------------------------===//
+
+def VLdStIdxOpcodeInfoTable : GenericTable {
+ let FilterClass = "VLdStIdxOpcodeInfo";
+ let CppTypeName = "VLdStIdxOpcodeInfo";
+ let Fields = ["Opcode", "BitWidth", "IsStore"];
+ let PrimaryKey = ["Opcode"];
+ let PrimaryKeyName = "getVLdStIdxOpcodeInfoByOpcodeImpl";
+}
+
+def getVLdStIdxOpcodeInfoByKeyImpl : SearchIndex {
+ let Table = VLdStIdxOpcodeInfoTable;
+ let Key = ["BitWidth", "IsStore"];
+}
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp b/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp
index b07905a71b16f..e944df1bbb9a2 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp
@@ -666,6 +666,7 @@ extern "C" LLVM_ABI LLVM_EXTERNAL_VISIBILITY void LLVMInitializeAMDGPUTarget() {
initializeAMDGPURegBankLegalizePass(*PR);
initializeSILowerWWMCopiesLegacyPass(*PR);
initializeAMDGPUMarkLastScratchLoadLegacyPass(*PR);
+ initializeAMDGPUAssignIdxToM0LegacyPass(*PR);
initializeSILowerSGPRSpillsLegacyPass(*PR);
initializeSIFixSGPRCopiesLegacyPass(*PR);
initializeSIFixVGPRCopiesLegacyPass(*PR);
@@ -1734,6 +1735,10 @@ void GCNPassConfig::addFastRegAlloc() {
}
void GCNPassConfig::addPreRegAlloc() {
+ // Copy the VGPR "as memory" load/store index into M0 before register
+ // allocation; the movrel emitted later by AMDGPULowerVGPREncoding reads it.
+ addPass(&AMDGPUAssignIdxToM0ID);
+
if (getOptLevel() != CodeGenOptLevel::None)
addPass(&AMDGPUPrepareAGPRAllocLegacyID);
}
@@ -2550,6 +2555,10 @@ Error AMDGPUCodeGenPassBuilder::addOptimizedRegAlloc(
}
void AMDGPUCodeGenPassBuilder::addPreRegAlloc(PassManagerWrapper &PMW) const {
+ // Set up M0 for the movrel that expands a VGPR "as memory" indexed access.
+ // Run before allocation so the index computation coalesces into M0.
+ addMachineFunctionPass(AMDGPUAssignIdxToM0Pass(), PMW);
+
if (getOptLevel() != CodeGenOptLevel::None)
addMachineFunctionPass(AMDGPUPrepareAGPRAllocPass(), PMW);
}
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.cpp b/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.cpp
index 5125735faddd6..c90498d02e4d7 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.cpp
@@ -1122,13 +1122,16 @@ bool GCNTTIImpl::isSourceOfDivergence(const Value *V) const {
// Loads from the private and flat address spaces are divergent, because
// threads can execute the load instruction with the same inputs and get
- // different results.
+ // different results. The same is true of the VGPR ("as memory") address
+ // space: it is a per-lane view of the vector registers, so an access at a
+ // uniform offset still yields a per-lane (divergent) value.
//
// All other loads are not divergent, because if threads issue loads with the
// same arguments, they will always get the same result.
if (const LoadInst *Load = dyn_cast<LoadInst>(V))
return Load->getPointerAddressSpace() == AMDGPUAS::PRIVATE_ADDRESS ||
- Load->getPointerAddressSpace() == AMDGPUAS::FLAT_ADDRESS;
+ Load->getPointerAddressSpace() == AMDGPUAS::FLAT_ADDRESS ||
+ Load->getPointerAddressSpace() == AMDGPUAS::VGPR;
// Atomics are divergent because they are executed sequentially: when an
// atomic operation refers to the same address in each thread, then each
diff --git a/llvm/lib/Target/AMDGPU/CMakeLists.txt b/llvm/lib/Target/AMDGPU/CMakeLists.txt
index 46edc44e2cc05..11f7427ab52f8 100644
--- a/llvm/lib/Target/AMDGPU/CMakeLists.txt
+++ b/llvm/lib/Target/AMDGPU/CMakeLists.txt
@@ -47,6 +47,7 @@ add_llvm_target(AMDGPUCodeGen
AMDGPUArgumentUsageInfo.cpp
AMDGPUAsanInstrumentation.cpp
AMDGPUAsmPrinter.cpp
+ AMDGPUAssignIdxToM0.cpp
AMDGPUAtomicOptimizer.cpp
AMDGPUAttributor.cpp
AMDGPUBarrierLatency.cpp
@@ -83,6 +84,7 @@ add_llvm_target(AMDGPUCodeGen
AMDGPULowerExecSync.cpp
AMDGPUSwLowerLDS.cpp
AMDGPUMachineFunctionInfo.cpp
+ AMDGPUMachineInstrs.cpp
AMDGPUMachineModuleInfo.cpp
AMDGPUMacroFusion.cpp
AMDGPUMCInstLower.cpp
diff --git a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
index bd087fae75403..4b03deb0cbe64 100644
--- a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
+++ b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
@@ -15,6 +15,7 @@
#include "AMDGPU.h"
#include "AMDGPUInstrInfo.h"
#include "AMDGPULaneMaskUtils.h"
+#include "AMDGPUMachineInstrs.h"
#include "AMDGPUMemoryUtils.h"
#include "AMDGPUSelectionDAGInfo.h"
#include "AMDGPUTargetMachine.h"
@@ -13110,6 +13111,87 @@ static bool addressMayBeAccessedAsPrivate(const MachineMemOperand *MMO,
return true;
}
+// Lower a load or store of the VGPR ("as memory") address space (13) to a
+// REG_LOAD / REG_STORE target node. The 32-bit pointer is a byte offset into
+// the wave's view of its vector registers; the target node carries the dword
+// index (pointer >> 2). Recognizing a constant dword offset is left to the
+// selection patterns, which fold an (add index, imm) shape into the pseudo.
+//
+// TODO: sub-dword (8/16-bit) accesses are not yet supported; they are
+// diagnosed as unsupported below.
+SDValue SITargetLowering::LowerLoadStoreVGPR(SDValue Op,
+ SelectionDAG &DAG) const {
+ SDLoc DL(Op);
+ MemSDNode *MemOp = cast<MemSDNode>(Op);
+ EVT MemVT = MemOp->getMemoryVT();
+ unsigned BitWidth = MemVT.getSizeInBits();
+
+ // Only whole-dword, non-extending/non-truncating accesses are implemented.
+ // Reject anything else with a diagnostic (replacing the value with poison)
+ // instead of failing instruction selection. Both callers - operation
+ // legalization and the pre-ISel combine - replace the node with this result,
+ // so the diagnostic is emitted exactly once.
+ auto reportUnsupported = [&]() -> SDValue {
+ const Function &F = DAG.getMachineFunction().getFunction();
+ DAG.getContext()->diagnose(DiagnosticInfoUnsupported(
+ F,
+ "unsupported access of VGPR 'as memory' address space (13); only "
+ "whole-dword loads and stores are implemented",
+ DL.getDebugLoc()));
+ if (isa<StoreSDNode>(MemOp))
+ return MemOp->getChain();
+ return DAG.getMergeValues(
+ {DAG.getPOISON(Op.getValueType()), MemOp->getChain()}, DL);
+ };
+
+ if (BitWidth < 32)
+ return reportUnsupported();
+ if (auto *Load = dyn_cast<LoadSDNode>(MemOp)) {
+ if (Load->getExtensionType() != ISD::NON_EXTLOAD)
+ return reportUnsupported();
+ if (AMDGPUMI::VLoadIdxInst::tryGetOpcodeForBitWidth(BitWidth) == -1)
+ return reportUnsupported();
+ } else {
+ auto *Store = cast<StoreSDNode>(MemOp);
+ if (Store->isTruncatingStore())
+ return reportUnsupported();
+ if (AMDGPUMI::VStoreIdxInst::tryGetOpcodeForBitWidth(BitWidth) == -1)
+ return reportUnsupported();
+ }
+
+ SDValue Chain = MemOp->getChain();
+ SDValue Index = DAG.getNode(ISD::SRL, DL, MVT::i32, MemOp->getBasePtr(),
+ DAG.getConstant(2, DL, MVT::i32));
+
+ // View the access as i32 / <N x i32> when the memory type is not register
+ // legal (e.g. v4i8), bitcasting the value across.
+ EVT RegVT = MemVT;
+ if (!isTypeLegal(RegVT)) {
+ unsigned NumDwords = BitWidth / 32;
+ RegVT = NumDwords == 1
+ ? EVT(MVT::i32)
+ : EVT::getVectorVT(*DAG.getContext(), MVT::i32, NumDwords);
+ }
+
+ if (auto *StoreOp = dyn_cast<StoreSDNode>(MemOp)) {
+ SDValue Value = StoreOp->getValue();
+ if (RegVT != MemVT)
+ Value = DAG.getNode(ISD::BITCAST, DL, RegVT, Value);
+ return DAG.getMemIntrinsicNode(
+ AMDGPUISD::REG_STORE, DL, DAG.getVTList(MVT::Other),
+ {Chain, Value, Index}, MemVT, StoreOp->getMemOperand());
+ }
+
+ auto *LoadOp = cast<LoadSDNode>(MemOp);
+ SDValue NewLoad = DAG.getMemIntrinsicNode(
+ AMDGPUISD::REG_LOAD, DL, DAG.getVTList(RegVT, MVT::Other), {Chain, Index},
+ MemVT, LoadOp->getMemOperand());
+ if (RegVT == MemVT)
+ return NewLoad;
+ SDValue Value = DAG.getNode(ISD::BITCAST, DL, MemVT, NewLoad);
+ return DAG.getMergeValues({Value, NewLoad.getValue(1)}, DL);
+}
+
SDValue SITargetLowering::LowerLOAD(SDValue Op, SelectionDAG &DAG) const {
SDLoc DL(Op);
LoadSDNode *Load = cast<LoadSDNode>(Op);
@@ -13117,6 +13199,9 @@ SDValue SITargetLowering::LowerLOAD(SDValue Op, SelectionDAG &DAG) const {
EVT MemVT = Load->getMemoryVT();
MachineMemOperand *MMO = Load->getMemOperand();
+ if (Load->getAddressSpace() == AMDGPUAS::VGPR)
+ return LowerLoadStoreVGPR(Op, DAG);
+
if (ExtType == ISD::NON_EXTLOAD && MemVT.getSizeInBits() < 32) {
if (MemVT == MVT::i16 && isTypeLegal(MVT::i16))
return SDValue();
@@ -13789,6 +13874,9 @@ SDValue SITargetLowering::LowerSTORE(SDValue Op, SelectionDAG &DAG) const {
StoreSDNode *Store = cast<StoreSDNode>(Op);
EVT VT = Store->getMemoryVT();
+ if (Store->getAddressSpace() == AMDGPUAS::VGPR)
+ return LowerLoadStoreVGPR(Op, DAG);
+
if (VT == MVT::i1) {
return DAG.getTruncStore(
Store->getChain(), DL,
@@ -18598,6 +18686,19 @@ SDValue SITargetLowering::PerformDAGCombine(SDNode *N,
if (auto Res = promoteUniformOpToI32(SDValue(N, 0), DCI))
return Res;
break;
+ case ISD::LOAD:
+ // Lower a VGPR ("as memory") address space (13) load to a REG_LOAD target
+ // node. Done here (not via operation legalization) so it also fires at -O0,
+ // where a scalar load is otherwise Legal and never reaches LowerLOAD.
+ if (cast<LoadSDNode>(N)->getAddressSpace() == AMDGPUAS::VGPR)
+ if (SDValue V = LowerLoadStoreVGPR(SDValue(N, 0), DCI.DAG))
+ return V;
+ break;
+ case ISD::STORE:
+ if (cast<StoreSDNode>(N)->getAddressSpace() == AMDGPUAS::VGPR)
+ if (SDValue V = LowerLoadStoreVGPR(SDValue(N, 0), DCI.DAG))
+ return V;
+ break;
default:
break;
}
@@ -19197,6 +19298,22 @@ void SITargetLowering::AdjustInstrPostInstrSelection(MachineInstr &MI,
return;
}
+ // A VGPR "as memory" indexed load/store with a register index expands (on
+ // movrel subtargets) to an M0-relative move. Add an implicit-def of $m0: it
+ // records that the eventual move clobbers M0, and - because an instruction
+ // defining a physical register is not hoisted/sunk - keeps a divergent access
+ // pinned inside its waterfall loop. AMDGPUAssignIdxToM0 removes this when it
+ // writes M0 for real.
+ if (auto *LdStIdx = dyn_cast<AMDGPUMI::VLoadStoreIdxInst>(&MI)) {
+ if (getSubtarget()->hasMovrel()) {
+ MachineOperand &IdxOp = LdStIdx->getIdxOp();
+ if (IdxOp.isReg())
+ MI.addOperand(MachineOperand::CreateReg(AMDGPU::M0, /*isDef=*/true,
+ /*isImp=*/true));
+ }
+ return;
+ }
+
if (TII->isImage(MI))
TII->enforceOperandRCAlignment(MI, AMDGPU::OpName::vaddr);
}
diff --git a/llvm/lib/Target/AMDGPU/SIISelLowering.h b/llvm/lib/Target/AMDGPU/SIISelLowering.h
index a181fa072d534..fec7510522db8 100644
--- a/llvm/lib/Target/AMDGPU/SIISelLowering.h
+++ b/llvm/lib/Target/AMDGPU/SIISelLowering.h
@@ -115,6 +115,9 @@ class SITargetLowering final : public AMDGPUTargetLowering {
SDValue widenLoad(LoadSDNode *Ld, DAGCombinerInfo &DCI) const;
SDValue LowerLOAD(SDValue Op, SelectionDAG &DAG) const;
+ // Lower a load/store of the VGPR ("as memory") address space (13) to a
+ // REG_LOAD/REG_STORE target node indexed by the pointer's dword offset.
+ SDValue LowerLoadStoreVGPR(SDValue Op, SelectionDAG &DAG) const;
SDValue LowerSELECT(SDValue Op, SelectionDAG &DAG) const;
SDValue lowerFastUnsafeFDIV(SDValue Op, SelectionDAG &DAG) const;
SDValue lowerFastUnsafeFDIV64(SDValue Op, SelectionDAG &DAG) const;
diff --git a/llvm/lib/Target/AMDGPU/SIInstrInfo.cpp b/llvm/lib/Target/AMDGPU/SIInstrInfo.cpp
index e945db946c158..f8d7feaeaf69f 100644
--- a/llvm/lib/Target/AMDGPU/SIInstrInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/SIInstrInfo.cpp
@@ -15,6 +15,7 @@
#include "AMDGPU.h"
#include "AMDGPUInstrInfo.h"
#include "AMDGPULaneMaskUtils.h"
+#include "AMDGPUMachineInstrs.h"
#include "GCNHazardRecognizer.h"
#include "GCNSubtarget.h"
#include "SIMachineFunctionInfo.h"
@@ -508,6 +509,18 @@ bool SIInstrInfo::getMemOperandsWithOffsetWidth(
return true;
}
+ if (auto *LdStIdx = dyn_cast<AMDGPUMI::VLoadStoreIdxInst>(&LdSt)) {
+ BaseOp = &LdStIdx->getIdxOp();
+ OffsetOp = &LdStIdx->getOffsetOp();
+
+ BaseOps.push_back(BaseOp);
+ Offset = OffsetOp->getImm() * 4; // Offset has units of dwords.
+
+ // Get appropriate operand, and compute width accordingly.
+ Width = LocationSize::precise(LdStIdx->getBitWidth() / 8);
+ return true;
+ }
+
return false;
}
@@ -1913,6 +1926,7 @@ bool SIInstrInfo::expandPostRAPseudo(MachineInstr &MI) const {
MachineBasicBlock &MBB = *MI.getParent();
DebugLoc DL = MBB.findDebugLoc(MI);
const AMDGPU::LaneMaskConstants &LMC = AMDGPU::LaneMaskConstants::get(ST);
+
switch (MI.getOpcode()) {
default: return TargetInstrInfo::expandPostRAPseudo(MI);
case AMDGPU::S_MOV_B64_term:
@@ -3964,10 +3978,20 @@ bool SIInstrInfo::areMemAccessesTriviallyDisjoint(const MachineInstr &MIa,
if (MIa.hasOrderedMemoryRef() || MIb.hasOrderedMemoryRef())
return false;
- if (isLDSDMA(MIa) || isLDSDMA(MIb))
+ if (MIa.isBundle() || MIb.isBundle())
return false;
- if (MIa.isBundle() || MIb.isBundle())
+ // VGPR "as memory" indexed accesses only alias each other, and then only
+ // when their [idx+offset, idx+offset+width) dword ranges overlap.
+ const bool IsLdStIdxA = isa<AMDGPUMI::VLoadStoreIdxInst>(MIa);
+ const bool IsLdStIdxB = isa<AMDGPUMI::VLoadStoreIdxInst>(MIb);
+ if (IsLdStIdxA || IsLdStIdxB) {
+ if (IsLdStIdxA && IsLdStIdxB)
+ return checkInstOffsetsDoNotOverlap(MIa, MIb);
+ return true;
+ }
+
+ if (isLDSDMA(MIa) || isLDSDMA(MIb))
return false;
// TODO: Should we check the address space from the MachineMemOperand? That
@@ -5589,6 +5613,7 @@ bool SIInstrInfo::verifyInstruction(const MachineInstr &MI,
Desc.getOpcode() == AMDGPU::V_MOVRELD_B32_e64) {
const bool IsDst = Desc.getOpcode() == AMDGPU::V_MOVRELD_B32_e32 ||
Desc.getOpcode() == AMDGPU::V_MOVRELD_B32_e64;
+ const bool IsPreRA = !MI.getMF()->getProperties().hasNoVRegs();
const unsigned StaticNumOps =
Desc.getNumOperands() + Desc.implicit_uses().size();
@@ -5598,7 +5623,7 @@ bool SIInstrInfo::verifyInstruction(const MachineInstr &MI,
// post RA scheduler where the main implicit operand is killed and
// implicit-defs are added for sub-registers that remain live after this
// instruction.
- if (MI.getNumOperands() < StaticNumOps + NumImplicitOps) {
+ if (IsPreRA && MI.getNumOperands() < StaticNumOps + NumImplicitOps) {
ErrInfo = "missing implicit register operands";
return false;
}
@@ -5611,20 +5636,22 @@ bool SIInstrInfo::verifyInstruction(const MachineInstr &MI,
}
unsigned UseOpIdx;
- if (!MI.isRegTiedToUseOperand(StaticNumOps, &UseOpIdx) ||
- UseOpIdx != StaticNumOps + 1) {
+ if (IsPreRA && (!MI.isRegTiedToUseOperand(StaticNumOps, &UseOpIdx) ||
+ UseOpIdx != StaticNumOps + 1)) {
ErrInfo = "movrel implicit operands should be tied";
return false;
}
}
- const MachineOperand &Src0 = MI.getOperand(Src0Idx);
- const MachineOperand &ImpUse
- = MI.getOperand(StaticNumOps + NumImplicitOps - 1);
- if (!ImpUse.isReg() || !ImpUse.isUse() ||
- !isSubRegOf(RI, ImpUse, IsDst ? *Dst : Src0)) {
- ErrInfo = "src0 should be subreg of implicit vector use";
- return false;
+ if (IsPreRA) {
+ const MachineOperand &Src0 = MI.getOperand(Src0Idx);
+ const MachineOperand &ImpUse =
+ MI.getOperand(StaticNumOps + NumImplicitOps - 1);
+ if (!ImpUse.isReg() || !ImpUse.isUse() ||
+ !isSubRegOf(RI, ImpUse, IsDst ? *Dst : Src0)) {
+ ErrInfo = "src0 should be subreg of implicit vector use";
+ return false;
+ }
}
}
@@ -7434,6 +7461,17 @@ SIInstrInfo::legalizeOperands(MachineInstr &MI,
return CreatedBB;
}
+ // A VGPR "as memory" indexed load/store needs its dword index in an SGPR (it
+ // becomes M0). A divergent (VGPR) index is made uniform with a waterfall
+ // loop that executes the access once per unique index across the wave.
+ if (auto *LdStIdx = dyn_cast<AMDGPUMI::VLoadStoreIdxInst>(&MI)) {
+ MachineOperand *Idx = &LdStIdx->getIdxOp();
+ if (Idx->isReg() && Idx->getReg().isVirtual() &&
+ !RI.isSGPRClass(MRI.getRegClass(Idx->getReg())))
+ CreatedBB = generateWaterFallLoop(*this, MI, {Idx}, MDT);
+ return CreatedBB;
+ }
+
// Legalize PHI
// The register class of the operands must be the same type as the register
// class of the output.
@@ -10905,9 +10943,16 @@ SIInstrInfo::getGenericValueUniformity(const MachineInstr &MI) const {
return ValueUniformity::Default;
}
+ // A VGPR ("as memory") indexed load is always divergent: it reads the wave's
+ // per-lane view of its vector registers, so even a uniform index yields a
+ // per-lane (divergent) value.
+ if (Opcode == AMDGPU::G_AMDGPU_REG_LOAD)
+ return ValueUniformity::NeverUniform;
+
// Loads from the private and flat address spaces are divergent, because
// threads can execute the load instruction with the same inputs and get
- // different results.
+ // different results. The VGPR address space is likewise divergent (see
+ // above; this covers a G_LOAD not yet legalized to G_AMDGPU_REG_LOAD).
//
// All other loads are not divergent, because if threads issue loads with the
// same arguments, they will always get the same result.
@@ -10918,7 +10963,8 @@ SIInstrInfo::getGenericValueUniformity(const MachineInstr &MI) const {
if (llvm::any_of(MI.memoperands(), [](const MachineMemOperand *mmo) {
return mmo->getAddrSpace() == AMDGPUAS::PRIVATE_ADDRESS ||
- mmo->getAddrSpace() == AMDGPUAS::FLAT_ADDRESS;
+ mmo->getAddrSpace() == AMDGPUAS::FLAT_ADDRESS ||
+ mmo->getAddrSpace() == AMDGPUAS::VGPR;
})) {
// At least one MMO in a non-global address space.
return ValueUniformity::NeverUniform;
diff --git a/llvm/lib/Target/AMDGPU/SIInstrInfo.td b/llvm/lib/Target/AMDGPU/SIInstrInfo.td
index 7be067c085b5f..c6bbfc1ec14b4 100644
--- a/llvm/lib/Target/AMDGPU/SIInstrInfo.td
+++ b/llvm/lib/Target/AMDGPU/SIInstrInfo.td
@@ -59,6 +59,17 @@ def GFX10Gen : GFXGen<isGFX10Only, "GFX10", "_gfx10", SIEncodingFamily.G
// modifier behavior with dx10_enable.
def AMDGPUclamp : SDNode<"AMDGPUISD::CLAMP", SDTFPUnaryOp>;
+// VGPR address space (13) load/store with a dword index operand. The index is
+// the byte offset into the wave's view of its vector registers, divided by 4.
+def SDTRegIdxLoad : SDTypeProfile<1, 1,
+ [SDTCisVT<1, i32>]>; // dword_index
+def SDTRegIdxStore : SDTypeProfile<0, 2,
+ [SDTCisVT<1, i32>]>; // data, dword_index
+def SIreg_load : SDNode<"AMDGPUISD::REG_LOAD", SDTRegIdxLoad,
+ [SDNPHasChain, SDNPMayLoad, SDNPMemOperand]>;
+def SIreg_store : SDNode<"AMDGPUISD::REG_STORE", SDTRegIdxStore,
+ [SDNPHasChain, SDNPMayStore, SDNPMemOperand]>;
+
def SDTSBufferLoad : SDTypeProfile<1, 3,
[ // vdata
SDTCisVT<1, v4i32>, // rsrc
diff --git a/llvm/lib/Target/AMDGPU/SIInstructions.td b/llvm/lib/Target/AMDGPU/SIInstructions.td
index 49441b86d59eb..c2854d07da1bb 100644
--- a/llvm/lib/Target/AMDGPU/SIInstructions.td
+++ b/llvm/lib/Target/AMDGPU/SIInstructions.td
@@ -1061,6 +1061,88 @@ def SI_INDIRECT_DST_V32 : SI_INDIRECT_DST<VReg_1024>;
} // End Uses = [EXEC], Defs = [M0, EXEC]
+//===----------------------------------------------------------------------===//
+// VGPR "as memory" indexed load/store pseudos (address space 13)
+//===----------------------------------------------------------------------===//
+
+// V_LOAD_IDX_B<N> / V_STORE_IDX_B<N> load or store N bits from/to the wave's
+// view of its vector registers, at a dword index of ($idx + $offset). $idx is
+// a 32-bit value that may be uniform (SGPR) or divergent (VGPR); $offset is a
+// constant dword offset folded in at selection.
+//
+// AMDGPUAssignIdxToM0 copies $idx into M0 before register allocation, then
+// AMDGPULowerVGPREncoding lowers each into v_movrels_b32 (load) /
+// v_movreld_b32 (store) over the wave's vector registers.
+
+// Populates the VLdStIdxOpcodeInfo searchable table (see AMDGPUMachineInstrs.h
+// and AMDGPU::getVLdStIdxOpcodeInfo*), mapping each pseudo to its bit width and
+// load/store direction.
+class VLdStIdxOpcodeInfo<int size, bit isStore> {
+ Instruction Opcode = !cast<Instruction>(NAME);
+ bits<12> BitWidth = size;
+ bit IsStore = isStore;
+}
+
+foreach rc = [VGPR_32, VReg_64, VReg_96, VReg_128, VReg_160, VReg_192,
+ VReg_224, VReg_256, VReg_288, VReg_320, VReg_352, VReg_384,
+ VReg_512, VReg_1024] in {
+ // The units of $idx and $offset are in dwords.
+ //
+ // VALU adds an implicit $exec use; together with the implicit-def $m0 added
+ // by AdjustInstrPostInstrSelection (see hasPostISelHook), this keeps a
+ // divergent-index access pinned inside its waterfall loop rather than being
+ // hoisted/sunk out.
+ def V_LOAD_IDX_B#rc.Size : VPseudoInstSI <
+ (outs rc:$data),
+ (ins SReg_32:$idx, i32imm:$offset)>,
+ VLdStIdxOpcodeInfo<rc.Size, 0> {
+ let mayLoad = 1;
+ let VALU = 1;
+ let UseNamedOperandTable = 1;
+ let hasSideEffects = 0;
+ let hasPostISelHook = 1;
+ }
+ def V_STORE_IDX_B#rc.Size : VPseudoInstSI <
+ (outs),
+ (ins rc:$data, SReg_32:$idx, i32imm:$offset)>,
+ VLdStIdxOpcodeInfo<rc.Size, 1> {
+ let mayStore = 1;
+ let VALU = 1;
+ let UseNamedOperandTable = 1;
+ let hasSideEffects = 0;
+ let hasPostISelHook = 1;
+ }
+}
+
+// Select the REG_LOAD/REG_STORE target nodes into the sized indexed pseudos.
+// The pointer is a byte offset into the file; SIreg_load/SIreg_store carry the
+// dword index (ptr >> 2), and an (add idx, imm) shape folds a constant dword
+// offset into the pseudo's $offset operand.
+multiclass VRegIdxLoadStorePat<ValueType vt> {
+ defvar load_inst = !cast<Instruction>("V_LOAD_IDX_B"#vt.Size);
+ defvar store_inst = !cast<Instruction>("V_STORE_IDX_B"#vt.Size);
+
+ def : GCNPat<
+ (vt (SIreg_load (add i32:$idx, (i32 imm:$offset)))),
+ (load_inst $idx, imm:$offset)>;
+ def : GCNPat<
+ (vt (SIreg_load i32:$idx)),
+ (load_inst $idx, 0)>;
+ def : GCNPat<
+ (SIreg_store vt:$data, (add i32:$idx, (i32 imm:$offset))),
+ (store_inst $data, $idx, imm:$offset)>;
+ def : GCNPat<
+ (SIreg_store vt:$data, i32:$idx),
+ (store_inst $data, $idx, 0)>;
+}
+
+foreach vt = !listconcat(
+ Reg32Types.types, Reg64Types.types, Reg96Types.types, Reg128Types.types,
+ Reg160Types.types, Reg192Types.types, Reg224Types.types, Reg256Types.types,
+ Reg288Types.types, Reg320Types.types, Reg352Types.types, Reg384Types.types,
+ Reg512Types.types, Reg1024Types.types) in
+defm : VRegIdxLoadStorePat<vt>;
+
// This is a pseudo variant of the v_movreld_b32 instruction in which the
// vector operand appears only twice, once as def and once as use. Using this
// pseudo avoids problems with the Two Address instructions pass.
@@ -4804,6 +4886,24 @@ def G_AMDGPU_BUFFER_STORE_FORMAT_D16 : BufferStoreGenericInstruction;
def G_AMDGPU_TBUFFER_STORE_FORMAT : TBufferStoreGenericInstruction;
def G_AMDGPU_TBUFFER_STORE_FORMAT_D16 : TBufferStoreGenericInstruction;
+// GlobalISel equivalents of the REG_LOAD / REG_STORE target nodes: a load or
+// store of the VGPR ("as memory") address space, indexed by the pointer's
+// dword offset. They select to the V_LOAD_IDX_B<N> / V_STORE_IDX_B<N> pseudos
+// through the same TableGen patterns (see GINodeEquiv in AMDGPUGISel.td).
+def G_AMDGPU_REG_LOAD : AMDGPUGenericInstruction {
+ let OutOperandList = (outs type0:$dst);
+ let InOperandList = (ins type1:$dword_index);
+ let hasSideEffects = 0;
+ let mayLoad = 1;
+}
+
+def G_AMDGPU_REG_STORE : AMDGPUGenericInstruction {
+ let OutOperandList = (outs);
+ let InOperandList = (ins type0:$data, type1:$dword_index);
+ let hasSideEffects = 0;
+ let mayStore = 1;
+}
+
def G_AMDGPU_FMIN_LEGACY : AMDGPUGenericInstruction {
let OutOperandList = (outs type0:$dst);
let InOperandList = (ins type0:$src0, type0:$src1);
diff --git a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
index 3ae1ae40ca34a..40e8e1f364178 100644
--- a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
@@ -478,9 +478,20 @@ struct FP4FP8DstByteSelInfo {
#define GET_getMFMA_F8F6F4_WithSize_IMPL
#define GET_isMFMA_F8F6F4Table_IMPL
#define GET_isCvtScaleF32_F32F16ToF8F4Table_IMPL
+#define GET_VLdStIdxOpcodeInfoTable_DECL
+#define GET_VLdStIdxOpcodeInfoTable_IMPL
#include "AMDGPUGenSearchableTables.inc"
+const VLdStIdxOpcodeInfo *getVLdStIdxOpcodeInfoByOpcode(unsigned Opc) {
+ return getVLdStIdxOpcodeInfoByOpcodeImpl(Opc);
+}
+
+const VLdStIdxOpcodeInfo *getVLdStIdxOpcodeInfoByKey(uint16_t BitWidth,
+ bool IsStore) {
+ return getVLdStIdxOpcodeInfoByKeyImpl(BitWidth, IsStore);
+}
+
int getMTBUFBaseOpcode(unsigned Opc) {
const MTBUFInfo *Info = getMTBUFInfoFromOpcode(Opc);
return Info ? Info->BaseOpcode : -1;
diff --git a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h
index 4b08c08c5a60d..fe9c66882941e 100644
--- a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h
+++ b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h
@@ -500,6 +500,19 @@ struct MIMGInfo {
LLVM_READONLY
const MIMGInfo *getMIMGInfo(unsigned Opc);
+struct VLdStIdxOpcodeInfo {
+ unsigned Opcode;
+ uint16_t BitWidth;
+ bool IsStore;
+};
+
+LLVM_READONLY
+const VLdStIdxOpcodeInfo *getVLdStIdxOpcodeInfoByOpcode(unsigned Opc);
+
+LLVM_READONLY
+const VLdStIdxOpcodeInfo *getVLdStIdxOpcodeInfoByKey(uint16_t BitWidth,
+ bool IsStore);
+
LLVM_READONLY
int getMTBUFBaseOpcode(unsigned Opc);
diff --git a/llvm/test/CodeGen/AMDGPU/AddressSpaceVGPR/as-vgpr-basic.ll b/llvm/test/CodeGen/AMDGPU/AddressSpaceVGPR/as-vgpr-basic.ll
new file mode 100644
index 0000000000000..08d3095de3561
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/AddressSpaceVGPR/as-vgpr-basic.ll
@@ -0,0 +1,384 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
+; RUN: llc -global-isel=0 -verify-machineinstrs -mtriple=amdgcn -mcpu=gfx1200 -o - %s | FileCheck %s --check-prefixes=GFX12,GFX12-SDAG
+; RUN: llc -global-isel=1 -verify-machineinstrs -mtriple=amdgcn -mcpu=gfx1200 -o - %s | FileCheck %s --check-prefixes=GFX12,GFX12-GISEL
+
+; End-to-end lowering of the VGPR "as memory" address space (13) on a
+; movrel-capable subtarget (gfx12). A load/store of a uniform (SGPR) pointer
+; lowers to an M0-relative move (v_movrels_b32 / v_movreld_b32) over the wave's
+; vector registers, with the dword index (pointer >> 2) placed in M0.
+
+define i32 @load_i32(ptr addrspace(13) inreg %p) {
+; GFX12-LABEL: load_i32:
+; GFX12: ; %bb.0:
+; GFX12-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-NEXT: s_wait_expcnt 0x0
+; GFX12-NEXT: s_wait_samplecnt 0x0
+; GFX12-NEXT: s_wait_bvhcnt 0x0
+; GFX12-NEXT: s_wait_kmcnt 0x0
+; GFX12-NEXT: s_lshr_b32 m0, s0, 2
+; GFX12-NEXT: v_movrels_b32_e32 v0, v0
+; GFX12-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-NEXT: v_add_nc_u32_e32 v0, 1, v0
+; GFX12-NEXT: s_setpc_b64 s[30:31]
+ %x = load i32, ptr addrspace(13) %p
+ %y = add i32 %x, 1
+ ret i32 %y
+}
+
+define i64 @load_i64(ptr addrspace(13) inreg %p) {
+; GFX12-LABEL: load_i64:
+; GFX12: ; %bb.0:
+; GFX12-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-NEXT: s_wait_expcnt 0x0
+; GFX12-NEXT: s_wait_samplecnt 0x0
+; GFX12-NEXT: s_wait_bvhcnt 0x0
+; GFX12-NEXT: s_wait_kmcnt 0x0
+; GFX12-NEXT: s_lshr_b32 m0, s0, 2
+; GFX12-NEXT: v_movrels_b32_e32 v0, v0
+; GFX12-NEXT: v_movrels_b32_e32 v1, v1
+; GFX12-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(SKIP_1) | instid1(VALU_DEP_2)
+; GFX12-NEXT: v_add_co_u32 v0, vcc_lo, v0, 1
+; GFX12-NEXT: s_wait_alu depctr_va_vcc(0)
+; GFX12-NEXT: v_add_co_ci_u32_e64 v1, null, 0, v1, vcc_lo
+; GFX12-NEXT: s_setpc_b64 s[30:31]
+ %x = load i64, ptr addrspace(13) %p
+ %y = add i64 %x, 1
+ ret i64 %y
+}
+
+define <2 x float> @load_v2f32(ptr addrspace(13) inreg %p) {
+; GFX12-LABEL: load_v2f32:
+; GFX12: ; %bb.0:
+; GFX12-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-NEXT: s_wait_expcnt 0x0
+; GFX12-NEXT: s_wait_samplecnt 0x0
+; GFX12-NEXT: s_wait_bvhcnt 0x0
+; GFX12-NEXT: s_wait_kmcnt 0x0
+; GFX12-NEXT: s_lshr_b32 m0, s0, 2
+; GFX12-NEXT: v_movrels_b32_e32 v0, v0
+; GFX12-NEXT: v_movrels_b32_e32 v1, v1
+; GFX12-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-NEXT: v_dual_add_f32 v0, 0x42280000, v0 :: v_dual_add_f32 v1, 0x42280000, v1
+; GFX12-NEXT: s_setpc_b64 s[30:31]
+ %x = load <2 x float>, ptr addrspace(13) %p
+ %y = fadd <2 x float> %x, <float 42.0, float 42.0>
+ ret <2 x float> %y
+}
+
+define <3 x float> @load_v3f32(ptr addrspace(13) inreg %p) {
+; GFX12-SDAG-LABEL: load_v3f32:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-SDAG-NEXT: s_wait_expcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: s_wait_bvhcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_kmcnt 0x0
+; GFX12-SDAG-NEXT: s_lshr_b32 m0, s0, 2
+; GFX12-SDAG-NEXT: s_add_co_i32 s0, s0, 64
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v0, v0
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v1, v1
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v2, v2
+; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-SDAG-NEXT: s_lshr_b32 m0, s0, 2
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v3, v0
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v4, v1
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v5, v2
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX12-SDAG-NEXT: v_dual_add_f32 v0, v0, v3 :: v_dual_add_f32 v1, v1, v4
+; GFX12-SDAG-NEXT: v_add_f32_e32 v2, v2, v5
+; GFX12-SDAG-NEXT: s_setpc_b64 s[30:31]
+;
+; GFX12-GISEL-LABEL: load_v3f32:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-GISEL-NEXT: s_wait_expcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: s_wait_bvhcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_kmcnt 0x0
+; GFX12-GISEL-NEXT: s_lshr_b32 m0, s0, 2
+; GFX12-GISEL-NEXT: s_add_co_u32 s0, s0, 64
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v0, v0
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v1, v1
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v2, v2
+; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-GISEL-NEXT: s_lshr_b32 m0, s0, 2
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v3, v0
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v4, v1
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v5, v2
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX12-GISEL-NEXT: v_dual_add_f32 v0, v0, v3 :: v_dual_add_f32 v1, v1, v4
+; GFX12-GISEL-NEXT: v_add_f32_e32 v2, v2, v5
+; GFX12-GISEL-NEXT: s_setpc_b64 s[30:31]
+ %p.2 = getelementptr i32, ptr addrspace(13) %p, i32 16
+ %x = load <3 x float>, ptr addrspace(13) %p
+ %y = load <3 x float>, ptr addrspace(13) %p.2
+ %z = fadd <3 x float> %x, %y
+ ret <3 x float> %z
+}
+
+define void @store_i32(ptr addrspace(13) inreg %p, i32 %x) {
+; GFX12-LABEL: store_i32:
+; GFX12: ; %bb.0:
+; GFX12-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-NEXT: s_wait_expcnt 0x0
+; GFX12-NEXT: s_wait_samplecnt 0x0
+; GFX12-NEXT: s_wait_bvhcnt 0x0
+; GFX12-NEXT: s_wait_kmcnt 0x0
+; GFX12-NEXT: v_add_nc_u32_e32 v0, 1, v0
+; GFX12-NEXT: s_lshr_b32 m0, s0, 2
+; GFX12-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-NEXT: v_movreld_b32_e32 v0, v0
+; GFX12-NEXT: s_setpc_b64 s[30:31]
+ %y = add i32 %x, 1
+ store i32 %y, ptr addrspace(13) %p
+ ret void
+}
+
+define void @store_i64(ptr addrspace(13) inreg %p, i64 %x, i64 %y) {
+; GFX12-LABEL: store_i64:
+; GFX12: ; %bb.0:
+; GFX12-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-NEXT: s_wait_expcnt 0x0
+; GFX12-NEXT: s_wait_samplecnt 0x0
+; GFX12-NEXT: s_wait_bvhcnt 0x0
+; GFX12-NEXT: s_wait_kmcnt 0x0
+; GFX12-NEXT: v_add_co_u32 v0, vcc_lo, v0, v2
+; GFX12-NEXT: s_wait_alu depctr_va_vcc(0)
+; GFX12-NEXT: v_add_co_ci_u32_e64 v1, null, v1, v3, vcc_lo
+; GFX12-NEXT: s_lshr_b32 m0, s0, 2
+; GFX12-NEXT: v_movreld_b32_e32 v0, v0
+; GFX12-NEXT: v_movreld_b32_e32 v1, v1
+; GFX12-NEXT: s_setpc_b64 s[30:31]
+ %z = add i64 %x, %y
+ store i64 %z, ptr addrspace(13) %p
+ ret void
+}
+
+define void @store_v3i32(ptr addrspace(13) inreg %p, <3 x i32> %x, <3 x i32> %y) {
+; GFX12-SDAG-LABEL: store_v3i32:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-SDAG-NEXT: s_wait_expcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: s_wait_bvhcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_kmcnt 0x0
+; GFX12-SDAG-NEXT: v_add_nc_u32_e32 v2, v2, v5
+; GFX12-SDAG-NEXT: v_add_nc_u32_e32 v1, v1, v4
+; GFX12-SDAG-NEXT: v_add_nc_u32_e32 v0, v0, v3
+; GFX12-SDAG-NEXT: s_lshr_b32 m0, s0, 2
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_3)
+; GFX12-SDAG-NEXT: v_movreld_b32_e32 v0, v0
+; GFX12-SDAG-NEXT: v_movreld_b32_e32 v1, v1
+; GFX12-SDAG-NEXT: v_movreld_b32_e32 v2, v2
+; GFX12-SDAG-NEXT: s_setpc_b64 s[30:31]
+;
+; GFX12-GISEL-LABEL: store_v3i32:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-GISEL-NEXT: s_wait_expcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: s_wait_bvhcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_kmcnt 0x0
+; GFX12-GISEL-NEXT: v_add_nc_u32_e32 v0, v0, v3
+; GFX12-GISEL-NEXT: v_add_nc_u32_e32 v1, v1, v4
+; GFX12-GISEL-NEXT: v_add_nc_u32_e32 v2, v2, v5
+; GFX12-GISEL-NEXT: s_lshr_b32 m0, s0, 2
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_3) | instskip(NEXT) | instid1(VALU_DEP_3)
+; GFX12-GISEL-NEXT: v_movreld_b32_e32 v0, v0
+; GFX12-GISEL-NEXT: v_movreld_b32_e32 v1, v1
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_3)
+; GFX12-GISEL-NEXT: v_movreld_b32_e32 v2, v2
+; GFX12-GISEL-NEXT: s_setpc_b64 s[30:31]
+ %z = add <3 x i32> %x, %y
+ store <3 x i32> %z, ptr addrspace(13) %p
+ ret void
+}
+
+define void @store_v8f16(ptr addrspace(13) inreg %p, <8 x half> %x, <8 x half> %y) {
+; GFX12-SDAG-LABEL: store_v8f16:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-SDAG-NEXT: s_wait_expcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: s_wait_bvhcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_kmcnt 0x0
+; GFX12-SDAG-NEXT: v_pk_add_f16 v3, v3, v7
+; GFX12-SDAG-NEXT: v_pk_add_f16 v2, v2, v6
+; GFX12-SDAG-NEXT: v_pk_add_f16 v1, v1, v5
+; GFX12-SDAG-NEXT: v_pk_add_f16 v0, v0, v4
+; GFX12-SDAG-NEXT: s_lshr_b32 m0, s0, 2
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_3)
+; GFX12-SDAG-NEXT: v_movreld_b32_e32 v0, v0
+; GFX12-SDAG-NEXT: v_movreld_b32_e32 v1, v1
+; GFX12-SDAG-NEXT: v_movreld_b32_e32 v2, v2
+; GFX12-SDAG-NEXT: v_movreld_b32_e32 v3, v3
+; GFX12-SDAG-NEXT: s_setpc_b64 s[30:31]
+;
+; GFX12-GISEL-LABEL: store_v8f16:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-GISEL-NEXT: s_wait_expcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: s_wait_bvhcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_kmcnt 0x0
+; GFX12-GISEL-NEXT: v_pk_add_f16 v0, v0, v4
+; GFX12-GISEL-NEXT: v_pk_add_f16 v1, v1, v5
+; GFX12-GISEL-NEXT: v_pk_add_f16 v2, v2, v6
+; GFX12-GISEL-NEXT: v_pk_add_f16 v3, v3, v7
+; GFX12-GISEL-NEXT: s_lshr_b32 m0, s0, 2
+; GFX12-GISEL-NEXT: v_movreld_b32_e32 v0, v0
+; GFX12-GISEL-NEXT: v_movreld_b32_e32 v1, v1
+; GFX12-GISEL-NEXT: v_movreld_b32_e32 v2, v2
+; GFX12-GISEL-NEXT: v_movreld_b32_e32 v3, v3
+; GFX12-GISEL-NEXT: s_setpc_b64 s[30:31]
+ %z = fadd <8 x half> %x, %y
+ store <8 x half> %z, ptr addrspace(13) %p
+ ret void
+}
+
+define void @copy_i32(ptr addrspace(13) inreg %dst, ptr addrspace(13) inreg %src) {
+; GFX12-LABEL: copy_i32:
+; GFX12: ; %bb.0:
+; GFX12-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-NEXT: s_wait_expcnt 0x0
+; GFX12-NEXT: s_wait_samplecnt 0x0
+; GFX12-NEXT: s_wait_bvhcnt 0x0
+; GFX12-NEXT: s_wait_kmcnt 0x0
+; GFX12-NEXT: s_lshr_b32 m0, s1, 2
+; GFX12-NEXT: v_movrels_b32_e32 v0, v0
+; GFX12-NEXT: s_lshr_b32 m0, s0, 2
+; GFX12-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-NEXT: v_movreld_b32_e32 v0, v0
+; GFX12-NEXT: s_setpc_b64 s[30:31]
+ %x = load i32, ptr addrspace(13) %src
+ store i32 %x, ptr addrspace(13) %dst
+ ret void
+}
+
+define void @copy_v2i32_unaligned(ptr addrspace(13) inreg %dst, ptr addrspace(13) inreg %src) {
+; GFX12-LABEL: copy_v2i32_unaligned:
+; GFX12: ; %bb.0:
+; GFX12-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-NEXT: s_wait_expcnt 0x0
+; GFX12-NEXT: s_wait_samplecnt 0x0
+; GFX12-NEXT: s_wait_bvhcnt 0x0
+; GFX12-NEXT: s_wait_kmcnt 0x0
+; GFX12-NEXT: s_lshr_b32 m0, s1, 2
+; GFX12-NEXT: v_movrels_b32_e32 v0, v0
+; GFX12-NEXT: v_movrels_b32_e32 v1, v1
+; GFX12-NEXT: s_lshr_b32 m0, s0, 2
+; GFX12-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX12-NEXT: v_movreld_b32_e32 v0, v0
+; GFX12-NEXT: v_movreld_b32_e32 v1, v1
+; GFX12-NEXT: s_setpc_b64 s[30:31]
+ %x = load <2 x i32>, ptr addrspace(13) %src, align 4
+ store <2 x i32> %x, ptr addrspace(13) %dst, align 4
+ ret void
+}
+
+define void @copy_i64_aligned(ptr addrspace(13) inreg %dst, ptr addrspace(13) inreg %src) {
+; GFX12-LABEL: copy_i64_aligned:
+; GFX12: ; %bb.0:
+; GFX12-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-NEXT: s_wait_expcnt 0x0
+; GFX12-NEXT: s_wait_samplecnt 0x0
+; GFX12-NEXT: s_wait_bvhcnt 0x0
+; GFX12-NEXT: s_wait_kmcnt 0x0
+; GFX12-NEXT: s_lshr_b32 m0, s1, 2
+; GFX12-NEXT: v_movrels_b32_e32 v0, v0
+; GFX12-NEXT: v_movrels_b32_e32 v1, v1
+; GFX12-NEXT: s_lshr_b32 m0, s0, 2
+; GFX12-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX12-NEXT: v_movreld_b32_e32 v0, v0
+; GFX12-NEXT: v_movreld_b32_e32 v1, v1
+; GFX12-NEXT: s_setpc_b64 s[30:31]
+ %x = load i64, ptr addrspace(13) %src, align 8
+ store i64 %x, ptr addrspace(13) %dst, align 8
+ ret void
+}
+
+; Null and poison pointers must be accepted (produce valid code) rather than
+; crash or fail the machine verifier. The specific null-pointer value is
+; defined by the parent change that introduces the address space.
+
+define i32 @load_null() {
+; GFX12-LABEL: load_null:
+; GFX12: ; %bb.0:
+; GFX12-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-NEXT: s_wait_expcnt 0x0
+; GFX12-NEXT: s_wait_samplecnt 0x0
+; GFX12-NEXT: s_wait_bvhcnt 0x0
+; GFX12-NEXT: s_wait_kmcnt 0x0
+; GFX12-NEXT: s_mov_b32 m0, 0
+; GFX12-NEXT: v_movrels_b32_e32 v0, v0
+; GFX12-NEXT: s_setpc_b64 s[30:31]
+ %x = load i32, ptr addrspace(13) null
+ ret i32 %x
+}
+
+define void @store_null(i32 %v) {
+; GFX12-LABEL: store_null:
+; GFX12: ; %bb.0:
+; GFX12-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-NEXT: s_wait_expcnt 0x0
+; GFX12-NEXT: s_wait_samplecnt 0x0
+; GFX12-NEXT: s_wait_bvhcnt 0x0
+; GFX12-NEXT: s_wait_kmcnt 0x0
+; GFX12-NEXT: s_mov_b32 m0, 0
+; GFX12-NEXT: v_movreld_b32_e32 v0, v0
+; GFX12-NEXT: s_setpc_b64 s[30:31]
+ store i32 %v, ptr addrspace(13) null
+ ret void
+}
+
+define i32 @load_poison() {
+; GFX12-SDAG-LABEL: load_poison:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-SDAG-NEXT: s_wait_expcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: s_wait_bvhcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_kmcnt 0x0
+; GFX12-SDAG-NEXT: s_mov_b32 m0, 0
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v0, v0
+; GFX12-SDAG-NEXT: s_setpc_b64 s[30:31]
+;
+; GFX12-GISEL-LABEL: load_poison:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-GISEL-NEXT: s_wait_expcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: s_wait_bvhcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_kmcnt 0x0
+; GFX12-GISEL-NEXT: s_lshr_b32 m0, s0, 2
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v0, v0
+; GFX12-GISEL-NEXT: s_setpc_b64 s[30:31]
+ %x = load i32, ptr addrspace(13) poison
+ ret i32 %x
+}
+
+define void @store_poison(i32 %v) {
+; GFX12-SDAG-LABEL: store_poison:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-SDAG-NEXT: s_wait_expcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: s_wait_bvhcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_kmcnt 0x0
+; GFX12-SDAG-NEXT: s_mov_b32 m0, 0
+; GFX12-SDAG-NEXT: v_movreld_b32_e32 v0, v0
+; GFX12-SDAG-NEXT: s_setpc_b64 s[30:31]
+;
+; GFX12-GISEL-LABEL: store_poison:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-GISEL-NEXT: s_wait_expcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: s_wait_bvhcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_kmcnt 0x0
+; GFX12-GISEL-NEXT: s_lshr_b32 m0, s0, 2
+; GFX12-GISEL-NEXT: v_movreld_b32_e32 v0, v0
+; GFX12-GISEL-NEXT: s_setpc_b64 s[30:31]
+ store i32 %v, ptr addrspace(13) poison
+ ret void
+}
diff --git a/llvm/test/CodeGen/AMDGPU/AddressSpaceVGPR/as-vgpr-copy.ll b/llvm/test/CodeGen/AMDGPU/AddressSpaceVGPR/as-vgpr-copy.ll
new file mode 100644
index 0000000000000..48d90ecaf2eff
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/AddressSpaceVGPR/as-vgpr-copy.ll
@@ -0,0 +1,461 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
+; RUN: llc -global-isel=0 -verify-machineinstrs -mtriple=amdgcn -mcpu=gfx1200 -o - %s | FileCheck %s --check-prefixes=GFX12,GFX12-SDAG
+; RUN: llc -global-isel=1 -verify-machineinstrs -mtriple=amdgcn -mcpu=gfx1200 -o - %s | FileCheck %s --check-prefixes=GFX12,GFX12-GISEL
+
+; Copy from VGPR "as memory" (address space 13) to global memory across the
+; range of legal whole-dword access sizes (32 up to 1024 bits). Each uniform
+; load lowers to a sequence of per-dword M0-relative moves (v_movrels_b32).
+
+define void @copy_i32(ptr addrspace(1) inreg %out, ptr addrspace(13) inreg %in) {
+; GFX12-SDAG-LABEL: copy_i32:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-SDAG-NEXT: s_wait_expcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: s_wait_bvhcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_kmcnt 0x0
+; GFX12-SDAG-NEXT: s_lshr_b32 m0, s2, 2
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v1, v0
+; GFX12-SDAG-NEXT: global_store_b32 v0, v1, s[0:1]
+; GFX12-SDAG-NEXT: s_setpc_b64 s[30:31]
+;
+; GFX12-GISEL-LABEL: copy_i32:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-GISEL-NEXT: s_wait_expcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: s_wait_bvhcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_kmcnt 0x0
+; GFX12-GISEL-NEXT: s_lshr_b32 m0, s2, 2
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v1, 0
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v0, v0
+; GFX12-GISEL-NEXT: global_store_b32 v1, v0, s[0:1]
+; GFX12-GISEL-NEXT: s_setpc_b64 s[30:31]
+ %x = load i32, ptr addrspace(13) %in
+ store i32 %x, ptr addrspace(1) %out
+ ret void
+}
+
+define void @copy_v2i32(ptr addrspace(1) inreg %out, ptr addrspace(13) inreg %in) {
+; GFX12-LABEL: copy_v2i32:
+; GFX12: ; %bb.0:
+; GFX12-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-NEXT: s_wait_expcnt 0x0
+; GFX12-NEXT: s_wait_samplecnt 0x0
+; GFX12-NEXT: s_wait_bvhcnt 0x0
+; GFX12-NEXT: s_wait_kmcnt 0x0
+; GFX12-NEXT: s_lshr_b32 m0, s2, 2
+; GFX12-NEXT: v_mov_b32_e32 v2, 0
+; GFX12-NEXT: v_movrels_b32_e32 v0, v0
+; GFX12-NEXT: v_movrels_b32_e32 v1, v1
+; GFX12-NEXT: global_store_b64 v2, v[0:1], s[0:1]
+; GFX12-NEXT: s_setpc_b64 s[30:31]
+ %x = load <2 x i32>, ptr addrspace(13) %in
+ store <2 x i32> %x, ptr addrspace(1) %out
+ ret void
+}
+
+define void @copy_v3i32(ptr addrspace(1) inreg %out, ptr addrspace(13) inreg %in) {
+; GFX12-LABEL: copy_v3i32:
+; GFX12: ; %bb.0:
+; GFX12-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-NEXT: s_wait_expcnt 0x0
+; GFX12-NEXT: s_wait_samplecnt 0x0
+; GFX12-NEXT: s_wait_bvhcnt 0x0
+; GFX12-NEXT: s_wait_kmcnt 0x0
+; GFX12-NEXT: s_lshr_b32 m0, s2, 2
+; GFX12-NEXT: v_mov_b32_e32 v3, 0
+; GFX12-NEXT: v_movrels_b32_e32 v0, v0
+; GFX12-NEXT: v_movrels_b32_e32 v1, v1
+; GFX12-NEXT: v_movrels_b32_e32 v2, v2
+; GFX12-NEXT: global_store_b96 v3, v[0:2], s[0:1]
+; GFX12-NEXT: s_setpc_b64 s[30:31]
+ %x = load <3 x i32>, ptr addrspace(13) %in
+ store <3 x i32> %x, ptr addrspace(1) %out
+ ret void
+}
+
+define void @copy_v4i32(ptr addrspace(1) inreg %out, ptr addrspace(13) inreg %in) {
+; GFX12-LABEL: copy_v4i32:
+; GFX12: ; %bb.0:
+; GFX12-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-NEXT: s_wait_expcnt 0x0
+; GFX12-NEXT: s_wait_samplecnt 0x0
+; GFX12-NEXT: s_wait_bvhcnt 0x0
+; GFX12-NEXT: s_wait_kmcnt 0x0
+; GFX12-NEXT: s_lshr_b32 m0, s2, 2
+; GFX12-NEXT: v_mov_b32_e32 v4, 0
+; GFX12-NEXT: v_movrels_b32_e32 v0, v0
+; GFX12-NEXT: v_movrels_b32_e32 v1, v1
+; GFX12-NEXT: v_movrels_b32_e32 v2, v2
+; GFX12-NEXT: v_movrels_b32_e32 v3, v3
+; GFX12-NEXT: global_store_b128 v4, v[0:3], s[0:1]
+; GFX12-NEXT: s_setpc_b64 s[30:31]
+ %x = load <4 x i32>, ptr addrspace(13) %in
+ store <4 x i32> %x, ptr addrspace(1) %out
+ ret void
+}
+
+define void @copy_v5i32(ptr addrspace(1) inreg %out, ptr addrspace(13) inreg %in) {
+; GFX12-SDAG-LABEL: copy_v5i32:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-SDAG-NEXT: s_wait_expcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: s_wait_bvhcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_kmcnt 0x0
+; GFX12-SDAG-NEXT: s_lshr_b32 m0, s2, 2
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v5, 0
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v0, v0
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v1, v1
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v2, v2
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v3, v3
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v4, v4
+; GFX12-SDAG-NEXT: s_clause 0x1
+; GFX12-SDAG-NEXT: global_store_b32 v5, v4, s[0:1] offset:16
+; GFX12-SDAG-NEXT: global_store_b128 v5, v[0:3], s[0:1]
+; GFX12-SDAG-NEXT: s_setpc_b64 s[30:31]
+;
+; GFX12-GISEL-LABEL: copy_v5i32:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-GISEL-NEXT: s_wait_expcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: s_wait_bvhcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_kmcnt 0x0
+; GFX12-GISEL-NEXT: s_lshr_b32 m0, s2, 2
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v5, 0
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v0, v0
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v1, v1
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v2, v2
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v3, v3
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v4, v4
+; GFX12-GISEL-NEXT: s_clause 0x1
+; GFX12-GISEL-NEXT: global_store_b128 v5, v[0:3], s[0:1]
+; GFX12-GISEL-NEXT: global_store_b32 v5, v4, s[0:1] offset:16
+; GFX12-GISEL-NEXT: s_setpc_b64 s[30:31]
+ %x = load <5 x i32>, ptr addrspace(13) %in
+ store <5 x i32> %x, ptr addrspace(1) %out
+ ret void
+}
+
+define void @copy_v6i32(ptr addrspace(1) inreg %out, ptr addrspace(13) inreg %in) {
+; GFX12-SDAG-LABEL: copy_v6i32:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-SDAG-NEXT: s_wait_expcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: s_wait_bvhcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_kmcnt 0x0
+; GFX12-SDAG-NEXT: s_lshr_b32 m0, s2, 2
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v6, 0
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v0, v0
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v1, v1
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v2, v2
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v3, v3
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v4, v4
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v5, v5
+; GFX12-SDAG-NEXT: s_clause 0x1
+; GFX12-SDAG-NEXT: global_store_b64 v6, v[4:5], s[0:1] offset:16
+; GFX12-SDAG-NEXT: global_store_b128 v6, v[0:3], s[0:1]
+; GFX12-SDAG-NEXT: s_setpc_b64 s[30:31]
+;
+; GFX12-GISEL-LABEL: copy_v6i32:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-GISEL-NEXT: s_wait_expcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: s_wait_bvhcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_kmcnt 0x0
+; GFX12-GISEL-NEXT: s_lshr_b32 m0, s2, 2
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v6, 0
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v0, v0
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v1, v1
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v2, v2
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v3, v3
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v4, v4
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v5, v5
+; GFX12-GISEL-NEXT: s_clause 0x1
+; GFX12-GISEL-NEXT: global_store_b128 v6, v[0:3], s[0:1]
+; GFX12-GISEL-NEXT: global_store_b64 v6, v[4:5], s[0:1] offset:16
+; GFX12-GISEL-NEXT: s_setpc_b64 s[30:31]
+ %x = load <6 x i32>, ptr addrspace(13) %in
+ store <6 x i32> %x, ptr addrspace(1) %out
+ ret void
+}
+
+define void @copy_v7i32(ptr addrspace(1) inreg %out, ptr addrspace(13) inreg %in) {
+; GFX12-SDAG-LABEL: copy_v7i32:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-SDAG-NEXT: s_wait_expcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: s_wait_bvhcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_kmcnt 0x0
+; GFX12-SDAG-NEXT: s_lshr_b32 m0, s2, 2
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v7, 0
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v0, v0
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v1, v1
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v2, v2
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v3, v3
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v4, v4
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v5, v5
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v6, v6
+; GFX12-SDAG-NEXT: s_clause 0x1
+; GFX12-SDAG-NEXT: global_store_b96 v7, v[4:6], s[0:1] offset:16
+; GFX12-SDAG-NEXT: global_store_b128 v7, v[0:3], s[0:1]
+; GFX12-SDAG-NEXT: s_setpc_b64 s[30:31]
+;
+; GFX12-GISEL-LABEL: copy_v7i32:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-GISEL-NEXT: s_wait_expcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: s_wait_bvhcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_kmcnt 0x0
+; GFX12-GISEL-NEXT: s_lshr_b32 m0, s2, 2
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v7, 0
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v0, v0
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v1, v1
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v2, v2
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v3, v3
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v4, v4
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v5, v5
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v6, v6
+; GFX12-GISEL-NEXT: s_clause 0x1
+; GFX12-GISEL-NEXT: global_store_b128 v7, v[0:3], s[0:1]
+; GFX12-GISEL-NEXT: global_store_b96 v7, v[4:6], s[0:1] offset:16
+; GFX12-GISEL-NEXT: s_setpc_b64 s[30:31]
+ %x = load <7 x i32>, ptr addrspace(13) %in
+ store <7 x i32> %x, ptr addrspace(1) %out
+ ret void
+}
+
+define void @copy_v8i32(ptr addrspace(1) inreg %out, ptr addrspace(13) inreg %in) {
+; GFX12-SDAG-LABEL: copy_v8i32:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-SDAG-NEXT: s_wait_expcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: s_wait_bvhcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_kmcnt 0x0
+; GFX12-SDAG-NEXT: s_lshr_b32 m0, s2, 2
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v8, 0
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v0, v0
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v1, v1
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v2, v2
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v3, v3
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v4, v4
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v5, v5
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v6, v6
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v7, v7
+; GFX12-SDAG-NEXT: s_clause 0x1
+; GFX12-SDAG-NEXT: global_store_b128 v8, v[4:7], s[0:1] offset:16
+; GFX12-SDAG-NEXT: global_store_b128 v8, v[0:3], s[0:1]
+; GFX12-SDAG-NEXT: s_setpc_b64 s[30:31]
+;
+; GFX12-GISEL-LABEL: copy_v8i32:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-GISEL-NEXT: s_wait_expcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: s_wait_bvhcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_kmcnt 0x0
+; GFX12-GISEL-NEXT: s_lshr_b32 m0, s2, 2
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v8, 0
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v0, v0
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v1, v1
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v2, v2
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v3, v3
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v4, v4
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v5, v5
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v6, v6
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v7, v7
+; GFX12-GISEL-NEXT: s_clause 0x1
+; GFX12-GISEL-NEXT: global_store_b128 v8, v[0:3], s[0:1]
+; GFX12-GISEL-NEXT: global_store_b128 v8, v[4:7], s[0:1] offset:16
+; GFX12-GISEL-NEXT: s_setpc_b64 s[30:31]
+ %x = load <8 x i32>, ptr addrspace(13) %in
+ store <8 x i32> %x, ptr addrspace(1) %out
+ ret void
+}
+
+define void @copy_v16i32(ptr addrspace(1) inreg %out, ptr addrspace(13) inreg %in) {
+; GFX12-SDAG-LABEL: copy_v16i32:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-SDAG-NEXT: s_wait_expcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: s_wait_bvhcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_kmcnt 0x0
+; GFX12-SDAG-NEXT: s_lshr_b32 m0, s2, 2
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v16, 0
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v0, v0
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v1, v1
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v2, v2
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v3, v3
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v4, v4
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v5, v5
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v6, v6
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v7, v7
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v8, v8
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v9, v9
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v10, v10
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v11, v11
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v12, v12
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v13, v13
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v14, v14
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v15, v15
+; GFX12-SDAG-NEXT: s_clause 0x3
+; GFX12-SDAG-NEXT: global_store_b128 v16, v[12:15], s[0:1] offset:48
+; GFX12-SDAG-NEXT: global_store_b128 v16, v[8:11], s[0:1] offset:32
+; GFX12-SDAG-NEXT: global_store_b128 v16, v[4:7], s[0:1] offset:16
+; GFX12-SDAG-NEXT: global_store_b128 v16, v[0:3], s[0:1]
+; GFX12-SDAG-NEXT: s_setpc_b64 s[30:31]
+;
+; GFX12-GISEL-LABEL: copy_v16i32:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-GISEL-NEXT: s_wait_expcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: s_wait_bvhcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_kmcnt 0x0
+; GFX12-GISEL-NEXT: s_lshr_b32 m0, s2, 2
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v16, 0
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v0, v0
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v1, v1
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v2, v2
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v3, v3
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v4, v4
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v5, v5
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v6, v6
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v7, v7
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v8, v8
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v9, v9
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v10, v10
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v11, v11
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v12, v12
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v13, v13
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v14, v14
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v15, v15
+; GFX12-GISEL-NEXT: s_clause 0x3
+; GFX12-GISEL-NEXT: global_store_b128 v16, v[0:3], s[0:1]
+; GFX12-GISEL-NEXT: global_store_b128 v16, v[4:7], s[0:1] offset:16
+; GFX12-GISEL-NEXT: global_store_b128 v16, v[8:11], s[0:1] offset:32
+; GFX12-GISEL-NEXT: global_store_b128 v16, v[12:15], s[0:1] offset:48
+; GFX12-GISEL-NEXT: s_setpc_b64 s[30:31]
+ %x = load <16 x i32>, ptr addrspace(13) %in
+ store <16 x i32> %x, ptr addrspace(1) %out
+ ret void
+}
+
+define void @copy_v32i32(ptr addrspace(1) inreg %out, ptr addrspace(13) inreg %in) {
+; GFX12-SDAG-LABEL: copy_v32i32:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-SDAG-NEXT: s_wait_expcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: s_wait_bvhcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_kmcnt 0x0
+; GFX12-SDAG-NEXT: s_lshr_b32 m0, s2, 2
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v32, 0
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v0, v0
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v1, v1
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v2, v2
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v3, v3
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v4, v4
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v5, v5
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v6, v6
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v7, v7
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v8, v8
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v9, v9
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v10, v10
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v11, v11
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v12, v12
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v13, v13
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v14, v14
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v15, v15
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v16, v16
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v17, v17
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v18, v18
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v19, v19
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v20, v20
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v21, v21
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v22, v22
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v23, v23
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v24, v24
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v25, v25
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v26, v26
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v27, v27
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v28, v28
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v29, v29
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v30, v30
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v31, v31
+; GFX12-SDAG-NEXT: s_clause 0x7
+; GFX12-SDAG-NEXT: global_store_b128 v32, v[28:31], s[0:1] offset:112
+; GFX12-SDAG-NEXT: global_store_b128 v32, v[24:27], s[0:1] offset:96
+; GFX12-SDAG-NEXT: global_store_b128 v32, v[20:23], s[0:1] offset:80
+; GFX12-SDAG-NEXT: global_store_b128 v32, v[16:19], s[0:1] offset:64
+; GFX12-SDAG-NEXT: global_store_b128 v32, v[12:15], s[0:1] offset:48
+; GFX12-SDAG-NEXT: global_store_b128 v32, v[8:11], s[0:1] offset:32
+; GFX12-SDAG-NEXT: global_store_b128 v32, v[4:7], s[0:1] offset:16
+; GFX12-SDAG-NEXT: global_store_b128 v32, v[0:3], s[0:1]
+; GFX12-SDAG-NEXT: s_setpc_b64 s[30:31]
+;
+; GFX12-GISEL-LABEL: copy_v32i32:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-GISEL-NEXT: s_wait_expcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: s_wait_bvhcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_kmcnt 0x0
+; GFX12-GISEL-NEXT: s_lshr_b32 m0, s2, 2
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v32, 0
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v0, v0
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v1, v1
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v2, v2
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v3, v3
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v4, v4
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v5, v5
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v6, v6
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v7, v7
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v8, v8
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v9, v9
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v10, v10
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v11, v11
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v12, v12
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v13, v13
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v14, v14
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v15, v15
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v16, v16
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v17, v17
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v18, v18
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v19, v19
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v20, v20
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v21, v21
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v22, v22
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v23, v23
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v24, v24
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v25, v25
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v26, v26
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v27, v27
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v28, v28
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v29, v29
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v30, v30
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v31, v31
+; GFX12-GISEL-NEXT: s_clause 0x7
+; GFX12-GISEL-NEXT: global_store_b128 v32, v[0:3], s[0:1]
+; GFX12-GISEL-NEXT: global_store_b128 v32, v[4:7], s[0:1] offset:16
+; GFX12-GISEL-NEXT: global_store_b128 v32, v[8:11], s[0:1] offset:32
+; GFX12-GISEL-NEXT: global_store_b128 v32, v[12:15], s[0:1] offset:48
+; GFX12-GISEL-NEXT: global_store_b128 v32, v[16:19], s[0:1] offset:64
+; GFX12-GISEL-NEXT: global_store_b128 v32, v[20:23], s[0:1] offset:80
+; GFX12-GISEL-NEXT: global_store_b128 v32, v[24:27], s[0:1] offset:96
+; GFX12-GISEL-NEXT: global_store_b128 v32, v[28:31], s[0:1] offset:112
+; GFX12-GISEL-NEXT: s_setpc_b64 s[30:31]
+ %x = load <32 x i32>, ptr addrspace(13) %in
+ store <32 x i32> %x, ptr addrspace(1) %out
+ ret void
+}
diff --git a/llvm/test/CodeGen/AMDGPU/AddressSpaceVGPR/as-vgpr-divergent.ll b/llvm/test/CodeGen/AMDGPU/AddressSpaceVGPR/as-vgpr-divergent.ll
new file mode 100644
index 0000000000000..28aa0ff7f0fb9
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/AddressSpaceVGPR/as-vgpr-divergent.ll
@@ -0,0 +1,130 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
+; RUN: llc -global-isel=0 -verify-machineinstrs -mtriple=amdgcn -mcpu=gfx1200 -o - %s | FileCheck %s --check-prefixes=GFX12,GFX12-SDAG
+; RUN: llc -global-isel=1 -verify-machineinstrs -mtriple=amdgcn -mcpu=gfx1200 -o - %s | FileCheck %s --check-prefixes=GFX12,GFX12-GISEL
+
+; A divergent (per-lane) dword index into VGPR "as memory" (address space 13)
+; is handled with a waterfall loop: for each unique index across the wave, set
+; M0 and do the M0-relative move under a matching-lane EXEC subset. The pointer
+; arrives in a VGPR (no inreg), so the index (pointer >> 2) is divergent.
+
+define i32 @load_i32(ptr addrspace(13) %p) {
+; GFX12-SDAG-LABEL: load_i32:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-SDAG-NEXT: s_wait_expcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: s_wait_bvhcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_kmcnt 0x0
+; GFX12-SDAG-NEXT: v_lshrrev_b32_e32 v0, 2, v0
+; GFX12-SDAG-NEXT: s_mov_b32 s0, exec_lo
+; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-SDAG-NEXT: s_mov_b32 s1, s0
+; GFX12-SDAG-NEXT: .LBB0_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s2, v0
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e32 s2, v0
+; GFX12-SDAG-NEXT: s_mov_b32 m0, s2
+; GFX12-SDAG-NEXT: v_movrels_b32_e32 v1, v0
+; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-SDAG-NEXT: s_and_not1_wrexec_b32 s1, s1
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr0
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB0_1
+; GFX12-SDAG-NEXT: ; %bb.2:
+; GFX12-SDAG-NEXT: s_mov_b32 exec_lo, s0
+; GFX12-SDAG-NEXT: v_add_nc_u32_e32 v0, 1, v1
+; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-SDAG-NEXT: s_setpc_b64 s[30:31]
+;
+; GFX12-GISEL-LABEL: load_i32:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-GISEL-NEXT: s_wait_expcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: s_wait_bvhcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_kmcnt 0x0
+; GFX12-GISEL-NEXT: v_lshrrev_b32_e32 v0, 2, v0
+; GFX12-GISEL-NEXT: s_mov_b32 s0, exec_lo
+; GFX12-GISEL-NEXT: .LBB0_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s2, v0
+; GFX12-GISEL-NEXT: s_mov_b32 s1, exec_lo
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e32 s2, v0
+; GFX12-GISEL-NEXT: s_mov_b32 m0, s2
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr0
+; GFX12-GISEL-NEXT: v_movrels_b32_e32 v1, v0
+; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-GISEL-NEXT: s_xor_b32 exec_lo, exec_lo, s1
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB0_1
+; GFX12-GISEL-NEXT: ; %bb.2:
+; GFX12-GISEL-NEXT: s_mov_b32 exec_lo, s0
+; GFX12-GISEL-NEXT: v_add_nc_u32_e32 v0, 1, v1
+; GFX12-GISEL-NEXT: s_setpc_b64 s[30:31]
+ %x = load i32, ptr addrspace(13) %p
+ %y = add i32 %x, 1
+ ret i32 %y
+}
+
+define void @store_i32(ptr addrspace(13) %p, i32 %x) {
+; GFX12-SDAG-LABEL: store_i32:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-SDAG-NEXT: s_wait_expcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: s_wait_bvhcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_kmcnt 0x0
+; GFX12-SDAG-NEXT: v_add_nc_u32_e32 v1, 1, v1
+; GFX12-SDAG-NEXT: v_lshrrev_b32_e32 v0, 2, v0
+; GFX12-SDAG-NEXT: s_mov_b32 s0, exec_lo
+; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-SDAG-NEXT: s_mov_b32 s1, s0
+; GFX12-SDAG-NEXT: .LBB1_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s2, v0
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e32 s2, v0
+; GFX12-SDAG-NEXT: s_mov_b32 m0, s2
+; GFX12-SDAG-NEXT: v_movreld_b32_e32 v0, v1
+; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-SDAG-NEXT: s_and_not1_wrexec_b32 s1, s1
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr0
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr1
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB1_1
+; GFX12-SDAG-NEXT: ; %bb.2:
+; GFX12-SDAG-NEXT: s_mov_b32 exec_lo, s0
+; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-SDAG-NEXT: s_setpc_b64 s[30:31]
+;
+; GFX12-GISEL-LABEL: store_i32:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-GISEL-NEXT: s_wait_expcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: s_wait_bvhcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_kmcnt 0x0
+; GFX12-GISEL-NEXT: v_add_nc_u32_e32 v1, 1, v1
+; GFX12-GISEL-NEXT: v_lshrrev_b32_e32 v0, 2, v0
+; GFX12-GISEL-NEXT: s_mov_b32 s0, exec_lo
+; GFX12-GISEL-NEXT: .LBB1_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s2, v0
+; GFX12-GISEL-NEXT: s_mov_b32 s1, exec_lo
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e32 s2, v0
+; GFX12-GISEL-NEXT: s_mov_b32 m0, s2
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr0
+; GFX12-GISEL-NEXT: v_movreld_b32_e32 v0, v1
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr1
+; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-GISEL-NEXT: s_xor_b32 exec_lo, exec_lo, s1
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB1_1
+; GFX12-GISEL-NEXT: ; %bb.2:
+; GFX12-GISEL-NEXT: s_mov_b32 exec_lo, s0
+; GFX12-GISEL-NEXT: s_setpc_b64 s[30:31]
+ %y = add i32 %x, 1
+ store i32 %y, ptr addrspace(13) %p
+ ret void
+}
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GFX12: {{.*}}
diff --git a/llvm/test/CodeGen/AMDGPU/AddressSpaceVGPR/as-vgpr-inttoptr.ll b/llvm/test/CodeGen/AMDGPU/AddressSpaceVGPR/as-vgpr-inttoptr.ll
new file mode 100644
index 0000000000000..393928d359f96
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/AddressSpaceVGPR/as-vgpr-inttoptr.ll
@@ -0,0 +1,48 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
+; RUN: llc -global-isel=0 -verify-machineinstrs -mtriple=amdgcn -mcpu=gfx1200 -o - %s | FileCheck %s --check-prefixes=GFX12,GFX12-SDAG
+; RUN: llc -global-isel=1 -verify-machineinstrs -mtriple=amdgcn -mcpu=gfx1200 -o - %s | FileCheck %s --check-prefixes=GFX12,GFX12-GISEL
+
+; A constant VGPR "as memory" (address space 13) pointer formed with inttoptr:
+; the dword index (256 >> 2 = 64) is a compile-time constant, so M0 is set from
+; that constant and the access is a single M0-relative move.
+
+define i32 @load_i32() {
+; GFX12-LABEL: load_i32:
+; GFX12: ; %bb.0:
+; GFX12-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-NEXT: s_wait_expcnt 0x0
+; GFX12-NEXT: s_wait_samplecnt 0x0
+; GFX12-NEXT: s_wait_bvhcnt 0x0
+; GFX12-NEXT: s_wait_kmcnt 0x0
+; GFX12-NEXT: s_mov_b32 m0, 64
+; GFX12-NEXT: v_movrels_b32_e32 v0, v0
+; GFX12-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-NEXT: v_add_nc_u32_e32 v0, 1, v0
+; GFX12-NEXT: s_setpc_b64 s[30:31]
+ %p = inttoptr i32 256 to ptr addrspace(13)
+ %x = load i32, ptr addrspace(13) %p
+ %y = add i32 %x, 1
+ ret i32 %y
+}
+
+define void @store_i32(i32 %x) {
+; GFX12-LABEL: store_i32:
+; GFX12: ; %bb.0:
+; GFX12-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-NEXT: s_wait_expcnt 0x0
+; GFX12-NEXT: s_wait_samplecnt 0x0
+; GFX12-NEXT: s_wait_bvhcnt 0x0
+; GFX12-NEXT: s_wait_kmcnt 0x0
+; GFX12-NEXT: v_add_nc_u32_e32 v0, 1, v0
+; GFX12-NEXT: s_mov_b32 m0, 64
+; GFX12-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-NEXT: v_movreld_b32_e32 v0, v0
+; GFX12-NEXT: s_setpc_b64 s[30:31]
+ %p = inttoptr i32 256 to ptr addrspace(13)
+ %y = add i32 %x, 1
+ store i32 %y, ptr addrspace(13) %p
+ ret void
+}
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GFX12-GISEL: {{.*}}
+; GFX12-SDAG: {{.*}}
diff --git a/llvm/test/CodeGen/AMDGPU/AddressSpaceVGPR/as-vgpr-unsupported.ll b/llvm/test/CodeGen/AMDGPU/AddressSpaceVGPR/as-vgpr-unsupported.ll
new file mode 100644
index 0000000000000..40263e1c975ad
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/AddressSpaceVGPR/as-vgpr-unsupported.ll
@@ -0,0 +1,31 @@
+; RUN: not llc -global-isel=0 -mtriple=amdgcn -mcpu=gfx1200 -filetype=null %s 2>&1 | FileCheck %s
+; RUN: not llc -global-isel=1 -mtriple=amdgcn -mcpu=gfx1200 -filetype=null %s 2>&1 | FileCheck %s
+
+; Sub-dword (8/16-bit) accesses of the VGPR "as memory" address space (13) are
+; not yet implemented. They must be rejected with a clean diagnostic on both
+; SelectionDAG and GlobalISel, rather than failing with "cannot select" /
+; "unable to legalize".
+
+; CHECK: error: {{.*}}unsupported access of VGPR 'as memory' address space (13); only whole-dword loads and stores are implemented
+define i8 @load_i8(ptr addrspace(13) inreg %p) {
+ %x = load i8, ptr addrspace(13) %p
+ ret i8 %x
+}
+
+; CHECK: error: {{.*}}unsupported access of VGPR 'as memory' address space (13); only whole-dword loads and stores are implemented
+define i16 @load_i16(ptr addrspace(13) inreg %p) {
+ %x = load i16, ptr addrspace(13) %p
+ ret i16 %x
+}
+
+; CHECK: error: {{.*}}unsupported access of VGPR 'as memory' address space (13); only whole-dword loads and stores are implemented
+define void @store_i8(ptr addrspace(13) inreg %p, i8 %v) {
+ store i8 %v, ptr addrspace(13) %p
+ ret void
+}
+
+; CHECK: error: {{.*}}unsupported access of VGPR 'as memory' address space (13); only whole-dword loads and stores are implemented
+define void @store_i16(ptr addrspace(13) inreg %p, i16 %v) {
+ store i16 %v, ptr addrspace(13) %p
+ ret void
+}
diff --git a/llvm/test/CodeGen/AMDGPU/llc-pipeline-npm.ll b/llvm/test/CodeGen/AMDGPU/llc-pipeline-npm.ll
index 865058fa1d92c..63eef9ca10f89 100644
--- a/llvm/test/CodeGen/AMDGPU/llc-pipeline-npm.ll
+++ b/llvm/test/CodeGen/AMDGPU/llc-pipeline-npm.ll
@@ -69,6 +69,7 @@
; GCN-O0-NEXT: function
; GCN-O0-NEXT: machine-function
; GCN-O0-NEXT: reg-usage-propagation
+; GCN-O0-NEXT: amdgpu-assign-idx-to-m0
; GCN-O0-NEXT: phi-node-elimination
; GCN-O0-NEXT: si-lower-control-flow
; GCN-O0-NEXT: two-address-instruction
@@ -218,6 +219,7 @@
; GCN-O2-NEXT: function
; GCN-O2-NEXT: machine-function
; GCN-O2-NEXT: reg-usage-propagation
+; GCN-O2-NEXT: amdgpu-assign-idx-to-m0
; GCN-O2-NEXT: amdgpu-prepare-agpr-alloc
; GCN-O2-NEXT: detect-dead-lanes
; GCN-O2-NEXT: dead-mi-elimination
@@ -407,6 +409,7 @@
; GCN-O3-NEXT: function
; GCN-O3-NEXT: machine-function
; GCN-O3-NEXT: reg-usage-propagation
+; GCN-O3-NEXT: amdgpu-assign-idx-to-m0
; GCN-O3-NEXT: amdgpu-prepare-agpr-alloc
; GCN-O3-NEXT: detect-dead-lanes
; GCN-O3-NEXT: dead-mi-elimination
diff --git a/llvm/test/CodeGen/AMDGPU/llc-pipeline.ll b/llvm/test/CodeGen/AMDGPU/llc-pipeline.ll
index 97cf8eecf3c1f..e99ab7e996fb6 100644
--- a/llvm/test/CodeGen/AMDGPU/llc-pipeline.ll
+++ b/llvm/test/CodeGen/AMDGPU/llc-pipeline.ll
@@ -113,6 +113,7 @@
; GCN-O0-NEXT: Finalize ISel and expand pseudo-instructions
; GCN-O0-NEXT: Local Stack Slot Allocation
; GCN-O0-NEXT: Register Usage Information Propagation
+; GCN-O0-NEXT: AMDGPU Assign Idx To M0
; GCN-O0-NEXT: Eliminate PHI nodes for register allocation
; GCN-O0-NEXT: SI Lower control flow pseudo instructions
; GCN-O0-NEXT: Two-Address instruction pass
@@ -351,6 +352,7 @@
; GCN-O1-NEXT: Remove dead machine instructions
; GCN-O1-NEXT: SI Shrink Instructions
; GCN-O1-NEXT: Register Usage Information Propagation
+; GCN-O1-NEXT: AMDGPU Assign Idx To M0
; GCN-O1-NEXT: AMDGPU Prepare AGPR Alloc
; GCN-O1-NEXT: Detect Dead Lanes
; GCN-O1-NEXT: Remove dead machine instructions
@@ -673,6 +675,7 @@
; GCN-O1-OPTS-NEXT: Remove dead machine instructions
; GCN-O1-OPTS-NEXT: SI Shrink Instructions
; GCN-O1-OPTS-NEXT: Register Usage Information Propagation
+; GCN-O1-OPTS-NEXT: AMDGPU Assign Idx To M0
; GCN-O1-OPTS-NEXT: AMDGPU Prepare AGPR Alloc
; GCN-O1-OPTS-NEXT: Detect Dead Lanes
; GCN-O1-OPTS-NEXT: Remove dead machine instructions
@@ -999,6 +1002,7 @@
; GCN-O2-NEXT: Remove dead machine instructions
; GCN-O2-NEXT: SI Shrink Instructions
; GCN-O2-NEXT: Register Usage Information Propagation
+; GCN-O2-NEXT: AMDGPU Assign Idx To M0
; GCN-O2-NEXT: AMDGPU Prepare AGPR Alloc
; GCN-O2-NEXT: Detect Dead Lanes
; GCN-O2-NEXT: Remove dead machine instructions
@@ -1339,6 +1343,7 @@
; GCN-O3-NEXT: Remove dead machine instructions
; GCN-O3-NEXT: SI Shrink Instructions
; GCN-O3-NEXT: Register Usage Information Propagation
+; GCN-O3-NEXT: AMDGPU Assign Idx To M0
; GCN-O3-NEXT: AMDGPU Prepare AGPR Alloc
; GCN-O3-NEXT: Detect Dead Lanes
; GCN-O3-NEXT: Remove dead machine instructions
More information about the cfe-commits
mailing list