[llvm] [AMDGPU] Handle ds_write2 in AMDGPURewriteAGPRCopyMFMA (PR #215791)
Ketan Roy via llvm-commits
llvm-commits at lists.llvm.org
Wed Aug 12 05:44:19 PDT 2026
https://github.com/MrRoy09 created https://github.com/llvm/llvm-project/pull/215791
Modified the pass to treat any instruction with an AGPRFormTable entry as
a candidate for rewrite and to derive the coupled operand set from the
AGPR form's operand list.
Tagged ds_write2 to be part of the AGPRFormTable so it can benefit from
this rewrite. Its two data operands share a single acc bit and hence must
both be VGPRs or both be AGPRs.
Separately, tryFoldCopiesFromAGPR passed the copy destination's physreg,
a VGPR, as the preferred AGPR, so the hint never applied. AssignedAGPR
was already an unused parameter. ds_write2 always takes this path, so
without the fix the rewrite did not reuse the AGPR already assigned to
the copy source, which in turn affected spill slot assignment.
Assisted by Claude Opus 5
Fixes #168983
Related to #155769
>From a35d331d53baee9d29023a2d02f6715ed31fb156 Mon Sep 17 00:00:00 2001
From: MrRoy09 <royketan09 at gmail.com>
Date: Wed, 12 Aug 2026 17:04:18 +0530
Subject: [PATCH] [AMDGPU] Handle ds_write2 in AMDGPURewriteAGPRCopyMFMA
Modified the pass to treat any instruction with an AGPRFormTable entry as
a candidate for rewrite and to derive the coupled operand set from the
AGPR form's operand list.
Tagged ds_write2 to be part of the AGPRFormTable so it can benefit from
this rewrite. Its two data operands share a single acc bit and hence must
both be VGPRs or both be AGPRs.
Separately, tryFoldCopiesFromAGPR passed the copy destination's physreg,
a VGPR, as the preferred AGPR, so the hint never applied. AssignedAGPR
was already an unused parameter. ds_write2 always takes this path, so
without the fix the rewrite did not reuse the AGPR already assigned to
the copy source, which in turn affected spill slot assignment.
Assisted by Claude Opus 5
Fixes #168983
Related to #155769
---
.../AMDGPU/AMDGPURewriteAGPRCopyMFMA.cpp | 182 +++++++++---------
llvm/lib/Target/AMDGPU/DSInstructions.td | 6 +-
llvm/test/CodeGen/AMDGPU/ds_write2_a_v.ll | 10 +-
.../AMDGPU/rewrite-vgpr-ds-write2-to-agpr.mir | 153 +++++++++++++++
4 files changed, 250 insertions(+), 101 deletions(-)
create mode 100644 llvm/test/CodeGen/AMDGPU/rewrite-vgpr-ds-write2-to-agpr.mir
diff --git a/llvm/lib/Target/AMDGPU/AMDGPURewriteAGPRCopyMFMA.cpp b/llvm/lib/Target/AMDGPU/AMDGPURewriteAGPRCopyMFMA.cpp
index 6510c07358e01..57e840e9eeeea 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPURewriteAGPRCopyMFMA.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPURewriteAGPRCopyMFMA.cpp
@@ -6,12 +6,13 @@
//
//===----------------------------------------------------------------------===//
//
-/// \file \brief Try to replace MFMA instructions using VGPRs with MFMA
-/// instructions using AGPRs. We expect MFMAs to be selected using VGPRs, and
-/// only use AGPRs if it helps avoid spilling. In this case, the MFMA will have
-/// copies between AGPRs and VGPRs and the AGPR variant of an MFMA pseudo. This
-/// pass will attempt to delete the cross register bank copy and replace the
-/// MFMA opcode.
+/// \file \brief Try to replace instructions using VGPRs with equivalent
+/// instructions using AGPRs. Some instructions have paired pseudos which
+/// differ only in the register bank of certain operands, such as MFMA and
+/// ds_write2. We expect these to be selected using VGPRs, and only use AGPRs
+/// if it helps avoid spilling. In this case, the instruction will have copies
+/// between AGPRs and VGPRs and the AGPR variant of the pseudo. This pass will
+/// attempt to delete the cross register bank copy and replace the opcode.
///
/// TODO:
/// - Handle rewrites of phis. This must be more careful than normal about the
@@ -42,12 +43,12 @@ using namespace llvm;
#define DEBUG_TYPE "amdgpu-rewrite-agpr-copy-mfma"
DEBUG_COUNTER(RewriteAGPRCopyMFMACounter, DEBUG_TYPE,
- "Controls which MFMA chains are rewritten to AGPR form");
+ "Controls which chains are rewritten to AGPR form");
namespace {
-STATISTIC(NumMFMAsRewrittenToAGPR,
- "Number of MFMA instructions rewritten to use AGPR form");
+STATISTIC(NumInstsRewrittenToAGPR,
+ "Number of instructions rewritten to use AGPR form");
/// Map from spill slot frame index to list of instructions which reference it.
using SpillReferenceMap = DenseMap<int, SmallVector<MachineInstr *, 4>>;
@@ -77,7 +78,7 @@ class AMDGPURewriteAGPRCopyMFMAImpl {
LIS(LIS), LSS(LSS), RegClassInfo(RegClassInfo) {}
bool isRewriteCandidate(const MachineInstr &MI) const {
- return TII.isMAI(MI) && AMDGPU::getAGPRFormOp(MI.getOpcode()) != -1;
+ return AMDGPU::getAGPRFormOp(MI.getOpcode()) != -1;
}
/// Find AV_* registers assigned to AGPRs (or virtual registers which were
@@ -96,19 +97,20 @@ class AMDGPURewriteAGPRCopyMFMAImpl {
return TRI.isAGPRClass(AssignedRC) ? PhysReg : MCRegister();
}
- bool tryReassigningMFMAChain(MachineInstr &MFMA, Register MFMAHintReg,
- MCPhysReg PhysRegHint) const;
+ bool tryReassigningChain(MachineInstr &MI, Register HintReg,
+ MCPhysReg PhysRegHint) const;
/// Compute the register class constraints based on the uses of \p Reg,
- /// excluding MFMA uses from which can be rewritten to change the register
- /// class constraint. MFMA scale operands need to be constraint checked.
- /// This should be nearly identical to MachineRegisterInfo::recomputeRegClass.
-
- /// \p RewriteCandidates will collect the set of MFMA instructions that need
+ /// excluding uses from instructions which can be rewritten to change the
+ /// register class constraint. MFMA scale operands need to be constraint
+ /// checked. This should be nearly identical to
+ /// MachineRegisterInfo::recomputeRegClass.
+ ///
+ /// \p RewriteCandidates will collect the set of instructions that need
/// to have the opcode mutated to perform the replacement.
///
- /// \p RewriteRegs will accumulate the set of register used by those MFMAs
- /// that need to have the register classes adjusted.
+ /// \p RewriteRegs will accumulate the set of registers used by those
+ /// instructions that need to have the register classes adjusted.
bool recomputeRegClassExceptRewritable(
Register Reg, SmallVectorImpl<MachineInstr *> &RewriteCandidates,
SmallSetVector<Register, 4> &RewriteRegs) const;
@@ -139,7 +141,7 @@ bool AMDGPURewriteAGPRCopyMFMAImpl::recomputeRegClassExceptRewritable(
SmallSetVector<Register, 4> &RewriteRegs) const {
SmallVector<Register, 8> Worklist = {StartReg};
- // Recursively visit all transitive MFMA users
+ // Recursively visit all transitive users
while (!Worklist.empty()) {
Register Reg = Worklist.pop_back_val();
const TargetRegisterClass *OldRC = MRI.getRegClass(Reg);
@@ -154,48 +156,46 @@ bool AMDGPURewriteAGPRCopyMFMAImpl::recomputeRegClassExceptRewritable(
// Apply the effect of the given operand to NewRC.
MachineInstr *MI = MO.getParent();
- // We can swap the classes of dst + src2 as a pair to AGPR, so ignore the
- // effects of rewrite candidates. It just so happens that we can use
- // either AGPR or VGPR in src0/src1. We still need to check constraint
- // effects for scale variant, which does not allow AGPR.
+ // We can swap the classes of the coupled operands to AGPR together
+ // ({vdst, src2} for MFMA, {data0, data1} for ds_write2), so ignore the
+ // effects of rewrite candidates. We still need to check constraint
+ // effects for operands which do not allow AGPR.
if (isRewriteCandidate(*MI)) {
int AGPROp = AMDGPU::getAGPRFormOp(MI->getOpcode());
const MCInstrDesc &AGPRDesc = TII.get(AGPROp);
- const TargetRegisterClass *NewRC =
+ assert(MI->getDesc().getNumOperands() == AGPRDesc.getNumOperands() &&
+ "AGPR form must have the same operand list");
+
+ const TargetRegisterClass *AGPRFormRC =
TII.getRegClass(AGPRDesc, MO.getOperandNo());
- if (!TRI.hasAGPRs(NewRC))
+ if (!TRI.hasAGPRs(AGPRFormRC))
return false;
- const MachineOperand *VDst =
- TII.getNamedOperand(*MI, AMDGPU::OpName::vdst);
- const MachineOperand *Src2 =
- TII.getNamedOperand(*MI, AMDGPU::OpName::src2);
- for (const MachineOperand *Op : {VDst, Src2}) {
- if (!Op->isReg())
+ SmallVector<Register, 4> CoupledRegs;
+ for (unsigned I = 0, E = AGPRDesc.getNumOperands(); I != E; ++I) {
+ const TargetRegisterClass *OpRC = TII.getRegClass(AGPRDesc, I);
+ if (!OpRC || !TRI.isAGPRClass(OpRC))
continue;
-
- Register OtherReg = Op->getReg();
- if (OtherReg.isPhysical())
+ const MachineOperand &Op = MI->getOperand(I);
+ if (!Op.isReg())
+ continue;
+ Register CoupledReg = Op.getReg();
+ if (CoupledReg.isPhysical())
return false;
-
- if (OtherReg != Reg && RewriteRegs.insert(OtherReg))
- Worklist.push_back(OtherReg);
+ CoupledRegs.push_back(CoupledReg);
+ if (CoupledReg != Reg && RewriteRegs.insert(CoupledReg))
+ Worklist.push_back(CoupledReg);
}
if (!is_contained(RewriteCandidates, MI)) {
LLVM_DEBUG({
- Register VDstPhysReg = VRM.getPhys(VDst->getReg());
- dbgs() << "Attempting to replace VGPR MFMA with AGPR version:"
- << " Dst=[" << printReg(VDst->getReg()) << " => "
- << printReg(VDstPhysReg, &TRI);
-
- if (Src2->isReg()) {
- Register Src2PhysReg = VRM.getPhys(Src2->getReg());
- dbgs() << "], Src2=[" << printReg(Src2->getReg(), &TRI) << " => "
- << printReg(Src2PhysReg, &TRI);
- }
-
- dbgs() << "]: " << MI;
+ dbgs() << "Attempting to replace VGPR instruction with AGPR "
+ "version: [";
+ ListSeparator LS;
+ for (Register CoupledReg : CoupledRegs)
+ dbgs() << LS << printReg(CoupledReg, &TRI) << " => "
+ << printReg(VRM.getPhys(CoupledReg), &TRI);
+ dbgs() << "]: " << *MI;
});
RewriteCandidates.push_back(MI);
@@ -219,48 +219,49 @@ bool AMDGPURewriteAGPRCopyMFMAImpl::recomputeRegClassExceptRewritable(
return true;
}
-bool AMDGPURewriteAGPRCopyMFMAImpl::tryReassigningMFMAChain(
- MachineInstr &MFMA, Register MFMAHintReg, MCPhysReg PhysRegHint) const {
- // src2 and dst have the same physical class constraint; try to preserve
- // the original src2 subclass if one were to exist.
- SmallVector<MachineInstr *, 4> RewriteCandidates = {&MFMA};
+bool AMDGPURewriteAGPRCopyMFMAImpl::tryReassigningChain(
+ MachineInstr &MI, Register HintReg, MCPhysReg PhysRegHint) const {
+ // The coupled operands have the same physical class constraint; try to
+ // preserve the original subclass if one were to exist.
+ SmallVector<MachineInstr *, 4> RewriteCandidates = {&MI};
SmallSetVector<Register, 4> RewriteRegs;
- // Make sure we reassign the MFMA we found the copy from first. We want
- // to ensure dst ends up in the physreg we were originally copying to.
- RewriteRegs.insert(MFMAHintReg);
-
- // We've found av = COPY (MFMA) (or MFMA (v = COPY av)) and need to verify
- // that we can trivially rewrite src2 to use the new AGPR. If we can't
- // trivially replace it, we're going to induce as many copies as we would have
- // emitted in the first place, as well as need to assign another register, and
- // need to figure out where to put them. The live range splitting is smarter
- // than anything we're doing here, so trust it did something reasonable.
+ // Make sure we reassign the register we found the copy from first. We want
+ // to ensure it ends up in the physreg we were originally copying to.
+ RewriteRegs.insert(HintReg);
+
+ // We've found av = COPY (candidate) (or candidate (v = COPY av)) and need to
+ // verify that we can trivially rewrite the coupled operands to use the new
+ // AGPR. If we can't trivially replace them, we're going to induce as many
+ // copies as we would have emitted in the first place, as well as need to
+ // assign another register, and need to figure out where to put them. The
+ // live range splitting is smarter than anything we're doing here, so trust
+ // it did something reasonable.
//
// Note recomputeRegClassExceptRewritable will consider the constraints of
- // this MFMA's src2 as well as the src2/dst of any transitive MFMA users.
- if (!recomputeRegClassExceptRewritable(MFMAHintReg, RewriteCandidates,
+ // this instruction's coupled operands as well as those of any transitive
+ // users.
+ if (!recomputeRegClassExceptRewritable(HintReg, RewriteCandidates,
RewriteRegs)) {
LLVM_DEBUG(dbgs() << "Could not recompute the regclass of dst reg "
- << printReg(MFMAHintReg, &TRI) << '\n');
+ << printReg(HintReg, &TRI) << '\n');
return false;
}
- // If src2 and dst are different registers, we need to also reassign the
- // input to an available AGPR if it is compatible with all other uses.
+ // If the coupled operands are different registers, we need to also reassign
+ // the input to an available AGPR if it is compatible with all other uses.
//
// If we can't reassign it, we'd need to introduce a different copy
// which is likely worse than the copy we'd be saving.
//
- // It's likely that the MFMA is used in sequence with other MFMAs; if we
- // cannot migrate the full use/def chain of MFMAs, we would need to
- // introduce intermediate copies somewhere. So we only make the
- // transform if all the interfering MFMAs can also be migrated. Collect
- // the set of rewritable MFMAs and check if we can assign an AGPR at
- // that point.
+ // It's likely that the instruction is used in sequence with others; if we
+ // cannot migrate the full use/def chain, we would need to introduce
+ // intermediate copies somewhere. So we only make the transform if all the
+ // interfering instructions can also be migrated. Collect the set of
+ // rewritable instructions and check if we can assign an AGPR at that point.
//
- // If any of the MFMAs aren't reassignable, we give up and rollback to
- // the original register assignments.
+ // If any of them aren't reassignable, we give up and rollback to the
+ // original register assignments.
using RecoloringStack =
SmallVector<std::pair<const LiveInterval *, MCRegister>, 8>;
@@ -293,9 +294,9 @@ bool AMDGPURewriteAGPRCopyMFMAImpl::tryReassigningMFMAChain(
}
for (MachineInstr *RewriteCandidate : RewriteCandidates) {
- int NewMFMAOp = AMDGPU::getAGPRFormOp(RewriteCandidate->getOpcode());
- RewriteCandidate->setDesc(TII.get(NewMFMAOp));
- ++NumMFMAsRewrittenToAGPR;
+ int AGPROp = AMDGPU::getAGPRFormOp(RewriteCandidate->getOpcode());
+ RewriteCandidate->setDesc(TII.get(AGPROp));
+ ++NumInstsRewrittenToAGPR;
}
return true;
@@ -358,7 +359,7 @@ bool AMDGPURewriteAGPRCopyMFMAImpl::attemptReassignmentsToAGPR(
/// %agpr = COPY %vgpr
///
/// Then try to replace the transitive uses of %src2 and %vdst with the AGPR
-/// versions of the MFMA. This should cover the common case.
+/// versions of the instruction. This should cover the common case.
bool AMDGPURewriteAGPRCopyMFMAImpl::tryFoldCopiesToAGPR(
Register VReg, MCRegister AssignedAGPR) const {
bool MadeChange = false;
@@ -385,8 +386,8 @@ bool AMDGPURewriteAGPRCopyMFMAImpl::tryFoldCopiesToAGPR(
for (MachineInstr &CopySrcDefMI : MRI.def_instructions(CopySrcReg)) {
if (isRewriteCandidate(CopySrcDefMI) &&
- tryReassigningMFMAChain(
- CopySrcDefMI, CopySrcDefMI.getOperand(0).getReg(), AssignedAGPR))
+ tryReassigningChain(CopySrcDefMI, CopySrcDefMI.getOperand(0).getReg(),
+ AssignedAGPR))
MadeChange = true;
}
}
@@ -399,8 +400,8 @@ bool AMDGPURewriteAGPRCopyMFMAImpl::tryFoldCopiesToAGPR(
/// %vdst:vgpr = V_MFMA_... %src0:av, %src1:av, %src:vgpr
///
/// Then try to replace the transitive uses of %src2 and %vdst with the AGPR
-/// versions of the MFMA. This should cover rarer cases, and will generally be
-/// redundant with tryFoldCopiesToAGPR.
+/// versions of the instruction. This should cover rarer cases, and will
+/// generally be redundant with tryFoldCopiesToAGPR.
bool AMDGPURewriteAGPRCopyMFMAImpl::tryFoldCopiesFromAGPR(
Register VReg, MCRegister AssignedAGPR) const {
bool MadeChange = false;
@@ -417,8 +418,7 @@ bool AMDGPURewriteAGPRCopyMFMAImpl::tryFoldCopiesFromAGPR(
MachineInstr &CopyUseMI = *CopyUseMO.getParent();
if (isRewriteCandidate(CopyUseMI)) {
- if (tryReassigningMFMAChain(CopyUseMI, CopyDstReg,
- VRM.getPhys(CopyDstReg)))
+ if (tryReassigningChain(CopyUseMI, CopyDstReg, AssignedAGPR))
MadeChange = true;
}
}
@@ -605,9 +605,9 @@ bool AMDGPURewriteAGPRCopyMFMAImpl::run(MachineFunction &MF) const {
MadeChange = true;
}
- // If we've successfully rewritten some MFMAs, we've alleviated some VGPR
- // pressure. See if we can eliminate some spills now that those registers are
- // more available.
+ // If we've successfully rewritten some instructions, we've alleviated some
+ // VGPR pressure. See if we can eliminate some spills now that those
+ // registers are more available.
if (MadeChange)
eliminateSpillsOfReassignedVGPRs();
diff --git a/llvm/lib/Target/AMDGPU/DSInstructions.td b/llvm/lib/Target/AMDGPU/DSInstructions.td
index 90713dfc1de7b..dbb38f2f90d81 100644
--- a/llvm/lib/Target/AMDGPU/DSInstructions.td
+++ b/llvm/lib/Target/AMDGPU/DSInstructions.td
@@ -205,10 +205,12 @@ multiclass DS_1A2D_Off8_NORET_mc <string opName, RegisterOperand rc = VGPROp_32>
def "" : DS_1A2D_Off8_NORET<opName, rc>;
let has_m0_read = 0 in {
- def _gfx9 : DS_1A2D_Off8_NORET<opName, rc>;
+ def _gfx9 : DS_1A2D_Off8_NORET<opName, rc>,
+ AGPRFormTable<"VGPR", NAME # "_agpr">;
let SubtargetPredicate = isGFX90APlus in {
- def _agpr : DS_1A2D_Off8_NORET<opName, getEquivalentAGPROperand<rc>.ret>;
+ def _agpr : DS_1A2D_Off8_NORET<opName, getEquivalentAGPROperand<rc>.ret>,
+ AGPRFormTable<"AGPR">;
}
}
}
diff --git a/llvm/test/CodeGen/AMDGPU/ds_write2_a_v.ll b/llvm/test/CodeGen/AMDGPU/ds_write2_a_v.ll
index f1da690dd6fb9..e3e7c2673fda7 100644
--- a/llvm/test/CodeGen/AMDGPU/ds_write2_a_v.ll
+++ b/llvm/test/CodeGen/AMDGPU/ds_write2_a_v.ll
@@ -474,9 +474,7 @@ define void @ds_write2_b32_av_av_no_vgprs(ptr addrspace(3) %lds) #0 {
; GCN-NEXT: buffer_store_dword v18, off, s[0:3], s32 offset:72 ; 4-byte Folded Spill
; GCN-NEXT: v_accvgpr_write_b32 a19, v31 ; Reload Reuse
; GCN-NEXT: v_accvgpr_read_b32 v0, a0
-; GCN-NEXT: v_accvgpr_read_b32 v1, a1
-; GCN-NEXT: v_accvgpr_read_b32 v2, a2
-; GCN-NEXT: ds_write2_b32 v0, v1, v2 offset0:10 offset1:24
+; GCN-NEXT: ds_write2_b32 v0, a1, a2 offset0:10 offset1:24
; GCN-NEXT: v_accvgpr_write_b32 a31, v19 ; Reload Reuse
; GCN-NEXT: v_accvgpr_write_b32 a30, v20 ; Reload Reuse
; GCN-NEXT: v_accvgpr_write_b32 a29, v21 ; Reload Reuse
@@ -1025,11 +1023,7 @@ define void @ds_write2_b64_av_av_no_vgprs(ptr addrspace(3) %lds) #0 {
; GCN-NEXT: buffer_store_dword v20, off, s[0:3], s32 offset:80 ; 4-byte Folded Spill
; GCN-NEXT: v_accvgpr_write_b32 a21, v31 ; Reload Reuse
; GCN-NEXT: v_accvgpr_read_b32 v0, a0
-; GCN-NEXT: v_accvgpr_read_b32 v2, a2
-; GCN-NEXT: v_accvgpr_read_b32 v3, a3
-; GCN-NEXT: v_accvgpr_read_b32 v4, a4
-; GCN-NEXT: v_accvgpr_read_b32 v5, a5
-; GCN-NEXT: ds_write2_b64 v0, v[2:3], v[4:5] offset0:10 offset1:24
+; GCN-NEXT: ds_write2_b64 v0, a[2:3], a[4:5] offset0:10 offset1:24
; GCN-NEXT: v_accvgpr_write_b32 a31, v21 ; Reload Reuse
; GCN-NEXT: v_accvgpr_write_b32 a30, v22 ; Reload Reuse
; GCN-NEXT: v_accvgpr_write_b32 a29, v23 ; Reload Reuse
diff --git a/llvm/test/CodeGen/AMDGPU/rewrite-vgpr-ds-write2-to-agpr.mir b/llvm/test/CodeGen/AMDGPU/rewrite-vgpr-ds-write2-to-agpr.mir
new file mode 100644
index 0000000000000..8e406c9045f37
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/rewrite-vgpr-ds-write2-to-agpr.mir
@@ -0,0 +1,153 @@
+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py UTC_ARGS: --version 6
+# RUN: llc -mtriple=amdgpu9.42-amd-amdhsa -run-pass=greedy,amdgpu-rewrite-agpr-copy-mfma -verify-machineinstrs -o - %s | FileCheck %s
+
+# The two data operands of ds_write2 share a single acc bit, so they must
+# both be VGPRs or both be AGPRs. Rewriting to the _agpr form therefore has
+# to conscript the second data operand even when nothing about it required
+# an AGPR.
+
+# %2 reaches the write2 as a copy from an AGPR, so it is the start of the
+# chain. %3 is an unrelated VGPR with no AGPR affinity; it must be dragged
+# into the rewrite anyway, or the _agpr form would have a VGPR in an
+# AGPR-only operand.
+---
+name: ds_write2_b32_coupled_data1_conscripted
+tracksRegLiveness: true
+body: |
+ bb.0:
+ liveins: $vgpr0, $vgpr1, $vgpr2
+
+ ; CHECK-LABEL: name: ds_write2_b32_coupled_data1_conscripted
+ ; CHECK: liveins: $vgpr0, $vgpr1, $vgpr2
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
+ ; CHECK-NEXT: [[COPY1:%[0-9]+]]:agpr_32 = COPY $vgpr1
+ ; CHECK-NEXT: [[COPY2:%[0-9]+]]:agpr_32 = COPY [[COPY1]]
+ ; CHECK-NEXT: [[COPY3:%[0-9]+]]:agpr_32 = COPY $vgpr2
+ ; CHECK-NEXT: DS_WRITE2_B32_agpr [[COPY]], [[COPY2]], [[COPY3]], 0, 4, 0, implicit $exec
+ ; CHECK-NEXT: SI_RETURN
+ %0:vgpr_32 = COPY $vgpr0
+ %1:agpr_32 = COPY $vgpr1
+ %2:vgpr_32 = COPY %1
+ %3:vgpr_32 = COPY $vgpr2
+ DS_WRITE2_B32_gfx9 %0, %2, %3, 0, 4, 0, implicit $exec
+ SI_RETURN
+...
+
+# Same function, except the conscripted operand %3 is pinned to a VGPR by an
+# inline asm use. The rewrite must be abandoned entirely: there is no form of
+# ds_write2 that can take an AGPR in data0 and a VGPR in data1.
+---
+name: ds_write2_b32_coupled_data1_unrewritable_use
+tracksRegLiveness: true
+body: |
+ bb.0:
+ liveins: $vgpr0, $vgpr1, $vgpr2
+
+ ; CHECK-LABEL: name: ds_write2_b32_coupled_data1_unrewritable_use
+ ; CHECK: liveins: $vgpr0, $vgpr1, $vgpr2
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
+ ; CHECK-NEXT: [[COPY1:%[0-9]+]]:agpr_32 = COPY $vgpr1
+ ; CHECK-NEXT: [[COPY2:%[0-9]+]]:vgpr_32 = COPY [[COPY1]]
+ ; CHECK-NEXT: [[COPY3:%[0-9]+]]:vgpr_32 = COPY $vgpr2
+ ; CHECK-NEXT: DS_WRITE2_B32_gfx9 [[COPY]], [[COPY2]], [[COPY3]], 0, 4, 0, implicit $exec
+ ; CHECK-NEXT: INLINEASM &"; use $0", sideeffect attdialect, reguse:VGPR_32, [[COPY3]]
+ ; CHECK-NEXT: SI_RETURN
+ %0:vgpr_32 = COPY $vgpr0
+ %1:agpr_32 = COPY $vgpr1
+ %2:vgpr_32 = COPY %1
+ %3:vgpr_32 = COPY $vgpr2
+ DS_WRITE2_B32_gfx9 %0, %2, %3, 0, 4, 0, implicit $exec
+ INLINEASM &"; use $0", sideeffect attdialect, reguse:VGPR_32, %3
+ SI_RETURN
+...
+
+# Here the AGPR reaches the $addr operand rather than a data operand. $addr is
+# VGPR_32 in both forms, so the _agpr form offers nowhere for the register to
+# go and the chain must be abandoned before any operand is collected. Rewriting
+# anyway would leave an AGPR in a VGPR-only operand.
+---
+name: ds_write2_b32_agpr_reaches_addr_operand
+tracksRegLiveness: true
+body: |
+ bb.0:
+ liveins: $vgpr0, $vgpr1, $vgpr2
+
+ ; CHECK-LABEL: name: ds_write2_b32_agpr_reaches_addr_operand
+ ; CHECK: liveins: $vgpr0, $vgpr1, $vgpr2
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: [[COPY:%[0-9]+]]:agpr_32 = COPY $vgpr0
+ ; CHECK-NEXT: [[COPY1:%[0-9]+]]:vgpr_32 = COPY [[COPY]]
+ ; CHECK-NEXT: [[COPY2:%[0-9]+]]:vgpr_32 = COPY $vgpr1
+ ; CHECK-NEXT: [[COPY3:%[0-9]+]]:vgpr_32 = COPY $vgpr2
+ ; CHECK-NEXT: DS_WRITE2_B32_gfx9 [[COPY1]], [[COPY2]], [[COPY3]], 0, 4, 0, implicit $exec
+ ; CHECK-NEXT: SI_RETURN
+ %0:agpr_32 = COPY $vgpr0
+ %1:vgpr_32 = COPY %0
+ %2:vgpr_32 = COPY $vgpr1
+ %3:vgpr_32 = COPY $vgpr2
+ DS_WRITE2_B32_gfx9 %1, %2, %3, 0, 4, 0, implicit $exec
+ SI_RETURN
+...
+
+# The stride-64 form shares the multiclass, and so the coupling, with the
+# plain form.
+---
+name: ds_write2st64_b32_coupled_data1_conscripted
+tracksRegLiveness: true
+body: |
+ bb.0:
+ liveins: $vgpr0, $vgpr1, $vgpr2
+
+ ; CHECK-LABEL: name: ds_write2st64_b32_coupled_data1_conscripted
+ ; CHECK: liveins: $vgpr0, $vgpr1, $vgpr2
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
+ ; CHECK-NEXT: [[COPY1:%[0-9]+]]:agpr_32 = COPY $vgpr1
+ ; CHECK-NEXT: [[COPY2:%[0-9]+]]:agpr_32 = COPY [[COPY1]]
+ ; CHECK-NEXT: [[COPY3:%[0-9]+]]:agpr_32 = COPY $vgpr2
+ ; CHECK-NEXT: DS_WRITE2ST64_B32_agpr [[COPY]], [[COPY2]], [[COPY3]], 0, 1, 0, implicit $exec
+ ; CHECK-NEXT: SI_RETURN
+ %0:vgpr_32 = COPY $vgpr0
+ %1:agpr_32 = COPY $vgpr1
+ %2:vgpr_32 = COPY %1
+ %3:vgpr_32 = COPY $vgpr2
+ DS_WRITE2ST64_B32_gfx9 %0, %2, %3, 0, 1, 0, implicit $exec
+ SI_RETURN
+...
+
+# One chain spanning both instruction families: the MFMA result is the data0
+# operand of the write2, with no copy in between. Before ds_write2 became a
+# rewrite candidate its VGPR-only data operands were a hard constraint, so the
+# MFMA could not be rewritten either. Now both are rewritten together, and the
+# unrelated data1 operand %6 is conscripted along with them.
+---
+name: mfma_to_ds_write2_b64_chain
+tracksRegLiveness: true
+body: |
+ bb.0:
+ liveins: $vgpr0_vgpr1, $vgpr2_vgpr3, $vgpr4_vgpr5, $vgpr6_vgpr7, $vgpr8
+
+ ; CHECK-LABEL: name: mfma_to_ds_write2_b64_chain
+ ; CHECK: liveins: $vgpr0_vgpr1, $vgpr2_vgpr3, $vgpr4_vgpr5, $vgpr6_vgpr7, $vgpr8
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr8
+ ; CHECK-NEXT: [[COPY1:%[0-9]+]]:av_64_align2 = COPY $vgpr0_vgpr1
+ ; CHECK-NEXT: [[COPY2:%[0-9]+]]:av_64_align2 = COPY $vgpr2_vgpr3
+ ; CHECK-NEXT: [[COPY3:%[0-9]+]]:areg_64_align2 = COPY $vgpr4_vgpr5
+ ; CHECK-NEXT: [[COPY4:%[0-9]+]]:areg_64_align2 = COPY [[COPY3]]
+ ; CHECK-NEXT: [[V_MFMA_F64_4X4X4F64_e64_:%[0-9]+]]:areg_64_align2 = V_MFMA_F64_4X4X4F64_e64 [[COPY1]], [[COPY2]], [[COPY4]], 0, 0, 0, implicit $mode, implicit $exec
+ ; CHECK-NEXT: [[COPY5:%[0-9]+]]:areg_64_align2 = COPY $vgpr6_vgpr7
+ ; CHECK-NEXT: DS_WRITE2_B64_agpr [[COPY]], [[V_MFMA_F64_4X4X4F64_e64_]], [[COPY5]], 0, 1, 0, implicit $exec
+ ; CHECK-NEXT: SI_RETURN
+ %0:vgpr_32 = COPY $vgpr8
+ %1:av_64_align2 = COPY $vgpr0_vgpr1
+ %2:av_64_align2 = COPY $vgpr2_vgpr3
+ %3:areg_64_align2 = COPY $vgpr4_vgpr5
+ %4:vreg_64_align2 = COPY %3
+ %5:vreg_64_align2 = V_MFMA_F64_4X4X4F64_vgprcd_e64 %1, %2, %4, 0, 0, 0, implicit $mode, implicit $exec
+ %6:vreg_64_align2 = COPY $vgpr6_vgpr7
+ DS_WRITE2_B64_gfx9 %0, %5, %6, 0, 1, 0, implicit $exec
+ SI_RETURN
+...
More information about the llvm-commits
mailing list