[llvm] [AArch64] Expand FORM_TRANSPOSED_REG_TUPLE to copies before regalloc (PR #207205)
Benjamin Maxwell via llvm-commits
llvm-commits at lists.llvm.org
Mon Jul 6 05:27:53 PDT 2026
https://github.com/MacDue updated https://github.com/llvm/llvm-project/pull/207205
>From 3a81ad9d7ae32c5851e67698711c1d1533d8eb28 Mon Sep 17 00:00:00 2001
From: Benjamin Maxwell <benjamin.maxwell at arm.com>
Date: Thu, 2 Jul 2026 09:13:37 +0000
Subject: [PATCH 1/7] [AArch64] Expand FORM_TRANSPOSED_REG_TUPLE to copies
before regalloc
Previously, we'd keep the `FORM_TRANSPOSED_REG_TUPLE` nodes around
during register allocation. The problem with this, is it does not model
the potential overlap of live ranges between the destination and source
operands.
This means the register allocator thinks it has complete freedom to
allocate registers to the operands. For example, there's nothing stopping
it from allocating:
z0, z1, z2, z3 FORM_TRANSPOSED_X4 z3, z2, z1, z0
However, such cases are hard to expand later (either requiring spills or
complex shuffles). The current expansions of `FORM_TRANSPOSED_REG_TUPLE`
will miscompile when handling cases like this, as when naively expanded
to copies (post-register allocation), earlier copies can clobber later
uses.
For the above case, the incorrect expansion would be:
z0 = COPY z3 // z0 clobbered
z1 = COPY z2 // z1 clobbered
z2 = COPY z1 // reads wrong value of z1
z3 = COPY Z0 // reads wrong value of z0
The patch fixes this issue by expanding FORM_TRANSPOSED_REG_TUPLEs to
copy sequences immediately before register allocation (in
aarch64-post-coalescer).
So for example:
%v4:zpr4mul4 = FORM_TRANSPOSED_X4 %v0:0, %v1:0, %v2:0, %v3:0
Expands to:
undef %v4.zsub0:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE %v0:0
%v4.zsub1:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE %v1:0
%v4.zsub2:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE %v2:0
%v4.zsub3:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE %v3:0
This is similar to how REG_SEQUENCE is expanded, and allows the register
allocator to reason about how the copies could interfere with eachother.
To ensure our register allocator hints still apply, we encourage the
scheduler to place `FORM_TRANSPOSED_REG_TUPLE` nodes immediately before
their users. This makes the live ranges for the hint nodes
(COPY_INTO_TRANSPOSED_TUPLE) short (and the live ranges for their
operands long). This means the register allocator is more likely to
allocate registers to the sources of the copies first, which works best
for our allocation hints.
---
.../AArch64/AArch64ExpandPseudoInsts.cpp | 42 ++----
.../AArch64/AArch64PostCoalescerPass.cpp | 56 +++++++-
.../Target/AArch64/AArch64RegisterInfo.cpp | 124 +++++++++---------
.../Target/AArch64/AArch64TargetMachine.cpp | 34 +++++
llvm/lib/Target/AArch64/SMEInstrFormats.td | 9 ++
.../AArch64/expand-form-transposed-tuple.mir | 69 ++++++++++
.../AArch64/sme2-multivec-regalloc.mir | 61 ++++-----
7 files changed, 273 insertions(+), 122 deletions(-)
create mode 100644 llvm/test/CodeGen/AArch64/expand-form-transposed-tuple.mir
diff --git a/llvm/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp b/llvm/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp
index 5fa93da1544fc..16b7b2e74fcd2 100644
--- a/llvm/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp
+++ b/llvm/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp
@@ -59,10 +59,8 @@ class AArch64ExpandPseudoImpl {
TargetRegisterClass ContiguousClass,
TargetRegisterClass StridedClass,
unsigned ContiguousOpc, unsigned StridedOpc);
- bool expandFormTuplePseudo(MachineBasicBlock &MBB,
- MachineBasicBlock::iterator MBBI,
- MachineBasicBlock::iterator &NextMBBI,
- unsigned Size);
+ bool expandCopyIntoTuplePseudo(MachineInstr &MI, MachineBasicBlock &MBB,
+ MachineBasicBlock::iterator MBBI);
bool expandMOVImm(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
unsigned BitSize);
@@ -1282,27 +1280,17 @@ bool AArch64ExpandPseudoImpl::expandMultiVecPseudo(
return true;
}
-bool AArch64ExpandPseudoImpl::expandFormTuplePseudo(
- MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
- MachineBasicBlock::iterator &NextMBBI, unsigned Size) {
- assert((Size == 2 || Size == 4) && "Invalid Tuple Size");
- MachineInstr &MI = *MBBI;
- Register ReturnTuple = MI.getOperand(0).getReg();
+bool AArch64ExpandPseudoImpl::expandCopyIntoTuplePseudo(
+ MachineInstr &MI, MachineBasicBlock &MBB,
+ MachineBasicBlock::iterator MBBI) {
+ Register Src = MI.getOperand(1).getReg();
+ Register Dest = MI.getOperand(0).getReg();
- const TargetRegisterInfo *TRI =
- MBB.getParent()->getSubtarget().getRegisterInfo();
- for (unsigned I = 0; I < Size; ++I) {
- Register FormTupleOpReg = MI.getOperand(I + 1).getReg();
- Register ReturnTupleSubReg =
- TRI->getSubReg(ReturnTuple, AArch64::zsub0 + I);
- // Add copies to ensure the subregisters remain in the correct order
- // for any contigious operation they are used by.
- if (FormTupleOpReg != ReturnTupleSubReg)
- BuildMI(MBB, MBBI, MI.getDebugLoc(), TII->get(AArch64::ORR_ZZZ))
- .addReg(ReturnTupleSubReg, RegState::Define)
- .addReg(FormTupleOpReg)
- .addReg(FormTupleOpReg);
- }
+ if (Src != Dest)
+ BuildMI(MBB, MBBI, MI.getDebugLoc(), TII->get(AArch64::ORR_ZZZ))
+ .addReg(Dest, RegState::Define)
+ .addReg(Src)
+ .addReg(Src);
MI.eraseFromParent();
return true;
@@ -1940,10 +1928,8 @@ bool AArch64ExpandPseudoImpl::expandMI(MachineBasicBlock &MBB,
return expandMultiVecPseudo(MBB, MBBI, AArch64::ZPR4RegClass,
AArch64::ZPR4StridedRegClass,
AArch64::LDNT1D_4Z, AArch64::LDNT1D_4Z_STRIDED);
- case AArch64::FORM_TRANSPOSED_REG_TUPLE_X2_PSEUDO:
- return expandFormTuplePseudo(MBB, MBBI, NextMBBI, 2);
- case AArch64::FORM_TRANSPOSED_REG_TUPLE_X4_PSEUDO:
- return expandFormTuplePseudo(MBB, MBBI, NextMBBI, 4);
+ case AArch64::COPY_INTO_TRANSPOSED_TUPLE:
+ return expandCopyIntoTuplePseudo(MI, MBB, MBBI);
case AArch64::EON_ZZZ:
case AArch64::NAND_ZZZ:
case AArch64::NOR_ZZZ:
diff --git a/llvm/lib/Target/AArch64/AArch64PostCoalescerPass.cpp b/llvm/lib/Target/AArch64/AArch64PostCoalescerPass.cpp
index 7815b7526ca4b..56e271d38440d 100644
--- a/llvm/lib/Target/AArch64/AArch64PostCoalescerPass.cpp
+++ b/llvm/lib/Target/AArch64/AArch64PostCoalescerPass.cpp
@@ -5,7 +5,6 @@
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
//===----------------------------------------------------------------------===//
-//===----------------------------------------------------------------------===//
#include "AArch64.h"
#include "AArch64MachineFunctionInfo.h"
@@ -21,9 +20,58 @@ using namespace llvm;
namespace {
+static bool expandFormTransposedRegTuple(MachineBasicBlock &MBB,
+ MachineInstr &MI, LiveIntervals &LIS) {
+ const TargetInstrInfo *TII =
+ MBB.getParent()->getSubtarget<AArch64Subtarget>().getInstrInfo();
+ unsigned TupleSize =
+ MI.getOpcode() == AArch64::FORM_TRANSPOSED_REG_TUPLE_X2_PSEUDO ? 2 : 4;
+
+ DebugLoc DL = MI.getDebugLoc();
+ Register TupleReg = MI.getOperand(0).getReg();
+ SmallVector<Register, 5> OrigRegs{TupleReg};
+ MachineBasicBlock::iterator FirstCopyMBBI;
+
+ for (unsigned I = 0; I < TupleSize; ++I) {
+ MachineOperand &SrcOp = MI.getOperand(I + 1);
+ OrigRegs.push_back(SrcOp.getReg());
+
+ // Ensure that an if operand is killed the kill flag is placed on the final
+ // copy for that operand. TODO: Can we remove this? Requesting the live
+ // intervals seems to clear the kill flags anyway.
+ if (SrcOp.isKill()) {
+ for (unsigned J = I + 2; J < MI.getNumOperands(); ++J) {
+ MachineOperand &LaterOp = MI.getOperand(J);
+ if (LaterOp.getReg() == SrcOp.getReg()) {
+ LaterOp.setIsKill();
+ SrcOp.setIsKill(false);
+ }
+ }
+ }
+
+ RegState DefState = I == 0 ? RegState::Undef : RegState::NoFlags;
+ MachineInstr *CopyMI =
+ BuildMI(MBB, MI, DL, TII->get(AArch64::COPY_INTO_TRANSPOSED_TUPLE))
+ .addDef(TupleReg, DefState, AArch64::zsub0 + I)
+ .add(SrcOp)
+ .addImm(TupleSize);
+
+ if (I == 0)
+ FirstCopyMBBI = CopyMI;
+ }
+
+ MachineBasicBlock::iterator EndMBBI = std::next(MI.getIterator());
+ LIS.RemoveMachineInstrFromMaps(MI);
+ MI.eraseFromParent();
+
+ LIS.repairIntervalsInRange(&MBB, FirstCopyMBBI, EndMBBI, OrigRegs);
+ return true;
+}
+
bool runAArch64PostCoalescer(MachineFunction &MF, LiveIntervals &LIS) {
AArch64FunctionInfo *FuncInfo = MF.getInfo<AArch64FunctionInfo>();
- if (!FuncInfo->hasStreamingModeChanges())
+ if (!FuncInfo->hasStreamingModeChanges() &&
+ !MF.getSubtarget<AArch64Subtarget>().isStreaming())
return false;
MachineRegisterInfo &MRI = MF.getRegInfo();
@@ -34,6 +82,10 @@ bool runAArch64PostCoalescer(MachineFunction &MF, LiveIntervals &LIS) {
switch (MI.getOpcode()) {
default:
break;
+ case AArch64::FORM_TRANSPOSED_REG_TUPLE_X2_PSEUDO:
+ case AArch64::FORM_TRANSPOSED_REG_TUPLE_X4_PSEUDO:
+ Changed |= expandFormTransposedRegTuple(MBB, MI, LIS);
+ break;
case AArch64::COALESCER_BARRIER_FPR16:
case AArch64::COALESCER_BARRIER_FPR32:
case AArch64::COALESCER_BARRIER_FPR64:
diff --git a/llvm/lib/Target/AArch64/AArch64RegisterInfo.cpp b/llvm/lib/Target/AArch64/AArch64RegisterInfo.cpp
index 9e55b29426dd2..9417e3d2df75f 100644
--- a/llvm/lib/Target/AArch64/AArch64RegisterInfo.cpp
+++ b/llvm/lib/Target/AArch64/AArch64RegisterInfo.cpp
@@ -1176,7 +1176,7 @@ static bool HandleDestructivePredicateHint(
// * Improve register allocation for SME multi-vector instructions where we can
// benefit from the strided- and contiguous register multi-vector tuples.
//
-// Here FORM_TRANSPOSED_REG_TUPLE nodes are created to improve register
+// Here COPY_INTO_TRANSPOSED_TUPLE nodes are created to improve register
// allocation where a consecutive multi-vector tuple is constructed from the
// same indices of multiple strided loads. This may still result in
// unnecessary copies between the loads and the tuple. Here we try to return a
@@ -1274,20 +1274,18 @@ bool AArch64RegisterInfo::getRegAllocationHints(
// callee-saved registers and so by default these will be pushed to the back
// of the allocation order for the ZPRStridedOrContiguous classes.
// If any of the instructions which define VirtReg are used by the
- // FORM_TRANSPOSED_REG_TUPLE pseudo, we want to favour reducing copy
+ // COPY_INTO_TRANSPOSED_TUPLE pseudos, we want to favour reducing copy
// instructions over reducing the number of clobbered callee-save registers,
// so we add the strided registers as a hint.
unsigned RegID = RegRC->getID();
if (RegID == AArch64::ZPR2StridedOrContiguousRegClassID ||
RegID == AArch64::ZPR4StridedOrContiguousRegClassID) {
- // Look through uses of the register for FORM_TRANSPOSED_REG_TUPLE.
+ // Look through uses of the register for COPY_INTO_TRANSPOSED_TUPLE.
for (const MachineInstr &Use : MRI.use_nodbg_instructions(VirtReg)) {
- if (Use.getOpcode() != AArch64::FORM_TRANSPOSED_REG_TUPLE_X2_PSEUDO &&
- Use.getOpcode() != AArch64::FORM_TRANSPOSED_REG_TUPLE_X4_PSEUDO)
+ if (Use.getOpcode() != AArch64::COPY_INTO_TRANSPOSED_TUPLE)
continue;
- unsigned UseOps = Use.getNumOperands() - 1;
const TargetRegisterClass *StridedRC;
switch (RegID) {
case AArch64::ZPR2StridedOrContiguousRegClassID:
@@ -1305,17 +1303,19 @@ bool AArch64RegisterInfo::getRegAllocationHints(
if (StridedRC->contains(Reg))
StridedOrder.push_back(Reg);
- int OpIdx = Use.findRegisterUseOperandIdx(VirtReg, this);
- assert(OpIdx != -1 && "Expected operand index from register use.");
+ unsigned TupleSize = Use.getOperand(2).getImm();
+ unsigned TupIdx = Use.getOperand(0).getSubReg() - AArch64::zsub0;
unsigned TupleID = MRI.getRegClass(Use.getOperand(0).getReg())->getID();
bool IsMulZPR = TupleID == AArch64::ZPR2Mul2RegClassID ||
TupleID == AArch64::ZPR4Mul4RegClassID;
- const MachineOperand *AssignedRegOp = llvm::find_if(
- make_range(Use.operands_begin() + 1, Use.operands_end()),
- [&VRM](const MachineOperand &Op) {
- return VRM->hasPhys(Op.getReg());
+ auto Copies = MRI.def_instructions(Use.getOperand(0).getReg());
+ auto CopyWithAssignedSrc =
+ llvm::find_if(Copies, [&](const MachineInstr &Def) {
+ auto &Src = Def.getOperand(1);
+ return Def.getOpcode() == Use.getOpcode() &&
+ VRM->hasPhys(Src.getReg());
});
// Example:
@@ -1326,7 +1326,10 @@ bool AArch64RegisterInfo::getRegAllocationHints(
// %v1:zpr2stridedorcontiguous = ld1 p0/z, [...]
// %v2:zpr2stridedorcontiguous = ld1 p0/z, [...]
// %v3:zpr2stridedorcontiguous = ld1 p0/z, [...]
- // %v4:zpr4mul4 = FORM_TRANSPOSED_X4 %v0:0, %v1:0, %v2:0, %v3:0
+ // %v4.zsub0:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE %v0:0
+ // %v4.zsub1:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE %v1:0
+ // %v4.zsub2:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE %v2:0
+ // %v4.zsub3:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE %v3:0
//
// One such suitable allocation would be:
//
@@ -1334,58 +1337,62 @@ bool AArch64RegisterInfo::getRegAllocationHints(
// { z1, z9 } = ld1 p0/z, [...]
// { z2, z10 } = ld1 p0/z, [...]
// { z3, z11 } = ld1 p0/z, [...]
- // { z0, z1, z2, z3 } =
- // FORM_TRANSPOSED_X4 {z0, z8}:0, {z1, z9}:0, {z2, z10}:0, {z3, z11}:0
+ // z0 = COPY_INTO_TRANSPOSED_TUPLE {z0, z8}:0
+ // z1 = COPY_INTO_TRANSPOSED_TUPLE {z1, z9}:0
+ // z2 = COPY_INTO_TRANSPOSED_TUPLE {z2, z10}:0
+ // z3 = COPY_INTO_TRANSPOSED_TUPLE {z3, z11}:0
//
// Below we distinguish two cases when trying to find a register:
- // * None of the registers used by FORM_TRANSPOSED_X4 have been assigned
- // yet. In this case the code muse ensure that there are at least UseOps
+ // * None of the sources of the copies have been assigned a register yet.
+ // In this case the code must ensure that there are at least TupleSize
// free consecutive registers. If IsMulZPR is true, then the first of
- // registers must also be a multiple of UseOps, e.g. { z0, z1, z2, z3 }
- // is valid but { z1, z2, z3, z5 } is not.
- // * One or more of the registers used by FORM_TRANSPOSED_X4 is already
- // assigned a physical register, which means only checking that a
- // consecutive range of free tuple registers exists which includes
- // the assigned register.
+ // registers must also be a multiple of TupleSize, e.g.
+ // { z0, z1, z2, z3 } is valid but { z1, z2, z3, z5 } is not.
+ // * One or more copies already have registers assigned to their sources,
+ // which means only checking that a consecutive range of free tuple
+ // registers exists which includes the assigned register.
// e.g. in the example above, if { z0, z8 } is already allocated for
// %v0, we just need to ensure that { z1, z9 }, { z2, z10 } and
// { z3, z11 } are also free. If so, we add { z2, z10 }.
- if (AssignedRegOp == Use.operands_end()) {
+ if (CopyWithAssignedSrc == Copies.end()) {
// There are no registers already assigned to any of the pseudo
// operands. Look for a valid starting register for the group.
for (unsigned I = 0; I < StridedOrder.size(); ++I) {
MCPhysReg Reg = StridedOrder[I];
- // If the FORM_TRANSPOSE nodes use the ZPRMul classes, the starting
- // register of the first load should be a multiple of 2 or 4.
- unsigned SubRegIdx = Use.getOperand(OpIdx).getSubReg();
- if (IsMulZPR && (getSubReg(Reg, SubRegIdx) - AArch64::Z0) % UseOps !=
- ((unsigned)OpIdx - 1))
+ // If the COPY_INTO_TRANSPOSED_TUPLE nodes use the ZPRMul classes, the
+ // starting register of the first load should be a multiple of 2 or 4.
+ unsigned SubRegIdx = Use.getOperand(1).getSubReg();
+ if (IsMulZPR &&
+ (getSubReg(Reg, SubRegIdx) - AArch64::Z0) % TupleSize != TupIdx)
continue;
// In the example above, if VirtReg is the third operand of the
// tuple (%v2) and Reg == Z2_Z10, then we need to make sure that
// Z0_Z8, Z1_Z9 and Z3_Z11 are also available.
- auto IsFreeConsecutiveReg = [&](unsigned UseOp) {
- unsigned R = Reg - (OpIdx - 1) + UseOp;
+ auto IsFreeConsecutiveReg = [&](unsigned I) {
+ unsigned R = Reg - TupIdx + I;
return StridedRC->contains(R) &&
- (UseOp == 0 ||
+ (I == 0 ||
((getSubReg(R, AArch64::zsub0) - AArch64::Z0) ==
(getSubReg(R - 1, AArch64::zsub0) - AArch64::Z0) + 1)) &&
!Matrix->isPhysRegUsed(R);
};
- if (all_of(iota_range<unsigned>(0U, UseOps, /*Inclusive=*/false),
+ if (all_of(iota_range<unsigned>(0U, TupleSize, /*Inclusive=*/false),
IsFreeConsecutiveReg))
Hints.push_back(Reg);
}
} else {
- // At least one operand already has a physical register assigned.
+ // At least copy already has a physical register assigned to its source.
// Find the starting sub-register of this and use it to work out the
// correct strided register to suggest based on the current op index.
+ unsigned AssignedTupIdx =
+ CopyWithAssignedSrc->getOperand(0).getSubReg() - AArch64::zsub0;
MCPhysReg TargetStartReg =
- getSubReg(VRM->getPhys(AssignedRegOp->getReg()), AArch64::zsub0) +
- (OpIdx - AssignedRegOp->getOperandNo());
+ getSubReg(VRM->getPhys(CopyWithAssignedSrc->getOperand(1).getReg()),
+ AArch64::zsub0) +
+ (TupIdx - AssignedTupIdx);
for (unsigned I = 0; I < StridedOrder.size(); ++I)
if (getSubReg(StridedOrder[I], AArch64::zsub0) == TargetStartReg)
@@ -1398,34 +1405,31 @@ bool AArch64RegisterInfo::getRegAllocationHints(
}
}
- for (MachineInstr &MI : MRI.def_instructions(VirtReg)) {
- if (MI.getOpcode() != AArch64::FORM_TRANSPOSED_REG_TUPLE_X2_PSEUDO &&
- MI.getOpcode() != AArch64::FORM_TRANSPOSED_REG_TUPLE_X4_PSEUDO)
- return TargetRegisterInfo::getRegAllocationHints(VirtReg, Order, Hints,
- MF, VRM);
-
- unsigned FirstOpSubReg = MI.getOperand(1).getSubReg();
- switch (FirstOpSubReg) {
- case AArch64::zsub0:
- case AArch64::zsub1:
- case AArch64::zsub2:
- case AArch64::zsub3:
- break;
- default:
+ for (auto &Def : MRI.def_instructions(VirtReg)) {
+ if (Def.getOpcode() != AArch64::COPY_INTO_TRANSPOSED_TUPLE)
continue;
- }
- // Look up the physical register mapped to the first operand of the pseudo.
- Register FirstOpVirtReg = MI.getOperand(1).getReg();
- if (!VRM->hasPhys(FirstOpVirtReg))
+ MachineOperand &Src = Def.getOperand(1);
+ MachineOperand &Dst = Def.getOperand(0);
+
+ if (!Src.getSubReg() || !Dst.getSubReg())
continue;
- MCRegister TupleStartReg =
- getSubReg(VRM->getPhys(FirstOpVirtReg), FirstOpSubReg);
- for (unsigned I = 0; I < Order.size(); ++I)
- if (MCRegister R = getSubReg(Order[I], AArch64::zsub0))
- if (R == TupleStartReg)
- Hints.push_back(Order[I]);
+ // FIXME: This is fragile. If we allocate a register to the Dst before Src,
+ // our hints are won't have any effect... This is currently mitigated by
+ // by trying to schedule copies immediately before their uses. This gives
+ // them a short live range (so they're low priority to allocate).
+ if (!VRM->hasPhys(Src.getReg()))
+ continue;
+
+ // Find the ZPR register mapped to the source of the copy.
+ MCPhysReg SrcZPR = getSubReg(VRM->getPhys(Src.getReg()), Src.getSubReg());
+
+ // Try to pick a tuple register for Dst with Src as a member.
+ for (MCPhysReg R : Order) {
+ if (getSubReg(R, Dst.getSubReg()) == SrcZPR)
+ Hints.push_back(R);
+ }
}
return TargetRegisterInfo::getRegAllocationHints(VirtReg, Order, Hints, MF,
diff --git a/llvm/lib/Target/AArch64/AArch64TargetMachine.cpp b/llvm/lib/Target/AArch64/AArch64TargetMachine.cpp
index 898d92e74c85c..5b2c48c860206 100644
--- a/llvm/lib/Target/AArch64/AArch64TargetMachine.cpp
+++ b/llvm/lib/Target/AArch64/AArch64TargetMachine.cpp
@@ -510,6 +510,37 @@ AArch64TargetMachine::getSubtargetImpl(const Function &F) const {
return I.get();
}
+// Encourage placing FORM_TRANSPOSED_REG immediately before the instruction that
+// uses/consumes it. This ensures it has a short live range, which means we're
+// more likely to allocate registers its operands first (which works best for
+// the hints in AArch64RegisterInfo::getRegAllocationHints).
+static bool scheduleFormTransposedTupleAdjacentToUsers(
+ const TargetInstrInfo &TII, const TargetSubtargetInfo &TSI,
+ const MachineInstr *FirstMI, const MachineInstr &SecondMI) {
+
+ auto *TRI = TSI.getRegisterInfo();
+ if (!FirstMI) {
+ // The SecondMI must be a multi-vector operation. So limit this to
+ // instructions that use full tuple registers (not a sub-register).
+ const MachineRegisterInfo &MRI = SecondMI.getMF()->getRegInfo();
+ for (const MachineOperand &Use : SecondMI.uses()) {
+ if (Use.isReg() && Use.getReg().isVirtual() && !Use.getSubReg() &&
+ TRI->isSubRegValidForRegClass(MRI.getRegClass(Use.getReg()),
+ AArch64::zsub0))
+ return true;
+ }
+ return false;
+ }
+
+ if (FirstMI->getOpcode() != AArch64::FORM_TRANSPOSED_REG_TUPLE_X2_PSEUDO &&
+ FirstMI->getOpcode() != AArch64::FORM_TRANSPOSED_REG_TUPLE_X4_PSEUDO)
+ return false;
+
+ Register TupleDef = FirstMI->getOperand(0).getReg();
+ return SecondMI.findRegisterUseOperandIdx(TupleDef, TSI.getRegisterInfo()) !=
+ -1;
+}
+
ScheduleDAGInstrs *
AArch64TargetMachine::createMachineScheduler(MachineSchedContext *C) const {
const AArch64Subtarget &ST = C->MF->getSubtarget<AArch64Subtarget>();
@@ -518,6 +549,9 @@ AArch64TargetMachine::createMachineScheduler(MachineSchedContext *C) const {
DAG->addMutation(createStoreClusterDAGMutation(DAG->TII, DAG->TRI));
if (ST.hasFusion())
DAG->addMutation(createAArch64MacroFusionDAGMutation());
+ if (ST.hasSME() && ST.isStreaming())
+ DAG->addMutation(createMacroFusionDAGMutation(
+ scheduleFormTransposedTupleAdjacentToUsers));
return DAG;
}
diff --git a/llvm/lib/Target/AArch64/SMEInstrFormats.td b/llvm/lib/Target/AArch64/SMEInstrFormats.td
index f07fb8ad81f63..041e5d8973e8a 100644
--- a/llvm/lib/Target/AArch64/SMEInstrFormats.td
+++ b/llvm/lib/Target/AArch64/SMEInstrFormats.td
@@ -58,6 +58,15 @@ def FORM_TRANSPOSED_REG_TUPLE_X4_PSEUDO :
let hasSideEffects = 0;
}
+// Expanded form of FORM_TRANSPOSED_REG_TUPLE. Each FORM_TRANSPOSED_REG_TUPLE is
+// expanded to one or more copies immediately before register allocation. This
+// ensures the live ranges of each member of the tuple are correctly modelled
+// during register allocation.
+def COPY_INTO_TRANSPOSED_TUPLE :
+ Pseudo<(outs ZPR:$tup), (ins ZPR:$zn, imm0_31: $size), []>, Sched<[]>{
+ let hasSideEffects = 0;
+}
+
def SDTZALoadStore : SDTypeProfile<0, 3, [SDTCisInt<0>, SDTCisPtrTy<1>, SDTCisInt<2>]>;
// SME ZA loads and stores
def AArch64SMELdr : SDNode<"AArch64ISD::SME_ZA_LDR", SDTZALoadStore,
diff --git a/llvm/test/CodeGen/AArch64/expand-form-transposed-tuple.mir b/llvm/test/CodeGen/AArch64/expand-form-transposed-tuple.mir
new file mode 100644
index 0000000000000..ddee51fb2787e
--- /dev/null
+++ b/llvm/test/CodeGen/AArch64/expand-form-transposed-tuple.mir
@@ -0,0 +1,69 @@
+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py UTC_ARGS: --version 6
+# RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sme2 -force-streaming -verify-machineinstrs -run-pass=aarch64-post-coalescer %s -o - | FileCheck %s
+
+---
+name: expand_x4_tuples
+tracksRegLiveness: true
+isSSA: false
+body: |
+ bb.0.entry:
+ ; CHECK-LABEL: name: expand_x4_tuples
+ ; CHECK: [[DEF:%[0-9]+]]:zpr4stridedorcontiguous = IMPLICIT_DEF
+ ; CHECK-NEXT: [[DEF1:%[0-9]+]]:zpr4stridedorcontiguous = IMPLICIT_DEF
+ ; CHECK-NEXT: [[DEF2:%[0-9]+]]:zpr4stridedorcontiguous = IMPLICIT_DEF
+ ; CHECK-NEXT: [[DEF3:%[0-9]+]]:zpr4stridedorcontiguous = IMPLICIT_DEF
+ ; CHECK-NEXT: undef [[COPY_INTO_TRANSPOSED_TUPLE:%[0-9]+]].zsub0:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE [[DEF]].zsub0, 4
+ ; CHECK-NEXT: [[COPY_INTO_TRANSPOSED_TUPLE:%[0-9]+]].zsub1:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE [[DEF1]].zsub0, 4
+ ; CHECK-NEXT: [[COPY_INTO_TRANSPOSED_TUPLE:%[0-9]+]].zsub2:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE [[DEF2]].zsub0, 4
+ ; CHECK-NEXT: [[COPY_INTO_TRANSPOSED_TUPLE:%[0-9]+]].zsub3:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE [[DEF3]].zsub0, 4
+ ; CHECK-NEXT: undef [[COPY_INTO_TRANSPOSED_TUPLE1:%[0-9]+]].zsub0:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE [[DEF]].zsub1, 4
+ ; CHECK-NEXT: [[COPY_INTO_TRANSPOSED_TUPLE1:%[0-9]+]].zsub1:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE [[DEF1]].zsub1, 4
+ ; CHECK-NEXT: [[COPY_INTO_TRANSPOSED_TUPLE1:%[0-9]+]].zsub2:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE [[DEF2]].zsub1, 4
+ ; CHECK-NEXT: [[COPY_INTO_TRANSPOSED_TUPLE1:%[0-9]+]].zsub3:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE [[DEF3]].zsub1, 4
+ ; CHECK-NEXT: undef [[COPY_INTO_TRANSPOSED_TUPLE2:%[0-9]+]].zsub0:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE [[DEF]].zsub2, 4
+ ; CHECK-NEXT: [[COPY_INTO_TRANSPOSED_TUPLE2:%[0-9]+]].zsub1:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE [[DEF1]].zsub2, 4
+ ; CHECK-NEXT: [[COPY_INTO_TRANSPOSED_TUPLE2:%[0-9]+]].zsub2:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE [[DEF2]].zsub2, 4
+ ; CHECK-NEXT: [[COPY_INTO_TRANSPOSED_TUPLE2:%[0-9]+]].zsub3:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE [[DEF3]].zsub2, 4
+ ; CHECK-NEXT: undef [[COPY_INTO_TRANSPOSED_TUPLE3:%[0-9]+]].zsub0:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE [[DEF]].zsub3, 4
+ ; CHECK-NEXT: [[COPY_INTO_TRANSPOSED_TUPLE3:%[0-9]+]].zsub1:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE [[DEF1]].zsub3, 4
+ ; CHECK-NEXT: [[COPY_INTO_TRANSPOSED_TUPLE3:%[0-9]+]].zsub2:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE [[DEF2]].zsub3, 4
+ ; CHECK-NEXT: [[COPY_INTO_TRANSPOSED_TUPLE3:%[0-9]+]].zsub3:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE [[DEF3]].zsub3, 4
+ ; CHECK-NEXT: FAKE_USE implicit [[COPY_INTO_TRANSPOSED_TUPLE]], implicit [[COPY_INTO_TRANSPOSED_TUPLE1]], implicit [[COPY_INTO_TRANSPOSED_TUPLE2]], implicit [[COPY_INTO_TRANSPOSED_TUPLE3]]
+ ; CHECK-NEXT: RET_ReallyLR
+ %0:zpr4stridedorcontiguous = IMPLICIT_DEF
+ %1:zpr4stridedorcontiguous = IMPLICIT_DEF
+ %2:zpr4stridedorcontiguous = IMPLICIT_DEF
+ %3:zpr4stridedorcontiguous = IMPLICIT_DEF
+
+ %4:zpr4mul4 = FORM_TRANSPOSED_REG_TUPLE_X4_PSEUDO %0.zsub0, %1.zsub0, %2.zsub0, %3.zsub0
+ %5:zpr4mul4 = FORM_TRANSPOSED_REG_TUPLE_X4_PSEUDO %0.zsub1, %1.zsub1, %2.zsub1, %3.zsub1
+ %6:zpr4mul4 = FORM_TRANSPOSED_REG_TUPLE_X4_PSEUDO %0.zsub2, %1.zsub2, %2.zsub2, %3.zsub2
+ %7:zpr4mul4 = FORM_TRANSPOSED_REG_TUPLE_X4_PSEUDO %0.zsub3, %1.zsub3, %2.zsub3, %3.zsub3
+
+ FAKE_USE implicit %4, implicit %5, implicit %6, implicit %7
+ RET_ReallyLR
+...
+---
+name: expand_x2_tuples
+tracksRegLiveness: true
+isSSA: false
+body: |
+ bb.0.entry:
+ ; CHECK-LABEL: name: expand_x2_tuples
+ ; CHECK: [[DEF:%[0-9]+]]:zpr2stridedorcontiguous = IMPLICIT_DEF
+ ; CHECK-NEXT: [[DEF1:%[0-9]+]]:zpr2stridedorcontiguous = IMPLICIT_DEF
+ ; CHECK-NEXT: undef [[COPY_INTO_TRANSPOSED_TUPLE:%[0-9]+]].zsub0:zpr2 = COPY_INTO_TRANSPOSED_TUPLE [[DEF]].zsub0, 2
+ ; CHECK-NEXT: [[COPY_INTO_TRANSPOSED_TUPLE:%[0-9]+]].zsub1:zpr2 = COPY_INTO_TRANSPOSED_TUPLE [[DEF1]].zsub0, 2
+ ; CHECK-NEXT: undef [[COPY_INTO_TRANSPOSED_TUPLE1:%[0-9]+]].zsub0:zpr2 = COPY_INTO_TRANSPOSED_TUPLE [[DEF]].zsub1, 2
+ ; CHECK-NEXT: [[COPY_INTO_TRANSPOSED_TUPLE1:%[0-9]+]].zsub1:zpr2 = COPY_INTO_TRANSPOSED_TUPLE [[DEF1]].zsub1, 2
+ ; CHECK-NEXT: FAKE_USE implicit [[COPY_INTO_TRANSPOSED_TUPLE]], implicit [[COPY_INTO_TRANSPOSED_TUPLE1]]
+ ; CHECK-NEXT: RET_ReallyLR
+ %0:zpr2stridedorcontiguous = IMPLICIT_DEF
+ %1:zpr2stridedorcontiguous = IMPLICIT_DEF
+
+ %4:zpr2 = FORM_TRANSPOSED_REG_TUPLE_X2_PSEUDO %0.zsub0, %1.zsub0
+ %5:zpr2 = FORM_TRANSPOSED_REG_TUPLE_X2_PSEUDO %0.zsub1, %1.zsub1
+
+ FAKE_USE implicit %4, implicit %5
+ RET_ReallyLR
+...
diff --git a/llvm/test/CodeGen/AArch64/sme2-multivec-regalloc.mir b/llvm/test/CodeGen/AArch64/sme2-multivec-regalloc.mir
index c3338b14522cb..964e0cd84e111 100644
--- a/llvm/test/CodeGen/AArch64/sme2-multivec-regalloc.mir
+++ b/llvm/test/CodeGen/AArch64/sme2-multivec-regalloc.mir
@@ -1,4 +1,4 @@
-# RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sme2 -force-streaming -verify-machineinstrs -enable-subreg-liveness -start-before=greedy %s -o - | FileCheck %s
+# RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sme2 -force-streaming -verify-machineinstrs -enable-subreg-liveness -start-after=aarch64-sme-peephole-opt %s -o - | FileCheck %s
# No available group of four strided x4 registers, fall back on default allocation order
---
@@ -12,7 +12,7 @@ body: |
bb.0.entry:
liveins: $x0, $x1, $z0, $z17
- ; CHECK-LABEL: form_4x_tuple_many_live
+ ; CHECK-LABEL: form_4x_tuple_many_live:
; CHECK: stp d11, d10, [sp, #-48]!
; CHECK-NEXT: stp d9, d8, [sp, #16]
; CHECK-NEXT: str x29, [sp, #32]
@@ -26,29 +26,32 @@ body: |
; CHECK-NEXT: lsl x9, x1, #1
; CHECK-NEXT: ptrue pn8.b
; CHECK-NEXT: mov w8, wzr
- ; CHECK-NEXT: ld1b { z16.b, z20.b, z24.b, z28.b }, pn8/z, [x0]
- ; CHECK-NEXT: ld1b { z18.b, z22.b, z26.b, z30.b }, pn8/z, [x0, x1]
+ ; CHECK-NEXT: ld1b { z19.b, z23.b, z27.b, z31.b }, pn8/z, [x0]
+ ; CHECK-NEXT: ld1b { z4.b - z7.b }, pn8/z, [x0, x1]
; CHECK-NEXT: ptrue p0.b
; CHECK-NEXT: add x10, x9, x1
- ; CHECK-NEXT: ld1b { z19.b, z23.b, z27.b, z31.b }, pn8/z, [x0, x9]
- ; CHECK-NEXT: ld1b { z4.b - z7.b }, pn8/z, [x0, x10]
- ; CHECK-NEXT: mov z8.d, z16.d
- ; CHECK-NEXT: mov z9.d, z18.d
- ; CHECK-NEXT: mov z21.d, z22.d
- ; CHECK-NEXT: mov z10.d, z19.d
- ; CHECK-NEXT: mov z22.d, z23.d
- ; CHECK-NEXT: mov z25.d, z26.d
- ; CHECK-NEXT: mov z11.d, z4.d
- ; CHECK-NEXT: mov z23.d, z5.d
- ; CHECK-NEXT: mov z26.d, z27.d
- ; CHECK-NEXT: mov z27.d, z6.d
- ; CHECK-NEXT: mov z29.d, z30.d
- ; CHECK-NEXT: mov z30.d, z31.d
- ; CHECK-NEXT: mov z31.d, z7.d
+ ; CHECK-NEXT: ld1b { z18.b, z22.b, z26.b, z30.b }, pn8/z, [x0, x9]
+ ; CHECK-NEXT: ld1b { z16.b, z20.b, z24.b, z28.b }, pn8/z, [x0, x10]
+ ; CHECK-NEXT: mov z8.d, z19.d
+ ; CHECK-NEXT: mov z9.d, z4.d
+ ; CHECK-NEXT: mov z21.d, z6.d
+ ; CHECK-NEXT: mov z10.d, z18.d
+ ; CHECK-NEXT: mov z4.d, z31.d
+ ; CHECK-NEXT: mov z6.d, z30.d
+ ; CHECK-NEXT: mov z11.d, z16.d
+ ; CHECK-NEXT: udot za.s[w8, 0, vgx4], { z8.b - z11.b }, z0.b[0]
+ ; CHECK-NEXT: mov z8.d, z23.d
+ ; CHECK-NEXT: mov z9.d, z5.d
+ ; CHECK-NEXT: mov z10.d, z22.d
+ ; CHECK-NEXT: mov z11.d, z20.d
+ ; CHECK-NEXT: mov z20.d, z27.d
+ ; CHECK-NEXT: mov z22.d, z26.d
+ ; CHECK-NEXT: mov z23.d, z24.d
+ ; CHECK-NEXT: mov z5.d, z7.d
+ ; CHECK-NEXT: mov z7.d, z28.d
; CHECK-NEXT: udot za.s[w8, 0, vgx4], { z8.b - z11.b }, z0.b[0]
; CHECK-NEXT: udot za.s[w8, 0, vgx4], { z20.b - z23.b }, z0.b[0]
- ; CHECK-NEXT: udot za.s[w8, 0, vgx4], { z24.b - z27.b }, z0.b[0]
- ; CHECK-NEXT: udot za.s[w8, 0, vgx4], { z28.b - z31.b }, z0.b[0]
+ ; CHECK-NEXT: udot za.s[w8, 0, vgx4], { z4.b - z7.b }, z0.b[0]
; CHECK-NEXT: st1b { z0.b }, p0, [x0]
; CHECK-NEXT: st1b { z17.b }, p0, [x0]
; CHECK-NEXT: addvl sp, sp, #2
@@ -150,18 +153,12 @@ body: |
bb.0.entry:
liveins: $x0, $x1, $z16, $z17, $z18, $z19, $z20, $z21, $z22
- ; CHECK: stp d9, d8, [sp, #-16]!
- ; CHECK-NEXT: .cfi_def_cfa_offset 16
- ; CHECK-NEXT: .cfi_offset b8, -8
- ; CHECK-NEXT: .cfi_offset b9, -16
- ; CHECK-NEXT: ptrue pn8.b
+ ; CHECK: ptrue pn8.b
; CHECK-NEXT: mov w8, wzr
- ; CHECK-NEXT: ld1b { z0.b, z8.b }, pn8/z, [x0]
- ; CHECK-NEXT: ld1b { z1.b, z9.b }, pn8/z, [x0, x1]
- ; CHECK-NEXT: udot za.s[w8, 0, vgx2], { z0.b, z1.b }, z0.b
- ; CHECK-NEXT: udot za.s[w8, 0, vgx2], { z8.b, z9.b }, z0.b
- ; CHECK-NEXT: ldp d9, d8, [sp], #16
- ; CHECK-NEXT: ret
+ ; CHECK-NEXT: ld1b { z16.b, z24.b }, pn8/z, [x0]
+ ; CHECK-NEXT: ld1b { z17.b, z25.b }, pn8/z, [x0, x1]
+ ; CHECK-NEXT: udot za.s[w8, 0, vgx2], { z16.b, z17.b }, z0.b
+ ; CHECK-NEXT: udot za.s[w8, 0, vgx2], { z24.b, z25.b }, z0.b
%0:gpr64 = COPY $x1
%1:gpr64common = COPY $x0
>From 11ba58727709432c38ba7106c50457bb551f48d1 Mon Sep 17 00:00:00 2001
From: Benjamin Maxwell <benjamin.maxwell at arm.com>
Date: Fri, 3 Jul 2026 10:44:55 +0000
Subject: [PATCH 2/7] Fixups
---
llvm/lib/Target/AArch64/AArch64PostCoalescerPass.cpp | 12 ++++++++++++
llvm/lib/Target/AArch64/SMEInstrFormats.td | 5 ++++-
2 files changed, 16 insertions(+), 1 deletion(-)
diff --git a/llvm/lib/Target/AArch64/AArch64PostCoalescerPass.cpp b/llvm/lib/Target/AArch64/AArch64PostCoalescerPass.cpp
index 56e271d38440d..fe0c7efc82c30 100644
--- a/llvm/lib/Target/AArch64/AArch64PostCoalescerPass.cpp
+++ b/llvm/lib/Target/AArch64/AArch64PostCoalescerPass.cpp
@@ -20,6 +20,18 @@ using namespace llvm;
namespace {
+/// Expands FORM_TRANSPOSED_REG_TUPLE_{X2|X4}_PSEUDO instructions into a
+/// copy sequences. Note: This expansion occurs immediately before greedy
+/// regalloc and after the pre-RA scheduler.
+///
+/// Example:
+///
+/// %v2:zpr2 = FORM_TRANSPOSED_REG_TUPLE_X2_PSEUDO %v0.zsub0, %v1.zsub0
+///
+/// Expands to:
+///
+/// undef %v2.zsub0:zpr2 = COPY_INTO_TRANSPOSED_TUPLE %v0.zsub0, 2
+/// %v2.zsub1:zpr2 = COPY_INTO_TRANSPOSED_TUPLE %v1.zsub0, 2
static bool expandFormTransposedRegTuple(MachineBasicBlock &MBB,
MachineInstr &MI, LiveIntervals &LIS) {
const TargetInstrInfo *TII =
diff --git a/llvm/lib/Target/AArch64/SMEInstrFormats.td b/llvm/lib/Target/AArch64/SMEInstrFormats.td
index 041e5d8973e8a..084bb386feac4 100644
--- a/llvm/lib/Target/AArch64/SMEInstrFormats.td
+++ b/llvm/lib/Target/AArch64/SMEInstrFormats.td
@@ -61,10 +61,13 @@ def FORM_TRANSPOSED_REG_TUPLE_X4_PSEUDO :
// Expanded form of FORM_TRANSPOSED_REG_TUPLE. Each FORM_TRANSPOSED_REG_TUPLE is
// expanded to one or more copies immediately before register allocation. This
// ensures the live ranges of each member of the tuple are correctly modelled
-// during register allocation.
+// during register allocation. These pseudos are used to guide the hints in
+// AArch64RegisterInfo::getRegAllocationHints().
def COPY_INTO_TRANSPOSED_TUPLE :
Pseudo<(outs ZPR:$tup), (ins ZPR:$zn, imm0_31: $size), []>, Sched<[]>{
let hasSideEffects = 0;
+ let isAsCheapAsAMove = 1;
+ let isReMaterializable = 1;
}
def SDTZALoadStore : SDTypeProfile<0, 3, [SDTCisInt<0>, SDTCisPtrTy<1>, SDTCisInt<2>]>;
>From f3a185e189fe751432717c4074ee5c89cbb06bed Mon Sep 17 00:00:00 2001
From: Benjamin Maxwell <benjamin.maxwell at arm.com>
Date: Mon, 6 Jul 2026 10:32:15 +0000
Subject: [PATCH 3/7] Fixups
---
llvm/lib/Target/AArch64/AArch64PostCoalescerPass.cpp | 8 ++++----
.../test/CodeGen/AArch64/expand-form-transposed-tuple.mir | 1 +
2 files changed, 5 insertions(+), 4 deletions(-)
diff --git a/llvm/lib/Target/AArch64/AArch64PostCoalescerPass.cpp b/llvm/lib/Target/AArch64/AArch64PostCoalescerPass.cpp
index fe0c7efc82c30..569cb8f01e39b 100644
--- a/llvm/lib/Target/AArch64/AArch64PostCoalescerPass.cpp
+++ b/llvm/lib/Target/AArch64/AArch64PostCoalescerPass.cpp
@@ -20,9 +20,9 @@ using namespace llvm;
namespace {
-/// Expands FORM_TRANSPOSED_REG_TUPLE_{X2|X4}_PSEUDO instructions into a
-/// copy sequences. Note: This expansion occurs immediately before greedy
-/// regalloc and after the pre-RA scheduler.
+/// Expands FORM_TRANSPOSED_REG_TUPLE_{X2|X4}_PSEUDO instructions into copy
+/// sequences. Note: This expansion occurs immediately before greedy regalloc
+/// and after the coalescer and pre-RA scheduler.
///
/// Example:
///
@@ -48,7 +48,7 @@ static bool expandFormTransposedRegTuple(MachineBasicBlock &MBB,
MachineOperand &SrcOp = MI.getOperand(I + 1);
OrigRegs.push_back(SrcOp.getReg());
- // Ensure that an if operand is killed the kill flag is placed on the final
+ // Ensure that if operand is killed, the kill flag is placed on the final
// copy for that operand. TODO: Can we remove this? Requesting the live
// intervals seems to clear the kill flags anyway.
if (SrcOp.isKill()) {
diff --git a/llvm/test/CodeGen/AArch64/expand-form-transposed-tuple.mir b/llvm/test/CodeGen/AArch64/expand-form-transposed-tuple.mir
index ddee51fb2787e..52eeea589e8bf 100644
--- a/llvm/test/CodeGen/AArch64/expand-form-transposed-tuple.mir
+++ b/llvm/test/CodeGen/AArch64/expand-form-transposed-tuple.mir
@@ -1,5 +1,6 @@
# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py UTC_ARGS: --version 6
# RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sme2 -force-streaming -verify-machineinstrs -run-pass=aarch64-post-coalescer %s -o - | FileCheck %s
+# RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sme2 -enable-subreg-liveness -force-streaming -verify-machineinstrs -run-pass=aarch64-post-coalescer %s -o - | FileCheck %s
---
name: expand_x4_tuples
>From fc229a2b95408cd7d6750c25711d4bd3dc0ae262 Mon Sep 17 00:00:00 2001
From: Benjamin Maxwell <benjamin.maxwell at arm.com>
Date: Mon, 6 Jul 2026 10:43:20 +0000
Subject: [PATCH 4/7] Fixups
---
llvm/lib/Target/AArch64/AArch64TargetMachine.cpp | 9 ++-------
1 file changed, 2 insertions(+), 7 deletions(-)
diff --git a/llvm/lib/Target/AArch64/AArch64TargetMachine.cpp b/llvm/lib/Target/AArch64/AArch64TargetMachine.cpp
index 5b2c48c860206..a97d3caf23d45 100644
--- a/llvm/lib/Target/AArch64/AArch64TargetMachine.cpp
+++ b/llvm/lib/Target/AArch64/AArch64TargetMachine.cpp
@@ -532,13 +532,8 @@ static bool scheduleFormTransposedTupleAdjacentToUsers(
return false;
}
- if (FirstMI->getOpcode() != AArch64::FORM_TRANSPOSED_REG_TUPLE_X2_PSEUDO &&
- FirstMI->getOpcode() != AArch64::FORM_TRANSPOSED_REG_TUPLE_X4_PSEUDO)
- return false;
-
- Register TupleDef = FirstMI->getOperand(0).getReg();
- return SecondMI.findRegisterUseOperandIdx(TupleDef, TSI.getRegisterInfo()) !=
- -1;
+ return FirstMI->getOpcode() == AArch64::FORM_TRANSPOSED_REG_TUPLE_X2_PSEUDO ||
+ FirstMI->getOpcode() == AArch64::FORM_TRANSPOSED_REG_TUPLE_X4_PSEUDO;
}
ScheduleDAGInstrs *
>From 39b3b43b432b8b371db336eaec31a65ed452ae76 Mon Sep 17 00:00:00 2001
From: Benjamin Maxwell <benjamin.maxwell at arm.com>
Date: Mon, 6 Jul 2026 11:04:13 +0000
Subject: [PATCH 5/7] Fixups
---
.../Target/AArch64/AArch64TargetMachine.cpp | 18 ++----------------
1 file changed, 2 insertions(+), 16 deletions(-)
diff --git a/llvm/lib/Target/AArch64/AArch64TargetMachine.cpp b/llvm/lib/Target/AArch64/AArch64TargetMachine.cpp
index a97d3caf23d45..1b53f450619fe 100644
--- a/llvm/lib/Target/AArch64/AArch64TargetMachine.cpp
+++ b/llvm/lib/Target/AArch64/AArch64TargetMachine.cpp
@@ -517,22 +517,8 @@ AArch64TargetMachine::getSubtargetImpl(const Function &F) const {
static bool scheduleFormTransposedTupleAdjacentToUsers(
const TargetInstrInfo &TII, const TargetSubtargetInfo &TSI,
const MachineInstr *FirstMI, const MachineInstr &SecondMI) {
-
- auto *TRI = TSI.getRegisterInfo();
- if (!FirstMI) {
- // The SecondMI must be a multi-vector operation. So limit this to
- // instructions that use full tuple registers (not a sub-register).
- const MachineRegisterInfo &MRI = SecondMI.getMF()->getRegInfo();
- for (const MachineOperand &Use : SecondMI.uses()) {
- if (Use.isReg() && Use.getReg().isVirtual() && !Use.getSubReg() &&
- TRI->isSubRegValidForRegClass(MRI.getRegClass(Use.getReg()),
- AArch64::zsub0))
- return true;
- }
- return false;
- }
-
- return FirstMI->getOpcode() == AArch64::FORM_TRANSPOSED_REG_TUPLE_X2_PSEUDO ||
+ return !FirstMI ||
+ FirstMI->getOpcode() == AArch64::FORM_TRANSPOSED_REG_TUPLE_X2_PSEUDO ||
FirstMI->getOpcode() == AArch64::FORM_TRANSPOSED_REG_TUPLE_X4_PSEUDO;
}
>From 996bd356318f689c41ac0aaa6586e2e6b8700035 Mon Sep 17 00:00:00 2001
From: Benjamin Maxwell <benjamin.maxwell at arm.com>
Date: Mon, 6 Jul 2026 11:07:46 +0000
Subject: [PATCH 6/7] Fixups
---
llvm/lib/Target/AArch64/AArch64TargetMachine.cpp | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/llvm/lib/Target/AArch64/AArch64TargetMachine.cpp b/llvm/lib/Target/AArch64/AArch64TargetMachine.cpp
index 1b53f450619fe..0841f50c2a424 100644
--- a/llvm/lib/Target/AArch64/AArch64TargetMachine.cpp
+++ b/llvm/lib/Target/AArch64/AArch64TargetMachine.cpp
@@ -511,9 +511,9 @@ AArch64TargetMachine::getSubtargetImpl(const Function &F) const {
}
// Encourage placing FORM_TRANSPOSED_REG immediately before the instruction that
-// uses/consumes it. This ensures it has a short live range, which means we're
-// more likely to allocate registers its operands first (which works best for
-// the hints in AArch64RegisterInfo::getRegAllocationHints).
+// uses/consumes it. This ensures its def has a short live range, which means
+// we're more likely to allocate registers its operands first (which works best
+// for the hints in AArch64RegisterInfo::getRegAllocationHints).
static bool scheduleFormTransposedTupleAdjacentToUsers(
const TargetInstrInfo &TII, const TargetSubtargetInfo &TSI,
const MachineInstr *FirstMI, const MachineInstr &SecondMI) {
>From a5abdf0f192789e315096dad687f37947d490b46 Mon Sep 17 00:00:00 2001
From: Benjamin Maxwell <benjamin.maxwell at arm.com>
Date: Mon, 6 Jul 2026 12:27:36 +0000
Subject: [PATCH 7/7] Fixups
---
.../AArch64/AArch64PostCoalescerPass.cpp | 29 +++++++++++--------
.../AArch64/expand-form-transposed-tuple.mir | 21 ++++++++++++++
2 files changed, 38 insertions(+), 12 deletions(-)
diff --git a/llvm/lib/Target/AArch64/AArch64PostCoalescerPass.cpp b/llvm/lib/Target/AArch64/AArch64PostCoalescerPass.cpp
index 569cb8f01e39b..bdef5017bc879 100644
--- a/llvm/lib/Target/AArch64/AArch64PostCoalescerPass.cpp
+++ b/llvm/lib/Target/AArch64/AArch64PostCoalescerPass.cpp
@@ -33,7 +33,7 @@ namespace {
/// undef %v2.zsub0:zpr2 = COPY_INTO_TRANSPOSED_TUPLE %v0.zsub0, 2
/// %v2.zsub1:zpr2 = COPY_INTO_TRANSPOSED_TUPLE %v1.zsub0, 2
static bool expandFormTransposedRegTuple(MachineBasicBlock &MBB,
- MachineInstr &MI, LiveIntervals &LIS) {
+ MachineInstr &MI, LiveIntervals *LIS) {
const TargetInstrInfo *TII =
MBB.getParent()->getSubtarget<AArch64Subtarget>().getInstrInfo();
unsigned TupleSize =
@@ -49,8 +49,7 @@ static bool expandFormTransposedRegTuple(MachineBasicBlock &MBB,
OrigRegs.push_back(SrcOp.getReg());
// Ensure that if operand is killed, the kill flag is placed on the final
- // copy for that operand. TODO: Can we remove this? Requesting the live
- // intervals seems to clear the kill flags anyway.
+ // copy for that operand.
if (SrcOp.isKill()) {
for (unsigned J = I + 2; J < MI.getNumOperands(); ++J) {
MachineOperand &LaterOp = MI.getOperand(J);
@@ -73,14 +72,16 @@ static bool expandFormTransposedRegTuple(MachineBasicBlock &MBB,
}
MachineBasicBlock::iterator EndMBBI = std::next(MI.getIterator());
- LIS.RemoveMachineInstrFromMaps(MI);
+ if (LIS)
+ LIS->RemoveMachineInstrFromMaps(MI);
MI.eraseFromParent();
- LIS.repairIntervalsInRange(&MBB, FirstCopyMBBI, EndMBBI, OrigRegs);
+ if (LIS)
+ LIS->repairIntervalsInRange(&MBB, FirstCopyMBBI, EndMBBI, OrigRegs);
return true;
}
-bool runAArch64PostCoalescer(MachineFunction &MF, LiveIntervals &LIS) {
+bool runAArch64PostCoalescer(MachineFunction &MF, LiveIntervals *LIS) {
AArch64FunctionInfo *FuncInfo = MF.getInfo<AArch64FunctionInfo>();
if (!FuncInfo->hasStreamingModeChanges() &&
!MF.getSubtarget<AArch64Subtarget>().isStreaming())
@@ -113,11 +114,14 @@ bool runAArch64PostCoalescer(MachineFunction &MF, LiveIntervals &LIS) {
// MI must be erased from the basic block before recalculating the live
// interval.
- LIS.RemoveMachineInstrFromMaps(MI);
+ if (LIS)
+ LIS->RemoveMachineInstrFromMaps(MI);
MI.eraseFromParent();
- LIS.removeInterval(Src);
- LIS.createAndComputeVirtRegInterval(Src);
+ if (LIS) {
+ LIS->removeInterval(Src);
+ LIS->createAndComputeVirtRegInterval(Src);
+ }
Changed = true;
break;
@@ -142,7 +146,7 @@ struct AArch64PostCoalescerLegacy : public MachineFunctionPass {
void getAnalysisUsage(AnalysisUsage &AU) const override {
AU.setPreservesCFG();
- AU.addRequired<LiveIntervalsWrapperPass>();
+ AU.addUsedIfAvailable<LiveIntervalsWrapperPass>();
AU.addPreserved<LiveIntervalsWrapperPass>();
AU.addPreserved<SlotIndexesWrapperPass>();
MachineFunctionPass::getAnalysisUsage(AU);
@@ -163,14 +167,15 @@ bool AArch64PostCoalescerLegacy::runOnMachineFunction(MachineFunction &MF) {
if (skipFunction(MF.getFunction()))
return false;
- auto &LIS = getAnalysis<LiveIntervalsWrapperPass>().getLIS();
+ auto *LISWrapper = getAnalysisIfAvailable<LiveIntervalsWrapperPass>();
+ auto *LIS = LISWrapper ? &LISWrapper->getLIS() : nullptr;
return runAArch64PostCoalescer(MF, LIS);
}
PreservedAnalyses
AArch64PostCoalescerPass::run(MachineFunction &MF,
MachineFunctionAnalysisManager &MFAM) {
- auto &LIS = MFAM.getResult<LiveIntervalsAnalysis>(MF);
+ auto *LIS = MFAM.getCachedResult<LiveIntervalsAnalysis>(MF);
const bool Changed = runAArch64PostCoalescer(MF, LIS);
if (!Changed)
return PreservedAnalyses::all();
diff --git a/llvm/test/CodeGen/AArch64/expand-form-transposed-tuple.mir b/llvm/test/CodeGen/AArch64/expand-form-transposed-tuple.mir
index 52eeea589e8bf..0e4ae9a587849 100644
--- a/llvm/test/CodeGen/AArch64/expand-form-transposed-tuple.mir
+++ b/llvm/test/CodeGen/AArch64/expand-form-transposed-tuple.mir
@@ -68,3 +68,24 @@ body: |
FAKE_USE implicit %4, implicit %5
RET_ReallyLR
...
+---
+name: expand_tuple_set_kill_flag_on_last_copy
+tracksRegLiveness: true
+isSSA: false
+body: |
+ bb.0.entry:
+ ; CHECK-LABEL: name: expand_tuple_set_kill_flag_on_last_copy
+ ; CHECK: [[DEF:%[0-9]+]]:zpr4stridedorcontiguous = IMPLICIT_DEF
+ ; CHECK-NEXT: undef [[COPY_INTO_TRANSPOSED_TUPLE:%[0-9]+]].zsub0:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE [[DEF]].zsub0, 4
+ ; CHECK-NEXT: [[COPY_INTO_TRANSPOSED_TUPLE:%[0-9]+]].zsub1:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE [[DEF]].zsub1, 4
+ ; CHECK-NEXT: [[COPY_INTO_TRANSPOSED_TUPLE:%[0-9]+]].zsub2:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE [[DEF]].zsub2, 4
+ ; CHECK-NEXT: [[COPY_INTO_TRANSPOSED_TUPLE:%[0-9]+]].zsub3:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE killed [[DEF]].zsub3, 4
+ ; CHECK-NEXT: FAKE_USE implicit [[COPY_INTO_TRANSPOSED_TUPLE]]
+ ; CHECK-NEXT: RET_ReallyLR
+ %0:zpr4stridedorcontiguous = IMPLICIT_DEF
+
+ ; Reuses %0 multiple times. Only the last copy in the expansion should have the "killed" flag.
+ %4:zpr4mul4 = FORM_TRANSPOSED_REG_TUPLE_X4_PSEUDO killed %0.zsub0, killed %0.zsub1, killed %0.zsub2, killed %0.zsub3
+ FAKE_USE implicit %4
+ RET_ReallyLR
+...
More information about the llvm-commits
mailing list