[llvm] [AArch64][SME] Implement the SME ABI (ZA state management) in Machine IR (PR #149062)
Benjamin Maxwell via llvm-commits
llvm-commits at lists.llvm.org
Wed Jul 16 03:39:16 PDT 2025
https://github.com/MacDue created https://github.com/llvm/llvm-project/pull/149062
## Short Summary
This patch adds a new pass `aarch64-machine-sme-abi` to handle the ABI for ZA state (e.g., lazy saves and agnostic ZA functions). This is currently not enabled by default (but aims to be by LLVM 22). The goal is for this new pass to more optimally place ZA saves/restores and to work with exception handling.
## Long Description
This patch reimplements management of ZA state for functions with private and shared ZA state. Agnostic ZA functions will be handled in a later patch. For now, this is under the flag `-aarch64-new-sme-abi`, however, we intend for this to replace the current SelectionDAG implementation once complete.
The approach taken here is to mark instructions as needing ZA to be in a specific ("ACTIVE" or "LOCAL_SAVED"). Machine instructions implicitly defining or using ZA registers (such as $zt0 or $zab0) require the "ACTIVE" state. Function calls may need the "LOCAL_SAVED" or "ACTIVE" state depending on the callee (having shared or private ZA).
We already add ZA register uses/definitions to machine instructions, so no extra work is needed to mark these.
Calls need to be marked by glueing Arch64ISD::INOUT_ZA_USE or Arch64ISD::REQUIRES_ZA_SAVE to the CALLSEQ_START.
These markers are then used by the MachineSMEABIPass to find instructions where there is a transition between required ZA states. These are the points we need to insert code to set up or restore a ZA save (or initialize ZA).
To handle control flow between blocks (which may have different ZA state requirements), we bundle the incoming and outgoing edges of blocks. Bundles are formed by assigning each block an incoming and outgoing bundle (initially, all blocks have their own two bundles). Bundles are then combined by joining the outgoing bundle of a block with the incoming bundle of all successors.
These bundles are then assigned a ZA state based on the blocks that participate in the bundle. Blocks whose incoming edges are in a bundle "vote" for a ZA state that matches the state required at the first instruction in the block, and likewise, blocks whose outgoing edges are in a bundle vote for the ZA state that matches the last instruction in the block. The ZA state with the most votes is used, which aims to minimize the number of state transitions.
>From dfb3f42eb650e3e3bcf1dc59d003624ead3f8a81 Mon Sep 17 00:00:00 2001
From: Benjamin Maxwell <benjamin.maxwell at arm.com>
Date: Tue, 15 Jul 2025 10:50:56 +0000
Subject: [PATCH] [AArch64][SME] Implement the SME ABI (ZA state management) in
Machine IR
<h1>Short Summary</h1>
This patch adds a new pass `aarch64-machine-sme-abi` to handle the ABI
for ZA state (e.g., lazy saves and agnostic ZA functions). This is
currently not enabled by default (but aims to be by LLVM 22). The goal
is for this new pass to more optimally place ZA saves/restores and to
work with exception handling.
<h1>Long Description</h1>
This patch reimplements management of ZA state for functions with
private and shared ZA state. Agnostic ZA functions will be handled in
a later patch. For now, this is under the flag `-aarch64-new-sme-abi`,
however, we intend for this to replace the current SelectionDAG
implementation once complete.
The approach taken here is to mark instructions as needing ZA to be in a
specific ("ACTIVE" or "LOCAL_SAVED"). Machine instructions implicitly
defining or using ZA registers (such as $zt0 or $zab0) require the
"ACTIVE" state. Function calls may need the "LOCAL_SAVED" or "ACTIVE"
state depending on the callee (having shared or private ZA).
We already add ZA register uses/definitions to machine instructions,
so no extra work is needed to mark these.
Calls need to be marked by glueing Arch64ISD::INOUT_ZA_USE or
Arch64ISD::REQUIRES_ZA_SAVE to the CALLSEQ_START.
These markers are then used by the MachineSMEABIPass to find
instructions where there is a transition between required ZA states.
These are the points we need to insert code to set up or restore a ZA
save (or initialize ZA).
To handle control flow between blocks (which may have different ZA state
requirements), we bundle the incoming and outgoing edges of blocks.
Bundles are formed by assigning each block an incoming and outgoing
bundle (initially, all blocks have their own two bundles). Bundles are
then combined by joining the outgoing bundle of a block with the
incoming bundle of all successors.
These bundles are then assigned a ZA state based on the blocks that
participate in the bundle. Blocks whose incoming edges are in a bundle
"vote" for a ZA state that matches the state required at the first
instruction in the block, and likewise, blocks whose outgoing edges are
in a bundle vote for the ZA state that matches the last instruction in
the block. The ZA state with the most votes is used, which aims to
minimize the number of state transitions.
Change-Id: Iced4a3f329deab3ff8f3fd449a2337f7bbfa71ec
---
llvm/lib/Target/AArch64/AArch64.h | 2 +
.../AArch64/AArch64ExpandPseudoInsts.cpp | 39 +-
.../Target/AArch64/AArch64ISelLowering.cpp | 147 ++-
llvm/lib/Target/AArch64/AArch64ISelLowering.h | 4 +
.../AArch64/AArch64MachineFunctionInfo.h | 36 +-
.../lib/Target/AArch64/AArch64SMEInstrInfo.td | 28 +
llvm/lib/Target/AArch64/AArch64Subtarget.cpp | 3 +-
llvm/lib/Target/AArch64/AArch64Subtarget.h | 6 +-
.../Target/AArch64/AArch64TargetMachine.cpp | 25 +-
llvm/lib/Target/AArch64/CMakeLists.txt | 1 +
llvm/lib/Target/AArch64/MachineSMEABIPass.cpp | 642 ++++++++++
.../AArch64/Utils/AArch64SMEAttributes.h | 2 +-
llvm/test/CodeGen/AArch64/sme-agnostic-za.ll | 39 +
.../CodeGen/AArch64/sme-lazy-save-call.ll | 629 ++++++++-
.../AArch64/sme-lazy-sve-nzcv-live.mir | 132 ++
.../CodeGen/AArch64/sme-new-za-function.ll | 195 ++-
.../AArch64/sme-shared-za-interface.ll | 53 +-
.../CodeGen/AArch64/sme-za-control-flow.ll | 1131 +++++++++++++++++
.../test/CodeGen/AArch64/sme-za-exceptions.ll | 288 +++++
.../AArch64/sme-za-lazy-save-buffer.ll | 84 +-
llvm/test/CodeGen/AArch64/sme-zt0-state.ll | 229 +++-
.../CodeGen/AArch64/sve-stack-frame-layout.ll | 668 +++++-----
.../Target/AArch64/SMEAttributesTest.cpp | 4 +-
23 files changed, 3884 insertions(+), 503 deletions(-)
create mode 100644 llvm/lib/Target/AArch64/MachineSMEABIPass.cpp
create mode 100644 llvm/test/CodeGen/AArch64/sme-lazy-sve-nzcv-live.mir
create mode 100644 llvm/test/CodeGen/AArch64/sme-za-control-flow.ll
create mode 100644 llvm/test/CodeGen/AArch64/sme-za-exceptions.ll
diff --git a/llvm/lib/Target/AArch64/AArch64.h b/llvm/lib/Target/AArch64/AArch64.h
index 5496ebd495a55..8d0ff41fc8c08 100644
--- a/llvm/lib/Target/AArch64/AArch64.h
+++ b/llvm/lib/Target/AArch64/AArch64.h
@@ -60,6 +60,7 @@ FunctionPass *createAArch64CleanupLocalDynamicTLSPass();
FunctionPass *createAArch64CollectLOHPass();
FunctionPass *createSMEABIPass();
FunctionPass *createSMEPeepholeOptPass();
+FunctionPass *createMachineSMEABIPass();
ModulePass *createSVEIntrinsicOptsPass();
InstructionSelector *
createAArch64InstructionSelector(const AArch64TargetMachine &,
@@ -111,6 +112,7 @@ void initializeFalkorMarkStridedAccessesLegacyPass(PassRegistry&);
void initializeLDTLSCleanupPass(PassRegistry&);
void initializeSMEABIPass(PassRegistry &);
void initializeSMEPeepholeOptPass(PassRegistry &);
+void initializeMachineSMEABIPass(PassRegistry &);
void initializeSVEIntrinsicOptsPass(PassRegistry &);
void initializeAArch64Arm64ECCallLoweringPass(PassRegistry &);
} // end namespace llvm
diff --git a/llvm/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp b/llvm/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp
index 7de66ccbf6f29..c0d118aa3afed 100644
--- a/llvm/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp
+++ b/llvm/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp
@@ -92,8 +92,8 @@ class AArch64ExpandPseudo : public MachineFunctionPass {
bool expandCALL_BTI(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI);
bool expandStoreSwiftAsyncContext(MachineBasicBlock &MBB,
MachineBasicBlock::iterator MBBI);
- MachineBasicBlock *expandRestoreZA(MachineBasicBlock &MBB,
- MachineBasicBlock::iterator MBBI);
+ MachineBasicBlock *expandCommitOrRestoreZA(MachineBasicBlock &MBB,
+ MachineBasicBlock::iterator MBBI);
MachineBasicBlock *expandCondSMToggle(MachineBasicBlock &MBB,
MachineBasicBlock::iterator MBBI);
};
@@ -974,9 +974,13 @@ bool AArch64ExpandPseudo::expandStoreSwiftAsyncContext(
}
MachineBasicBlock *
-AArch64ExpandPseudo::expandRestoreZA(MachineBasicBlock &MBB,
- MachineBasicBlock::iterator MBBI) {
+AArch64ExpandPseudo::expandCommitOrRestoreZA(MachineBasicBlock &MBB,
+ MachineBasicBlock::iterator MBBI) {
MachineInstr &MI = *MBBI;
+ bool IsRestoreZA = MI.getOpcode() == AArch64::RestoreZAPseudo;
+ assert((MI.getOpcode() == AArch64::RestoreZAPseudo ||
+ MI.getOpcode() == AArch64::CommitZAPseudo) &&
+ "Expected ZA commit or restore");
assert((std::next(MBBI) != MBB.end() ||
MI.getParent()->successors().begin() !=
MI.getParent()->successors().end()) &&
@@ -984,21 +988,23 @@ AArch64ExpandPseudo::expandRestoreZA(MachineBasicBlock &MBB,
// Compare TPIDR2_EL0 value against 0.
DebugLoc DL = MI.getDebugLoc();
- MachineInstrBuilder Cbz = BuildMI(MBB, MBBI, DL, TII->get(AArch64::CBZX))
- .add(MI.getOperand(0));
+ MachineInstrBuilder Branch =
+ BuildMI(MBB, MBBI, DL,
+ TII->get(IsRestoreZA ? AArch64::CBZX : AArch64::CBNZX))
+ .add(MI.getOperand(0));
// Split MBB and create two new blocks:
// - MBB now contains all instructions before RestoreZAPseudo.
- // - SMBB contains the RestoreZAPseudo instruction only.
- // - EndBB contains all instructions after RestoreZAPseudo.
+ // - SMBB contains the [Commit|RestoreZA]Pseudo instruction only.
+ // - EndBB contains all instructions after [Commit|RestoreZA]Pseudo.
MachineInstr &PrevMI = *std::prev(MBBI);
MachineBasicBlock *SMBB = MBB.splitAt(PrevMI, /*UpdateLiveIns*/ true);
MachineBasicBlock *EndBB = std::next(MI.getIterator()) == SMBB->end()
? *SMBB->successors().begin()
: SMBB->splitAt(MI, /*UpdateLiveIns*/ true);
- // Add the SMBB label to the TB[N]Z instruction & create a branch to EndBB.
- Cbz.addMBB(SMBB);
+ // Add the SMBB label to the CB[N]Z instruction & create a branch to EndBB.
+ Branch.addMBB(SMBB);
BuildMI(&MBB, DL, TII->get(AArch64::B))
.addMBB(EndBB);
MBB.addSuccessor(EndBB);
@@ -1006,8 +1012,12 @@ AArch64ExpandPseudo::expandRestoreZA(MachineBasicBlock &MBB,
// Replace the pseudo with a call (BL).
MachineInstrBuilder MIB =
BuildMI(*SMBB, SMBB->end(), DL, TII->get(AArch64::BL));
- MIB.addReg(MI.getOperand(1).getReg(), RegState::Implicit);
- for (unsigned I = 2; I < MI.getNumOperands(); ++I)
+ unsigned FirstBLOperand = 1;
+ if (IsRestoreZA) {
+ MIB.addReg(MI.getOperand(1).getReg(), RegState::Implicit);
+ FirstBLOperand = 2;
+ }
+ for (unsigned I = FirstBLOperand; I < MI.getNumOperands(); ++I)
MIB.add(MI.getOperand(I));
BuildMI(SMBB, DL, TII->get(AArch64::B)).addMBB(EndBB);
@@ -1617,8 +1627,9 @@ bool AArch64ExpandPseudo::expandMI(MachineBasicBlock &MBB,
return expandCALL_BTI(MBB, MBBI);
case AArch64::StoreSwiftAsyncContext:
return expandStoreSwiftAsyncContext(MBB, MBBI);
+ case AArch64::CommitZAPseudo:
case AArch64::RestoreZAPseudo: {
- auto *NewMBB = expandRestoreZA(MBB, MBBI);
+ auto *NewMBB = expandCommitOrRestoreZA(MBB, MBBI);
if (NewMBB != &MBB)
NextMBBI = MBB.end(); // The NextMBBI iterator is invalidated.
return true;
@@ -1629,6 +1640,8 @@ bool AArch64ExpandPseudo::expandMI(MachineBasicBlock &MBB,
NextMBBI = MBB.end(); // The NextMBBI iterator is invalidated.
return true;
}
+ case AArch64::InOutZAUsePseudo:
+ case AArch64::RequiresZASavePseudo:
case AArch64::COALESCER_BARRIER_FPR16:
case AArch64::COALESCER_BARRIER_FPR32:
case AArch64::COALESCER_BARRIER_FPR64:
diff --git a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
index 4f13a14d24649..49135d05b689b 100644
--- a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
+++ b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
@@ -8154,53 +8154,54 @@ SDValue AArch64TargetLowering::LowerFormalArguments(
if (Subtarget->hasCustomCallingConv())
Subtarget->getRegisterInfo()->UpdateCustomCalleeSavedRegs(MF);
- // Create a 16 Byte TPIDR2 object. The dynamic buffer
- // will be expanded and stored in the static object later using a pseudonode.
- if (Attrs.hasZAState()) {
- TPIDR2Object &TPIDR2 = FuncInfo->getTPIDR2Obj();
- TPIDR2.FrameIndex = MFI.CreateStackObject(16, Align(16), false);
- SDValue SVL = DAG.getNode(AArch64ISD::RDSVL, DL, MVT::i64,
- DAG.getConstant(1, DL, MVT::i32));
-
- SDValue Buffer;
- if (!Subtarget->isTargetWindows() && !hasInlineStackProbe(MF)) {
- Buffer = DAG.getNode(AArch64ISD::ALLOCATE_ZA_BUFFER, DL,
- DAG.getVTList(MVT::i64, MVT::Other), {Chain, SVL});
- } else {
- SDValue Size = DAG.getNode(ISD::MUL, DL, MVT::i64, SVL, SVL);
- Buffer = DAG.getNode(ISD::DYNAMIC_STACKALLOC, DL,
- DAG.getVTList(MVT::i64, MVT::Other),
- {Chain, Size, DAG.getConstant(1, DL, MVT::i64)});
- MFI.CreateVariableSizedObject(Align(16), nullptr);
- }
- Chain = DAG.getNode(
- AArch64ISD::INIT_TPIDR2OBJ, DL, DAG.getVTList(MVT::Other),
- {/*Chain*/ Buffer.getValue(1), /*Buffer ptr*/ Buffer.getValue(0)});
- } else if (Attrs.hasAgnosticZAInterface()) {
- // Call __arm_sme_state_size().
- SDValue BufferSize =
- DAG.getNode(AArch64ISD::GET_SME_SAVE_SIZE, DL,
- DAG.getVTList(MVT::i64, MVT::Other), Chain);
- Chain = BufferSize.getValue(1);
-
- SDValue Buffer;
- if (!Subtarget->isTargetWindows() && !hasInlineStackProbe(MF)) {
- Buffer =
- DAG.getNode(AArch64ISD::ALLOC_SME_SAVE_BUFFER, DL,
- DAG.getVTList(MVT::i64, MVT::Other), {Chain, BufferSize});
- } else {
- // Allocate space dynamically.
- Buffer = DAG.getNode(
- ISD::DYNAMIC_STACKALLOC, DL, DAG.getVTList(MVT::i64, MVT::Other),
- {Chain, BufferSize, DAG.getConstant(1, DL, MVT::i64)});
- MFI.CreateVariableSizedObject(Align(16), nullptr);
+ if (!Subtarget->useNewSMEABILowering() || Attrs.hasAgnosticZAInterface()) {
+ // Old SME ABI lowering (deprecated):
+ // Create a 16 Byte TPIDR2 object. The dynamic buffer
+ // will be expanded and stored in the static object later using a
+ // pseudonode.
+ if (Attrs.hasZAState()) {
+ TPIDR2Object &TPIDR2 = FuncInfo->getTPIDR2Obj();
+ TPIDR2.FrameIndex = MFI.CreateStackObject(16, Align(16), false);
+ SDValue SVL = DAG.getNode(AArch64ISD::RDSVL, DL, MVT::i64,
+ DAG.getConstant(1, DL, MVT::i32));
+ SDValue Buffer;
+ if (!Subtarget->isTargetWindows() && !hasInlineStackProbe(MF)) {
+ Buffer = DAG.getNode(AArch64ISD::ALLOCATE_ZA_BUFFER, DL,
+ DAG.getVTList(MVT::i64, MVT::Other), {Chain, SVL});
+ } else {
+ SDValue Size = DAG.getNode(ISD::MUL, DL, MVT::i64, SVL, SVL);
+ Buffer = DAG.getNode(ISD::DYNAMIC_STACKALLOC, DL,
+ DAG.getVTList(MVT::i64, MVT::Other),
+ {Chain, Size, DAG.getConstant(1, DL, MVT::i64)});
+ MFI.CreateVariableSizedObject(Align(16), nullptr);
+ }
+ Chain = DAG.getNode(
+ AArch64ISD::INIT_TPIDR2OBJ, DL, DAG.getVTList(MVT::Other),
+ {/*Chain*/ Buffer.getValue(1), /*Buffer ptr*/ Buffer.getValue(0)});
+ } else if (Attrs.hasAgnosticZAInterface()) {
+ // Call __arm_sme_state_size().
+ SDValue BufferSize =
+ DAG.getNode(AArch64ISD::GET_SME_SAVE_SIZE, DL,
+ DAG.getVTList(MVT::i64, MVT::Other), Chain);
+ Chain = BufferSize.getValue(1);
+ SDValue Buffer;
+ if (!Subtarget->isTargetWindows() && !hasInlineStackProbe(MF)) {
+ Buffer = DAG.getNode(AArch64ISD::ALLOC_SME_SAVE_BUFFER, DL,
+ DAG.getVTList(MVT::i64, MVT::Other),
+ {Chain, BufferSize});
+ } else {
+ // Allocate space dynamically.
+ Buffer = DAG.getNode(
+ ISD::DYNAMIC_STACKALLOC, DL, DAG.getVTList(MVT::i64, MVT::Other),
+ {Chain, BufferSize, DAG.getConstant(1, DL, MVT::i64)});
+ MFI.CreateVariableSizedObject(Align(16), nullptr);
+ }
+ // Copy the value to a virtual register, and save that in FuncInfo.
+ Register BufferPtr =
+ MF.getRegInfo().createVirtualRegister(&AArch64::GPR64RegClass);
+ FuncInfo->setSMESaveBufferAddr(BufferPtr);
+ Chain = DAG.getCopyToReg(Chain, DL, BufferPtr, Buffer);
}
-
- // Copy the value to a virtual register, and save that in FuncInfo.
- Register BufferPtr =
- MF.getRegInfo().createVirtualRegister(&AArch64::GPR64RegClass);
- FuncInfo->setSMESaveBufferAddr(BufferPtr);
- Chain = DAG.getCopyToReg(Chain, DL, BufferPtr, Buffer);
}
if (CallConv == CallingConv::PreserveNone) {
@@ -8217,6 +8218,15 @@ SDValue AArch64TargetLowering::LowerFormalArguments(
}
}
+ if (Subtarget->useNewSMEABILowering()) {
+ // Clear new ZT0 state. TODO: Move this to the SME ABI pass.
+ if (Attrs.isNewZT0())
+ Chain = DAG.getNode(
+ ISD::INTRINSIC_VOID, DL, MVT::Other, Chain,
+ DAG.getConstant(Intrinsic::aarch64_sme_zero_zt, DL, MVT::i32),
+ DAG.getTargetConstant(0, DL, MVT::i32));
+ }
+
return Chain;
}
@@ -8781,14 +8791,12 @@ static SDValue emitSMEStateSaveRestore(const AArch64TargetLowering &TLI,
MachineFunction &MF = DAG.getMachineFunction();
AArch64FunctionInfo *FuncInfo = MF.getInfo<AArch64FunctionInfo>();
FuncInfo->setSMESaveBufferUsed();
-
TargetLowering::ArgListTy Args;
TargetLowering::ArgListEntry Entry;
Entry.Ty = PointerType::getUnqual(*DAG.getContext());
Entry.Node =
DAG.getCopyFromReg(Chain, DL, Info->getSMESaveBufferAddr(), MVT::i64);
Args.push_back(Entry);
-
SDValue Callee =
DAG.getExternalSymbol(IsSave ? "__arm_sme_save" : "__arm_sme_restore",
TLI.getPointerTy(DAG.getDataLayout()));
@@ -8906,6 +8914,9 @@ AArch64TargetLowering::LowerCall(CallLoweringInfo &CLI,
*DAG.getContext());
RetCCInfo.AnalyzeCallResult(Ins, RetCC);
+ // Determine whether we need any streaming mode changes.
+ SMECallAttrs CallAttrs = getSMECallAttrs(MF.getFunction(), CLI);
+
// Check callee args/returns for SVE registers and set calling convention
// accordingly.
if (CallConv == CallingConv::C || CallConv == CallingConv::Fast) {
@@ -8919,14 +8930,26 @@ AArch64TargetLowering::LowerCall(CallLoweringInfo &CLI,
CallConv = CallingConv::AArch64_SVE_VectorCall;
}
+ bool UseNewSMEABILowering = Subtarget->useNewSMEABILowering();
+ bool IsAgnosticZAFunction = CallAttrs.caller().hasAgnosticZAInterface();
+ auto ZAMarkerNode = [&]() -> std::optional<unsigned> {
+ // TODO: Handle agnostic ZA functions.
+ if (!UseNewSMEABILowering || IsAgnosticZAFunction)
+ return std::nullopt;
+ if (!CallAttrs.caller().hasZAState() && !CallAttrs.caller().hasZT0State())
+ return std::nullopt;
+ return CallAttrs.requiresLazySave() ? AArch64ISD::REQUIRES_ZA_SAVE
+ : AArch64ISD::INOUT_ZA_USE;
+ }();
+
if (IsTailCall) {
// Check if it's really possible to do a tail call.
IsTailCall = isEligibleForTailCallOptimization(CLI);
// A sibling call is one where we're under the usual C ABI and not planning
// to change that but can still do a tail call:
- if (!TailCallOpt && IsTailCall && CallConv != CallingConv::Tail &&
- CallConv != CallingConv::SwiftTail)
+ if (!ZAMarkerNode.has_value() && !TailCallOpt && IsTailCall &&
+ CallConv != CallingConv::Tail && CallConv != CallingConv::SwiftTail)
IsSibCall = true;
if (IsTailCall)
@@ -8978,9 +9001,6 @@ AArch64TargetLowering::LowerCall(CallLoweringInfo &CLI,
assert(FPDiff % 16 == 0 && "unaligned stack on tail call");
}
- // Determine whether we need any streaming mode changes.
- SMECallAttrs CallAttrs = getSMECallAttrs(MF.getFunction(), CLI);
-
auto DescribeCallsite =
[&](OptimizationRemarkAnalysis &R) -> OptimizationRemarkAnalysis & {
R << "call from '" << ore::NV("Caller", MF.getName()) << "' to '";
@@ -8994,7 +9014,7 @@ AArch64TargetLowering::LowerCall(CallLoweringInfo &CLI,
return R;
};
- bool RequiresLazySave = CallAttrs.requiresLazySave();
+ bool RequiresLazySave = !UseNewSMEABILowering && CallAttrs.requiresLazySave();
bool RequiresSaveAllZA = CallAttrs.requiresPreservingAllZAState();
if (RequiresLazySave) {
const TPIDR2Object &TPIDR2 = FuncInfo->getTPIDR2Obj();
@@ -9076,10 +9096,21 @@ AArch64TargetLowering::LowerCall(CallLoweringInfo &CLI,
AArch64ISD::SMSTOP, DL, DAG.getVTList(MVT::Other, MVT::Glue), Chain,
DAG.getTargetConstant((int32_t)(AArch64SVCR::SVCRZA), DL, MVT::i32));
- // Adjust the stack pointer for the new arguments...
+ // Adjust the stack pointer for the new arguments... and mark ZA uses.
// These operations are automatically eliminated by the prolog/epilog pass
- if (!IsSibCall)
+ assert((!IsSibCall || !ZAMarkerNode.has_value()) &&
+ "ZA markers require CALLSEQ_START");
+ if (!IsSibCall) {
Chain = DAG.getCALLSEQ_START(Chain, IsTailCall ? 0 : NumBytes, 0, DL);
+ if (ZAMarkerNode) {
+ // Note: We need the CALLSEQ_START to glue the ZAMarkerNode to, simply
+ // using a chain can result in incorrect scheduling. The markers referer
+ // to the position just before the CALLSEQ_START (though occur after as
+ // CALLSEQ_START lacks in-glue).
+ Chain = DAG.getNode(*ZAMarkerNode, DL, DAG.getVTList(MVT::Other),
+ {Chain, Chain.getValue(1)});
+ }
+ }
SDValue StackPtr = DAG.getCopyFromReg(Chain, DL, AArch64::SP,
getPointerTy(DAG.getDataLayout()));
@@ -9551,7 +9582,7 @@ AArch64TargetLowering::LowerCall(CallLoweringInfo &CLI,
}
}
- if (CallAttrs.requiresEnablingZAAfterCall())
+ if (RequiresLazySave || CallAttrs.requiresEnablingZAAfterCall())
// Unconditionally resume ZA.
Result = DAG.getNode(
AArch64ISD::SMSTART, DL, DAG.getVTList(MVT::Other, MVT::Glue), Result,
@@ -9572,7 +9603,6 @@ AArch64TargetLowering::LowerCall(CallLoweringInfo &CLI,
SDValue TPIDR2_EL0 = DAG.getNode(
ISD::INTRINSIC_W_CHAIN, DL, MVT::i64, Result,
DAG.getConstant(Intrinsic::aarch64_sme_get_tpidr2, DL, MVT::i32));
-
// Copy the address of the TPIDR2 block into X0 before 'calling' the
// RESTORE_ZA pseudo.
SDValue Glue;
@@ -9584,7 +9614,6 @@ AArch64TargetLowering::LowerCall(CallLoweringInfo &CLI,
DAG.getNode(AArch64ISD::RESTORE_ZA, DL, MVT::Other,
{Result, TPIDR2_EL0, DAG.getRegister(AArch64::X0, MVT::i64),
RestoreRoutine, RegMask, Result.getValue(1)});
-
// Finally reset the TPIDR2_EL0 register to 0.
Result = DAG.getNode(
ISD::INTRINSIC_VOID, DL, MVT::Other, Result,
diff --git a/llvm/lib/Target/AArch64/AArch64ISelLowering.h b/llvm/lib/Target/AArch64/AArch64ISelLowering.h
index 6afb3c330d25b..72897a0446ca2 100644
--- a/llvm/lib/Target/AArch64/AArch64ISelLowering.h
+++ b/llvm/lib/Target/AArch64/AArch64ISelLowering.h
@@ -173,6 +173,10 @@ class AArch64TargetLowering : public TargetLowering {
MachineBasicBlock *EmitZTInstr(MachineInstr &MI, MachineBasicBlock *BB,
unsigned Opcode, bool Op0IsDef) const;
MachineBasicBlock *EmitZero(MachineInstr &MI, MachineBasicBlock *BB) const;
+
+ // Note: The following group of functions are only used as part of the old SME
+ // ABI lowering. They will be removed once -aarch64-new-sme-abi=true is the
+ // default.
MachineBasicBlock *EmitInitTPIDR2Object(MachineInstr &MI,
MachineBasicBlock *BB) const;
MachineBasicBlock *EmitAllocateZABuffer(MachineInstr &MI,
diff --git a/llvm/lib/Target/AArch64/AArch64MachineFunctionInfo.h b/llvm/lib/Target/AArch64/AArch64MachineFunctionInfo.h
index 800787cc0b4f5..3f6980fe11aea 100644
--- a/llvm/lib/Target/AArch64/AArch64MachineFunctionInfo.h
+++ b/llvm/lib/Target/AArch64/AArch64MachineFunctionInfo.h
@@ -213,9 +213,6 @@ class AArch64FunctionInfo final : public MachineFunctionInfo {
/// or return type
bool IsSVECC = false;
- /// The frame-index for the TPIDR2 object used for lazy saves.
- TPIDR2Object TPIDR2;
-
/// Whether this function changes streaming mode within the function.
bool HasStreamingModeChanges = false;
@@ -231,14 +228,6 @@ class AArch64FunctionInfo final : public MachineFunctionInfo {
// on function entry to record the initial pstate of a function.
Register PStateSMReg = MCRegister::NoRegister;
- // Holds a pointer to a buffer that is large enough to represent
- // all SME ZA state and any additional state required by the
- // __arm_sme_save/restore support routines.
- Register SMESaveBufferAddr = MCRegister::NoRegister;
-
- // true if SMESaveBufferAddr is used.
- bool SMESaveBufferUsed = false;
-
// Has the PNReg used to build PTRUE instruction.
// The PTRUE is used for the LD/ST of ZReg pairs in save and restore.
unsigned PredicateRegForFillSpill = 0;
@@ -250,6 +239,16 @@ class AArch64FunctionInfo final : public MachineFunctionInfo {
// Holds the SME function attributes (streaming mode, ZA/ZT0 state).
SMEAttrs SMEFnAttrs;
+ // Note: The following properties are only used for the old SME ABI lowering:
+ /// The frame-index for the TPIDR2 object used for lazy saves.
+ TPIDR2Object TPIDR2;
+ // Holds a pointer to a buffer that is large enough to represent
+ // all SME ZA state and any additional state required by the
+ // __arm_sme_save/restore support routines.
+ Register SMESaveBufferAddr = MCRegister::NoRegister;
+ // true if SMESaveBufferAddr is used.
+ bool SMESaveBufferUsed = false;
+
public:
AArch64FunctionInfo(const Function &F, const AArch64Subtarget *STI);
@@ -258,6 +257,13 @@ class AArch64FunctionInfo final : public MachineFunctionInfo {
const DenseMap<MachineBasicBlock *, MachineBasicBlock *> &Src2DstMBB)
const override;
+ // Old SME ABI lowering state getters/setters:
+ Register getSMESaveBufferAddr() const { return SMESaveBufferAddr; };
+ void setSMESaveBufferAddr(Register Reg) { SMESaveBufferAddr = Reg; };
+ unsigned isSMESaveBufferUsed() const { return SMESaveBufferUsed; };
+ void setSMESaveBufferUsed(bool Used = true) { SMESaveBufferUsed = Used; };
+ TPIDR2Object &getTPIDR2Obj() { return TPIDR2; }
+
void setPredicateRegForFillSpill(unsigned Reg) {
PredicateRegForFillSpill = Reg;
}
@@ -265,12 +271,6 @@ class AArch64FunctionInfo final : public MachineFunctionInfo {
return PredicateRegForFillSpill;
}
- Register getSMESaveBufferAddr() const { return SMESaveBufferAddr; };
- void setSMESaveBufferAddr(Register Reg) { SMESaveBufferAddr = Reg; };
-
- unsigned isSMESaveBufferUsed() const { return SMESaveBufferUsed; };
- void setSMESaveBufferUsed(bool Used = true) { SMESaveBufferUsed = Used; };
-
Register getPStateSMReg() const { return PStateSMReg; };
void setPStateSMReg(Register Reg) { PStateSMReg = Reg; };
@@ -283,8 +283,6 @@ class AArch64FunctionInfo final : public MachineFunctionInfo {
bool isSVECC() const { return IsSVECC; };
void setIsSVECC(bool s) { IsSVECC = s; };
- TPIDR2Object &getTPIDR2Obj() { return TPIDR2; }
-
void initializeBaseYamlFields(const yaml::AArch64FunctionInfo &YamlMFI);
unsigned getBytesInStackArgArea() const { return BytesInStackArgArea; }
diff --git a/llvm/lib/Target/AArch64/AArch64SMEInstrInfo.td b/llvm/lib/Target/AArch64/AArch64SMEInstrInfo.td
index db27ca978980f..d50dcd8ebd815 100644
--- a/llvm/lib/Target/AArch64/AArch64SMEInstrInfo.td
+++ b/llvm/lib/Target/AArch64/AArch64SMEInstrInfo.td
@@ -45,6 +45,10 @@ def AArch64VGSave : SDNode<"AArch64ISD::VG_SAVE", SDTypeProfile<0, 0, []>,
def AArch64VGRestore : SDNode<"AArch64ISD::VG_RESTORE", SDTypeProfile<0, 0, []>,
[SDNPHasChain, SDNPSideEffect, SDNPOptInGlue, SDNPOutGlue]>;
+//===----------------------------------------------------------------------===//
+// Old SME ABI lowering ISD nodes/pseudos (deprecated)
+//===----------------------------------------------------------------------===//
+
def AArch64AllocateZABuffer : SDNode<"AArch64ISD::ALLOCATE_ZA_BUFFER", SDTypeProfile<1, 1,
[SDTCisInt<0>, SDTCisInt<1>]>,
[SDNPHasChain, SDNPSideEffect]>;
@@ -77,6 +81,30 @@ let usesCustomInserter = 1, Defs = [SP] in {
def : Pat<(i64 (AArch64AllocateSMESaveBuffer GPR64:$size)),
(AllocateSMESaveBuffer $size)>;
+//===----------------------------------------------------------------------===//
+// New SME ABI lowering ISD nodes/pseudos (-aarch64-new-sme-abi)
+//===----------------------------------------------------------------------===//
+
+let hasSideEffects = 1 in {
+ def InOutZAUsePseudo : Pseudo<(outs), (ins), []>, Sched<[]>;
+ def RequiresZASavePseudo : Pseudo<(outs), (ins), []>, Sched<[]>;
+}
+
+def CommitZAPseudo
+ : Pseudo<(outs),
+ (ins GPR64:$tpidr2_el0, i64imm:$restore_routine, variable_ops), []>,
+ Sched<[]>;
+
+def AArch64_inout_za_use
+ : SDNode<"AArch64ISD::INOUT_ZA_USE", SDTypeProfile<0, 0,[]>,
+ [SDNPHasChain, SDNPInGlue]>;
+def : Pat<(AArch64_inout_za_use), (InOutZAUsePseudo)>;
+
+def AArch64_requires_za_save
+ : SDNode<"AArch64ISD::REQUIRES_ZA_SAVE", SDTypeProfile<0, 0,[]>,
+ [SDNPHasChain, SDNPInGlue]>;
+def : Pat<(AArch64_requires_za_save), (RequiresZASavePseudo)>;
+
//===----------------------------------------------------------------------===//
// Instruction naming conventions.
//===----------------------------------------------------------------------===//
diff --git a/llvm/lib/Target/AArch64/AArch64Subtarget.cpp b/llvm/lib/Target/AArch64/AArch64Subtarget.cpp
index 2409cc862f21c..d36333057303e 100644
--- a/llvm/lib/Target/AArch64/AArch64Subtarget.cpp
+++ b/llvm/lib/Target/AArch64/AArch64Subtarget.cpp
@@ -372,13 +372,14 @@ AArch64Subtarget::AArch64Subtarget(const Triple &TT, StringRef CPU,
unsigned MinSVEVectorSizeInBitsOverride,
unsigned MaxSVEVectorSizeInBitsOverride,
bool IsStreaming, bool IsStreamingCompatible,
- bool HasMinSize)
+ bool HasMinSize, bool UseNewSMEABILowering)
: AArch64GenSubtargetInfo(TT, CPU, TuneCPU, FS),
ReserveXRegister(AArch64::GPR64commonRegClass.getNumRegs()),
ReserveXRegisterForRA(AArch64::GPR64commonRegClass.getNumRegs()),
CustomCallSavedXRegs(AArch64::GPR64commonRegClass.getNumRegs()),
IsLittle(LittleEndian), IsStreaming(IsStreaming),
IsStreamingCompatible(IsStreamingCompatible),
+ UseNewSMEABILowering(UseNewSMEABILowering),
StreamingHazardSize(
AArch64StreamingHazardSize.getNumOccurrences() > 0
? std::optional<unsigned>(AArch64StreamingHazardSize)
diff --git a/llvm/lib/Target/AArch64/AArch64Subtarget.h b/llvm/lib/Target/AArch64/AArch64Subtarget.h
index 154db3c074f71..95b33346f773a 100644
--- a/llvm/lib/Target/AArch64/AArch64Subtarget.h
+++ b/llvm/lib/Target/AArch64/AArch64Subtarget.h
@@ -85,6 +85,7 @@ class AArch64Subtarget final : public AArch64GenSubtargetInfo {
bool IsStreaming;
bool IsStreamingCompatible;
+ bool UseNewSMEABILowering = false;
std::optional<unsigned> StreamingHazardSize;
unsigned MinSVEVectorSizeInBits;
unsigned MaxSVEVectorSizeInBits;
@@ -128,7 +129,7 @@ class AArch64Subtarget final : public AArch64GenSubtargetInfo {
unsigned MinSVEVectorSizeInBitsOverride = 0,
unsigned MaxSVEVectorSizeInBitsOverride = 0,
bool IsStreaming = false, bool IsStreamingCompatible = false,
- bool HasMinSize = false);
+ bool HasMinSize = false, bool UseNewSMEABILowering = false);
virtual unsigned getHwModeSet() const override;
@@ -212,6 +213,9 @@ class AArch64Subtarget final : public AArch64GenSubtargetInfo {
return hasSVE() || isStreamingSVEAvailable();
}
+ /// Returns true if the new SME ABI lowering should be used.
+ bool useNewSMEABILowering() const { return UseNewSMEABILowering; }
+
unsigned getMinVectorRegisterBitWidth() const {
// Don't assume any minimum vector size when PSTATE.SM may not be 0, because
// we don't yet support streaming-compatible codegen support that we trust
diff --git a/llvm/lib/Target/AArch64/AArch64TargetMachine.cpp b/llvm/lib/Target/AArch64/AArch64TargetMachine.cpp
index 95eab16511e5a..2c1edecd0b48d 100644
--- a/llvm/lib/Target/AArch64/AArch64TargetMachine.cpp
+++ b/llvm/lib/Target/AArch64/AArch64TargetMachine.cpp
@@ -224,6 +224,11 @@ static cl::opt<bool>
cl::desc("Enable Machine Pipeliner for AArch64"),
cl::init(false), cl::Hidden);
+static cl::opt<bool>
+ EnableNewSMEABILowering("aarch64-new-sme-abi",
+ cl::desc("Enable new lowering for the SME ABI"),
+ cl::init(false), cl::Hidden);
+
extern "C" LLVM_ABI LLVM_EXTERNAL_VISIBILITY void
LLVMInitializeAArch64Target() {
// Register the target.
@@ -263,6 +268,7 @@ LLVMInitializeAArch64Target() {
initializeLDTLSCleanupPass(PR);
initializeKCFIPass(PR);
initializeSMEABIPass(PR);
+ initializeMachineSMEABIPass(PR);
initializeSMEPeepholeOptPass(PR);
initializeSVEIntrinsicOptsPass(PR);
initializeAArch64SpeculationHardeningPass(PR);
@@ -478,7 +484,8 @@ AArch64TargetMachine::getSubtargetImpl(const Function &F) const {
resetTargetOptions(F);
I = std::make_unique<AArch64Subtarget>(
TargetTriple, CPU, TuneCPU, FS, *this, isLittle, MinSVEVectorSize,
- MaxSVEVectorSize, IsStreaming, IsStreamingCompatible, HasMinSize);
+ MaxSVEVectorSize, IsStreaming, IsStreamingCompatible, HasMinSize,
+ EnableNewSMEABILowering);
}
if (IsStreaming && !I->hasSME())
@@ -668,10 +675,12 @@ void AArch64PassConfig::addIRPasses() {
addPass(createInterleavedAccessPass());
}
- // Expand any functions marked with SME attributes which require special
- // changes for the calling convention or that require the lazy-saving
- // mechanism specified in the SME ABI.
- addPass(createSMEABIPass());
+ if (!EnableNewSMEABILowering) {
+ // Expand any functions marked with SME attributes which require special
+ // changes for the calling convention or that require the lazy-saving
+ // mechanism specified in the SME ABI.
+ addPass(createSMEABIPass());
+ }
// Add Control Flow Guard checks.
if (TM->getTargetTriple().isOSWindows()) {
@@ -782,6 +791,9 @@ bool AArch64PassConfig::addGlobalInstructionSelect() {
}
void AArch64PassConfig::addMachineSSAOptimization() {
+ if (EnableNewSMEABILowering && TM->getOptLevel() != CodeGenOptLevel::None)
+ addPass(createMachineSMEABIPass());
+
if (TM->getOptLevel() != CodeGenOptLevel::None && EnableSMEPeepholeOpt)
addPass(createSMEPeepholeOptPass());
@@ -812,6 +824,9 @@ bool AArch64PassConfig::addILPOpts() {
}
void AArch64PassConfig::addPreRegAlloc() {
+ if (EnableNewSMEABILowering && TM->getOptLevel() == CodeGenOptLevel::None)
+ addPass(createMachineSMEABIPass());
+
// Change dead register definitions to refer to the zero register.
if (TM->getOptLevel() != CodeGenOptLevel::None &&
EnableDeadRegisterElimination)
diff --git a/llvm/lib/Target/AArch64/CMakeLists.txt b/llvm/lib/Target/AArch64/CMakeLists.txt
index 66136a464f05d..803943fd57c4d 100644
--- a/llvm/lib/Target/AArch64/CMakeLists.txt
+++ b/llvm/lib/Target/AArch64/CMakeLists.txt
@@ -89,6 +89,7 @@ add_llvm_target(AArch64CodeGen
SMEABIPass.cpp
SMEPeepholeOpt.cpp
SVEIntrinsicOpts.cpp
+ MachineSMEABIPass.cpp
AArch64SIMDInstrOpt.cpp
DEPENDS
diff --git a/llvm/lib/Target/AArch64/MachineSMEABIPass.cpp b/llvm/lib/Target/AArch64/MachineSMEABIPass.cpp
new file mode 100644
index 0000000000000..9ef4d71d04568
--- /dev/null
+++ b/llvm/lib/Target/AArch64/MachineSMEABIPass.cpp
@@ -0,0 +1,642 @@
+//===- MachineSMEABIPass.cpp ----------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This pass implements the SME ABI requirements for ZA state. This includes
+// implementing the lazy ZA state save schemes around calls.
+//
+//===----------------------------------------------------------------------===//
+
+#include "AArch64InstrInfo.h"
+#include "AArch64MachineFunctionInfo.h"
+#include "AArch64Subtarget.h"
+#include "MCTargetDesc/AArch64AddressingModes.h"
+#include "llvm/ADT/BitmaskEnum.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/CodeGen/EdgeBundles.h"
+#include "llvm/CodeGen/LivePhysRegs.h"
+#include "llvm/CodeGen/MachineBasicBlock.h"
+#include "llvm/CodeGen/MachineFunctionPass.h"
+#include "llvm/CodeGen/MachineRegisterInfo.h"
+#include "llvm/CodeGen/TargetRegisterInfo.h"
+
+using namespace llvm;
+
+#define DEBUG_TYPE "aarch64-machine-sme-abi"
+
+namespace {
+
+enum ZAState {
+ ANY = 0,
+ ACTIVE,
+ LOCAL_SAVED,
+ CALLER_DORMANT,
+ OFF,
+ NUM_ZA_STATE
+};
+
+enum LiveRegs : uint8_t {
+ None = 0,
+ NZCV = 1 << 0,
+ W0 = 1 << 1,
+ W0_HI = 1 << 2,
+ X0 = W0 | W0_HI,
+ LLVM_MARK_AS_BITMASK_ENUM(/* LargestValue = */ W0_HI)
+};
+
+static bool isLegalEdgeBundleZAState(ZAState State) {
+ switch (State) {
+ case ZAState::ACTIVE:
+ case ZAState::LOCAL_SAVED:
+ return true;
+ default:
+ return false;
+ }
+}
+struct TPIDR2State {
+ int FrameIndex = -1;
+};
+
+StringRef getZAStateString(ZAState State) {
+#define MAKE_CASE(V) \
+ case V: \
+ return #V;
+ switch (State) {
+ MAKE_CASE(ZAState::ANY)
+ MAKE_CASE(ZAState::ACTIVE)
+ MAKE_CASE(ZAState::LOCAL_SAVED)
+ MAKE_CASE(ZAState::CALLER_DORMANT)
+ MAKE_CASE(ZAState::OFF)
+ default:
+ llvm_unreachable("Unexpected ZAState");
+ }
+#undef MAKE_CASE
+}
+
+static bool isZAorZT0RegOp(const TargetRegisterInfo &TRI,
+ const MachineOperand &MO) {
+ if (!MO.isReg() || !MO.getReg().isPhysical())
+ return false;
+ return any_of(TRI.subregs_inclusive(MO.getReg()), [](const MCPhysReg &SR) {
+ return AArch64::MPR128RegClass.contains(SR) ||
+ AArch64::ZTRRegClass.contains(SR);
+ });
+}
+
+static std::pair<ZAState, MachineBasicBlock::iterator>
+getInstNeededZAState(const TargetRegisterInfo &TRI, MachineInstr &MI,
+ bool ZALiveAtReturn) {
+ MachineBasicBlock::iterator InsertPt(MI);
+
+ if (MI.getOpcode() == AArch64::InOutZAUsePseudo)
+ return {ZAState::ACTIVE, std::prev(InsertPt)};
+
+ if (MI.getOpcode() == AArch64::RequiresZASavePseudo)
+ return {ZAState::LOCAL_SAVED, std::prev(InsertPt)};
+
+ if (MI.isReturn())
+ return {ZALiveAtReturn ? ZAState::ACTIVE : ZAState::OFF, InsertPt};
+
+ for (auto &MO : MI.operands()) {
+ if (isZAorZT0RegOp(TRI, MO))
+ return {ZAState::ACTIVE, InsertPt};
+ }
+
+ return {ZAState::ANY, InsertPt};
+}
+
+struct MachineSMEABI : public MachineFunctionPass {
+ inline static char ID = 0;
+
+ MachineSMEABI() : MachineFunctionPass(ID) {}
+
+ bool runOnMachineFunction(MachineFunction &MF) override;
+
+ StringRef getPassName() const override { return "Machine SME ABI pass"; }
+
+ void getAnalysisUsage(AnalysisUsage &AU) const override {
+ AU.setPreservesCFG();
+ AU.addRequired<EdgeBundlesWrapperLegacy>();
+ AU.addPreservedID(MachineLoopInfoID);
+ AU.addPreservedID(MachineDominatorsID);
+ MachineFunctionPass::getAnalysisUsage(AU);
+ }
+
+ void collectNeededZAStates(MachineFunction &MF, SMEAttrs);
+ void pickBundleZAStates(MachineFunction &MF);
+ void insertStateChanges(MachineFunction &MF);
+
+ // Emission routines for private and shared ZA functions (using lazy saves).
+ void emitNewZAPrologue(MachineBasicBlock &MBB,
+ MachineBasicBlock::iterator MBBI);
+ void emitRestoreLazySave(MachineBasicBlock &MBB,
+ MachineBasicBlock::iterator MBBI,
+ LiveRegs PhysLiveRegs);
+ void emitSetupLazySave(MachineBasicBlock &MBB,
+ MachineBasicBlock::iterator MBBI);
+ void emitAllocateLazySaveBuffer(MachineBasicBlock &MBB,
+ MachineBasicBlock::iterator MBBI);
+ void emitZAOff(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
+ bool ClearTPIDR2);
+
+ void emitStateChange(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
+ ZAState From, ZAState To, LiveRegs PhysLiveRegs);
+
+ TPIDR2State getTPIDR2Block(MachineFunction &MF);
+
+private:
+ struct InstInfo {
+ ZAState NeededState{ZAState::ANY};
+ MachineBasicBlock::iterator InsertPt;
+ LiveRegs PhysLiveRegs = LiveRegs::None;
+ };
+
+ struct BlockInfo {
+ ZAState FixedEntryState{ZAState::ANY};
+ SmallVector<InstInfo> Insts;
+ LiveRegs PhysLiveRegsAtExit = LiveRegs::None;
+ };
+
+ // All pass state that must be cleared between functions.
+ struct PassState {
+ SmallVector<BlockInfo> Blocks;
+ SmallVector<ZAState> BundleStates;
+ std::optional<TPIDR2State> TPIDR2Block;
+ } State;
+
+ EdgeBundles *Bundles = nullptr;
+};
+
+void MachineSMEABI::collectNeededZAStates(MachineFunction &MF,
+ SMEAttrs SMEFnAttrs) {
+ const TargetRegisterInfo &TRI = *MF.getSubtarget().getRegisterInfo();
+ assert((SMEFnAttrs.hasZT0State() || SMEFnAttrs.hasZAState()) &&
+ "Expected function to have ZA/ZT0 state!");
+
+ State.Blocks.resize(MF.getNumBlockIDs());
+ for (MachineBasicBlock &MBB : MF) {
+ BlockInfo &Block = State.Blocks[MBB.getNumber()];
+ if (&MBB == &MF.front()) {
+ // Entry block:
+ Block.FixedEntryState = SMEFnAttrs.hasPrivateZAInterface()
+ ? ZAState::CALLER_DORMANT
+ : ZAState::ACTIVE;
+ } else if (MBB.isEHPad()) {
+ // EH entry block:
+ Block.FixedEntryState = ZAState::LOCAL_SAVED;
+ }
+
+ LiveRegUnits LiveUnits(TRI);
+ LiveUnits.addLiveOuts(MBB);
+
+ auto GetPhysLiveRegs = [&] {
+ LiveRegs PhysLiveRegs = LiveRegs::None;
+ if (!LiveUnits.available(AArch64::NZCV))
+ PhysLiveRegs |= LiveRegs::NZCV;
+ // We have to track W0 and X0 separately as otherwise things can get
+ // confused we attempt to preserve X0 but only W0 was defined.
+ if (!LiveUnits.available(AArch64::W0))
+ PhysLiveRegs |= LiveRegs::W0;
+ if (!LiveUnits.available(AArch64::W0_HI))
+ PhysLiveRegs |= LiveRegs::W0_HI;
+ return PhysLiveRegs;
+ };
+
+ Block.PhysLiveRegsAtExit = GetPhysLiveRegs();
+ auto FirstTerminatorInsertPt = MBB.getFirstTerminator();
+ for (MachineInstr &MI : reverse(MBB)) {
+ MachineBasicBlock::iterator MBBI(MI);
+ LiveUnits.stepBackward(MI);
+ LiveRegs PhysLiveRegs = GetPhysLiveRegs();
+ auto [NeededState, InsertPt] = getInstNeededZAState(
+ TRI, MI, /*ZALiveAtReturn=*/SMEFnAttrs.hasSharedZAInterface());
+ assert((InsertPt == MBBI ||
+ InsertPt->getOpcode() == AArch64::ADJCALLSTACKDOWN) &&
+ "Unexpected state change insertion point!");
+ // TODO: Do something to avoid state changes where NZCV is live.
+ if (MBBI == FirstTerminatorInsertPt)
+ Block.PhysLiveRegsAtExit = PhysLiveRegs;
+ if (NeededState != ZAState::ANY)
+ Block.Insts.push_back({NeededState, InsertPt, PhysLiveRegs});
+ }
+
+ // Reverse vector (as we had to iterate backwards for liveness).
+ std::reverse(Block.Insts.begin(), Block.Insts.end());
+ }
+}
+
+void MachineSMEABI::pickBundleZAStates(MachineFunction &MF) {
+ State.BundleStates.resize(Bundles->getNumBundles());
+ for (unsigned I = 0, E = Bundles->getNumBundles(); I != E; ++I) {
+ LLVM_DEBUG(dbgs() << "Picking ZA state for edge bundle: " << I << '\n');
+
+ // Attempt to pick a ZA state for this bundle that minimizes state
+ // transitions. Edges within loops are given a higher weight as we assume
+ // they will be executed more than once.
+ // TODO: We should propagate desired incoming/outgoing states through blocks
+ // that have the "ANY" state first to make better global decisions.
+ int EdgeStateCounts[ZAState::NUM_ZA_STATE] = {0};
+ for (unsigned BlockID : Bundles->getBlocks(I)) {
+ LLVM_DEBUG(dbgs() << "- bb." << BlockID);
+
+ BlockInfo &Block = State.Blocks[BlockID];
+ if (Block.Insts.empty()) {
+ LLVM_DEBUG(dbgs() << " (no state preference)\n");
+ continue;
+ }
+ bool InEdge = Bundles->getBundle(BlockID, /*Out=*/false) == I;
+ bool OutEdge = Bundles->getBundle(BlockID, /*Out=*/true) == I;
+
+ ZAState DesiredIncomingState = Block.Insts.front().NeededState;
+ if (InEdge && isLegalEdgeBundleZAState(DesiredIncomingState)) {
+ EdgeStateCounts[DesiredIncomingState]++;
+ LLVM_DEBUG(dbgs() << " DesiredIncomingState: "
+ << getZAStateString(DesiredIncomingState));
+ }
+ ZAState DesiredOutgoingState = Block.Insts.front().NeededState;
+ if (OutEdge && isLegalEdgeBundleZAState(DesiredOutgoingState)) {
+ EdgeStateCounts[DesiredOutgoingState]++;
+ LLVM_DEBUG(dbgs() << " DesiredOutgoingState: "
+ << getZAStateString(DesiredOutgoingState));
+ }
+ LLVM_DEBUG(dbgs() << '\n');
+ }
+
+ ZAState BundleState =
+ ZAState(max_element(EdgeStateCounts) - EdgeStateCounts);
+
+ // Force ZA to be active in bundles that don't have a preferred state.
+ // TODO: Something better here (to avoid extra mode switches).
+ if (BundleState == ZAState::ANY)
+ BundleState = ZAState::ACTIVE;
+
+ LLVM_DEBUG({
+ dbgs() << "Chosen ZA state: " << getZAStateString(BundleState) << '\n'
+ << "Edge counts:";
+ for (auto [State, Count] : enumerate(EdgeStateCounts))
+ dbgs() << " " << getZAStateString(ZAState(State)) << ": " << Count;
+ dbgs() << "\n\n";
+ });
+
+ State.BundleStates[I] = BundleState;
+ }
+}
+
+void MachineSMEABI::insertStateChanges(MachineFunction &MF) {
+ for (MachineBasicBlock &MBB : MF) {
+ BlockInfo &Block = State.Blocks[MBB.getNumber()];
+ ZAState InState =
+ State.BundleStates[Bundles->getBundle(MBB.getNumber(), /*Out=*/false)];
+ ZAState OutState =
+ State.BundleStates[Bundles->getBundle(MBB.getNumber(), /*Out=*/true)];
+
+ ZAState CurrentState = Block.FixedEntryState;
+ if (CurrentState == ZAState::ANY)
+ CurrentState = InState;
+
+ for (auto &Inst : Block.Insts) {
+ if (CurrentState != Inst.NeededState)
+ emitStateChange(MBB, Inst.InsertPt, CurrentState, Inst.NeededState,
+ Inst.PhysLiveRegs);
+ CurrentState = Inst.NeededState;
+ }
+
+ if (MBB.succ_empty())
+ continue;
+
+ if (CurrentState != OutState)
+ emitStateChange(MBB, MBB.getFirstTerminator(), CurrentState, OutState,
+ Block.PhysLiveRegsAtExit);
+ }
+}
+
+TPIDR2State MachineSMEABI::getTPIDR2Block(MachineFunction &MF) {
+ if (State.TPIDR2Block)
+ return *State.TPIDR2Block;
+ MachineFrameInfo &MFI = MF.getFrameInfo();
+ State.TPIDR2Block = TPIDR2State{MFI.CreateStackObject(16, Align(16), false)};
+ return *State.TPIDR2Block;
+}
+
+static DebugLoc getDebugLoc(MachineBasicBlock &MBB,
+ MachineBasicBlock::iterator MBBI) {
+ if (MBBI != MBB.end())
+ return MBBI->getDebugLoc();
+ return DebugLoc();
+}
+
+void MachineSMEABI::emitSetupLazySave(MachineBasicBlock &MBB,
+ MachineBasicBlock::iterator MBBI) {
+ MachineFunction &MF = *MBB.getParent();
+ auto &Subtarget = MF.getSubtarget<AArch64Subtarget>();
+ const TargetInstrInfo &TII = *Subtarget.getInstrInfo();
+ MachineRegisterInfo &MRI = MF.getRegInfo();
+ DebugLoc DL = getDebugLoc(MBB, MBBI);
+
+ // Get pointer to TPIDR2 block.
+ Register TPIDR2 = MRI.createVirtualRegister(&AArch64::GPR64spRegClass);
+ Register TPIDR2Ptr = MRI.createVirtualRegister(&AArch64::GPR64RegClass);
+ BuildMI(MBB, MBBI, DL, TII.get(AArch64::ADDXri), TPIDR2)
+ .addFrameIndex(getTPIDR2Block(MF).FrameIndex)
+ .addImm(0)
+ .addImm(0);
+ BuildMI(MBB, MBBI, DL, TII.get(TargetOpcode::COPY), TPIDR2Ptr).addReg(TPIDR2);
+ // Set TPIDR2_EL0 to point to TPIDR2 block.
+ BuildMI(MBB, MBBI, DL, TII.get(AArch64::MSR))
+ .addImm(AArch64SysReg::TPIDR2_EL0)
+ .addReg(TPIDR2Ptr);
+}
+
+// Helper class for saving physical registers around calls.
+struct ScopedPhysRegSave {
+ ScopedPhysRegSave(MachineRegisterInfo &MRI, const TargetInstrInfo &TII,
+ DebugLoc DL, MachineBasicBlock &MBB,
+ MachineBasicBlock::iterator MBBI, LiveRegs PhysLiveRegs)
+ : TII(TII), DL(DL), MBB(MBB), MBBI(MBBI), PhysLiveRegs(PhysLiveRegs) {
+ if (PhysLiveRegs & LiveRegs::NZCV) {
+ StatusFlags = MRI.createVirtualRegister(&AArch64::GPR64RegClass);
+ BuildMI(MBB, MBBI, DL, TII.get(AArch64::MRS))
+ .addReg(StatusFlags, RegState::Define)
+ .addImm(AArch64SysReg::NZCV)
+ .addReg(AArch64::NZCV, RegState::Implicit);
+ }
+ // Note: Preserving X0 is "free" as this is before register allocation, so
+ // the register allocator is still able to optimize these copies.
+ if (PhysLiveRegs & LiveRegs::W0) {
+ X0Save = MRI.createVirtualRegister(PhysLiveRegs & LiveRegs::W0_HI
+ ? &AArch64::GPR64RegClass
+ : &AArch64::GPR32RegClass);
+ BuildMI(MBB, MBBI, DL, TII.get(TargetOpcode::COPY), X0Save)
+ .addReg(PhysLiveRegs & LiveRegs::W0_HI ? AArch64::X0 : AArch64::W0);
+ }
+ }
+
+ ~ScopedPhysRegSave() {
+ if (StatusFlags != AArch64::NoRegister)
+ BuildMI(MBB, MBBI, DL, TII.get(AArch64::MSR))
+ .addImm(AArch64SysReg::NZCV)
+ .addReg(StatusFlags)
+ .addReg(AArch64::NZCV, RegState::ImplicitDefine);
+
+ if (X0Save != AArch64::NoRegister)
+ BuildMI(MBB, MBBI, DL, TII.get(TargetOpcode::COPY),
+ PhysLiveRegs & LiveRegs::W0_HI ? AArch64::X0 : AArch64::W0)
+ .addReg(X0Save);
+ }
+
+ const TargetInstrInfo &TII;
+ DebugLoc DL;
+ MachineBasicBlock &MBB;
+ MachineBasicBlock::iterator MBBI;
+ LiveRegs PhysLiveRegs;
+ Register StatusFlags = AArch64::NoRegister;
+ Register X0Save = AArch64::NoRegister;
+};
+
+void MachineSMEABI::emitRestoreLazySave(MachineBasicBlock &MBB,
+ MachineBasicBlock::iterator MBBI,
+ LiveRegs PhysLiveRegs) {
+ MachineFunction &MF = *MBB.getParent();
+ auto &Subtarget = MF.getSubtarget<AArch64Subtarget>();
+ const AArch64RegisterInfo &TRI = *Subtarget.getRegisterInfo();
+ const TargetInstrInfo &TII = *Subtarget.getInstrInfo();
+ MachineRegisterInfo &MRI = MF.getRegInfo();
+
+ DebugLoc DL = getDebugLoc(MBB, MBBI);
+ Register TPIDR2EL0 = MRI.createVirtualRegister(&AArch64::GPR64RegClass);
+ Register TPIDR2 = AArch64::X0;
+
+ // TODO: Emit these within the restore MBB to prevent unnecessary saves.
+ ScopedPhysRegSave ScopedPhysRegSave(MRI, TII, DL, MBB, MBBI, PhysLiveRegs);
+
+ // Enable ZA.
+ BuildMI(MBB, MBBI, DL, TII.get(AArch64::MSRpstatesvcrImm1))
+ .addImm(AArch64SVCR::SVCRZA)
+ .addImm(1);
+ // Get current TPIDR2_EL0.
+ BuildMI(MBB, MBBI, DL, TII.get(AArch64::MRS))
+ .addReg(TPIDR2EL0, RegState::Define)
+ .addImm(AArch64SysReg::TPIDR2_EL0);
+ // Get pointer to TPIDR2 block.
+ BuildMI(MBB, MBBI, DL, TII.get(AArch64::ADDXri), TPIDR2)
+ .addFrameIndex(getTPIDR2Block(MF).FrameIndex)
+ .addImm(0)
+ .addImm(0);
+ // (Conditionally) restore ZA state.
+ BuildMI(MBB, MBBI, DL, TII.get(AArch64::RestoreZAPseudo))
+ .addReg(TPIDR2EL0)
+ .addReg(TPIDR2)
+ .addExternalSymbol("__arm_tpidr2_restore")
+ .addRegMask(TRI.SMEABISupportRoutinesCallPreservedMaskFromX0());
+ // Zero TPIDR2_EL0.
+ BuildMI(MBB, MBBI, DL, TII.get(AArch64::MSR))
+ .addImm(AArch64SysReg::TPIDR2_EL0)
+ .addReg(AArch64::XZR);
+}
+
+void MachineSMEABI::emitZAOff(MachineBasicBlock &MBB,
+ MachineBasicBlock::iterator MBBI,
+ bool ClearTPIDR2) {
+ MachineFunction &MF = *MBB.getParent();
+ auto &Subtarget = MF.getSubtarget<AArch64Subtarget>();
+ const TargetInstrInfo &TII = *Subtarget.getInstrInfo();
+ DebugLoc DL = getDebugLoc(MBB, MBBI);
+
+ // Clear TPIDR2.
+ if (ClearTPIDR2)
+ BuildMI(MBB, MBBI, DL, TII.get(AArch64::MSR))
+ .addImm(AArch64SysReg::TPIDR2_EL0)
+ .addReg(AArch64::XZR);
+
+ // Disable ZA.
+ BuildMI(MBB, MBBI, DL, TII.get(AArch64::MSRpstatesvcrImm1))
+ .addImm(AArch64SVCR::SVCRZA)
+ .addImm(0);
+}
+
+void MachineSMEABI::emitAllocateLazySaveBuffer(
+ MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI) {
+ MachineFunction &MF = *MBB.getParent();
+ MachineFrameInfo &MFI = MF.getFrameInfo();
+ auto &Subtarget = MF.getSubtarget<AArch64Subtarget>();
+ const TargetInstrInfo &TII = *Subtarget.getInstrInfo();
+ MachineRegisterInfo &MRI = MF.getRegInfo();
+
+ DebugLoc DL = getDebugLoc(MBB, MBBI);
+ Register SP = MRI.createVirtualRegister(&AArch64::GPR64RegClass);
+ Register SVL = MRI.createVirtualRegister(&AArch64::GPR64RegClass);
+ Register Buffer = MRI.createVirtualRegister(&AArch64::GPR64RegClass);
+
+ // Calculate SVL.
+ BuildMI(MBB, MBBI, DL, TII.get(AArch64::RDSVLI_XI), SVL).addImm(1);
+
+ // 1. Allocate the lazy save buffer.
+ {
+ // TODO This function grows the stack with a subtraction, which doesn't work
+ // on Windows. Some refactoring to share the functionality in
+ // LowerWindowsDYNAMIC_STACKALLOC will be required once the Windows ABI
+ // supports SME
+ assert(!Subtarget.isTargetWindows() &&
+ "Lazy ZA save is not yet supported on Windows");
+ // Get original stack pointer.
+ BuildMI(MBB, MBBI, DL, TII.get(TargetOpcode::COPY), SP).addReg(AArch64::SP);
+ // Allocate a lazy-save buffer object of the size given, normally SVL * SVL
+ BuildMI(MBB, MBBI, DL, TII.get(AArch64::MSUBXrrr), Buffer)
+ .addReg(SVL)
+ .addReg(SVL)
+ .addReg(SP);
+ BuildMI(MBB, MBBI, DL, TII.get(TargetOpcode::COPY), AArch64::SP)
+ .addReg(Buffer);
+ // We have just allocated a variable sized object, tell this to PEI.
+ MFI.CreateVariableSizedObject(Align(16), nullptr);
+ }
+
+ // 2. Setup the TPIDR2 block.
+ {
+ Register TPIDInitSaveSlicesReg = SVL;
+ if (!Subtarget.isLittleEndian()) {
+ Register TmpReg =
+ MF.getRegInfo().createVirtualRegister(&AArch64::GPR64RegClass);
+ // For big-endian targets move "num_za_save_slices" to the top two bytes.
+ BuildMI(MBB, MBBI, DL, TII.get(AArch64::UBFMXri), TmpReg)
+ .addReg(TPIDInitSaveSlicesReg)
+ .addImm(16)
+ .addImm(15);
+ TPIDInitSaveSlicesReg = TmpReg;
+ }
+ // Store buffer pointer and num_za_save_slices.
+ // Bytes 10-15 are implicitly zeroed.
+ BuildMI(MBB, MBBI, DL, TII.get(AArch64::STPXi))
+ .addReg(Buffer)
+ .addReg(TPIDInitSaveSlicesReg)
+ .addFrameIndex(getTPIDR2Block(MF).FrameIndex)
+ .addImm(0);
+ }
+}
+
+static void emitZeroZA(const TargetInstrInfo &TII, DebugLoc DL,
+ MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
+ unsigned Mask) {
+ MachineInstrBuilder MIB =
+ BuildMI(MBB, MBBI, DL, TII.get(AArch64::ZERO_M)).addImm(Mask);
+ for (unsigned I = 0; I < 8; I++) {
+ if (Mask & (1 << I))
+ MIB.addDef(AArch64::ZAD0 + I, RegState::ImplicitDefine);
+ }
+}
+
+void MachineSMEABI::emitNewZAPrologue(MachineBasicBlock &MBB,
+ MachineBasicBlock::iterator MBBI) {
+ MachineFunction &MF = *MBB.getParent();
+ auto &Subtarget = MF.getSubtarget<AArch64Subtarget>();
+ const TargetInstrInfo &TII = *Subtarget.getInstrInfo();
+ MachineRegisterInfo &MRI = MF.getRegInfo();
+ const AArch64RegisterInfo &TRI = *Subtarget.getRegisterInfo();
+ DebugLoc DL = getDebugLoc(MBB, MBBI);
+
+ // Get current TPIDR2_EL0.
+ Register TPIDR2EL0 = MRI.createVirtualRegister(&AArch64::GPR64RegClass);
+ BuildMI(MBB, MBBI, DL, TII.get(AArch64::MRS))
+ .addReg(TPIDR2EL0, RegState::Define)
+ .addImm(AArch64SysReg::TPIDR2_EL0);
+ // If TPIDR2_EL0 is non-zero, commit the lazy save.
+ BuildMI(MBB, MBBI, DL, TII.get(AArch64::CommitZAPseudo))
+ .addReg(TPIDR2EL0)
+ .addExternalSymbol("__arm_tpidr2_save")
+ .addRegMask(TRI.SMEABISupportRoutinesCallPreservedMaskFromX0());
+ // Clear TPIDR2_EL0.
+ BuildMI(MBB, MBBI, DL, TII.get(AArch64::MSR))
+ .addImm(AArch64SysReg::TPIDR2_EL0)
+ .addReg(AArch64::XZR);
+ // Enable ZA (as ZA could have previously been in the OFF state).
+ BuildMI(MBB, MBBI, DL, TII.get(AArch64::MSRpstatesvcrImm1))
+ .addImm(AArch64SVCR::SVCRZA)
+ .addImm(1);
+ // Zero ZA. Note: ZA state may new be needed for new ZT0 functions.
+ if (MF.getInfo<AArch64FunctionInfo>()->getSMEFnAttrs().hasZAState())
+ emitZeroZA(TII, DL, MBB, MBBI, /*Mask=*/0b11111111);
+}
+
+void MachineSMEABI::emitStateChange(MachineBasicBlock &MBB,
+ MachineBasicBlock::iterator InsertPt,
+ ZAState From, ZAState To,
+ LiveRegs PhysLiveRegs) {
+
+ // ZA not used.
+ if (From == ZAState::ANY || To == ZAState::ANY)
+ return;
+
+ // TODO: Avoid setting up the save buffer if there's no transition to
+ // LOCAL_SAVED.
+ if (From == ZAState::CALLER_DORMANT && To != ZAState::OFF) {
+ assert(MBB.getParent()
+ ->getInfo<AArch64FunctionInfo>()
+ ->getSMEFnAttrs()
+ .hasPrivateZAInterface() &&
+ "CALLER_DORMANT state requires private ZA interface");
+ assert(&MBB == &MBB.getParent()->front() &&
+ "CALLER_DORMANT state only valid in entry block");
+ emitNewZAPrologue(MBB, MBB.getFirstNonPHI());
+ if (To == ZAState::ACTIVE)
+ return; // Nothing more to to (ZA is active after the prologue).
+
+ // Note: "emitNewZAPrologue" zeros ZA, so we may need to setup a lazy save
+ // if "To" os "ZAState::LOCAL_SAVED". If may be possible to improve this
+ // case by changing the placement of the zero instruction.
+ From = ZAState::ACTIVE;
+ }
+
+ if (From == ZAState::ACTIVE && To == ZAState::LOCAL_SAVED)
+ emitSetupLazySave(MBB, InsertPt);
+ else if (From == ZAState::LOCAL_SAVED && To == ZAState::ACTIVE)
+ emitRestoreLazySave(MBB, InsertPt, PhysLiveRegs);
+ else if (To == ZAState::OFF) {
+ // If we're exiting from the CALLER_DORMANT state that means this new ZA
+ // function did not touch ZA (so ZA was never turned on).
+ if (From != ZAState::CALLER_DORMANT)
+ emitZAOff(MBB, InsertPt, /*ClearTPIDR2=*/From == ZAState::LOCAL_SAVED);
+ } else {
+ dbgs() << "Error: Transition from " << getZAStateString(From) << " to "
+ << getZAStateString(To) << '\n';
+ llvm_unreachable("Unimplemented state transition");
+ }
+}
+
+} // end anonymous namespace
+
+INITIALIZE_PASS(MachineSMEABI, "aarch64-machine-sme-abi", "Machine SME ABI",
+ false, false)
+
+bool MachineSMEABI::runOnMachineFunction(MachineFunction &MF) {
+ if (!MF.getSubtarget<AArch64Subtarget>().hasSME())
+ return false;
+
+ auto *AFI = MF.getInfo<AArch64FunctionInfo>();
+ SMEAttrs SMEFnAttrs = AFI->getSMEFnAttrs();
+ if (!SMEFnAttrs.hasZAState() && !SMEFnAttrs.hasZT0State())
+ return false;
+
+ assert(MF.getRegInfo().isSSA() && "Expected to be run on SSA form!");
+
+ // Reset pass state.
+ State = PassState{};
+ Bundles = &getAnalysis<EdgeBundlesWrapperLegacy>().getEdgeBundles();
+
+ collectNeededZAStates(MF, SMEFnAttrs);
+ pickBundleZAStates(MF);
+ insertStateChanges(MF);
+
+ // Allocate save buffer (if needed).
+ if (State.TPIDR2Block.has_value()) {
+ MachineBasicBlock &EntryBlock = MF.front();
+ emitAllocateLazySaveBuffer(EntryBlock, EntryBlock.getFirstNonPHI());
+ }
+
+ return true;
+}
+
+FunctionPass *llvm::createMachineSMEABIPass() { return new MachineSMEABI(); }
diff --git a/llvm/lib/Target/AArch64/Utils/AArch64SMEAttributes.h b/llvm/lib/Target/AArch64/Utils/AArch64SMEAttributes.h
index f1be0ecbee7ed..9e5801a28407c 100644
--- a/llvm/lib/Target/AArch64/Utils/AArch64SMEAttributes.h
+++ b/llvm/lib/Target/AArch64/Utils/AArch64SMEAttributes.h
@@ -194,7 +194,7 @@ class SMECallAttrs {
}
bool requiresEnablingZAAfterCall() const {
- return requiresLazySave() || requiresDisablingZABeforeCall();
+ return requiresDisablingZABeforeCall();
}
bool requiresPreservingAllZAState() const {
diff --git a/llvm/test/CodeGen/AArch64/sme-agnostic-za.ll b/llvm/test/CodeGen/AArch64/sme-agnostic-za.ll
index 1f68815411097..d1ec53f54c702 100644
--- a/llvm/test/CodeGen/AArch64/sme-agnostic-za.ll
+++ b/llvm/test/CodeGen/AArch64/sme-agnostic-za.ll
@@ -1,5 +1,6 @@
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5
; RUN: llc -mattr=+sme2 < %s | FileCheck %s
+; RUN: llc -mattr=+sme2 < %s -aarch64-new-sme-abi | FileCheck %s
target triple = "aarch64"
@@ -200,3 +201,41 @@ define i64 @streaming_compatible_agnostic_caller_nonstreaming_private_za_callee(
%res2 = call i64 @private_za_decl(i64 %res)
ret i64 %res2
}
+
+declare i64 @many_args_private_za_callee(
+ i64, i64, i64, i64, i64, i64, i64, i64, i64, i64)
+
+; In this example some arguments are passed on the stack, which decrements the
+; stack pointer before the call -- in this test the call to __arm_sme_save
+; should occur _before_ the stack decrement.
+define i64 @test_many_callee_arguments(
+; CHECK-LABEL: test_many_callee_arguments:
+; CHECK: // %bb.0:
+; CHECK-NEXT: stp x29, x30, [sp, #-32]! // 16-byte Folded Spill
+; CHECK-NEXT: str x19, [sp, #16] // 8-byte Folded Spill
+; CHECK-NEXT: mov x29, sp
+; CHECK-NEXT: mov x8, x0
+; CHECK-NEXT: bl __arm_sme_state_size
+; CHECK-NEXT: sub sp, sp, x0
+; CHECK-NEXT: ldp x9, x10, [x29, #32]
+; CHECK-NEXT: mov x19, sp
+; CHECK-NEXT: mov x0, x19
+; CHECK-NEXT: bl __arm_sme_save
+; CHECK-NEXT: stp x9, x10, [sp, #-16]!
+; CHECK-NEXT: mov x0, x8
+; CHECK-NEXT: bl many_args_private_za_callee
+; CHECK-NEXT: add sp, sp, #16
+; CHECK-NEXT: mov x1, x0
+; CHECK-NEXT: mov x0, x19
+; CHECK-NEXT: bl __arm_sme_restore
+; CHECK-NEXT: mov x0, x1
+; CHECK-NEXT: mov sp, x29
+; CHECK-NEXT: ldr x19, [sp, #16] // 8-byte Folded Reload
+; CHECK-NEXT: ldp x29, x30, [sp], #32 // 16-byte Folded Reload
+; CHECK-NEXT: ret
+ i64 %0, i64 %1, i64 %2, i64 %3, i64 %4, i64 %5, i64 %6, i64 %7, i64 %8, i64 %9
+) nounwind "aarch64_za_state_agnostic" {
+ %ret = call i64 @many_args_private_za_callee(
+ i64 %0, i64 %1, i64 %2, i64 %3, i64 %4, i64 %5, i64 %6, i64 %7, i64 %8, i64 %9)
+ ret i64 %ret
+}
diff --git a/llvm/test/CodeGen/AArch64/sme-lazy-save-call.ll b/llvm/test/CodeGen/AArch64/sme-lazy-save-call.ll
index e463e833bdbde..b8d3813810c6b 100644
--- a/llvm/test/CodeGen/AArch64/sme-lazy-save-call.ll
+++ b/llvm/test/CodeGen/AArch64/sme-lazy-save-call.ll
@@ -1,7 +1,11 @@
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=aarch64 -aarch64-streaming-hazard-size=0 -mattr=+sve -mattr=+sme < %s | FileCheck %s
+; RUN: llc -mtriple=aarch64 -aarch64-streaming-hazard-size=0 -mattr=+sve -mattr=+sme < %s | FileCheck %s --check-prefixes=CHECK-COMMON,CHECK
+; RUN: llc -mtriple=aarch64 -aarch64-streaming-hazard-size=0 -mattr=+sve -mattr=+sme -aarch64-new-sme-abi < %s | FileCheck %s --check-prefixes=CHECK-COMMON,CHECK-NEWLOWERING
declare void @private_za_callee()
+declare void @shared_za_callee() "aarch64_inout_za"
+declare void @preserves_za_callee() "aarch64_preserves_za"
+
declare float @llvm.cos.f32(float)
; Test lazy-save mechanism for a single callee.
@@ -35,6 +39,33 @@ define void @test_lazy_save_1_callee() nounwind "aarch64_inout_za" {
; CHECK-NEXT: ldr x19, [sp, #16] // 8-byte Folded Reload
; CHECK-NEXT: ldp x29, x30, [sp], #32 // 16-byte Folded Reload
; CHECK-NEXT: ret
+;
+; CHECK-NEWLOWERING-LABEL: test_lazy_save_1_callee:
+; CHECK-NEWLOWERING: // %bb.0:
+; CHECK-NEWLOWERING-NEXT: stp x29, x30, [sp, #-32]! // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: str x19, [sp, #16] // 8-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: mov x29, sp
+; CHECK-NEWLOWERING-NEXT: sub sp, sp, #16
+; CHECK-NEWLOWERING-NEXT: rdsvl x8, #1
+; CHECK-NEWLOWERING-NEXT: mov x9, sp
+; CHECK-NEWLOWERING-NEXT: msub x9, x8, x8, x9
+; CHECK-NEWLOWERING-NEXT: mov sp, x9
+; CHECK-NEWLOWERING-NEXT: sub x10, x29, #16
+; CHECK-NEWLOWERING-NEXT: stp x9, x8, [x29, #-16]
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, x10
+; CHECK-NEWLOWERING-NEXT: bl private_za_callee
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: sub x0, x29, #16
+; CHECK-NEWLOWERING-NEXT: cbnz x8, .LBB0_2
+; CHECK-NEWLOWERING-NEXT: // %bb.1:
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEWLOWERING-NEXT: .LBB0_2:
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: mov sp, x29
+; CHECK-NEWLOWERING-NEXT: ldr x19, [sp, #16] // 8-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ldp x29, x30, [sp], #32 // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ret
call void @private_za_callee()
ret void
}
@@ -83,6 +114,34 @@ define void @test_lazy_save_2_callees() nounwind "aarch64_inout_za" {
; CHECK-NEXT: ldr x21, [sp, #16] // 8-byte Folded Reload
; CHECK-NEXT: ldp x29, x30, [sp], #48 // 16-byte Folded Reload
; CHECK-NEXT: ret
+;
+; CHECK-NEWLOWERING-LABEL: test_lazy_save_2_callees:
+; CHECK-NEWLOWERING: // %bb.0:
+; CHECK-NEWLOWERING-NEXT: stp x29, x30, [sp, #-32]! // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: str x19, [sp, #16] // 8-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: mov x29, sp
+; CHECK-NEWLOWERING-NEXT: sub sp, sp, #16
+; CHECK-NEWLOWERING-NEXT: rdsvl x8, #1
+; CHECK-NEWLOWERING-NEXT: mov x9, sp
+; CHECK-NEWLOWERING-NEXT: msub x9, x8, x8, x9
+; CHECK-NEWLOWERING-NEXT: mov sp, x9
+; CHECK-NEWLOWERING-NEXT: sub x10, x29, #16
+; CHECK-NEWLOWERING-NEXT: stp x9, x8, [x29, #-16]
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, x10
+; CHECK-NEWLOWERING-NEXT: bl private_za_callee
+; CHECK-NEWLOWERING-NEXT: bl private_za_callee
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: sub x0, x29, #16
+; CHECK-NEWLOWERING-NEXT: cbnz x8, .LBB1_2
+; CHECK-NEWLOWERING-NEXT: // %bb.1:
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEWLOWERING-NEXT: .LBB1_2:
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: mov sp, x29
+; CHECK-NEWLOWERING-NEXT: ldr x19, [sp, #16] // 8-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ldp x29, x30, [sp], #32 // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ret
call void @private_za_callee()
call void @private_za_callee()
ret void
@@ -119,6 +178,33 @@ define float @test_lazy_save_expanded_intrinsic(float %a) nounwind "aarch64_inou
; CHECK-NEXT: ldr x19, [sp, #16] // 8-byte Folded Reload
; CHECK-NEXT: ldp x29, x30, [sp], #32 // 16-byte Folded Reload
; CHECK-NEXT: ret
+;
+; CHECK-NEWLOWERING-LABEL: test_lazy_save_expanded_intrinsic:
+; CHECK-NEWLOWERING: // %bb.0:
+; CHECK-NEWLOWERING-NEXT: stp x29, x30, [sp, #-32]! // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: str x19, [sp, #16] // 8-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: mov x29, sp
+; CHECK-NEWLOWERING-NEXT: sub sp, sp, #16
+; CHECK-NEWLOWERING-NEXT: rdsvl x8, #1
+; CHECK-NEWLOWERING-NEXT: mov x9, sp
+; CHECK-NEWLOWERING-NEXT: msub x9, x8, x8, x9
+; CHECK-NEWLOWERING-NEXT: mov sp, x9
+; CHECK-NEWLOWERING-NEXT: sub x10, x29, #16
+; CHECK-NEWLOWERING-NEXT: stp x9, x8, [x29, #-16]
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, x10
+; CHECK-NEWLOWERING-NEXT: bl cosf
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: sub x0, x29, #16
+; CHECK-NEWLOWERING-NEXT: cbnz x8, .LBB2_2
+; CHECK-NEWLOWERING-NEXT: // %bb.1:
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEWLOWERING-NEXT: .LBB2_2:
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: mov sp, x29
+; CHECK-NEWLOWERING-NEXT: ldr x19, [sp, #16] // 8-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ldp x29, x30, [sp], #32 // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ret
%res = call float @llvm.cos.f32(float %a)
ret float %res
}
@@ -174,6 +260,547 @@ define void @test_lazy_save_and_conditional_smstart() nounwind "aarch64_inout_za
; CHECK-NEXT: ldp d13, d12, [sp, #16] // 16-byte Folded Reload
; CHECK-NEXT: ldp d15, d14, [sp], #112 // 16-byte Folded Reload
; CHECK-NEXT: ret
+;
+; CHECK-NEWLOWERING-LABEL: test_lazy_save_and_conditional_smstart:
+; CHECK-NEWLOWERING: // %bb.0:
+; CHECK-NEWLOWERING-NEXT: stp d15, d14, [sp, #-112]! // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: cntd x9
+; CHECK-NEWLOWERING-NEXT: stp d13, d12, [sp, #16] // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: stp d11, d10, [sp, #32] // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: stp d9, d8, [sp, #48] // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: stp x29, x30, [sp, #64] // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: add x29, sp, #64
+; CHECK-NEWLOWERING-NEXT: str x9, [sp, #80] // 8-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: stp x20, x19, [sp, #96] // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: sub sp, sp, #16
+; CHECK-NEWLOWERING-NEXT: rdsvl x8, #1
+; CHECK-NEWLOWERING-NEXT: mov x9, sp
+; CHECK-NEWLOWERING-NEXT: msub x9, x8, x8, x9
+; CHECK-NEWLOWERING-NEXT: mov sp, x9
+; CHECK-NEWLOWERING-NEXT: stp x9, x8, [x29, #-80]
+; CHECK-NEWLOWERING-NEXT: bl __arm_sme_state
+; CHECK-NEWLOWERING-NEXT: sub x8, x29, #80
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, x8
+; CHECK-NEWLOWERING-NEXT: and x20, x0, #0x1
+; CHECK-NEWLOWERING-NEXT: tbz w20, #0, .LBB3_2
+; CHECK-NEWLOWERING-NEXT: // %bb.1:
+; CHECK-NEWLOWERING-NEXT: smstop sm
+; CHECK-NEWLOWERING-NEXT: .LBB3_2:
+; CHECK-NEWLOWERING-NEXT: bl private_za_callee
+; CHECK-NEWLOWERING-NEXT: tbz w20, #0, .LBB3_4
+; CHECK-NEWLOWERING-NEXT: // %bb.3:
+; CHECK-NEWLOWERING-NEXT: smstart sm
+; CHECK-NEWLOWERING-NEXT: .LBB3_4:
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: sub x0, x29, #80
+; CHECK-NEWLOWERING-NEXT: cbnz x8, .LBB3_6
+; CHECK-NEWLOWERING-NEXT: // %bb.5:
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEWLOWERING-NEXT: .LBB3_6:
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: sub sp, x29, #64
+; CHECK-NEWLOWERING-NEXT: ldp x20, x19, [sp, #96] // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ldp x29, x30, [sp, #64] // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ldp d9, d8, [sp, #48] // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ldp d11, d10, [sp, #32] // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ldp d13, d12, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ldp d15, d14, [sp], #112 // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ret
+ call void @private_za_callee()
+ ret void
+}
+
+; Note: For the final @private_za_callee() we setup a lazy save then don't
+; restore from it (since ZA is off on return). We could improve this case
+; by turning ZA off before the final private ZA call.
+define void @test_lazy_save_mixed_shared_and_private_callees() "aarch64_new_za"
+; CHECK-LABEL: test_lazy_save_mixed_shared_and_private_callees:
+; CHECK: // %bb.0: // %prelude
+; CHECK-NEXT: stp x29, x30, [sp, #-48]! // 16-byte Folded Spill
+; CHECK-NEXT: str x21, [sp, #16] // 8-byte Folded Spill
+; CHECK-NEXT: mov x29, sp
+; CHECK-NEXT: stp x20, x19, [sp, #32] // 16-byte Folded Spill
+; CHECK-NEXT: sub sp, sp, #16
+; CHECK-NEXT: .cfi_def_cfa w29, 48
+; CHECK-NEXT: .cfi_offset w19, -8
+; CHECK-NEXT: .cfi_offset w20, -16
+; CHECK-NEXT: .cfi_offset w21, -32
+; CHECK-NEXT: .cfi_offset w30, -40
+; CHECK-NEXT: .cfi_offset w29, -48
+; CHECK-NEXT: rdsvl x8, #1
+; CHECK-NEXT: mov x9, sp
+; CHECK-NEXT: msub x8, x8, x8, x9
+; CHECK-NEXT: mov sp, x8
+; CHECK-NEXT: stur x8, [x29, #-16]
+; CHECK-NEXT: sturh wzr, [x29, #-6]
+; CHECK-NEXT: stur wzr, [x29, #-4]
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: cbz x8, .LBB4_2
+; CHECK-NEXT: // %bb.1: // %save.za
+; CHECK-NEXT: bl __arm_tpidr2_save
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: .LBB4_2:
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: rdsvl x20, #1
+; CHECK-NEXT: sub x21, x29, #16
+; CHECK-NEXT: zero {za}
+; CHECK-NEXT: sturh w20, [x29, #-8]
+; CHECK-NEXT: msr TPIDR2_EL0, x21
+; CHECK-NEXT: bl private_za_callee
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: sub x0, x29, #16
+; CHECK-NEXT: cbnz x8, .LBB4_4
+; CHECK-NEXT: // %bb.3:
+; CHECK-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEXT: .LBB4_4:
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: bl shared_za_callee
+; CHECK-NEXT: bl preserves_za_callee
+; CHECK-NEXT: sturh w20, [x29, #-8]
+; CHECK-NEXT: msr TPIDR2_EL0, x21
+; CHECK-NEXT: bl private_za_callee
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: sub x0, x29, #16
+; CHECK-NEXT: cbnz x8, .LBB4_6
+; CHECK-NEXT: // %bb.5:
+; CHECK-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEXT: .LBB4_6:
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: smstop za
+; CHECK-NEXT: mov sp, x29
+; CHECK-NEXT: ldp x20, x19, [sp, #32] // 16-byte Folded Reload
+; CHECK-NEXT: ldr x21, [sp, #16] // 8-byte Folded Reload
+; CHECK-NEXT: ldp x29, x30, [sp], #48 // 16-byte Folded Reload
+; CHECK-NEXT: ret
+;
+; CHECK-NEWLOWERING-LABEL: test_lazy_save_mixed_shared_and_private_callees:
+; CHECK-NEWLOWERING: // %bb.0:
+; CHECK-NEWLOWERING-NEXT: stp x29, x30, [sp, #-32]! // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: str x19, [sp, #16] // 8-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: mov x29, sp
+; CHECK-NEWLOWERING-NEXT: sub sp, sp, #16
+; CHECK-NEWLOWERING-NEXT: .cfi_def_cfa w29, 32
+; CHECK-NEWLOWERING-NEXT: .cfi_offset w19, -16
+; CHECK-NEWLOWERING-NEXT: .cfi_offset w30, -24
+; CHECK-NEWLOWERING-NEXT: .cfi_offset w29, -32
+; CHECK-NEWLOWERING-NEXT: rdsvl x8, #1
+; CHECK-NEWLOWERING-NEXT: mov x9, sp
+; CHECK-NEWLOWERING-NEXT: msub x9, x8, x8, x9
+; CHECK-NEWLOWERING-NEXT: mov sp, x9
+; CHECK-NEWLOWERING-NEXT: stp x9, x8, [x29, #-16]
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: cbz x8, .LBB4_2
+; CHECK-NEWLOWERING-NEXT: // %bb.1:
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_save
+; CHECK-NEWLOWERING-NEXT: .LBB4_2:
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: sub x8, x29, #16
+; CHECK-NEWLOWERING-NEXT: zero {za}
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, x8
+; CHECK-NEWLOWERING-NEXT: bl private_za_callee
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: sub x0, x29, #16
+; CHECK-NEWLOWERING-NEXT: cbnz x8, .LBB4_4
+; CHECK-NEWLOWERING-NEXT: // %bb.3:
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEWLOWERING-NEXT: .LBB4_4:
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: bl shared_za_callee
+; CHECK-NEWLOWERING-NEXT: bl preserves_za_callee
+; CHECK-NEWLOWERING-NEXT: sub x8, x29, #16
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, x8
+; CHECK-NEWLOWERING-NEXT: bl private_za_callee
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: smstop za
+; CHECK-NEWLOWERING-NEXT: mov sp, x29
+; CHECK-NEWLOWERING-NEXT: ldr x19, [sp, #16] // 8-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ldp x29, x30, [sp], #32 // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ret
+{
+ call void @private_za_callee()
+ call void @shared_za_callee()
+ call void @preserves_za_callee()
+ call void @private_za_callee()
+ ret void
+}
+
+define void @test_many_back2back_private_za_calls() "aarch64_inout_za" {
+; CHECK-LABEL: test_many_back2back_private_za_calls:
+; CHECK: // %bb.0:
+; CHECK-NEXT: stp x29, x30, [sp, #-48]! // 16-byte Folded Spill
+; CHECK-NEXT: str x21, [sp, #16] // 8-byte Folded Spill
+; CHECK-NEXT: mov x29, sp
+; CHECK-NEXT: stp x20, x19, [sp, #32] // 16-byte Folded Spill
+; CHECK-NEXT: sub sp, sp, #16
+; CHECK-NEXT: .cfi_def_cfa w29, 48
+; CHECK-NEXT: .cfi_offset w19, -8
+; CHECK-NEXT: .cfi_offset w20, -16
+; CHECK-NEXT: .cfi_offset w21, -32
+; CHECK-NEXT: .cfi_offset w30, -40
+; CHECK-NEXT: .cfi_offset w29, -48
+; CHECK-NEXT: rdsvl x20, #1
+; CHECK-NEXT: mov x8, sp
+; CHECK-NEXT: msub x8, x20, x20, x8
+; CHECK-NEXT: mov sp, x8
+; CHECK-NEXT: stur x8, [x29, #-16]
+; CHECK-NEXT: sturh wzr, [x29, #-6]
+; CHECK-NEXT: stur wzr, [x29, #-4]
+; CHECK-NEXT: bl shared_za_callee
+; CHECK-NEXT: sub x21, x29, #16
+; CHECK-NEXT: sturh w20, [x29, #-8]
+; CHECK-NEXT: msr TPIDR2_EL0, x21
+; CHECK-NEXT: bl private_za_callee
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: sub x0, x29, #16
+; CHECK-NEXT: cbnz x8, .LBB5_2
+; CHECK-NEXT: // %bb.1:
+; CHECK-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEXT: .LBB5_2:
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: sturh w20, [x29, #-8]
+; CHECK-NEXT: msr TPIDR2_EL0, x21
+; CHECK-NEXT: bl private_za_callee
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: sub x0, x29, #16
+; CHECK-NEXT: cbnz x8, .LBB5_4
+; CHECK-NEXT: // %bb.3:
+; CHECK-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEXT: .LBB5_4:
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: sturh w20, [x29, #-8]
+; CHECK-NEXT: msr TPIDR2_EL0, x21
+; CHECK-NEXT: bl private_za_callee
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: sub x0, x29, #16
+; CHECK-NEXT: cbnz x8, .LBB5_6
+; CHECK-NEXT: // %bb.5:
+; CHECK-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEXT: .LBB5_6:
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: sturh w20, [x29, #-8]
+; CHECK-NEXT: msr TPIDR2_EL0, x21
+; CHECK-NEXT: bl private_za_callee
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: sub x0, x29, #16
+; CHECK-NEXT: cbnz x8, .LBB5_8
+; CHECK-NEXT: // %bb.7:
+; CHECK-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEXT: .LBB5_8:
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: sturh w20, [x29, #-8]
+; CHECK-NEXT: msr TPIDR2_EL0, x21
+; CHECK-NEXT: bl private_za_callee
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: sub x0, x29, #16
+; CHECK-NEXT: cbnz x8, .LBB5_10
+; CHECK-NEXT: // %bb.9:
+; CHECK-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEXT: .LBB5_10:
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: sturh w20, [x29, #-8]
+; CHECK-NEXT: msr TPIDR2_EL0, x21
+; CHECK-NEXT: bl private_za_callee
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: sub x0, x29, #16
+; CHECK-NEXT: cbnz x8, .LBB5_12
+; CHECK-NEXT: // %bb.11:
+; CHECK-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEXT: .LBB5_12:
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: bl shared_za_callee
+; CHECK-NEXT: mov sp, x29
+; CHECK-NEXT: ldp x20, x19, [sp, #32] // 16-byte Folded Reload
+; CHECK-NEXT: ldr x21, [sp, #16] // 8-byte Folded Reload
+; CHECK-NEXT: ldp x29, x30, [sp], #48 // 16-byte Folded Reload
+; CHECK-NEXT: ret
+;
+; CHECK-NEWLOWERING-LABEL: test_many_back2back_private_za_calls:
+; CHECK-NEWLOWERING: // %bb.0:
+; CHECK-NEWLOWERING-NEXT: stp x29, x30, [sp, #-32]! // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: str x19, [sp, #16] // 8-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: mov x29, sp
+; CHECK-NEWLOWERING-NEXT: sub sp, sp, #16
+; CHECK-NEWLOWERING-NEXT: .cfi_def_cfa w29, 32
+; CHECK-NEWLOWERING-NEXT: .cfi_offset w19, -16
+; CHECK-NEWLOWERING-NEXT: .cfi_offset w30, -24
+; CHECK-NEWLOWERING-NEXT: .cfi_offset w29, -32
+; CHECK-NEWLOWERING-NEXT: rdsvl x8, #1
+; CHECK-NEWLOWERING-NEXT: mov x9, sp
+; CHECK-NEWLOWERING-NEXT: msub x9, x8, x8, x9
+; CHECK-NEWLOWERING-NEXT: mov sp, x9
+; CHECK-NEWLOWERING-NEXT: stp x9, x8, [x29, #-16]
+; CHECK-NEWLOWERING-NEXT: bl shared_za_callee
+; CHECK-NEWLOWERING-NEXT: sub x8, x29, #16
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, x8
+; CHECK-NEWLOWERING-NEXT: bl private_za_callee
+; CHECK-NEWLOWERING-NEXT: bl private_za_callee
+; CHECK-NEWLOWERING-NEXT: bl private_za_callee
+; CHECK-NEWLOWERING-NEXT: bl private_za_callee
+; CHECK-NEWLOWERING-NEXT: bl private_za_callee
+; CHECK-NEWLOWERING-NEXT: bl private_za_callee
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: sub x0, x29, #16
+; CHECK-NEWLOWERING-NEXT: cbnz x8, .LBB5_2
+; CHECK-NEWLOWERING-NEXT: // %bb.1:
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEWLOWERING-NEXT: .LBB5_2:
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: bl shared_za_callee
+; CHECK-NEWLOWERING-NEXT: mov sp, x29
+; CHECK-NEWLOWERING-NEXT: ldr x19, [sp, #16] // 8-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ldp x29, x30, [sp], #32 // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ret
+ call void @shared_za_callee()
+ call void @private_za_callee()
+ call void @private_za_callee()
+ call void @private_za_callee()
+ call void @private_za_callee()
+ call void @private_za_callee()
+ call void @private_za_callee()
+ call void @shared_za_callee()
+ ret void
+}
+
+define void @test_shared_private_shared() nounwind "aarch64_inout_za" {
+; CHECK-LABEL: test_shared_private_shared:
+; CHECK: // %bb.0:
+; CHECK-NEXT: stp x29, x30, [sp, #-32]! // 16-byte Folded Spill
+; CHECK-NEXT: stp x20, x19, [sp, #16] // 16-byte Folded Spill
+; CHECK-NEXT: mov x29, sp
+; CHECK-NEXT: sub sp, sp, #16
+; CHECK-NEXT: rdsvl x20, #1
+; CHECK-NEXT: mov x8, sp
+; CHECK-NEXT: msub x8, x20, x20, x8
+; CHECK-NEXT: mov sp, x8
+; CHECK-NEXT: stur x8, [x29, #-16]
+; CHECK-NEXT: sturh wzr, [x29, #-6]
+; CHECK-NEXT: stur wzr, [x29, #-4]
+; CHECK-NEXT: bl shared_za_callee
+; CHECK-NEXT: sub x8, x29, #16
+; CHECK-NEXT: sturh w20, [x29, #-8]
+; CHECK-NEXT: msr TPIDR2_EL0, x8
+; CHECK-NEXT: bl private_za_callee
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: sub x0, x29, #16
+; CHECK-NEXT: cbnz x8, .LBB6_2
+; CHECK-NEXT: // %bb.1:
+; CHECK-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEXT: .LBB6_2:
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: bl shared_za_callee
+; CHECK-NEXT: mov sp, x29
+; CHECK-NEXT: ldp x20, x19, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEXT: ldp x29, x30, [sp], #32 // 16-byte Folded Reload
+; CHECK-NEXT: ret
+;
+; CHECK-NEWLOWERING-LABEL: test_shared_private_shared:
+; CHECK-NEWLOWERING: // %bb.0:
+; CHECK-NEWLOWERING-NEXT: stp x29, x30, [sp, #-32]! // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: str x19, [sp, #16] // 8-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: mov x29, sp
+; CHECK-NEWLOWERING-NEXT: sub sp, sp, #16
+; CHECK-NEWLOWERING-NEXT: rdsvl x8, #1
+; CHECK-NEWLOWERING-NEXT: mov x9, sp
+; CHECK-NEWLOWERING-NEXT: msub x9, x8, x8, x9
+; CHECK-NEWLOWERING-NEXT: mov sp, x9
+; CHECK-NEWLOWERING-NEXT: stp x9, x8, [x29, #-16]
+; CHECK-NEWLOWERING-NEXT: bl shared_za_callee
+; CHECK-NEWLOWERING-NEXT: sub x8, x29, #16
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, x8
+; CHECK-NEWLOWERING-NEXT: bl private_za_callee
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: sub x0, x29, #16
+; CHECK-NEWLOWERING-NEXT: cbnz x8, .LBB6_2
+; CHECK-NEWLOWERING-NEXT: // %bb.1:
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEWLOWERING-NEXT: .LBB6_2:
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: bl shared_za_callee
+; CHECK-NEWLOWERING-NEXT: mov sp, x29
+; CHECK-NEWLOWERING-NEXT: ldr x19, [sp, #16] // 8-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ldp x29, x30, [sp], #32 // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ret
+ call void @shared_za_callee()
call void @private_za_callee()
+ call void @shared_za_callee()
ret void
}
+
+define void @test_only_shared_za() nounwind "aarch64_inout_za" {
+; CHECK-COMMON-LABEL: test_only_shared_za:
+; CHECK-COMMON: // %bb.0:
+; CHECK-COMMON-NEXT: str x30, [sp, #-16]! // 8-byte Folded Spill
+; CHECK-COMMON-NEXT: bl shared_za_callee
+; CHECK-COMMON-NEXT: ldr x30, [sp], #16 // 8-byte Folded Reload
+; CHECK-COMMON-NEXT: ret
+ call void @shared_za_callee()
+ ret void
+}
+
+declare i64 @shared_za_callee_i64(i64) "aarch64_inout_za"
+declare i64 @private_za_callee_i64(i64)
+
+define i64 @test_shared_private_shared_i64(i64 %x) nounwind "aarch64_inout_za" {
+; CHECK-LABEL: test_shared_private_shared_i64:
+; CHECK: // %bb.0:
+; CHECK-NEXT: stp x29, x30, [sp, #-32]! // 16-byte Folded Spill
+; CHECK-NEXT: stp x20, x19, [sp, #16] // 16-byte Folded Spill
+; CHECK-NEXT: mov x29, sp
+; CHECK-NEXT: sub sp, sp, #16
+; CHECK-NEXT: rdsvl x20, #1
+; CHECK-NEXT: mov x8, sp
+; CHECK-NEXT: msub x8, x20, x20, x8
+; CHECK-NEXT: mov sp, x8
+; CHECK-NEXT: stur x8, [x29, #-16]
+; CHECK-NEXT: sturh wzr, [x29, #-6]
+; CHECK-NEXT: stur wzr, [x29, #-4]
+; CHECK-NEXT: bl shared_za_callee_i64
+; CHECK-NEXT: sub x8, x29, #16
+; CHECK-NEXT: sturh w20, [x29, #-8]
+; CHECK-NEXT: msr TPIDR2_EL0, x8
+; CHECK-NEXT: bl private_za_callee_i64
+; CHECK-NEXT: mov x1, x0
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: sub x0, x29, #16
+; CHECK-NEXT: cbnz x8, .LBB8_2
+; CHECK-NEXT: // %bb.1:
+; CHECK-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEXT: .LBB8_2:
+; CHECK-NEXT: mov x0, x1
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: bl shared_za_callee_i64
+; CHECK-NEXT: mov sp, x29
+; CHECK-NEXT: ldp x20, x19, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEXT: ldp x29, x30, [sp], #32 // 16-byte Folded Reload
+; CHECK-NEXT: ret
+;
+; CHECK-NEWLOWERING-LABEL: test_shared_private_shared_i64:
+; CHECK-NEWLOWERING: // %bb.0:
+; CHECK-NEWLOWERING-NEXT: stp x29, x30, [sp, #-32]! // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: str x19, [sp, #16] // 8-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: mov x29, sp
+; CHECK-NEWLOWERING-NEXT: sub sp, sp, #16
+; CHECK-NEWLOWERING-NEXT: rdsvl x8, #1
+; CHECK-NEWLOWERING-NEXT: mov x9, sp
+; CHECK-NEWLOWERING-NEXT: msub x9, x8, x8, x9
+; CHECK-NEWLOWERING-NEXT: mov sp, x9
+; CHECK-NEWLOWERING-NEXT: stp x9, x8, [x29, #-16]
+; CHECK-NEWLOWERING-NEXT: bl shared_za_callee_i64
+; CHECK-NEWLOWERING-NEXT: sub x8, x29, #16
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, x8
+; CHECK-NEWLOWERING-NEXT: bl private_za_callee_i64
+; CHECK-NEWLOWERING-NEXT: mov x1, x0
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: sub x0, x29, #16
+; CHECK-NEWLOWERING-NEXT: cbnz x8, .LBB8_2
+; CHECK-NEWLOWERING-NEXT: // %bb.1:
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEWLOWERING-NEXT: .LBB8_2:
+; CHECK-NEWLOWERING-NEXT: mov x0, x1
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: bl shared_za_callee_i64
+; CHECK-NEWLOWERING-NEXT: mov sp, x29
+; CHECK-NEWLOWERING-NEXT: ldr x19, [sp, #16] // 8-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ldp x29, x30, [sp], #32 // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ret
+ %a = call i64 @shared_za_callee_i64(i64 %x)
+ %b = call i64 @private_za_callee_i64(i64 %a)
+ %c = call i64 @shared_za_callee_i64(i64 %b)
+ ret i64 %c
+}
+
+declare i64 @many_args_private_za_callee(
+ i64, i64, i64, i64, i64, i64, i64, i64, i64, i64)
+
+; In this example some arguments are passed on the stack, which decrements the
+; stack pointer before the call -- in this test the lazy save should be setup
+; before the stack decrement.
+define i64 @test_many_callee_arguments(
+; CHECK-LABEL: test_many_callee_arguments:
+; CHECK: // %bb.0:
+; CHECK-NEXT: stp x29, x30, [sp, #-32]! // 16-byte Folded Spill
+; CHECK-NEXT: str x19, [sp, #16] // 8-byte Folded Spill
+; CHECK-NEXT: mov x29, sp
+; CHECK-NEXT: sub sp, sp, #16
+; CHECK-NEXT: mov x8, sp
+; CHECK-NEXT: rdsvl x9, #1
+; CHECK-NEXT: msub x8, x9, x9, x8
+; CHECK-NEXT: mov sp, x8
+; CHECK-NEXT: ldp x10, x11, [x29, #32]
+; CHECK-NEXT: stur x8, [x29, #-16]
+; CHECK-NEXT: sub x8, x29, #16
+; CHECK-NEXT: sturh wzr, [x29, #-6]
+; CHECK-NEXT: stur wzr, [x29, #-4]
+; CHECK-NEXT: sturh w9, [x29, #-8]
+; CHECK-NEXT: msr TPIDR2_EL0, x8
+; CHECK-NEXT: stp x10, x11, [sp, #-16]!
+; CHECK-NEXT: bl many_args_private_za_callee
+; CHECK-NEXT: add sp, sp, #16
+; CHECK-NEXT: mov x1, x0
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: sub x0, x29, #16
+; CHECK-NEXT: cbnz x8, .LBB9_2
+; CHECK-NEXT: // %bb.1:
+; CHECK-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEXT: .LBB9_2:
+; CHECK-NEXT: mov x0, x1
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: mov sp, x29
+; CHECK-NEXT: ldr x19, [sp, #16] // 8-byte Folded Reload
+; CHECK-NEXT: ldp x29, x30, [sp], #32 // 16-byte Folded Reload
+; CHECK-NEXT: ret
+;
+; CHECK-NEWLOWERING-LABEL: test_many_callee_arguments:
+; CHECK-NEWLOWERING: // %bb.0:
+; CHECK-NEWLOWERING-NEXT: stp x29, x30, [sp, #-32]! // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: str x19, [sp, #16] // 8-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: mov x29, sp
+; CHECK-NEWLOWERING-NEXT: sub sp, sp, #16
+; CHECK-NEWLOWERING-NEXT: rdsvl x8, #1
+; CHECK-NEWLOWERING-NEXT: mov x9, sp
+; CHECK-NEWLOWERING-NEXT: msub x9, x8, x8, x9
+; CHECK-NEWLOWERING-NEXT: mov sp, x9
+; CHECK-NEWLOWERING-NEXT: ldp x10, x11, [x29, #32]
+; CHECK-NEWLOWERING-NEXT: sub x12, x29, #16
+; CHECK-NEWLOWERING-NEXT: stp x9, x8, [x29, #-16]
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, x12
+; CHECK-NEWLOWERING-NEXT: stp x10, x11, [sp, #-16]!
+; CHECK-NEWLOWERING-NEXT: bl many_args_private_za_callee
+; CHECK-NEWLOWERING-NEXT: add sp, sp, #16
+; CHECK-NEWLOWERING-NEXT: mov x8, x0
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: mrs x9, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: sub x0, x29, #16
+; CHECK-NEWLOWERING-NEXT: cbnz x9, .LBB9_2
+; CHECK-NEWLOWERING-NEXT: // %bb.1:
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEWLOWERING-NEXT: .LBB9_2:
+; CHECK-NEWLOWERING-NEXT: mov x0, x8
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: mov sp, x29
+; CHECK-NEWLOWERING-NEXT: ldr x19, [sp, #16] // 8-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ldp x29, x30, [sp], #32 // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ret
+ i64 %0, i64 %1, i64 %2, i64 %3, i64 %4, i64 %5, i64 %6, i64 %7, i64 %8, i64 %9
+) nounwind "aarch64_inout_za" {
+ %ret = call i64 @many_args_private_za_callee(
+ i64 %0, i64 %1, i64 %2, i64 %3, i64 %4, i64 %5, i64 %6, i64 %7, i64 %8, i64 %9)
+ ret i64 %ret
+}
diff --git a/llvm/test/CodeGen/AArch64/sme-lazy-sve-nzcv-live.mir b/llvm/test/CodeGen/AArch64/sme-lazy-sve-nzcv-live.mir
new file mode 100644
index 0000000000000..18764d508d0fa
--- /dev/null
+++ b/llvm/test/CodeGen/AArch64/sme-lazy-sve-nzcv-live.mir
@@ -0,0 +1,132 @@
+# RUN: llc -mtriple=aarch64 -mattr=+sve -mattr=+sme -run-pass=aarch64-machine-sme-abi -verify-machineinstrs %s -o - | FileCheck %s
+# RUN: llc -mtriple=aarch64 -mattr=+sve -mattr=+sme -aarch64-new-sme-abi %s -o - | FileCheck %s --check-prefix=CHECK-ASM
+
+# This tests the unfortunate case the status flags ($nzcv) are live at the point
+# we want to restore ZA. Currently, this is handled by saving them to a scratch
+# register.
+
+--- |
+ define void @cmp_branch(i32 %0) "aarch64_inout_za" {
+ tail call void @clobber()
+ %2 = icmp sgt i32 %0, 100
+ br i1 %2, label %3, label %4
+
+ 3: ; preds = %1
+ tail call void @inout_call() #3
+ br label %4
+
+ 4: ; preds = %3, %1
+ tail call void @inout_call() #3
+ ret void
+ }
+
+ declare void @clobber()
+ declare void @inout_call() "aarch64_inout_za"
+...
+---
+name: cmp_branch
+alignment: 4
+tracksRegLiveness: true
+noPhis: false
+isSSA: true
+noVRegs: false
+hasFakeUses: false
+registers:
+ - { id: 0, class: gpr32common }
+ - { id: 1, class: gpr32 }
+liveins:
+ - { reg: '$w0' }
+frameInfo:
+ maxAlignment: 1
+ adjustsStack: true
+ hasCalls: true
+ maxCallFrameSize: 0
+ hasTailCall: true
+machineFunctionInfo: {}
+body: |
+ ; CHECK-LABEL: name: cmp_branch
+ ; CHECK: bb.0 (%ir-block.1):
+ ; CHECK-NEXT: successors: %bb.1(0x40000000), %bb.2(0x40000000)
+ ; CHECK-NEXT: liveins: $w0
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: [[RDSVLI_XI:%[0-9]+]]:gpr64 = RDSVLI_XI 1, implicit $vg
+ ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64 = COPY $sp
+ ; CHECK-NEXT: [[MSUBXrrr:%[0-9]+]]:gpr64 = MSUBXrrr [[RDSVLI_XI]], [[RDSVLI_XI]], [[COPY]]
+ ; CHECK-NEXT: $sp = COPY [[MSUBXrrr]]
+ ; CHECK-NEXT: STPXi [[MSUBXrrr]], [[RDSVLI_XI]], %stack.0, 0
+ ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr32common = COPY $w0
+ ; CHECK-NEXT: [[ADDXri:%[0-9]+]]:gpr64sp = ADDXri %stack.0, 0, 0
+ ; CHECK-NEXT: [[COPY2:%[0-9]+]]:gpr64 = COPY [[ADDXri]]
+ ; CHECK-NEXT: MSR 56965, [[COPY2]]
+ ; CHECK-NEXT: ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+ ; CHECK-NEXT: RequiresZASavePseudo
+ ; CHECK-NEXT: BL @clobber, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp
+ ; CHECK-NEXT: ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+ ; CHECK-NEXT: [[SUBSWri:%[0-9]+]]:gpr32 = SUBSWri [[COPY1]], 101, 0, implicit-def $nzcv
+ ; CHECK-NEXT: [[MRS:%[0-9]+]]:gpr64 = MRS 55824, implicit-def $nzcv, implicit $nzcv
+ ; CHECK-NEXT: MSRpstatesvcrImm1 2, 1, implicit-def $nzcv
+ ; CHECK-NEXT: [[MRS1:%[0-9]+]]:gpr64 = MRS 56965, implicit-def $nzcv
+ ; CHECK-NEXT: $x0 = ADDXri %stack.0, 0, 0
+ ; CHECK-NEXT: RestoreZAPseudo [[MRS1]], $x0, &__arm_tpidr2_restore, csr_aarch64_sme_abi_support_routines_preservemost_from_x0
+ ; CHECK-NEXT: MSR 56965, $xzr
+ ; CHECK-NEXT: MSR 55824, [[MRS]], implicit-def $nzcv
+ ; CHECK-NEXT: Bcc 11, %bb.2, implicit $nzcv
+ ; CHECK-NEXT: B %bb.1
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.1 (%ir-block.3):
+ ; CHECK-NEXT: successors: %bb.2(0x80000000)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+ ; CHECK-NEXT: InOutZAUsePseudo
+ ; CHECK-NEXT: BL @inout_call, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp
+ ; CHECK-NEXT: ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.2 (%ir-block.4):
+ ; CHECK-NEXT: ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+ ; CHECK-NEXT: InOutZAUsePseudo
+ ; CHECK-NEXT: ADJCALLSTACKUP 0, 0, implicit-def $sp, implicit $sp
+ ; CHECK-NEXT: TCRETURNdi @inout_call, 0, csr_aarch64_aapcs, implicit $sp
+
+ bb.0 (%ir-block.1):
+ successors: %bb.1, %bb.2
+ liveins: $w0
+
+ %0:gpr32common = COPY $w0
+ ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+ RequiresZASavePseudo
+ BL @clobber, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp
+ ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+ %1:gpr32 = SUBSWri %0, 101, 0, implicit-def $nzcv
+ Bcc 11, %bb.2, implicit $nzcv
+ B %bb.1
+
+ bb.1 (%ir-block.3):
+ ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+ InOutZAUsePseudo
+ BL @inout_call, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp
+ ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+
+ bb.2 (%ir-block.4):
+ ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+ InOutZAUsePseudo
+ ADJCALLSTACKUP 0, 0, implicit-def $sp, implicit $sp
+ TCRETURNdi @inout_call, 0, csr_aarch64_aapcs, implicit $sp
+...
+
+# CHECK-ASM-LABEL: cmp_branch
+# CHECK-ASM: msr TPIDR2_EL0, x10
+# CHECK-ASM-NEXT: bl clobber
+# CHECK-ASM-NEXT: cmp w20, #101
+# CHECK-ASM-NEXT: mrs x8, NZCV
+# CHECK-ASM-NEXT: smstart za
+# CHECK-ASM-NEXT: mrs x9, TPIDR2_EL0
+# CHECK-ASM-NEXT: sub x0, x29, #16
+# CHECK-ASM-NEXT: cbnz x9, .LBB0_2
+# CHECK-ASM: bl __arm_tpidr2_restore
+# CHECK-ASM-NEXT: .LBB0_2:
+# CHECK-ASM-NEXT: msr TPIDR2_EL0, xzr
+# CHECK-ASM-NEXT: msr NZCV, x8
+# CHECK-ASM-NEXT: b.lt .LBB0_4
+# CHECK-ASM: bl inout_call
+# CHECK-ASM-NEXT: .LBB0_4:
+# CHECK-ASM: b inout_call
diff --git a/llvm/test/CodeGen/AArch64/sme-new-za-function.ll b/llvm/test/CodeGen/AArch64/sme-new-za-function.ll
index 04d26902c536a..157fc4d88b8dc 100644
--- a/llvm/test/CodeGen/AArch64/sme-new-za-function.ll
+++ b/llvm/test/CodeGen/AArch64/sme-new-za-function.ll
@@ -1,53 +1,133 @@
-; RUN: opt -S -mtriple=aarch64-linux-gnu -aarch64-sme-abi %s | FileCheck %s
-; RUN: opt -S -mtriple=aarch64-linux-gnu -aarch64-sme-abi -aarch64-sme-abi %s | FileCheck %s
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5
+; RUN: llc -O0 -mtriple=aarch64-linux-gnu -mattr=+sme -verify-machineinstrs < %s | FileCheck %s
+; RUN: llc -O0 -mtriple=aarch64-linux-gnu -mattr=+sme -verify-machineinstrs -aarch64-new-sme-abi < %s | FileCheck %s --check-prefix=CHECK-NEWLOWERING
declare void @shared_za_callee() "aarch64_inout_za"
define void @private_za() "aarch64_new_za" {
-; CHECK-LABEL: @private_za(
-; CHECK-NEXT: prelude:
-; CHECK-NEXT: [[TPIDR2:%.*]] = call i64 @llvm.aarch64.sme.get.tpidr2()
-; CHECK-NEXT: [[CMP:%.*]] = icmp ne i64 [[TPIDR2]], 0
-; CHECK-NEXT: br i1 [[CMP]], label [[SAVE_ZA:%.*]], label [[TMP0:%.*]]
-; CHECK: save.za:
-; CHECK-NEXT: call aarch64_sme_preservemost_from_x0 void @__arm_tpidr2_save()
-; CHECK-NEXT: call void @llvm.aarch64.sme.set.tpidr2(i64 0)
-; CHECK-NEXT: br label [[TMP0]]
-; CHECK: 0:
-; CHECK-NEXT: call void @llvm.aarch64.sme.za.enable()
-; CHECK-NEXT: call void @llvm.aarch64.sme.zero(i32 255)
-; CHECK-NEXT: call void @shared_za_callee()
-; CHECK-NEXT: call void @llvm.aarch64.sme.za.disable()
-; CHECK-NEXT: ret void
+; CHECK-LABEL: private_za:
+; CHECK: // %bb.0: // %prelude
+; CHECK-NEXT: str x30, [sp, #-16]! // 8-byte Folded Spill
+; CHECK-NEXT: .cfi_def_cfa_offset 16
+; CHECK-NEXT: .cfi_offset w30, -16
+; CHECK-NEXT: rdsvl x8, #1
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: cbz x8, .LBB0_2
+; CHECK-NEXT: b .LBB0_1
+; CHECK-NEXT: .LBB0_1: // %save.za
+; CHECK-NEXT: bl __arm_tpidr2_save
+; CHECK-NEXT: mov x8, xzr
+; CHECK-NEXT: msr TPIDR2_EL0, x8
+; CHECK-NEXT: b .LBB0_2
+; CHECK-NEXT: .LBB0_2:
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: zero {za}
+; CHECK-NEXT: bl shared_za_callee
+; CHECK-NEXT: smstop za
+; CHECK-NEXT: ldr x30, [sp], #16 // 8-byte Folded Reload
+; CHECK-NEXT: ret
;
+; CHECK-NEWLOWERING-LABEL: private_za:
+; CHECK-NEWLOWERING: // %bb.0:
+; CHECK-NEWLOWERING-NEXT: str x30, [sp, #-16]! // 8-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: .cfi_def_cfa_offset 16
+; CHECK-NEWLOWERING-NEXT: .cfi_offset w30, -16
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: cbnz x8, .LBB0_1
+; CHECK-NEWLOWERING-NEXT: b .LBB0_2
+; CHECK-NEWLOWERING-NEXT: .LBB0_1:
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_save
+; CHECK-NEWLOWERING-NEXT: b .LBB0_2
+; CHECK-NEWLOWERING-NEXT: .LBB0_2:
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: zero {za}
+; CHECK-NEWLOWERING-NEXT: bl shared_za_callee
+; CHECK-NEWLOWERING-NEXT: smstop za
+; CHECK-NEWLOWERING-NEXT: ldr x30, [sp], #16 // 8-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ret
call void @shared_za_callee()
ret void
}
+; Note: This test must run at -O0 as otherwise the multiple exits are optimized out.
+; TODO: We should be able to omit the ZA save here (as this function does not use ZA).
define i32 @private_za_multiple_exit(i32 %a, i32 %b, i64 %cond) "aarch64_new_za" {
-; CHECK-LABEL: @private_za_multiple_exit(
-; CHECK-NEXT: prelude:
-; CHECK-NEXT: [[TPIDR2:%.*]] = call i64 @llvm.aarch64.sme.get.tpidr2()
-; CHECK-NEXT: [[CMP:%.*]] = icmp ne i64 [[TPIDR2]], 0
-; CHECK-NEXT: br i1 [[CMP]], label [[SAVE_ZA:%.*]], label [[ENTRY:%.*]]
-; CHECK: save.za:
-; CHECK-NEXT: call aarch64_sme_preservemost_from_x0 void @__arm_tpidr2_save()
-; CHECK-NEXT: call void @llvm.aarch64.sme.set.tpidr2(i64 0)
-; CHECK-NEXT: br label [[ENTRY]]
-; CHECK: entry:
-; CHECK-NEXT: call void @llvm.aarch64.sme.za.enable()
-; CHECK-NEXT: call void @llvm.aarch64.sme.zero(i32 255)
-; CHECK-NEXT: [[TOBOOL:%.*]] = icmp eq i64 [[COND:%.*]], 1
-; CHECK-NEXT: br i1 [[TOBOOL]], label [[IF_ELSE:%.*]], label [[IF_END:%.*]]
-; CHECK: if.else:
-; CHECK-NEXT: [[ADD:%.*]] = add i32 [[A:%.*]], [[B:%.*]]
-; CHECK-NEXT: call void @llvm.aarch64.sme.za.disable()
-; CHECK-NEXT: ret i32 [[ADD]]
-; CHECK: if.end:
-; CHECK-NEXT: [[SUB:%.*]] = sub i32 [[A]], [[B]]
-; CHECK-NEXT: call void @llvm.aarch64.sme.za.disable()
-; CHECK-NEXT: ret i32 [[SUB]]
+; CHECK-LABEL: private_za_multiple_exit:
+; CHECK: // %bb.0: // %prelude
+; CHECK-NEXT: sub sp, sp, #32
+; CHECK-NEXT: str x30, [sp, #16] // 8-byte Folded Spill
+; CHECK-NEXT: .cfi_def_cfa_offset 32
+; CHECK-NEXT: .cfi_offset w30, -16
+; CHECK-NEXT: str x2, [sp] // 8-byte Folded Spill
+; CHECK-NEXT: str w1, [sp, #8] // 4-byte Folded Spill
+; CHECK-NEXT: str w0, [sp, #12] // 4-byte Folded Spill
+; CHECK-NEXT: rdsvl x8, #1
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: cbz x8, .LBB1_2
+; CHECK-NEXT: b .LBB1_1
+; CHECK-NEXT: .LBB1_1: // %save.za
+; CHECK-NEXT: bl __arm_tpidr2_save
+; CHECK-NEXT: mov x8, xzr
+; CHECK-NEXT: msr TPIDR2_EL0, x8
+; CHECK-NEXT: b .LBB1_2
+; CHECK-NEXT: .LBB1_2: // %entry
+; CHECK-NEXT: ldr x8, [sp] // 8-byte Folded Reload
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: zero {za}
+; CHECK-NEXT: subs x8, x8, #1
+; CHECK-NEXT: b.ne .LBB1_4
+; CHECK-NEXT: b .LBB1_3
+; CHECK-NEXT: .LBB1_3: // %if.else
+; CHECK-NEXT: ldr w8, [sp, #12] // 4-byte Folded Reload
+; CHECK-NEXT: ldr w9, [sp, #8] // 4-byte Folded Reload
+; CHECK-NEXT: add w0, w8, w9
+; CHECK-NEXT: smstop za
+; CHECK-NEXT: ldr x30, [sp, #16] // 8-byte Folded Reload
+; CHECK-NEXT: add sp, sp, #32
+; CHECK-NEXT: ret
+; CHECK-NEXT: .LBB1_4: // %if.end
+; CHECK-NEXT: ldr w8, [sp, #12] // 4-byte Folded Reload
+; CHECK-NEXT: ldr w9, [sp, #8] // 4-byte Folded Reload
+; CHECK-NEXT: subs w0, w8, w9
+; CHECK-NEXT: smstop za
+; CHECK-NEXT: ldr x30, [sp, #16] // 8-byte Folded Reload
+; CHECK-NEXT: add sp, sp, #32
+; CHECK-NEXT: ret
;
+; CHECK-NEWLOWERING-LABEL: private_za_multiple_exit:
+; CHECK-NEWLOWERING: // %bb.0: // %entry
+; CHECK-NEWLOWERING-NEXT: sub sp, sp, #16
+; CHECK-NEWLOWERING-NEXT: .cfi_def_cfa_offset 16
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: cbnz x8, .LBB1_1
+; CHECK-NEWLOWERING-NEXT: b .LBB1_2
+; CHECK-NEWLOWERING-NEXT: .LBB1_1: // %entry
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_save
+; CHECK-NEWLOWERING-NEXT: b .LBB1_2
+; CHECK-NEWLOWERING-NEXT: .LBB1_2: // %entry
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: zero {za}
+; CHECK-NEWLOWERING-NEXT: str w1, [sp, #8] // 4-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: str w0, [sp, #12] // 4-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: subs x8, x2, #1
+; CHECK-NEWLOWERING-NEXT: b.ne .LBB1_4
+; CHECK-NEWLOWERING-NEXT: b .LBB1_3
+; CHECK-NEWLOWERING-NEXT: .LBB1_3: // %if.else
+; CHECK-NEWLOWERING-NEXT: ldr w8, [sp, #12] // 4-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ldr w9, [sp, #8] // 4-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: add w0, w8, w9
+; CHECK-NEWLOWERING-NEXT: smstop za
+; CHECK-NEWLOWERING-NEXT: add sp, sp, #16
+; CHECK-NEWLOWERING-NEXT: ret
+; CHECK-NEWLOWERING-NEXT: .LBB1_4: // %if.end
+; CHECK-NEWLOWERING-NEXT: ldr w8, [sp, #12] // 4-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ldr w9, [sp, #8] // 4-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: subs w0, w8, w9
+; CHECK-NEWLOWERING-NEXT: smstop za
+; CHECK-NEWLOWERING-NEXT: add sp, sp, #16
+; CHECK-NEWLOWERING-NEXT: ret
entry:
%tobool = icmp eq i64 %cond, 1
br i1 %tobool, label %if.else, label %if.end
@@ -61,5 +141,38 @@ if.end:
ret i32 %sub
}
-; CHECK: declare void @__arm_tpidr2_save() #[[ATTR:[0-9]+]]
-; CHECK: attributes #[[ATTR]] = { "aarch64_pstate_sm_compatible" }
+; In simple cases like this we should omit all ZA setup.
+define i32 @private_za_trivially_does_not_use_za(i32 %x) "aarch64_new_za" {
+; CHECK-LABEL: private_za_trivially_does_not_use_za:
+; CHECK: // %bb.0: // %prelude
+; CHECK-NEXT: sub sp, sp, #32
+; CHECK-NEXT: str x30, [sp, #16] // 8-byte Folded Spill
+; CHECK-NEXT: .cfi_def_cfa_offset 32
+; CHECK-NEXT: .cfi_offset w30, -16
+; CHECK-NEXT: str w0, [sp, #12] // 4-byte Folded Spill
+; CHECK-NEXT: rdsvl x8, #1
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: cbz x8, .LBB2_2
+; CHECK-NEXT: b .LBB2_1
+; CHECK-NEXT: .LBB2_1: // %save.za
+; CHECK-NEXT: bl __arm_tpidr2_save
+; CHECK-NEXT: mov x8, xzr
+; CHECK-NEXT: msr TPIDR2_EL0, x8
+; CHECK-NEXT: b .LBB2_2
+; CHECK-NEXT: .LBB2_2:
+; CHECK-NEXT: ldr w8, [sp, #12] // 4-byte Folded Reload
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: zero {za}
+; CHECK-NEXT: add w0, w8, w8
+; CHECK-NEXT: smstop za
+; CHECK-NEXT: ldr x30, [sp, #16] // 8-byte Folded Reload
+; CHECK-NEXT: add sp, sp, #32
+; CHECK-NEXT: ret
+;
+; CHECK-NEWLOWERING-LABEL: private_za_trivially_does_not_use_za:
+; CHECK-NEWLOWERING: // %bb.0:
+; CHECK-NEWLOWERING-NEXT: add w0, w0, w0
+; CHECK-NEWLOWERING-NEXT: ret
+ %ret = add i32 %x, %x
+ ret i32 %ret
+}
diff --git a/llvm/test/CodeGen/AArch64/sme-shared-za-interface.ll b/llvm/test/CodeGen/AArch64/sme-shared-za-interface.ll
index 393ff3b79aedf..b4ff8d085ff40 100644
--- a/llvm/test/CodeGen/AArch64/sme-shared-za-interface.ll
+++ b/llvm/test/CodeGen/AArch64/sme-shared-za-interface.ll
@@ -1,5 +1,6 @@
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sme -verify-machineinstrs < %s | FileCheck %s
+; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sme -verify-machineinstrs < %s | FileCheck %s
+; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sme -verify-machineinstrs -aarch64-new-sme-abi < %s | FileCheck %s --check-prefix=CHECK-NEWLOWERING
declare void @private_za_callee()
@@ -32,6 +33,31 @@ define void @disable_tailcallopt() "aarch64_inout_za" nounwind {
; CHECK-NEXT: mov sp, x29
; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
; CHECK-NEXT: ret
+;
+; CHECK-NEWLOWERING-LABEL: disable_tailcallopt:
+; CHECK-NEWLOWERING: // %bb.0:
+; CHECK-NEWLOWERING-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: mov x29, sp
+; CHECK-NEWLOWERING-NEXT: sub sp, sp, #16
+; CHECK-NEWLOWERING-NEXT: rdsvl x8, #1
+; CHECK-NEWLOWERING-NEXT: mov x9, sp
+; CHECK-NEWLOWERING-NEXT: msub x9, x8, x8, x9
+; CHECK-NEWLOWERING-NEXT: mov sp, x9
+; CHECK-NEWLOWERING-NEXT: sub x10, x29, #16
+; CHECK-NEWLOWERING-NEXT: stp x9, x8, [x29, #-16]
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, x10
+; CHECK-NEWLOWERING-NEXT: bl private_za_callee
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: sub x0, x29, #16
+; CHECK-NEWLOWERING-NEXT: cbnz x8, .LBB0_2
+; CHECK-NEWLOWERING-NEXT: // %bb.1:
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEWLOWERING-NEXT: .LBB0_2:
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: mov sp, x29
+; CHECK-NEWLOWERING-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ret
tail call void @private_za_callee()
ret void
}
@@ -65,6 +91,31 @@ define fp128 @f128_call_za(fp128 %a, fp128 %b) "aarch64_inout_za" nounwind {
; CHECK-NEXT: mov sp, x29
; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
; CHECK-NEXT: ret
+;
+; CHECK-NEWLOWERING-LABEL: f128_call_za:
+; CHECK-NEWLOWERING: // %bb.0:
+; CHECK-NEWLOWERING-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: mov x29, sp
+; CHECK-NEWLOWERING-NEXT: sub sp, sp, #16
+; CHECK-NEWLOWERING-NEXT: rdsvl x8, #1
+; CHECK-NEWLOWERING-NEXT: mov x9, sp
+; CHECK-NEWLOWERING-NEXT: msub x9, x8, x8, x9
+; CHECK-NEWLOWERING-NEXT: mov sp, x9
+; CHECK-NEWLOWERING-NEXT: sub x10, x29, #16
+; CHECK-NEWLOWERING-NEXT: stp x9, x8, [x29, #-16]
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, x10
+; CHECK-NEWLOWERING-NEXT: bl __addtf3
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: sub x0, x29, #16
+; CHECK-NEWLOWERING-NEXT: cbnz x8, .LBB1_2
+; CHECK-NEWLOWERING-NEXT: // %bb.1:
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEWLOWERING-NEXT: .LBB1_2:
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: mov sp, x29
+; CHECK-NEWLOWERING-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ret
%res = fadd fp128 %a, %b
ret fp128 %res
}
diff --git a/llvm/test/CodeGen/AArch64/sme-za-control-flow.ll b/llvm/test/CodeGen/AArch64/sme-za-control-flow.ll
new file mode 100644
index 0000000000000..d3d7e953bedfa
--- /dev/null
+++ b/llvm/test/CodeGen/AArch64/sme-za-control-flow.ll
@@ -0,0 +1,1131 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5
+; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sme < %s | FileCheck %s --check-prefixes=CHECK-COMMON,CHECK
+; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sme -aarch64-new-sme-abi < %s | FileCheck %s --check-prefixes=CHECK-COMMON,CHECK-NEWLOWERING
+
+declare void @private_za_call()
+declare void @shared_za_call() "aarch64_inout_za"
+
+define void @private_za_loop(i32 %n) "aarch64_inout_za" nounwind {
+; CHECK-LABEL: private_za_loop:
+; CHECK: // %bb.0: // %entry
+; CHECK-NEXT: stp x29, x30, [sp, #-48]! // 16-byte Folded Spill
+; CHECK-NEXT: str x21, [sp, #16] // 8-byte Folded Spill
+; CHECK-NEXT: mov x29, sp
+; CHECK-NEXT: stp x20, x19, [sp, #32] // 16-byte Folded Spill
+; CHECK-NEXT: sub sp, sp, #16
+; CHECK-NEXT: rdsvl x8, #1
+; CHECK-NEXT: mov x9, sp
+; CHECK-NEXT: msub x8, x8, x8, x9
+; CHECK-NEXT: mov sp, x8
+; CHECK-NEXT: cmp w0, #1
+; CHECK-NEXT: stur x8, [x29, #-16]
+; CHECK-NEXT: sturh wzr, [x29, #-6]
+; CHECK-NEXT: stur wzr, [x29, #-4]
+; CHECK-NEXT: b.lt .LBB0_5
+; CHECK-NEXT: // %bb.1: // %loop.preheader
+; CHECK-NEXT: mov w19, w0
+; CHECK-NEXT: rdsvl x20, #1
+; CHECK-NEXT: sub x21, x29, #16
+; CHECK-NEXT: b .LBB0_3
+; CHECK-NEXT: .LBB0_2: // %loop
+; CHECK-NEXT: // in Loop: Header=BB0_3 Depth=1
+; CHECK-NEXT: subs w19, w19, #1
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: b.eq .LBB0_5
+; CHECK-NEXT: .LBB0_3: // %loop
+; CHECK-NEXT: // =>This Inner Loop Header: Depth=1
+; CHECK-NEXT: sturh w20, [x29, #-8]
+; CHECK-NEXT: msr TPIDR2_EL0, x21
+; CHECK-NEXT: bl private_za_call
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: sub x0, x29, #16
+; CHECK-NEXT: cbnz x8, .LBB0_2
+; CHECK-NEXT: // %bb.4: // %loop
+; CHECK-NEXT: // in Loop: Header=BB0_3 Depth=1
+; CHECK-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEXT: b .LBB0_2
+; CHECK-NEXT: .LBB0_5: // %exit
+; CHECK-NEXT: mov sp, x29
+; CHECK-NEXT: ldp x20, x19, [sp, #32] // 16-byte Folded Reload
+; CHECK-NEXT: ldr x21, [sp, #16] // 8-byte Folded Reload
+; CHECK-NEXT: ldp x29, x30, [sp], #48 // 16-byte Folded Reload
+; CHECK-NEXT: ret
+;
+; CHECK-NEWLOWERING-LABEL: private_za_loop:
+; CHECK-NEWLOWERING: // %bb.0: // %entry
+; CHECK-NEWLOWERING-NEXT: stp x29, x30, [sp, #-32]! // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: str x19, [sp, #16] // 8-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: mov x29, sp
+; CHECK-NEWLOWERING-NEXT: sub sp, sp, #16
+; CHECK-NEWLOWERING-NEXT: rdsvl x8, #1
+; CHECK-NEWLOWERING-NEXT: mov x9, sp
+; CHECK-NEWLOWERING-NEXT: msub x9, x8, x8, x9
+; CHECK-NEWLOWERING-NEXT: mov sp, x9
+; CHECK-NEWLOWERING-NEXT: sub x10, x29, #16
+; CHECK-NEWLOWERING-NEXT: cmp w0, #1
+; CHECK-NEWLOWERING-NEXT: stp x9, x8, [x29, #-16]
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, x10
+; CHECK-NEWLOWERING-NEXT: b.lt .LBB0_3
+; CHECK-NEWLOWERING-NEXT: // %bb.1: // %loop.preheader
+; CHECK-NEWLOWERING-NEXT: mov w19, w0
+; CHECK-NEWLOWERING-NEXT: .LBB0_2: // %loop
+; CHECK-NEWLOWERING-NEXT: // =>This Inner Loop Header: Depth=1
+; CHECK-NEWLOWERING-NEXT: bl private_za_call
+; CHECK-NEWLOWERING-NEXT: subs w19, w19, #1
+; CHECK-NEWLOWERING-NEXT: b.ne .LBB0_2
+; CHECK-NEWLOWERING-NEXT: .LBB0_3: // %exit
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: sub x0, x29, #16
+; CHECK-NEWLOWERING-NEXT: cbnz x8, .LBB0_5
+; CHECK-NEWLOWERING-NEXT: // %bb.4: // %exit
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEWLOWERING-NEXT: .LBB0_5: // %exit
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: mov sp, x29
+; CHECK-NEWLOWERING-NEXT: ldr x19, [sp, #16] // 8-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ldp x29, x30, [sp], #32 // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ret
+entry:
+ %cmpgt = icmp sgt i32 %n, 0
+ br i1 %cmpgt, label %loop, label %exit
+
+loop:
+ %iv = phi i32 [ %next_iv, %loop ], [ 0, %entry ]
+ tail call void @private_za_call()
+ %next_iv = add nuw nsw i32 %iv, 1
+ %cmpeq = icmp eq i32 %next_iv, %n
+ br i1 %cmpeq, label %exit, label %loop
+
+exit:
+ ret void
+}
+
+; FIXME: In the new lowering we could weight edges to avoid doing the lazy save in the loop.
+define void @private_za_loop_active_entry_and_exit(i32 %n) "aarch64_inout_za" nounwind {
+; CHECK-LABEL: private_za_loop_active_entry_and_exit:
+; CHECK: // %bb.0: // %entry
+; CHECK-NEXT: stp x29, x30, [sp, #-48]! // 16-byte Folded Spill
+; CHECK-NEXT: str x21, [sp, #16] // 8-byte Folded Spill
+; CHECK-NEXT: mov x29, sp
+; CHECK-NEXT: stp x20, x19, [sp, #32] // 16-byte Folded Spill
+; CHECK-NEXT: sub sp, sp, #16
+; CHECK-NEXT: rdsvl x8, #1
+; CHECK-NEXT: mov x9, sp
+; CHECK-NEXT: mov w19, w0
+; CHECK-NEXT: msub x8, x8, x8, x9
+; CHECK-NEXT: mov sp, x8
+; CHECK-NEXT: stur x8, [x29, #-16]
+; CHECK-NEXT: sturh wzr, [x29, #-6]
+; CHECK-NEXT: stur wzr, [x29, #-4]
+; CHECK-NEXT: bl shared_za_call
+; CHECK-NEXT: cmp w19, #1
+; CHECK-NEXT: b.lt .LBB1_5
+; CHECK-NEXT: // %bb.1: // %loop.preheader
+; CHECK-NEXT: rdsvl x20, #1
+; CHECK-NEXT: sub x21, x29, #16
+; CHECK-NEXT: b .LBB1_3
+; CHECK-NEXT: .LBB1_2: // %loop
+; CHECK-NEXT: // in Loop: Header=BB1_3 Depth=1
+; CHECK-NEXT: subs w19, w19, #1
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: b.eq .LBB1_5
+; CHECK-NEXT: .LBB1_3: // %loop
+; CHECK-NEXT: // =>This Inner Loop Header: Depth=1
+; CHECK-NEXT: sturh w20, [x29, #-8]
+; CHECK-NEXT: msr TPIDR2_EL0, x21
+; CHECK-NEXT: bl private_za_call
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: sub x0, x29, #16
+; CHECK-NEXT: cbnz x8, .LBB1_2
+; CHECK-NEXT: // %bb.4: // %loop
+; CHECK-NEXT: // in Loop: Header=BB1_3 Depth=1
+; CHECK-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEXT: b .LBB1_2
+; CHECK-NEXT: .LBB1_5: // %exit
+; CHECK-NEXT: mov sp, x29
+; CHECK-NEXT: ldp x20, x19, [sp, #32] // 16-byte Folded Reload
+; CHECK-NEXT: ldr x21, [sp, #16] // 8-byte Folded Reload
+; CHECK-NEXT: ldp x29, x30, [sp], #48 // 16-byte Folded Reload
+; CHECK-NEXT: b shared_za_call
+;
+; CHECK-NEWLOWERING-LABEL: private_za_loop_active_entry_and_exit:
+; CHECK-NEWLOWERING: // %bb.0: // %entry
+; CHECK-NEWLOWERING-NEXT: stp x29, x30, [sp, #-32]! // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: stp x20, x19, [sp, #16] // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: mov x29, sp
+; CHECK-NEWLOWERING-NEXT: sub sp, sp, #16
+; CHECK-NEWLOWERING-NEXT: rdsvl x8, #1
+; CHECK-NEWLOWERING-NEXT: mov x9, sp
+; CHECK-NEWLOWERING-NEXT: msub x9, x8, x8, x9
+; CHECK-NEWLOWERING-NEXT: mov sp, x9
+; CHECK-NEWLOWERING-NEXT: mov w19, w0
+; CHECK-NEWLOWERING-NEXT: stp x9, x8, [x29, #-16]
+; CHECK-NEWLOWERING-NEXT: bl shared_za_call
+; CHECK-NEWLOWERING-NEXT: cmp w19, #1
+; CHECK-NEWLOWERING-NEXT: b.lt .LBB1_5
+; CHECK-NEWLOWERING-NEXT: // %bb.1: // %loop.preheader
+; CHECK-NEWLOWERING-NEXT: sub x20, x29, #16
+; CHECK-NEWLOWERING-NEXT: b .LBB1_3
+; CHECK-NEWLOWERING-NEXT: .LBB1_2: // %loop
+; CHECK-NEWLOWERING-NEXT: // in Loop: Header=BB1_3 Depth=1
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: cbz w19, .LBB1_5
+; CHECK-NEWLOWERING-NEXT: .LBB1_3: // %loop
+; CHECK-NEWLOWERING-NEXT: // =>This Inner Loop Header: Depth=1
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, x20
+; CHECK-NEWLOWERING-NEXT: bl private_za_call
+; CHECK-NEWLOWERING-NEXT: sub w19, w19, #1
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: sub x0, x29, #16
+; CHECK-NEWLOWERING-NEXT: cbnz x8, .LBB1_2
+; CHECK-NEWLOWERING-NEXT: // %bb.4: // %loop
+; CHECK-NEWLOWERING-NEXT: // in Loop: Header=BB1_3 Depth=1
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEWLOWERING-NEXT: b .LBB1_2
+; CHECK-NEWLOWERING-NEXT: .LBB1_5: // %exit
+; CHECK-NEWLOWERING-NEXT: mov sp, x29
+; CHECK-NEWLOWERING-NEXT: ldp x20, x19, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ldp x29, x30, [sp], #32 // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: b shared_za_call
+entry:
+ %cmpgt = icmp sgt i32 %n, 0
+ tail call void @shared_za_call()
+ br i1 %cmpgt, label %loop, label %exit
+
+loop:
+ %iv = phi i32 [ %next_iv, %loop ], [ 0, %entry ]
+ tail call void @private_za_call()
+ %next_iv = add nuw nsw i32 %iv, 1
+ %cmpeq = icmp eq i32 %next_iv, %n
+ br i1 %cmpeq, label %exit, label %loop
+
+exit:
+ tail call void @shared_za_call()
+ ret void
+}
+
+define void @shared_za_loop(i32 %n) "aarch64_inout_za" nounwind {
+; CHECK-COMMON-LABEL: shared_za_loop:
+; CHECK-COMMON: // %bb.0: // %entry
+; CHECK-COMMON-NEXT: cmp w0, #1
+; CHECK-COMMON-NEXT: b.lt .LBB2_4
+; CHECK-COMMON-NEXT: // %bb.1: // %loop.preheader
+; CHECK-COMMON-NEXT: stp x30, x19, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-COMMON-NEXT: mov w19, w0
+; CHECK-COMMON-NEXT: .LBB2_2: // %loop
+; CHECK-COMMON-NEXT: // =>This Inner Loop Header: Depth=1
+; CHECK-COMMON-NEXT: bl shared_za_call
+; CHECK-COMMON-NEXT: subs w19, w19, #1
+; CHECK-COMMON-NEXT: b.ne .LBB2_2
+; CHECK-COMMON-NEXT: // %bb.3:
+; CHECK-COMMON-NEXT: ldp x30, x19, [sp], #16 // 16-byte Folded Reload
+; CHECK-COMMON-NEXT: .LBB2_4: // %exit
+; CHECK-COMMON-NEXT: ret
+entry:
+ %cmpgt = icmp sgt i32 %n, 0
+ br i1 %cmpgt, label %loop, label %exit
+
+loop:
+ %iv = phi i32 [ %next_iv, %loop ], [ 0, %entry ]
+ tail call void @shared_za_call()
+ %next_iv = add nuw nsw i32 %iv, 1
+ %cmpeq = icmp eq i32 %next_iv, %n
+ br i1 %cmpeq, label %exit, label %loop
+
+exit:
+ ret void
+}
+
+; FIXME: The codegen for this case could be improved (by tuning weights).
+; Here the ZA save has been hoisted out of the conditional, but would be better
+; to sink it.
+define void @cond_private_za_call(i1 %cond) "aarch64_inout_za" nounwind {
+; CHECK-LABEL: cond_private_za_call:
+; CHECK: // %bb.0:
+; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT: mov x29, sp
+; CHECK-NEXT: sub sp, sp, #16
+; CHECK-NEXT: rdsvl x8, #1
+; CHECK-NEXT: mov x9, sp
+; CHECK-NEXT: msub x8, x8, x8, x9
+; CHECK-NEXT: mov sp, x8
+; CHECK-NEXT: stur x8, [x29, #-16]
+; CHECK-NEXT: sturh wzr, [x29, #-6]
+; CHECK-NEXT: stur wzr, [x29, #-4]
+; CHECK-NEXT: tbz w0, #0, .LBB3_4
+; CHECK-NEXT: // %bb.1: // %private_za_call
+; CHECK-NEXT: rdsvl x8, #1
+; CHECK-NEXT: sub x9, x29, #16
+; CHECK-NEXT: sturh w8, [x29, #-8]
+; CHECK-NEXT: msr TPIDR2_EL0, x9
+; CHECK-NEXT: bl private_za_call
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: sub x0, x29, #16
+; CHECK-NEXT: cbnz x8, .LBB3_3
+; CHECK-NEXT: // %bb.2: // %private_za_call
+; CHECK-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEXT: .LBB3_3: // %private_za_call
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: .LBB3_4: // %exit
+; CHECK-NEXT: mov sp, x29
+; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT: b shared_za_call
+;
+; CHECK-NEWLOWERING-LABEL: cond_private_za_call:
+; CHECK-NEWLOWERING: // %bb.0:
+; CHECK-NEWLOWERING-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: mov x29, sp
+; CHECK-NEWLOWERING-NEXT: sub sp, sp, #16
+; CHECK-NEWLOWERING-NEXT: rdsvl x8, #1
+; CHECK-NEWLOWERING-NEXT: mov x9, sp
+; CHECK-NEWLOWERING-NEXT: msub x9, x8, x8, x9
+; CHECK-NEWLOWERING-NEXT: mov sp, x9
+; CHECK-NEWLOWERING-NEXT: sub x10, x29, #16
+; CHECK-NEWLOWERING-NEXT: stp x9, x8, [x29, #-16]
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, x10
+; CHECK-NEWLOWERING-NEXT: tbz w0, #0, .LBB3_2
+; CHECK-NEWLOWERING-NEXT: // %bb.1: // %private_za_call
+; CHECK-NEWLOWERING-NEXT: bl private_za_call
+; CHECK-NEWLOWERING-NEXT: .LBB3_2: // %exit
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: sub x0, x29, #16
+; CHECK-NEWLOWERING-NEXT: cbnz x8, .LBB3_4
+; CHECK-NEWLOWERING-NEXT: // %bb.3: // %exit
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEWLOWERING-NEXT: .LBB3_4: // %exit
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: mov sp, x29
+; CHECK-NEWLOWERING-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: b shared_za_call
+ br i1 %cond, label %private_za_call, label %exit
+
+private_za_call:
+ tail call void @private_za_call()
+ br label %exit
+
+exit:
+ tail call void @shared_za_call()
+ ret void
+}
+
+define void @mixed_shared_private_za_loop(ptr %cond) "aarch64_inout_za" nounwind {
+; CHECK-LABEL: mixed_shared_private_za_loop:
+; CHECK: // %bb.0:
+; CHECK-NEXT: stp x29, x30, [sp, #-48]! // 16-byte Folded Spill
+; CHECK-NEXT: str x21, [sp, #16] // 8-byte Folded Spill
+; CHECK-NEXT: mov x29, sp
+; CHECK-NEXT: stp x20, x19, [sp, #32] // 16-byte Folded Spill
+; CHECK-NEXT: sub sp, sp, #16
+; CHECK-NEXT: rdsvl x20, #1
+; CHECK-NEXT: mov x8, sp
+; CHECK-NEXT: mov x19, x0
+; CHECK-NEXT: msub x8, x20, x20, x8
+; CHECK-NEXT: mov sp, x8
+; CHECK-NEXT: sub x21, x29, #16
+; CHECK-NEXT: stur x8, [x29, #-16]
+; CHECK-NEXT: sturh wzr, [x29, #-6]
+; CHECK-NEXT: stur wzr, [x29, #-4]
+; CHECK-NEXT: b .LBB4_2
+; CHECK-NEXT: .LBB4_1: // %loop
+; CHECK-NEXT: // in Loop: Header=BB4_2 Depth=1
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: ldrb w8, [x19]
+; CHECK-NEXT: tbz w8, #0, .LBB4_4
+; CHECK-NEXT: .LBB4_2: // %loop
+; CHECK-NEXT: // =>This Inner Loop Header: Depth=1
+; CHECK-NEXT: bl shared_za_call
+; CHECK-NEXT: sturh w20, [x29, #-8]
+; CHECK-NEXT: msr TPIDR2_EL0, x21
+; CHECK-NEXT: bl private_za_call
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: sub x0, x29, #16
+; CHECK-NEXT: cbnz x8, .LBB4_1
+; CHECK-NEXT: // %bb.3: // %loop
+; CHECK-NEXT: // in Loop: Header=BB4_2 Depth=1
+; CHECK-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEXT: b .LBB4_1
+; CHECK-NEXT: .LBB4_4: // %exit
+; CHECK-NEXT: bl shared_za_call
+; CHECK-NEXT: mov sp, x29
+; CHECK-NEXT: ldp x20, x19, [sp, #32] // 16-byte Folded Reload
+; CHECK-NEXT: ldr x21, [sp, #16] // 8-byte Folded Reload
+; CHECK-NEXT: ldp x29, x30, [sp], #48 // 16-byte Folded Reload
+; CHECK-NEXT: ret
+;
+; CHECK-NEWLOWERING-LABEL: mixed_shared_private_za_loop:
+; CHECK-NEWLOWERING: // %bb.0:
+; CHECK-NEWLOWERING-NEXT: stp x29, x30, [sp, #-32]! // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: stp x20, x19, [sp, #16] // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: mov x29, sp
+; CHECK-NEWLOWERING-NEXT: sub sp, sp, #16
+; CHECK-NEWLOWERING-NEXT: rdsvl x8, #1
+; CHECK-NEWLOWERING-NEXT: mov x9, sp
+; CHECK-NEWLOWERING-NEXT: msub x9, x8, x8, x9
+; CHECK-NEWLOWERING-NEXT: mov sp, x9
+; CHECK-NEWLOWERING-NEXT: mov x19, x0
+; CHECK-NEWLOWERING-NEXT: sub x20, x29, #16
+; CHECK-NEWLOWERING-NEXT: stp x9, x8, [x29, #-16]
+; CHECK-NEWLOWERING-NEXT: b .LBB4_2
+; CHECK-NEWLOWERING-NEXT: .LBB4_1: // %loop
+; CHECK-NEWLOWERING-NEXT: // in Loop: Header=BB4_2 Depth=1
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: tbz w8, #0, .LBB4_4
+; CHECK-NEWLOWERING-NEXT: .LBB4_2: // %loop
+; CHECK-NEWLOWERING-NEXT: // =>This Inner Loop Header: Depth=1
+; CHECK-NEWLOWERING-NEXT: bl shared_za_call
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, x20
+; CHECK-NEWLOWERING-NEXT: bl private_za_call
+; CHECK-NEWLOWERING-NEXT: ldrb w8, [x19]
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: mrs x9, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: sub x0, x29, #16
+; CHECK-NEWLOWERING-NEXT: cbnz x9, .LBB4_1
+; CHECK-NEWLOWERING-NEXT: // %bb.3: // %loop
+; CHECK-NEWLOWERING-NEXT: // in Loop: Header=BB4_2 Depth=1
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEWLOWERING-NEXT: b .LBB4_1
+; CHECK-NEWLOWERING-NEXT: .LBB4_4: // %exit
+; CHECK-NEWLOWERING-NEXT: bl shared_za_call
+; CHECK-NEWLOWERING-NEXT: mov sp, x29
+; CHECK-NEWLOWERING-NEXT: ldp x20, x19, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ldp x29, x30, [sp], #32 // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ret
+ br label %loop
+
+loop:
+ call void @shared_za_call()
+ call void @private_za_call()
+ br label %latch
+
+latch:
+ %bool = load volatile i8, ptr %cond, align 1
+ %trunc = trunc i8 %bool to i1
+ br i1 %trunc, label %loop, label %exit
+
+exit:
+ call void @shared_za_call()
+ ret void
+}
+
+
+define void @cond_clobber_followed_by_clobber(i1 %cond) "aarch64_inout_za" nounwind {
+; CHECK-LABEL: cond_clobber_followed_by_clobber:
+; CHECK: // %bb.0:
+; CHECK-NEXT: stp x29, x30, [sp, #-32]! // 16-byte Folded Spill
+; CHECK-NEXT: str x19, [sp, #16] // 8-byte Folded Spill
+; CHECK-NEXT: mov x29, sp
+; CHECK-NEXT: sub sp, sp, #16
+; CHECK-NEXT: rdsvl x8, #1
+; CHECK-NEXT: mov x9, sp
+; CHECK-NEXT: mov w19, w0
+; CHECK-NEXT: msub x8, x8, x8, x9
+; CHECK-NEXT: mov sp, x8
+; CHECK-NEXT: stur x8, [x29, #-16]
+; CHECK-NEXT: sturh wzr, [x29, #-6]
+; CHECK-NEXT: stur wzr, [x29, #-4]
+; CHECK-NEXT: bl shared_za_call
+; CHECK-NEXT: tbz w19, #0, .LBB5_4
+; CHECK-NEXT: // %bb.1: // %cond_clobber
+; CHECK-NEXT: rdsvl x8, #1
+; CHECK-NEXT: sub x9, x29, #16
+; CHECK-NEXT: sturh w8, [x29, #-8]
+; CHECK-NEXT: msr TPIDR2_EL0, x9
+; CHECK-NEXT: bl private_za_call
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: sub x0, x29, #16
+; CHECK-NEXT: cbnz x8, .LBB5_3
+; CHECK-NEXT: // %bb.2: // %cond_clobber
+; CHECK-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEXT: .LBB5_3: // %cond_clobber
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: .LBB5_4: // %exit
+; CHECK-NEXT: rdsvl x8, #1
+; CHECK-NEXT: sub x9, x29, #16
+; CHECK-NEXT: sturh w8, [x29, #-8]
+; CHECK-NEXT: msr TPIDR2_EL0, x9
+; CHECK-NEXT: bl private_za_call
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: sub x0, x29, #16
+; CHECK-NEXT: cbnz x8, .LBB5_6
+; CHECK-NEXT: // %bb.5: // %exit
+; CHECK-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEXT: .LBB5_6: // %exit
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: mov sp, x29
+; CHECK-NEXT: ldr x19, [sp, #16] // 8-byte Folded Reload
+; CHECK-NEXT: ldp x29, x30, [sp], #32 // 16-byte Folded Reload
+; CHECK-NEXT: b shared_za_call
+;
+; CHECK-NEWLOWERING-LABEL: cond_clobber_followed_by_clobber:
+; CHECK-NEWLOWERING: // %bb.0:
+; CHECK-NEWLOWERING-NEXT: stp x29, x30, [sp, #-32]! // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: str x19, [sp, #16] // 8-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: mov x29, sp
+; CHECK-NEWLOWERING-NEXT: sub sp, sp, #16
+; CHECK-NEWLOWERING-NEXT: rdsvl x8, #1
+; CHECK-NEWLOWERING-NEXT: mov x9, sp
+; CHECK-NEWLOWERING-NEXT: msub x9, x8, x8, x9
+; CHECK-NEWLOWERING-NEXT: mov sp, x9
+; CHECK-NEWLOWERING-NEXT: mov w19, w0
+; CHECK-NEWLOWERING-NEXT: stp x9, x8, [x29, #-16]
+; CHECK-NEWLOWERING-NEXT: bl shared_za_call
+; CHECK-NEWLOWERING-NEXT: sub x8, x29, #16
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, x8
+; CHECK-NEWLOWERING-NEXT: tbz w19, #0, .LBB5_2
+; CHECK-NEWLOWERING-NEXT: // %bb.1: // %cond_clobber
+; CHECK-NEWLOWERING-NEXT: bl private_za_call
+; CHECK-NEWLOWERING-NEXT: .LBB5_2: // %exit
+; CHECK-NEWLOWERING-NEXT: bl private_za_call
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: sub x0, x29, #16
+; CHECK-NEWLOWERING-NEXT: cbnz x8, .LBB5_4
+; CHECK-NEWLOWERING-NEXT: // %bb.3: // %exit
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEWLOWERING-NEXT: .LBB5_4: // %exit
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: mov sp, x29
+; CHECK-NEWLOWERING-NEXT: ldr x19, [sp, #16] // 8-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ldp x29, x30, [sp], #32 // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: b shared_za_call
+ tail call void @shared_za_call()
+ br i1 %cond, label %cond_clobber, label %exit
+
+cond_clobber:
+ tail call void @private_za_call()
+ br label %exit
+
+exit:
+ tail call void @private_za_call()
+ tail call void @shared_za_call()
+ ret void
+}
+
+define void @conditionally_use_za(i1 %cond) "aarch64_inout_za" nounwind {
+; CHECK-LABEL: conditionally_use_za:
+; CHECK: // %bb.0:
+; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT: mov x29, sp
+; CHECK-NEXT: sub sp, sp, #16
+; CHECK-NEXT: rdsvl x8, #1
+; CHECK-NEXT: mov x9, sp
+; CHECK-NEXT: msub x8, x8, x8, x9
+; CHECK-NEXT: mov sp, x8
+; CHECK-NEXT: stur x8, [x29, #-16]
+; CHECK-NEXT: sturh wzr, [x29, #-6]
+; CHECK-NEXT: stur wzr, [x29, #-4]
+; CHECK-NEXT: tbz w0, #0, .LBB6_4
+; CHECK-NEXT: // %bb.1: // %use_za
+; CHECK-NEXT: bl shared_za_call
+; CHECK-NEXT: rdsvl x8, #1
+; CHECK-NEXT: sub x9, x29, #16
+; CHECK-NEXT: sturh w8, [x29, #-8]
+; CHECK-NEXT: msr TPIDR2_EL0, x9
+; CHECK-NEXT: bl private_za_call
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: sub x0, x29, #16
+; CHECK-NEXT: cbnz x8, .LBB6_3
+; CHECK-NEXT: // %bb.2: // %use_za
+; CHECK-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEXT: .LBB6_3: // %use_za
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: .LBB6_4: // %exit
+; CHECK-NEXT: mov sp, x29
+; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT: ret
+;
+; CHECK-NEWLOWERING-LABEL: conditionally_use_za:
+; CHECK-NEWLOWERING: // %bb.0:
+; CHECK-NEWLOWERING-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: mov x29, sp
+; CHECK-NEWLOWERING-NEXT: sub sp, sp, #16
+; CHECK-NEWLOWERING-NEXT: rdsvl x8, #1
+; CHECK-NEWLOWERING-NEXT: mov x9, sp
+; CHECK-NEWLOWERING-NEXT: msub x9, x8, x8, x9
+; CHECK-NEWLOWERING-NEXT: mov sp, x9
+; CHECK-NEWLOWERING-NEXT: stp x9, x8, [x29, #-16]
+; CHECK-NEWLOWERING-NEXT: tbz w0, #0, .LBB6_4
+; CHECK-NEWLOWERING-NEXT: // %bb.1: // %use_za
+; CHECK-NEWLOWERING-NEXT: bl shared_za_call
+; CHECK-NEWLOWERING-NEXT: sub x8, x29, #16
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, x8
+; CHECK-NEWLOWERING-NEXT: bl private_za_call
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: sub x0, x29, #16
+; CHECK-NEWLOWERING-NEXT: cbnz x8, .LBB6_3
+; CHECK-NEWLOWERING-NEXT: // %bb.2: // %use_za
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEWLOWERING-NEXT: .LBB6_3: // %use_za
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: .LBB6_4: // %exit
+; CHECK-NEWLOWERING-NEXT: mov sp, x29
+; CHECK-NEWLOWERING-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ret
+ br i1 %cond, label %use_za, label %exit
+
+use_za:
+ tail call void @shared_za_call()
+ tail call void @private_za_call()
+ br label %exit
+
+exit:
+ ret void
+}
+
+
+define void @diamond_mixed_za_merge_shared(i1 %cond) "aarch64_inout_za" nounwind {
+; CHECK-LABEL: diamond_mixed_za_merge_shared:
+; CHECK: // %bb.0: // %entry
+; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT: mov x29, sp
+; CHECK-NEXT: sub sp, sp, #16
+; CHECK-NEXT: rdsvl x8, #1
+; CHECK-NEXT: mov x9, sp
+; CHECK-NEXT: msub x8, x8, x8, x9
+; CHECK-NEXT: mov sp, x8
+; CHECK-NEXT: stur x8, [x29, #-16]
+; CHECK-NEXT: sturh wzr, [x29, #-6]
+; CHECK-NEXT: stur wzr, [x29, #-4]
+; CHECK-NEXT: tbz w0, #0, .LBB7_2
+; CHECK-NEXT: // %bb.1: // %then
+; CHECK-NEXT: bl shared_za_call
+; CHECK-NEXT: b .LBB7_5
+; CHECK-NEXT: .LBB7_2: // %else
+; CHECK-NEXT: rdsvl x8, #1
+; CHECK-NEXT: sub x9, x29, #16
+; CHECK-NEXT: sturh w8, [x29, #-8]
+; CHECK-NEXT: msr TPIDR2_EL0, x9
+; CHECK-NEXT: bl private_za_call
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: sub x0, x29, #16
+; CHECK-NEXT: cbnz x8, .LBB7_4
+; CHECK-NEXT: // %bb.3: // %else
+; CHECK-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEXT: .LBB7_4: // %else
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: .LBB7_5: // %merge_shared
+; CHECK-NEXT: bl shared_za_call
+; CHECK-NEXT: mov sp, x29
+; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT: ret
+;
+; CHECK-NEWLOWERING-LABEL: diamond_mixed_za_merge_shared:
+; CHECK-NEWLOWERING: // %bb.0: // %entry
+; CHECK-NEWLOWERING-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: mov x29, sp
+; CHECK-NEWLOWERING-NEXT: sub sp, sp, #16
+; CHECK-NEWLOWERING-NEXT: rdsvl x8, #1
+; CHECK-NEWLOWERING-NEXT: mov x9, sp
+; CHECK-NEWLOWERING-NEXT: msub x9, x8, x8, x9
+; CHECK-NEWLOWERING-NEXT: mov sp, x9
+; CHECK-NEWLOWERING-NEXT: stp x9, x8, [x29, #-16]
+; CHECK-NEWLOWERING-NEXT: tbz w0, #0, .LBB7_2
+; CHECK-NEWLOWERING-NEXT: // %bb.1: // %then
+; CHECK-NEWLOWERING-NEXT: bl shared_za_call
+; CHECK-NEWLOWERING-NEXT: b .LBB7_5
+; CHECK-NEWLOWERING-NEXT: .LBB7_2: // %else
+; CHECK-NEWLOWERING-NEXT: sub x8, x29, #16
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, x8
+; CHECK-NEWLOWERING-NEXT: bl private_za_call
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: sub x0, x29, #16
+; CHECK-NEWLOWERING-NEXT: cbnz x8, .LBB7_4
+; CHECK-NEWLOWERING-NEXT: // %bb.3: // %else
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEWLOWERING-NEXT: .LBB7_4: // %else
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: .LBB7_5: // %merge_shared
+; CHECK-NEWLOWERING-NEXT: bl shared_za_call
+; CHECK-NEWLOWERING-NEXT: mov sp, x29
+; CHECK-NEWLOWERING-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ret
+entry:
+ br i1 %cond, label %then, label %else
+
+then:
+ call void @shared_za_call()
+ br label %merge_shared
+
+else:
+ call void @private_za_call()
+ br label %merge_shared
+
+merge_shared:
+ call void @shared_za_call()
+ ret void
+}
+
+
+define void @diamond_mixed_za_merge_private(i1 %cond) "aarch64_inout_za" nounwind {
+; CHECK-LABEL: diamond_mixed_za_merge_private:
+; CHECK: // %bb.0: // %entry
+; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT: mov x29, sp
+; CHECK-NEXT: sub sp, sp, #16
+; CHECK-NEXT: rdsvl x8, #1
+; CHECK-NEXT: mov x9, sp
+; CHECK-NEXT: msub x8, x8, x8, x9
+; CHECK-NEXT: mov sp, x8
+; CHECK-NEXT: stur x8, [x29, #-16]
+; CHECK-NEXT: sturh wzr, [x29, #-6]
+; CHECK-NEXT: stur wzr, [x29, #-4]
+; CHECK-NEXT: tbz w0, #0, .LBB8_2
+; CHECK-NEXT: // %bb.1: // %then
+; CHECK-NEXT: bl shared_za_call
+; CHECK-NEXT: b .LBB8_5
+; CHECK-NEXT: .LBB8_2: // %else
+; CHECK-NEXT: rdsvl x8, #1
+; CHECK-NEXT: sub x9, x29, #16
+; CHECK-NEXT: sturh w8, [x29, #-8]
+; CHECK-NEXT: msr TPIDR2_EL0, x9
+; CHECK-NEXT: bl private_za_call
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: sub x0, x29, #16
+; CHECK-NEXT: cbnz x8, .LBB8_4
+; CHECK-NEXT: // %bb.3: // %else
+; CHECK-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEXT: .LBB8_4: // %else
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: .LBB8_5: // %merge_private_za
+; CHECK-NEXT: rdsvl x8, #1
+; CHECK-NEXT: sub x9, x29, #16
+; CHECK-NEXT: sturh w8, [x29, #-8]
+; CHECK-NEXT: msr TPIDR2_EL0, x9
+; CHECK-NEXT: bl private_za_call
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: sub x0, x29, #16
+; CHECK-NEXT: cbnz x8, .LBB8_7
+; CHECK-NEXT: // %bb.6: // %merge_private_za
+; CHECK-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEXT: .LBB8_7: // %merge_private_za
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: mov sp, x29
+; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT: ret
+;
+; CHECK-NEWLOWERING-LABEL: diamond_mixed_za_merge_private:
+; CHECK-NEWLOWERING: // %bb.0: // %entry
+; CHECK-NEWLOWERING-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: mov x29, sp
+; CHECK-NEWLOWERING-NEXT: sub sp, sp, #16
+; CHECK-NEWLOWERING-NEXT: rdsvl x8, #1
+; CHECK-NEWLOWERING-NEXT: mov x9, sp
+; CHECK-NEWLOWERING-NEXT: msub x9, x8, x8, x9
+; CHECK-NEWLOWERING-NEXT: mov sp, x9
+; CHECK-NEWLOWERING-NEXT: stp x9, x8, [x29, #-16]
+; CHECK-NEWLOWERING-NEXT: tbz w0, #0, .LBB8_2
+; CHECK-NEWLOWERING-NEXT: // %bb.1: // %then
+; CHECK-NEWLOWERING-NEXT: bl shared_za_call
+; CHECK-NEWLOWERING-NEXT: sub x8, x29, #16
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, x8
+; CHECK-NEWLOWERING-NEXT: b .LBB8_3
+; CHECK-NEWLOWERING-NEXT: .LBB8_2: // %else
+; CHECK-NEWLOWERING-NEXT: sub x8, x29, #16
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, x8
+; CHECK-NEWLOWERING-NEXT: bl private_za_call
+; CHECK-NEWLOWERING-NEXT: .LBB8_3: // %merge_private_za
+; CHECK-NEWLOWERING-NEXT: bl private_za_call
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: sub x0, x29, #16
+; CHECK-NEWLOWERING-NEXT: cbnz x8, .LBB8_5
+; CHECK-NEWLOWERING-NEXT: // %bb.4: // %merge_private_za
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEWLOWERING-NEXT: .LBB8_5: // %merge_private_za
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: mov sp, x29
+; CHECK-NEWLOWERING-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ret
+entry:
+ br i1 %cond, label %then, label %else
+
+then:
+ call void @shared_za_call()
+ br label %merge_private_za
+
+else:
+ call void @private_za_call()
+ br label %merge_private_za
+
+merge_private_za:
+ call void @private_za_call()
+ ret void
+}
+
+define void @critical_edge_mixed_za(i1 %c1, i1 %c2) "aarch64_inout_za" nounwind {
+; CHECK-LABEL: critical_edge_mixed_za:
+; CHECK: // %bb.0: // %entry
+; CHECK-NEXT: stp x29, x30, [sp, #-32]! // 16-byte Folded Spill
+; CHECK-NEXT: str x19, [sp, #16] // 8-byte Folded Spill
+; CHECK-NEXT: mov x29, sp
+; CHECK-NEXT: sub sp, sp, #16
+; CHECK-NEXT: rdsvl x8, #1
+; CHECK-NEXT: mov x9, sp
+; CHECK-NEXT: mov w19, w1
+; CHECK-NEXT: msub x8, x8, x8, x9
+; CHECK-NEXT: mov sp, x8
+; CHECK-NEXT: stur x8, [x29, #-16]
+; CHECK-NEXT: sturh wzr, [x29, #-6]
+; CHECK-NEXT: stur wzr, [x29, #-4]
+; CHECK-NEXT: tbz w0, #0, .LBB9_5
+; CHECK-NEXT: // %bb.1: // %shared_path
+; CHECK-NEXT: bl shared_za_call
+; CHECK-NEXT: tbz w19, #0, .LBB9_8
+; CHECK-NEXT: .LBB9_2: // %exit_private
+; CHECK-NEXT: rdsvl x8, #1
+; CHECK-NEXT: sub x9, x29, #16
+; CHECK-NEXT: sturh w8, [x29, #-8]
+; CHECK-NEXT: msr TPIDR2_EL0, x9
+; CHECK-NEXT: bl private_za_call
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: sub x0, x29, #16
+; CHECK-NEXT: cbnz x8, .LBB9_4
+; CHECK-NEXT: // %bb.3: // %exit_private
+; CHECK-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEXT: .LBB9_4: // %exit_private
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: b .LBB9_9
+; CHECK-NEXT: .LBB9_5: // %private_path
+; CHECK-NEXT: rdsvl x8, #1
+; CHECK-NEXT: sub x9, x29, #16
+; CHECK-NEXT: sturh w8, [x29, #-8]
+; CHECK-NEXT: msr TPIDR2_EL0, x9
+; CHECK-NEXT: bl private_za_call
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: sub x0, x29, #16
+; CHECK-NEXT: cbnz x8, .LBB9_7
+; CHECK-NEXT: // %bb.6: // %private_path
+; CHECK-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEXT: .LBB9_7: // %private_path
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: tbnz w19, #0, .LBB9_2
+; CHECK-NEXT: .LBB9_8: // %exit_shared
+; CHECK-NEXT: bl shared_za_call
+; CHECK-NEXT: .LBB9_9: // %common.ret
+; CHECK-NEXT: mov sp, x29
+; CHECK-NEXT: ldr x19, [sp, #16] // 8-byte Folded Reload
+; CHECK-NEXT: ldp x29, x30, [sp], #32 // 16-byte Folded Reload
+; CHECK-NEXT: ret
+;
+; CHECK-NEWLOWERING-LABEL: critical_edge_mixed_za:
+; CHECK-NEWLOWERING: // %bb.0: // %entry
+; CHECK-NEWLOWERING-NEXT: stp x29, x30, [sp, #-32]! // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: str x19, [sp, #16] // 8-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: mov x29, sp
+; CHECK-NEWLOWERING-NEXT: sub sp, sp, #16
+; CHECK-NEWLOWERING-NEXT: rdsvl x8, #1
+; CHECK-NEWLOWERING-NEXT: mov x9, sp
+; CHECK-NEWLOWERING-NEXT: msub x9, x8, x8, x9
+; CHECK-NEWLOWERING-NEXT: mov sp, x9
+; CHECK-NEWLOWERING-NEXT: mov w19, w1
+; CHECK-NEWLOWERING-NEXT: stp x9, x8, [x29, #-16]
+; CHECK-NEWLOWERING-NEXT: tbz w0, #0, .LBB9_5
+; CHECK-NEWLOWERING-NEXT: // %bb.1: // %shared_path
+; CHECK-NEWLOWERING-NEXT: bl shared_za_call
+; CHECK-NEWLOWERING-NEXT: tbz w19, #0, .LBB9_8
+; CHECK-NEWLOWERING-NEXT: .LBB9_2: // %exit_private
+; CHECK-NEWLOWERING-NEXT: sub x8, x29, #16
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, x8
+; CHECK-NEWLOWERING-NEXT: bl private_za_call
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: sub x0, x29, #16
+; CHECK-NEWLOWERING-NEXT: cbnz x8, .LBB9_4
+; CHECK-NEWLOWERING-NEXT: // %bb.3: // %exit_private
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEWLOWERING-NEXT: .LBB9_4: // %exit_private
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: b .LBB9_9
+; CHECK-NEWLOWERING-NEXT: .LBB9_5: // %private_path
+; CHECK-NEWLOWERING-NEXT: sub x8, x29, #16
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, x8
+; CHECK-NEWLOWERING-NEXT: bl private_za_call
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: sub x0, x29, #16
+; CHECK-NEWLOWERING-NEXT: cbnz x8, .LBB9_7
+; CHECK-NEWLOWERING-NEXT: // %bb.6: // %private_path
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEWLOWERING-NEXT: .LBB9_7: // %private_path
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: tbnz w19, #0, .LBB9_2
+; CHECK-NEWLOWERING-NEXT: .LBB9_8: // %exit_shared
+; CHECK-NEWLOWERING-NEXT: bl shared_za_call
+; CHECK-NEWLOWERING-NEXT: .LBB9_9: // %common.ret
+; CHECK-NEWLOWERING-NEXT: mov sp, x29
+; CHECK-NEWLOWERING-NEXT: ldr x19, [sp, #16] // 8-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ldp x29, x30, [sp], #32 // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ret
+entry:
+ br i1 %c1, label %shared_path, label %private_path
+
+shared_path:
+ call void @shared_za_call()
+ br label %merge
+
+private_path:
+ call void @private_za_call()
+ br label %merge
+
+merge:
+ br i1 %c2, label %exit_private, label %exit_shared
+
+exit_private:
+ call void @private_za_call()
+ ret void
+
+exit_shared:
+ call void @shared_za_call()
+ ret void
+}
+
+define void @nested_cond_in_loop(i32 %n, i1 %cond) "aarch64_inout_za" nounwind {
+; CHECK-LABEL: nested_cond_in_loop:
+; CHECK: // %bb.0: // %entry
+; CHECK-NEXT: stp x29, x30, [sp, #-64]! // 16-byte Folded Spill
+; CHECK-NEXT: str x23, [sp, #16] // 8-byte Folded Spill
+; CHECK-NEXT: mov x29, sp
+; CHECK-NEXT: stp x22, x21, [sp, #32] // 16-byte Folded Spill
+; CHECK-NEXT: stp x20, x19, [sp, #48] // 16-byte Folded Spill
+; CHECK-NEXT: sub sp, sp, #16
+; CHECK-NEXT: rdsvl x8, #1
+; CHECK-NEXT: mov x9, sp
+; CHECK-NEXT: msub x8, x8, x8, x9
+; CHECK-NEXT: mov sp, x8
+; CHECK-NEXT: cmp w0, #1
+; CHECK-NEXT: stur x8, [x29, #-16]
+; CHECK-NEXT: sturh wzr, [x29, #-6]
+; CHECK-NEXT: stur wzr, [x29, #-4]
+; CHECK-NEXT: b.lt .LBB10_8
+; CHECK-NEXT: // %bb.1: // %loop.preheader
+; CHECK-NEXT: mov w19, w1
+; CHECK-NEXT: mov w20, w0
+; CHECK-NEXT: mov w21, wzr
+; CHECK-NEXT: rdsvl x22, #1
+; CHECK-NEXT: sub x23, x29, #16
+; CHECK-NEXT: b .LBB10_4
+; CHECK-NEXT: .LBB10_2: // %use_shared
+; CHECK-NEXT: // in Loop: Header=BB10_4 Depth=1
+; CHECK-NEXT: bl shared_za_call
+; CHECK-NEXT: .LBB10_3: // %latch
+; CHECK-NEXT: // in Loop: Header=BB10_4 Depth=1
+; CHECK-NEXT: add w21, w21, #1
+; CHECK-NEXT: cmp w21, w20
+; CHECK-NEXT: b.ge .LBB10_8
+; CHECK-NEXT: .LBB10_4: // %loop
+; CHECK-NEXT: // =>This Inner Loop Header: Depth=1
+; CHECK-NEXT: tbnz w19, #0, .LBB10_2
+; CHECK-NEXT: // %bb.5: // %use_private
+; CHECK-NEXT: // in Loop: Header=BB10_4 Depth=1
+; CHECK-NEXT: sturh w22, [x29, #-8]
+; CHECK-NEXT: msr TPIDR2_EL0, x23
+; CHECK-NEXT: bl private_za_call
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: sub x0, x29, #16
+; CHECK-NEXT: cbnz x8, .LBB10_7
+; CHECK-NEXT: // %bb.6: // %use_private
+; CHECK-NEXT: // in Loop: Header=BB10_4 Depth=1
+; CHECK-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEXT: .LBB10_7: // %use_private
+; CHECK-NEXT: // in Loop: Header=BB10_4 Depth=1
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: b .LBB10_3
+; CHECK-NEXT: .LBB10_8: // %exit
+; CHECK-NEXT: mov sp, x29
+; CHECK-NEXT: ldp x20, x19, [sp, #48] // 16-byte Folded Reload
+; CHECK-NEXT: ldr x23, [sp, #16] // 8-byte Folded Reload
+; CHECK-NEXT: ldp x22, x21, [sp, #32] // 16-byte Folded Reload
+; CHECK-NEXT: ldp x29, x30, [sp], #64 // 16-byte Folded Reload
+; CHECK-NEXT: ret
+;
+; CHECK-NEWLOWERING-LABEL: nested_cond_in_loop:
+; CHECK-NEWLOWERING: // %bb.0: // %entry
+; CHECK-NEWLOWERING-NEXT: stp x29, x30, [sp, #-48]! // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: stp x22, x21, [sp, #16] // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: mov x29, sp
+; CHECK-NEWLOWERING-NEXT: stp x20, x19, [sp, #32] // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: sub sp, sp, #16
+; CHECK-NEWLOWERING-NEXT: rdsvl x8, #1
+; CHECK-NEWLOWERING-NEXT: mov x9, sp
+; CHECK-NEWLOWERING-NEXT: msub x9, x8, x8, x9
+; CHECK-NEWLOWERING-NEXT: mov sp, x9
+; CHECK-NEWLOWERING-NEXT: cmp w0, #1
+; CHECK-NEWLOWERING-NEXT: stp x9, x8, [x29, #-16]
+; CHECK-NEWLOWERING-NEXT: b.lt .LBB10_8
+; CHECK-NEWLOWERING-NEXT: // %bb.1: // %loop.preheader
+; CHECK-NEWLOWERING-NEXT: mov w19, w1
+; CHECK-NEWLOWERING-NEXT: mov w20, w0
+; CHECK-NEWLOWERING-NEXT: mov w21, wzr
+; CHECK-NEWLOWERING-NEXT: sub x22, x29, #16
+; CHECK-NEWLOWERING-NEXT: b .LBB10_4
+; CHECK-NEWLOWERING-NEXT: .LBB10_2: // %use_shared
+; CHECK-NEWLOWERING-NEXT: // in Loop: Header=BB10_4 Depth=1
+; CHECK-NEWLOWERING-NEXT: bl shared_za_call
+; CHECK-NEWLOWERING-NEXT: .LBB10_3: // %latch
+; CHECK-NEWLOWERING-NEXT: // in Loop: Header=BB10_4 Depth=1
+; CHECK-NEWLOWERING-NEXT: add w21, w21, #1
+; CHECK-NEWLOWERING-NEXT: cmp w21, w20
+; CHECK-NEWLOWERING-NEXT: b.ge .LBB10_8
+; CHECK-NEWLOWERING-NEXT: .LBB10_4: // %loop
+; CHECK-NEWLOWERING-NEXT: // =>This Inner Loop Header: Depth=1
+; CHECK-NEWLOWERING-NEXT: tbnz w19, #0, .LBB10_2
+; CHECK-NEWLOWERING-NEXT: // %bb.5: // %use_private
+; CHECK-NEWLOWERING-NEXT: // in Loop: Header=BB10_4 Depth=1
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, x22
+; CHECK-NEWLOWERING-NEXT: bl private_za_call
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: sub x0, x29, #16
+; CHECK-NEWLOWERING-NEXT: cbnz x8, .LBB10_7
+; CHECK-NEWLOWERING-NEXT: // %bb.6: // %use_private
+; CHECK-NEWLOWERING-NEXT: // in Loop: Header=BB10_4 Depth=1
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEWLOWERING-NEXT: .LBB10_7: // %use_private
+; CHECK-NEWLOWERING-NEXT: // in Loop: Header=BB10_4 Depth=1
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: b .LBB10_3
+; CHECK-NEWLOWERING-NEXT: .LBB10_8: // %exit
+; CHECK-NEWLOWERING-NEXT: mov sp, x29
+; CHECK-NEWLOWERING-NEXT: ldp x20, x19, [sp, #32] // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ldp x22, x21, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ldp x29, x30, [sp], #48 // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ret
+entry:
+ %cmp = icmp sgt i32 %n, 0
+ br i1 %cmp, label %loop, label %exit
+
+loop:
+ %iv = phi i32 [ 0, %entry ], [ %inc, %latch ]
+ br i1 %cond, label %use_shared, label %use_private
+
+use_shared:
+ call void @shared_za_call()
+ br label %latch
+
+use_private:
+ call void @private_za_call()
+ br label %latch
+
+latch:
+ %inc = add i32 %iv, 1
+ %cmp2 = icmp slt i32 %inc, %n
+ br i1 %cmp2, label %loop, label %exit
+
+exit:
+ ret void
+}
+
+define void @loop_with_external_entry(i1 %c1, i1 %c2) "aarch64_inout_za" nounwind {
+; CHECK-LABEL: loop_with_external_entry:
+; CHECK: // %bb.0: // %entry
+; CHECK-NEXT: stp x29, x30, [sp, #-48]! // 16-byte Folded Spill
+; CHECK-NEXT: str x21, [sp, #16] // 8-byte Folded Spill
+; CHECK-NEXT: mov x29, sp
+; CHECK-NEXT: stp x20, x19, [sp, #32] // 16-byte Folded Spill
+; CHECK-NEXT: sub sp, sp, #16
+; CHECK-NEXT: rdsvl x8, #1
+; CHECK-NEXT: mov x9, sp
+; CHECK-NEXT: mov w19, w1
+; CHECK-NEXT: msub x8, x8, x8, x9
+; CHECK-NEXT: mov sp, x8
+; CHECK-NEXT: stur x8, [x29, #-16]
+; CHECK-NEXT: sturh wzr, [x29, #-6]
+; CHECK-NEXT: stur wzr, [x29, #-4]
+; CHECK-NEXT: tbz w0, #0, .LBB11_2
+; CHECK-NEXT: // %bb.1: // %init
+; CHECK-NEXT: bl shared_za_call
+; CHECK-NEXT: .LBB11_2: // %loop.preheader
+; CHECK-NEXT: rdsvl x20, #1
+; CHECK-NEXT: sub x21, x29, #16
+; CHECK-NEXT: b .LBB11_4
+; CHECK-NEXT: .LBB11_3: // %loop
+; CHECK-NEXT: // in Loop: Header=BB11_4 Depth=1
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: tbz w19, #0, .LBB11_6
+; CHECK-NEXT: .LBB11_4: // %loop
+; CHECK-NEXT: // =>This Inner Loop Header: Depth=1
+; CHECK-NEXT: sturh w20, [x29, #-8]
+; CHECK-NEXT: msr TPIDR2_EL0, x21
+; CHECK-NEXT: bl private_za_call
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: sub x0, x29, #16
+; CHECK-NEXT: cbnz x8, .LBB11_3
+; CHECK-NEXT: // %bb.5: // %loop
+; CHECK-NEXT: // in Loop: Header=BB11_4 Depth=1
+; CHECK-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEXT: b .LBB11_3
+; CHECK-NEXT: .LBB11_6: // %exit
+; CHECK-NEXT: mov sp, x29
+; CHECK-NEXT: ldp x20, x19, [sp, #32] // 16-byte Folded Reload
+; CHECK-NEXT: ldr x21, [sp, #16] // 8-byte Folded Reload
+; CHECK-NEXT: ldp x29, x30, [sp], #48 // 16-byte Folded Reload
+; CHECK-NEXT: ret
+;
+; CHECK-NEWLOWERING-LABEL: loop_with_external_entry:
+; CHECK-NEWLOWERING: // %bb.0: // %entry
+; CHECK-NEWLOWERING-NEXT: stp x29, x30, [sp, #-32]! // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: str x19, [sp, #16] // 8-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: mov x29, sp
+; CHECK-NEWLOWERING-NEXT: sub sp, sp, #16
+; CHECK-NEWLOWERING-NEXT: rdsvl x8, #1
+; CHECK-NEWLOWERING-NEXT: mov x9, sp
+; CHECK-NEWLOWERING-NEXT: msub x9, x8, x8, x9
+; CHECK-NEWLOWERING-NEXT: mov sp, x9
+; CHECK-NEWLOWERING-NEXT: mov w19, w1
+; CHECK-NEWLOWERING-NEXT: stp x9, x8, [x29, #-16]
+; CHECK-NEWLOWERING-NEXT: tbz w0, #0, .LBB11_2
+; CHECK-NEWLOWERING-NEXT: // %bb.1: // %init
+; CHECK-NEWLOWERING-NEXT: bl shared_za_call
+; CHECK-NEWLOWERING-NEXT: .LBB11_2: // %loop.preheader
+; CHECK-NEWLOWERING-NEXT: sub x8, x29, #16
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, x8
+; CHECK-NEWLOWERING-NEXT: .LBB11_3: // %loop
+; CHECK-NEWLOWERING-NEXT: // =>This Inner Loop Header: Depth=1
+; CHECK-NEWLOWERING-NEXT: bl private_za_call
+; CHECK-NEWLOWERING-NEXT: tbnz w19, #0, .LBB11_3
+; CHECK-NEWLOWERING-NEXT: // %bb.4: // %exit
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: sub x0, x29, #16
+; CHECK-NEWLOWERING-NEXT: cbnz x8, .LBB11_6
+; CHECK-NEWLOWERING-NEXT: // %bb.5: // %exit
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEWLOWERING-NEXT: .LBB11_6: // %exit
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: mov sp, x29
+; CHECK-NEWLOWERING-NEXT: ldr x19, [sp, #16] // 8-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ldp x29, x30, [sp], #32 // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ret
+entry:
+ br i1 %c1, label %init, label %loop
+
+init:
+ call void @shared_za_call()
+ br label %loop
+
+loop:
+ call void @private_za_call()
+ br i1 %c2, label %loop, label %exit
+
+exit:
+ ret void
+}
diff --git a/llvm/test/CodeGen/AArch64/sme-za-exceptions.ll b/llvm/test/CodeGen/AArch64/sme-za-exceptions.ll
new file mode 100644
index 0000000000000..8d40ee9bda11a
--- /dev/null
+++ b/llvm/test/CodeGen/AArch64/sme-za-exceptions.ll
@@ -0,0 +1,288 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5
+; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sme -aarch64-new-sme-abi < %s | FileCheck %s
+
+; A simple EH test case that corresponds to the following C++ source:
+;
+; struct ZAResource {
+; ~ZAResource() __arm_inout("za") {
+; shared_za_call(); // simulate cleanup in destructor
+; }
+; };
+;
+; void za_with_raii(bool fail) __arm_inout("za") {
+; ZAResource r;
+; if (fail)
+; throw "Unwinding needs ZA state reload";
+; }
+;
+; Here if an exception is thrown we must call the ~ZAResource destructor while
+; unwinding the stack. That requires us to restore ZA state before the
+; shared_za_call in the cleanup block.
+
+ at .str = private unnamed_addr constant [32 x i8] c"Unwinding needs ZA state reload\00", align 1
+ at typeinfo_for_char_const_ptr = external constant ptr
+
+define void @za_with_raii(i1 %fail) "aarch64_inout_za" personality ptr @__gxx_personality_v0 {
+; CHECK-LABEL: za_with_raii:
+; CHECK: .Lfunc_begin0:
+; CHECK-NEXT: .cfi_startproc
+; CHECK-NEXT: .cfi_personality 156, DW.ref.__gxx_personality_v0
+; CHECK-NEXT: .cfi_lsda 28, .Lexception0
+; CHECK-NEXT: // %bb.0:
+; CHECK-NEXT: stp x29, x30, [sp, #-32]! // 16-byte Folded Spill
+; CHECK-NEXT: str x19, [sp, #16] // 8-byte Folded Spill
+; CHECK-NEXT: mov x29, sp
+; CHECK-NEXT: sub sp, sp, #16
+; CHECK-NEXT: .cfi_def_cfa w29, 32
+; CHECK-NEXT: .cfi_offset w19, -16
+; CHECK-NEXT: .cfi_offset w30, -24
+; CHECK-NEXT: .cfi_offset w29, -32
+; CHECK-NEXT: rdsvl x8, #1
+; CHECK-NEXT: mov x9, sp
+; CHECK-NEXT: msub x9, x8, x8, x9
+; CHECK-NEXT: mov sp, x9
+; CHECK-NEXT: stp x9, x8, [x29, #-16]
+; CHECK-NEXT: tbnz w0, #0, .LBB0_2
+; CHECK-NEXT: // %bb.1: // %return_normally
+; CHECK-NEXT: mov sp, x29
+; CHECK-NEXT: ldr x19, [sp, #16] // 8-byte Folded Reload
+; CHECK-NEXT: ldp x29, x30, [sp], #32 // 16-byte Folded Reload
+; CHECK-NEXT: b shared_za_call
+; CHECK-NEXT: .LBB0_2: // %throw_exception
+; CHECK-NEXT: sub x8, x29, #16
+; CHECK-NEXT: mov w0, #8 // =0x8
+; CHECK-NEXT: msr TPIDR2_EL0, x8
+; CHECK-NEXT: bl __cxa_allocate_exception
+; CHECK-NEXT: adrp x8, .L.str
+; CHECK-NEXT: add x8, x8, :lo12:.L.str
+; CHECK-NEXT: str x8, [x0]
+; CHECK-NEXT: .Ltmp0:
+; CHECK-NEXT: adrp x1, :got:typeinfo_for_char_const_ptr
+; CHECK-NEXT: mov x2, xzr
+; CHECK-NEXT: ldr x1, [x1, :got_lo12:typeinfo_for_char_const_ptr]
+; CHECK-NEXT: bl __cxa_throw
+; CHECK-NEXT: .Ltmp1:
+; CHECK-NEXT: mov x8, x0
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: mrs x9, TPIDR2_EL0
+; CHECK-NEXT: sub x0, x29, #16
+; CHECK-NEXT: cbnz x9, .LBB0_4
+; CHECK-NEXT: // %bb.3: // %throw_exception
+; CHECK-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEXT: .LBB0_4: // %throw_exception
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: // kill: def $x0 killed $x8
+; CHECK-NEXT: // %bb.5: // %throw_fail
+; CHECK-NEXT: .LBB0_6: // %unwind_dtors
+; CHECK-NEXT: .Ltmp2:
+; CHECK-NEXT: mov x19, x0
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: sub x0, x29, #16
+; CHECK-NEXT: cbnz x8, .LBB0_8
+; CHECK-NEXT: // %bb.7: // %unwind_dtors
+; CHECK-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEXT: .LBB0_8: // %unwind_dtors
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: bl shared_za_call
+; CHECK-NEXT: sub x8, x29, #16
+; CHECK-NEXT: mov x0, x19
+; CHECK-NEXT: msr TPIDR2_EL0, x8
+; CHECK-NEXT: bl _Unwind_Resume
+ br i1 %fail, label %throw_exception, label %return_normally
+
+throw_exception:
+ %exception_ptr = tail call ptr @__cxa_allocate_exception(i64 8) #3
+ store ptr @.str, ptr %exception_ptr, align 16
+ invoke void @__cxa_throw(ptr nonnull %exception_ptr, ptr nonnull @typeinfo_for_char_const_ptr, ptr null)
+ to label %throw_fail unwind label %unwind_dtors
+
+unwind_dtors:
+ %5 = landingpad { ptr, i32 }
+ cleanup
+ tail call void @shared_za_call()
+ resume { ptr, i32 } %5
+
+return_normally:
+ tail call void @shared_za_call()
+ ret void
+
+throw_fail:
+ unreachable
+}
+
+
+; Another simple exception handling example. Here we need to restore ZA in two
+; places. After the may_throw() call to handle the case it does not throw, and
+; within the catch block for the shared_za_call(). We also need to setup the
+; lazy save around C++ exception ABI routines (to handle the _very_ unlikely
+; case they use ZA state).
+;
+; void za_try_catch() __arm_inout("za") {
+; try {
+; may_throw();
+; } catch (...) {
+; shared_za_call();
+; }
+; shared_za_call();
+; }
+define dso_local void @try_catch() "aarch64_inout_za" personality ptr @__gxx_personality_v0 {
+; CHECK-LABEL: try_catch:
+; CHECK: .Lfunc_begin1:
+; CHECK-NEXT: .cfi_startproc
+; CHECK-NEXT: .cfi_personality 156, DW.ref.__gxx_personality_v0
+; CHECK-NEXT: .cfi_lsda 28, .Lexception1
+; CHECK-NEXT: // %bb.0:
+; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT: mov x29, sp
+; CHECK-NEXT: sub sp, sp, #16
+; CHECK-NEXT: .cfi_def_cfa w29, 16
+; CHECK-NEXT: .cfi_offset w30, -8
+; CHECK-NEXT: .cfi_offset w29, -16
+; CHECK-NEXT: rdsvl x8, #1
+; CHECK-NEXT: mov x9, sp
+; CHECK-NEXT: msub x9, x8, x8, x9
+; CHECK-NEXT: mov sp, x9
+; CHECK-NEXT: stp x9, x8, [x29, #-16]
+; CHECK-NEXT: .Ltmp3:
+; CHECK-NEXT: sub x8, x29, #16
+; CHECK-NEXT: msr TPIDR2_EL0, x8
+; CHECK-NEXT: bl may_throw
+; CHECK-NEXT: .Ltmp4:
+; CHECK-NEXT: .LBB1_1: // %after_catch
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: sub x0, x29, #16
+; CHECK-NEXT: cbnz x8, .LBB1_3
+; CHECK-NEXT: // %bb.2: // %after_catch
+; CHECK-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEXT: .LBB1_3: // %after_catch
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: mov sp, x29
+; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT: b shared_za_call
+; CHECK-NEXT: .LBB1_4: // %catch
+; CHECK-NEXT: .Ltmp5:
+; CHECK-NEXT: bl __cxa_begin_catch
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: sub x0, x29, #16
+; CHECK-NEXT: cbnz x8, .LBB1_6
+; CHECK-NEXT: // %bb.5: // %catch
+; CHECK-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEXT: .LBB1_6: // %catch
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: bl shared_za_call
+; CHECK-NEXT: sub x8, x29, #16
+; CHECK-NEXT: msr TPIDR2_EL0, x8
+; CHECK-NEXT: bl __cxa_end_catch
+; CHECK-NEXT: b .LBB1_1
+ invoke void @may_throw()
+ to label %after_catch unwind label %catch
+
+catch: ; preds = %0
+ %eh_info = landingpad { ptr, i32 }
+ catch ptr null
+ %exception_ptr = extractvalue { ptr, i32 } %eh_info, 0
+ tail call ptr @__cxa_begin_catch(ptr %exception_ptr)
+ tail call void @shared_za_call()
+ tail call void @__cxa_end_catch()
+ br label %after_catch
+
+after_catch:
+ tail call void @shared_za_call()
+ ret void
+}
+
+; This example corresponds to:
+;
+; __arm_new("za") void try_catch_shared_za_callee()
+; {
+; try {
+; shared_za_call();
+; } catch(...) {
+; noexcept_shared_za_call();
+; }
+; }
+;
+; In this example we don't setup a lazy save before shared_za_call(), however,
+; we still enter the catch block in a ZA off state. This leads to use emitting
+; a restore of a uninitialized save buffer in the catch block. This is not ideal
+; but is valid in the SME ABI. Ideally, we would omit the save buffer and
+; restore and simply set ZA to "on" in the catch block.
+
+define void @try_catch_shared_za_callee() "aarch64_new_za" personality ptr @__gxx_personality_v0 {
+; CHECK-LABEL: try_catch_shared_za_callee:
+; CHECK: .Lfunc_begin2:
+; CHECK-NEXT: .cfi_startproc
+; CHECK-NEXT: .cfi_personality 156, DW.ref.__gxx_personality_v0
+; CHECK-NEXT: .cfi_lsda 28, .Lexception2
+; CHECK-NEXT: // %bb.0:
+; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT: mov x29, sp
+; CHECK-NEXT: sub sp, sp, #16
+; CHECK-NEXT: .cfi_def_cfa w29, 16
+; CHECK-NEXT: .cfi_offset w30, -8
+; CHECK-NEXT: .cfi_offset w29, -16
+; CHECK-NEXT: rdsvl x8, #1
+; CHECK-NEXT: mov x9, sp
+; CHECK-NEXT: msub x9, x8, x8, x9
+; CHECK-NEXT: mov sp, x9
+; CHECK-NEXT: stp x9, x8, [x29, #-16]
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: cbz x8, .LBB2_2
+; CHECK-NEXT: // %bb.1:
+; CHECK-NEXT: bl __arm_tpidr2_save
+; CHECK-NEXT: .LBB2_2:
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: zero {za}
+; CHECK-NEXT: .Ltmp6:
+; CHECK-NEXT: bl shared_za_call
+; CHECK-NEXT: .Ltmp7:
+; CHECK-NEXT: .LBB2_3: // %exit
+; CHECK-NEXT: smstop za
+; CHECK-NEXT: mov sp, x29
+; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT: ret
+; CHECK-NEXT: .LBB2_4: // %catch
+; CHECK-NEXT: .Ltmp8:
+; CHECK-NEXT: bl __cxa_begin_catch
+; CHECK-NEXT: smstart za
+; CHECK-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEXT: sub x0, x29, #16
+; CHECK-NEXT: cbnz x8, .LBB2_6
+; CHECK-NEXT: // %bb.5: // %catch
+; CHECK-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEXT: .LBB2_6: // %catch
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: bl noexcept_shared_za_call
+; CHECK-NEXT: sub x8, x29, #16
+; CHECK-NEXT: msr TPIDR2_EL0, x8
+; CHECK-NEXT: bl __cxa_end_catch
+; CHECK-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEXT: b .LBB2_3
+ invoke void @shared_za_call() #4
+ to label %exit unwind label %catch
+catch:
+ %eh_info = landingpad { ptr, i32 }
+ catch ptr null
+ %exception_ptr = extractvalue { ptr, i32 } %eh_info, 0
+ tail call ptr @__cxa_begin_catch(ptr %exception_ptr)
+ tail call void @noexcept_shared_za_call()
+ tail call void @__cxa_end_catch()
+ br label %exit
+
+exit:
+ ret void
+}
+
+declare ptr @__cxa_allocate_exception(i64)
+declare void @__cxa_throw(ptr, ptr, ptr)
+declare ptr @__cxa_begin_catch(ptr)
+declare void @__cxa_end_catch()
+declare i32 @__gxx_personality_v0(...)
+
+declare void @may_throw()
+declare void @shared_za_call() "aarch64_inout_za"
+declare void @noexcept_shared_za_call() "aarch64_inout_za"
diff --git a/llvm/test/CodeGen/AArch64/sme-za-lazy-save-buffer.ll b/llvm/test/CodeGen/AArch64/sme-za-lazy-save-buffer.ll
index ad3f7f5514d0e..a9ad6f695cf8f 100644
--- a/llvm/test/CodeGen/AArch64/sme-za-lazy-save-buffer.ll
+++ b/llvm/test/CodeGen/AArch64/sme-za-lazy-save-buffer.ll
@@ -1,11 +1,12 @@
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 4
-; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sme2 < %s | FileCheck %s
+; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sme2 < %s | FileCheck %s --check-prefixes=CHECK-COMMON,CHECK
+; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sme2 -aarch64-new-sme-abi < %s | FileCheck %s --check-prefixes=CHECK-COMMON,CHECK-NEWLOWERING
define i32 @no_tpidr2_save_required() "aarch64_inout_za" {
-; CHECK-LABEL: no_tpidr2_save_required:
-; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: mov w0, #42 // =0x2a
-; CHECK-NEXT: ret
+; CHECK-COMMON-LABEL: no_tpidr2_save_required:
+; CHECK-COMMON: // %bb.0: // %entry
+; CHECK-COMMON-NEXT: mov w0, #42 // =0x2a
+; CHECK-COMMON-NEXT: ret
entry:
ret i32 42
}
@@ -50,6 +51,42 @@ define float @multi_bb_stpidr2_save_required(i32 %a, float %b, float %c) "aarch6
; CHECK-NEXT: mov sp, x29
; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
; CHECK-NEXT: ret
+;
+; CHECK-NEWLOWERING-LABEL: multi_bb_stpidr2_save_required:
+; CHECK-NEWLOWERING: // %bb.0:
+; CHECK-NEWLOWERING-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: mov x29, sp
+; CHECK-NEWLOWERING-NEXT: sub sp, sp, #16
+; CHECK-NEWLOWERING-NEXT: .cfi_def_cfa w29, 16
+; CHECK-NEWLOWERING-NEXT: .cfi_offset w30, -8
+; CHECK-NEWLOWERING-NEXT: .cfi_offset w29, -16
+; CHECK-NEWLOWERING-NEXT: rdsvl x8, #1
+; CHECK-NEWLOWERING-NEXT: mov x9, sp
+; CHECK-NEWLOWERING-NEXT: msub x9, x8, x8, x9
+; CHECK-NEWLOWERING-NEXT: mov sp, x9
+; CHECK-NEWLOWERING-NEXT: sub x10, x29, #16
+; CHECK-NEWLOWERING-NEXT: stp x9, x8, [x29, #-16]
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, x10
+; CHECK-NEWLOWERING-NEXT: cbz w0, .LBB1_2
+; CHECK-NEWLOWERING-NEXT: // %bb.1: // %use_b
+; CHECK-NEWLOWERING-NEXT: fmov s1, #4.00000000
+; CHECK-NEWLOWERING-NEXT: fadd s0, s0, s1
+; CHECK-NEWLOWERING-NEXT: b .LBB1_3
+; CHECK-NEWLOWERING-NEXT: .LBB1_2: // %use_c
+; CHECK-NEWLOWERING-NEXT: fmov s0, s1
+; CHECK-NEWLOWERING-NEXT: bl cosf
+; CHECK-NEWLOWERING-NEXT: .LBB1_3: // %exit
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: sub x0, x29, #16
+; CHECK-NEWLOWERING-NEXT: cbnz x8, .LBB1_5
+; CHECK-NEWLOWERING-NEXT: // %bb.4: // %exit
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEWLOWERING-NEXT: .LBB1_5: // %exit
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: mov sp, x29
+; CHECK-NEWLOWERING-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ret
%cmp = icmp ne i32 %a, 0
br i1 %cmp, label %use_b, label %use_c
@@ -66,6 +103,7 @@ exit:
ret float %ret
}
+; FIXME: This is missing stack probes with -aarch64-new-sme-abi.
define float @multi_bb_stpidr2_save_required_stackprobe(i32 %a, float %b, float %c) "aarch64_inout_za" "probe-stack"="inline-asm" "stack-probe-size"="65536" {
; CHECK-LABEL: multi_bb_stpidr2_save_required_stackprobe:
; CHECK: // %bb.0:
@@ -115,6 +153,42 @@ define float @multi_bb_stpidr2_save_required_stackprobe(i32 %a, float %b, float
; CHECK-NEXT: mov sp, x29
; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
; CHECK-NEXT: ret
+;
+; CHECK-NEWLOWERING-LABEL: multi_bb_stpidr2_save_required_stackprobe:
+; CHECK-NEWLOWERING: // %bb.0:
+; CHECK-NEWLOWERING-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: mov x29, sp
+; CHECK-NEWLOWERING-NEXT: str xzr, [sp, #-16]!
+; CHECK-NEWLOWERING-NEXT: .cfi_def_cfa w29, 16
+; CHECK-NEWLOWERING-NEXT: .cfi_offset w30, -8
+; CHECK-NEWLOWERING-NEXT: .cfi_offset w29, -16
+; CHECK-NEWLOWERING-NEXT: rdsvl x8, #1
+; CHECK-NEWLOWERING-NEXT: mov x9, sp
+; CHECK-NEWLOWERING-NEXT: msub x9, x8, x8, x9
+; CHECK-NEWLOWERING-NEXT: mov sp, x9
+; CHECK-NEWLOWERING-NEXT: sub x10, x29, #16
+; CHECK-NEWLOWERING-NEXT: stp x9, x8, [x29, #-16]
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, x10
+; CHECK-NEWLOWERING-NEXT: cbz w0, .LBB2_2
+; CHECK-NEWLOWERING-NEXT: // %bb.1: // %use_b
+; CHECK-NEWLOWERING-NEXT: fmov s1, #4.00000000
+; CHECK-NEWLOWERING-NEXT: fadd s0, s0, s1
+; CHECK-NEWLOWERING-NEXT: b .LBB2_3
+; CHECK-NEWLOWERING-NEXT: .LBB2_2: // %use_c
+; CHECK-NEWLOWERING-NEXT: fmov s0, s1
+; CHECK-NEWLOWERING-NEXT: bl cosf
+; CHECK-NEWLOWERING-NEXT: .LBB2_3: // %exit
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: sub x0, x29, #16
+; CHECK-NEWLOWERING-NEXT: cbnz x8, .LBB2_5
+; CHECK-NEWLOWERING-NEXT: // %bb.4: // %exit
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEWLOWERING-NEXT: .LBB2_5: // %exit
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: mov sp, x29
+; CHECK-NEWLOWERING-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ret
%cmp = icmp ne i32 %a, 0
br i1 %cmp, label %use_b, label %use_c
diff --git a/llvm/test/CodeGen/AArch64/sme-zt0-state.ll b/llvm/test/CodeGen/AArch64/sme-zt0-state.ll
index 63577e4d217a8..1c5caf9fa47ef 100644
--- a/llvm/test/CodeGen/AArch64/sme-zt0-state.ll
+++ b/llvm/test/CodeGen/AArch64/sme-zt0-state.ll
@@ -1,5 +1,6 @@
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 4
-; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sme2 -start-after=simplifycfg -enable-tail-merge=false -verify-machineinstrs < %s | FileCheck %s
+; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sme2 -start-after=simplifycfg -enable-tail-merge=false -verify-machineinstrs < %s | FileCheck %s --check-prefixes=CHECK-COMMON,CHECK
+; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sme2 -start-after=simplifycfg -enable-tail-merge=false -aarch64-new-sme-abi -verify-machineinstrs < %s | FileCheck %s --check-prefixes=CHECK-COMMON,CHECK-NEWLOWERING
;
; Private-ZA Callee
@@ -8,19 +9,19 @@
; Expect spill & fill of ZT0 around call
; Expect smstop/smstart za around call
define void @zt0_in_caller_no_state_callee(ptr %callee) "aarch64_in_zt0" nounwind {
-; CHECK-LABEL: zt0_in_caller_no_state_callee:
-; CHECK: // %bb.0:
-; CHECK-NEXT: sub sp, sp, #80
-; CHECK-NEXT: stp x30, x19, [sp, #64] // 16-byte Folded Spill
-; CHECK-NEXT: mov x19, sp
-; CHECK-NEXT: str zt0, [x19]
-; CHECK-NEXT: smstop za
-; CHECK-NEXT: blr x0
-; CHECK-NEXT: smstart za
-; CHECK-NEXT: ldr zt0, [x19]
-; CHECK-NEXT: ldp x30, x19, [sp, #64] // 16-byte Folded Reload
-; CHECK-NEXT: add sp, sp, #80
-; CHECK-NEXT: ret
+; CHECK-COMMON-LABEL: zt0_in_caller_no_state_callee:
+; CHECK-COMMON: // %bb.0:
+; CHECK-COMMON-NEXT: sub sp, sp, #80
+; CHECK-COMMON-NEXT: stp x30, x19, [sp, #64] // 16-byte Folded Spill
+; CHECK-COMMON-NEXT: mov x19, sp
+; CHECK-COMMON-NEXT: str zt0, [x19]
+; CHECK-COMMON-NEXT: smstop za
+; CHECK-COMMON-NEXT: blr x0
+; CHECK-COMMON-NEXT: smstart za
+; CHECK-COMMON-NEXT: ldr zt0, [x19]
+; CHECK-COMMON-NEXT: ldp x30, x19, [sp, #64] // 16-byte Folded Reload
+; CHECK-COMMON-NEXT: add sp, sp, #80
+; CHECK-COMMON-NEXT: ret
call void %callee();
ret void;
}
@@ -61,6 +62,36 @@ define void @za_zt0_shared_caller_no_state_callee(ptr %callee) "aarch64_inout_za
; CHECK-NEXT: ldr x19, [sp, #16] // 8-byte Folded Reload
; CHECK-NEXT: ldp x29, x30, [sp], #32 // 16-byte Folded Reload
; CHECK-NEXT: ret
+;
+; CHECK-NEWLOWERING-LABEL: za_zt0_shared_caller_no_state_callee:
+; CHECK-NEWLOWERING: // %bb.0:
+; CHECK-NEWLOWERING-NEXT: stp x29, x30, [sp, #-32]! // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: str x19, [sp, #16] // 8-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: mov x29, sp
+; CHECK-NEWLOWERING-NEXT: sub sp, sp, #80
+; CHECK-NEWLOWERING-NEXT: rdsvl x8, #1
+; CHECK-NEWLOWERING-NEXT: mov x9, sp
+; CHECK-NEWLOWERING-NEXT: msub x9, x8, x8, x9
+; CHECK-NEWLOWERING-NEXT: mov sp, x9
+; CHECK-NEWLOWERING-NEXT: sub x19, x29, #64
+; CHECK-NEWLOWERING-NEXT: sub x10, x29, #80
+; CHECK-NEWLOWERING-NEXT: stp x9, x8, [x29, #-80]
+; CHECK-NEWLOWERING-NEXT: str zt0, [x19]
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, x10
+; CHECK-NEWLOWERING-NEXT: blr x0
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: sub x0, x29, #80
+; CHECK-NEWLOWERING-NEXT: cbnz x8, .LBB1_2
+; CHECK-NEWLOWERING-NEXT: // %bb.1:
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEWLOWERING-NEXT: .LBB1_2:
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: ldr zt0, [x19]
+; CHECK-NEWLOWERING-NEXT: mov sp, x29
+; CHECK-NEWLOWERING-NEXT: ldr x19, [sp, #16] // 8-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ldp x29, x30, [sp], #32 // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ret
call void %callee();
ret void;
}
@@ -71,41 +102,41 @@ define void @za_zt0_shared_caller_no_state_callee(ptr %callee) "aarch64_inout_za
; Caller and callee have shared ZT0 state, no spill/fill of ZT0 required
define void @zt0_shared_caller_zt0_shared_callee(ptr %callee) "aarch64_in_zt0" nounwind {
-; CHECK-LABEL: zt0_shared_caller_zt0_shared_callee:
-; CHECK: // %bb.0:
-; CHECK-NEXT: str x30, [sp, #-16]! // 8-byte Folded Spill
-; CHECK-NEXT: blr x0
-; CHECK-NEXT: ldr x30, [sp], #16 // 8-byte Folded Reload
-; CHECK-NEXT: ret
+; CHECK-COMMON-LABEL: zt0_shared_caller_zt0_shared_callee:
+; CHECK-COMMON: // %bb.0:
+; CHECK-COMMON-NEXT: str x30, [sp, #-16]! // 8-byte Folded Spill
+; CHECK-COMMON-NEXT: blr x0
+; CHECK-COMMON-NEXT: ldr x30, [sp], #16 // 8-byte Folded Reload
+; CHECK-COMMON-NEXT: ret
call void %callee() "aarch64_in_zt0";
ret void;
}
; Expect spill & fill of ZT0 around call
define void @za_zt0_shared_caller_za_shared_callee(ptr %callee) "aarch64_inout_za" "aarch64_in_zt0" nounwind {
-; CHECK-LABEL: za_zt0_shared_caller_za_shared_callee:
-; CHECK: // %bb.0:
-; CHECK-NEXT: sub sp, sp, #80
-; CHECK-NEXT: stp x30, x19, [sp, #64] // 16-byte Folded Spill
-; CHECK-NEXT: mov x19, sp
-; CHECK-NEXT: str zt0, [x19]
-; CHECK-NEXT: blr x0
-; CHECK-NEXT: ldr zt0, [x19]
-; CHECK-NEXT: ldp x30, x19, [sp, #64] // 16-byte Folded Reload
-; CHECK-NEXT: add sp, sp, #80
-; CHECK-NEXT: ret
+; CHECK-COMMON-LABEL: za_zt0_shared_caller_za_shared_callee:
+; CHECK-COMMON: // %bb.0:
+; CHECK-COMMON-NEXT: sub sp, sp, #80
+; CHECK-COMMON-NEXT: stp x30, x19, [sp, #64] // 16-byte Folded Spill
+; CHECK-COMMON-NEXT: mov x19, sp
+; CHECK-COMMON-NEXT: str zt0, [x19]
+; CHECK-COMMON-NEXT: blr x0
+; CHECK-COMMON-NEXT: ldr zt0, [x19]
+; CHECK-COMMON-NEXT: ldp x30, x19, [sp, #64] // 16-byte Folded Reload
+; CHECK-COMMON-NEXT: add sp, sp, #80
+; CHECK-COMMON-NEXT: ret
call void %callee() "aarch64_inout_za";
ret void;
}
; Caller and callee have shared ZA & ZT0
define void @za_zt0_shared_caller_za_zt0_shared_callee(ptr %callee) "aarch64_inout_za" "aarch64_in_zt0" nounwind {
-; CHECK-LABEL: za_zt0_shared_caller_za_zt0_shared_callee:
-; CHECK: // %bb.0:
-; CHECK-NEXT: str x30, [sp, #-16]! // 8-byte Folded Spill
-; CHECK-NEXT: blr x0
-; CHECK-NEXT: ldr x30, [sp], #16 // 8-byte Folded Reload
-; CHECK-NEXT: ret
+; CHECK-COMMON-LABEL: za_zt0_shared_caller_za_zt0_shared_callee:
+; CHECK-COMMON: // %bb.0:
+; CHECK-COMMON-NEXT: str x30, [sp, #-16]! // 8-byte Folded Spill
+; CHECK-COMMON-NEXT: blr x0
+; CHECK-COMMON-NEXT: ldr x30, [sp], #16 // 8-byte Folded Reload
+; CHECK-COMMON-NEXT: ret
call void %callee() "aarch64_inout_za" "aarch64_in_zt0";
ret void;
}
@@ -115,19 +146,19 @@ define void @za_zt0_shared_caller_za_zt0_shared_callee(ptr %callee) "aarch64_ino
; Expect spill & fill of ZT0 around call
; Expect smstop/smstart za around call
define void @zt0_in_caller_zt0_new_callee(ptr %callee) "aarch64_in_zt0" nounwind {
-; CHECK-LABEL: zt0_in_caller_zt0_new_callee:
-; CHECK: // %bb.0:
-; CHECK-NEXT: sub sp, sp, #80
-; CHECK-NEXT: stp x30, x19, [sp, #64] // 16-byte Folded Spill
-; CHECK-NEXT: mov x19, sp
-; CHECK-NEXT: str zt0, [x19]
-; CHECK-NEXT: smstop za
-; CHECK-NEXT: blr x0
-; CHECK-NEXT: smstart za
-; CHECK-NEXT: ldr zt0, [x19]
-; CHECK-NEXT: ldp x30, x19, [sp, #64] // 16-byte Folded Reload
-; CHECK-NEXT: add sp, sp, #80
-; CHECK-NEXT: ret
+; CHECK-COMMON-LABEL: zt0_in_caller_zt0_new_callee:
+; CHECK-COMMON: // %bb.0:
+; CHECK-COMMON-NEXT: sub sp, sp, #80
+; CHECK-COMMON-NEXT: stp x30, x19, [sp, #64] // 16-byte Folded Spill
+; CHECK-COMMON-NEXT: mov x19, sp
+; CHECK-COMMON-NEXT: str zt0, [x19]
+; CHECK-COMMON-NEXT: smstop za
+; CHECK-COMMON-NEXT: blr x0
+; CHECK-COMMON-NEXT: smstart za
+; CHECK-COMMON-NEXT: ldr zt0, [x19]
+; CHECK-COMMON-NEXT: ldp x30, x19, [sp, #64] // 16-byte Folded Reload
+; CHECK-COMMON-NEXT: add sp, sp, #80
+; CHECK-COMMON-NEXT: ret
call void %callee() "aarch64_new_zt0";
ret void;
}
@@ -161,6 +192,29 @@ define void @zt0_new_caller_zt0_new_callee(ptr %callee) "aarch64_new_zt0" nounwi
; CHECK-NEXT: ldp x30, x19, [sp, #64] // 16-byte Folded Reload
; CHECK-NEXT: add sp, sp, #80
; CHECK-NEXT: ret
+;
+; CHECK-NEWLOWERING-LABEL: zt0_new_caller_zt0_new_callee:
+; CHECK-NEWLOWERING: // %bb.0:
+; CHECK-NEWLOWERING-NEXT: sub sp, sp, #80
+; CHECK-NEWLOWERING-NEXT: stp x30, x19, [sp, #64] // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: cbz x8, .LBB6_2
+; CHECK-NEWLOWERING-NEXT: // %bb.1:
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_save
+; CHECK-NEWLOWERING-NEXT: .LBB6_2:
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: zero { zt0 }
+; CHECK-NEWLOWERING-NEXT: mov x19, sp
+; CHECK-NEWLOWERING-NEXT: str zt0, [x19]
+; CHECK-NEWLOWERING-NEXT: smstop za
+; CHECK-NEWLOWERING-NEXT: blr x0
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: ldr zt0, [x19]
+; CHECK-NEWLOWERING-NEXT: smstop za
+; CHECK-NEWLOWERING-NEXT: ldp x30, x19, [sp, #64] // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: add sp, sp, #80
+; CHECK-NEWLOWERING-NEXT: ret
call void %callee() "aarch64_new_zt0";
ret void;
}
@@ -191,6 +245,27 @@ define i64 @zt0_new_caller_abi_routine_callee() "aarch64_new_zt0" nounwind {
; CHECK-NEXT: ldp x30, x19, [sp, #64] // 16-byte Folded Reload
; CHECK-NEXT: add sp, sp, #80
; CHECK-NEXT: ret
+;
+; CHECK-NEWLOWERING-LABEL: zt0_new_caller_abi_routine_callee:
+; CHECK-NEWLOWERING: // %bb.0:
+; CHECK-NEWLOWERING-NEXT: sub sp, sp, #80
+; CHECK-NEWLOWERING-NEXT: stp x30, x19, [sp, #64] // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: cbz x8, .LBB7_2
+; CHECK-NEWLOWERING-NEXT: // %bb.1:
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_save
+; CHECK-NEWLOWERING-NEXT: .LBB7_2:
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: zero { zt0 }
+; CHECK-NEWLOWERING-NEXT: mov x19, sp
+; CHECK-NEWLOWERING-NEXT: str zt0, [x19]
+; CHECK-NEWLOWERING-NEXT: bl __arm_sme_state
+; CHECK-NEWLOWERING-NEXT: ldr zt0, [x19]
+; CHECK-NEWLOWERING-NEXT: smstop za
+; CHECK-NEWLOWERING-NEXT: ldp x30, x19, [sp, #64] // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: add sp, sp, #80
+; CHECK-NEWLOWERING-NEXT: ret
%res = call {i64, i64} @__arm_sme_state()
%res.0 = extractvalue {i64, i64} %res, 0
ret i64 %res.0
@@ -221,6 +296,22 @@ define void @zt0_new_caller(ptr %callee) "aarch64_new_zt0" nounwind {
; CHECK-NEXT: smstop za
; CHECK-NEXT: ldr x30, [sp], #16 // 8-byte Folded Reload
; CHECK-NEXT: ret
+;
+; CHECK-NEWLOWERING-LABEL: zt0_new_caller:
+; CHECK-NEWLOWERING: // %bb.0:
+; CHECK-NEWLOWERING-NEXT: str x30, [sp, #-16]! // 8-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: cbz x8, .LBB8_2
+; CHECK-NEWLOWERING-NEXT: // %bb.1:
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_save
+; CHECK-NEWLOWERING-NEXT: .LBB8_2:
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: zero { zt0 }
+; CHECK-NEWLOWERING-NEXT: blr x0
+; CHECK-NEWLOWERING-NEXT: smstop za
+; CHECK-NEWLOWERING-NEXT: ldr x30, [sp], #16 // 8-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ret
call void %callee() "aarch64_in_zt0";
ret void;
}
@@ -245,6 +336,23 @@ define void @new_za_zt0_caller(ptr %callee) "aarch64_new_za" "aarch64_new_zt0" n
; CHECK-NEXT: smstop za
; CHECK-NEXT: ldr x30, [sp], #16 // 8-byte Folded Reload
; CHECK-NEXT: ret
+;
+; CHECK-NEWLOWERING-LABEL: new_za_zt0_caller:
+; CHECK-NEWLOWERING: // %bb.0:
+; CHECK-NEWLOWERING-NEXT: str x30, [sp, #-16]! // 8-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: mrs x8, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: cbz x8, .LBB9_2
+; CHECK-NEWLOWERING-NEXT: // %bb.1:
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_save
+; CHECK-NEWLOWERING-NEXT: .LBB9_2:
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: zero {za}
+; CHECK-NEWLOWERING-NEXT: zero { zt0 }
+; CHECK-NEWLOWERING-NEXT: blr x0
+; CHECK-NEWLOWERING-NEXT: smstop za
+; CHECK-NEWLOWERING-NEXT: ldr x30, [sp], #16 // 8-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ret
call void %callee() "aarch64_inout_za" "aarch64_in_zt0";
ret void;
}
@@ -258,19 +366,26 @@ define void @new_za_shared_zt0_caller(ptr %callee) "aarch64_new_za" "aarch64_in_
; CHECK-NEXT: blr x0
; CHECK-NEXT: ldr x30, [sp], #16 // 8-byte Folded Reload
; CHECK-NEXT: ret
+;
+; CHECK-NEWLOWERING-LABEL: new_za_shared_zt0_caller:
+; CHECK-NEWLOWERING: // %bb.0:
+; CHECK-NEWLOWERING-NEXT: str x30, [sp, #-16]! // 8-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: blr x0
+; CHECK-NEWLOWERING-NEXT: ldr x30, [sp], #16 // 8-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ret
call void %callee() "aarch64_inout_za" "aarch64_in_zt0";
ret void;
}
; Expect clear ZT0 on entry
define void @shared_za_new_zt0(ptr %callee) "aarch64_inout_za" "aarch64_new_zt0" nounwind {
-; CHECK-LABEL: shared_za_new_zt0:
-; CHECK: // %bb.0:
-; CHECK-NEXT: str x30, [sp, #-16]! // 8-byte Folded Spill
-; CHECK-NEXT: zero { zt0 }
-; CHECK-NEXT: blr x0
-; CHECK-NEXT: ldr x30, [sp], #16 // 8-byte Folded Reload
-; CHECK-NEXT: ret
+; CHECK-COMMON-LABEL: shared_za_new_zt0:
+; CHECK-COMMON: // %bb.0:
+; CHECK-COMMON-NEXT: str x30, [sp, #-16]! // 8-byte Folded Spill
+; CHECK-COMMON-NEXT: zero { zt0 }
+; CHECK-COMMON-NEXT: blr x0
+; CHECK-COMMON-NEXT: ldr x30, [sp], #16 // 8-byte Folded Reload
+; CHECK-COMMON-NEXT: ret
call void %callee() "aarch64_inout_za" "aarch64_in_zt0";
ret void;
}
diff --git a/llvm/test/CodeGen/AArch64/sve-stack-frame-layout.ll b/llvm/test/CodeGen/AArch64/sve-stack-frame-layout.ll
index c5cf4593cc86d..8366487d01bb8 100644
--- a/llvm/test/CodeGen/AArch64/sve-stack-frame-layout.ll
+++ b/llvm/test/CodeGen/AArch64/sve-stack-frame-layout.ll
@@ -1,5 +1,6 @@
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5
-; RUN: llc < %s -mtriple=aarch64 -mattr=+sve2 -aarch64-streaming-hazard-size=0 | FileCheck %s --check-prefixes=CHECK
+; RUN: llc < %s -mtriple=aarch64 -mattr=+sve2 -aarch64-streaming-hazard-size=0 | FileCheck %s --check-prefixes=CHECK-COMMON,CHECK
+; RUN: llc < %s -mtriple=aarch64 -mattr=+sve2 -aarch64-streaming-hazard-size=0 -aarch64-new-sme-abi | FileCheck %s --check-prefixes=CHECK-COMMON,CHECK-NEWLOWERING
; RUN: llc < %s -mtriple=aarch64 -mattr=+sve2 -aarch64-streaming-hazard-size=0 -pass-remarks-analysis=stack-frame-layout 2>&1 >/dev/null | FileCheck %s --check-prefixes=CHECK-FRAMELAYOUT
; CHECK-FRAMELAYOUT-LABEL: Function: csr_d8_allocnxv4i32i32f64
@@ -10,28 +11,28 @@
; CHECK-FRAMELAYOUT-NEXT: Offset: [SP-32-16 x vscale], Type: Variable, Align: 8, Size: 8
define i32 @csr_d8_allocnxv4i32i32f64(double %d) "aarch64_pstate_sm_compatible" {
-; CHECK-LABEL: csr_d8_allocnxv4i32i32f64:
-; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: str d8, [sp, #-16]! // 8-byte Folded Spill
-; CHECK-NEXT: str x29, [sp, #8] // 8-byte Folded Spill
-; CHECK-NEXT: sub sp, sp, #16
-; CHECK-NEXT: addvl sp, sp, #-1
-; CHECK-NEXT: .cfi_escape 0x0f, 0x0c, 0x8f, 0x00, 0x11, 0x20, 0x22, 0x11, 0x08, 0x92, 0x2e, 0x00, 0x1e, 0x22 // sp + 32 + 8 * VG
-; CHECK-NEXT: .cfi_offset w29, -8
-; CHECK-NEXT: .cfi_offset b8, -16
-; CHECK-NEXT: mov z1.s, #0 // =0x0
-; CHECK-NEXT: add x8, sp, #16
-; CHECK-NEXT: mov w0, wzr
-; CHECK-NEXT: //APP
-; CHECK-NEXT: //NO_APP
-; CHECK-NEXT: str wzr, [sp, #12]
-; CHECK-NEXT: str d0, [sp]
-; CHECK-NEXT: str z1, [x8]
-; CHECK-NEXT: addvl sp, sp, #1
-; CHECK-NEXT: add sp, sp, #16
-; CHECK-NEXT: ldr x29, [sp, #8] // 8-byte Folded Reload
-; CHECK-NEXT: ldr d8, [sp], #16 // 8-byte Folded Reload
-; CHECK-NEXT: ret
+; CHECK-COMMON-LABEL: csr_d8_allocnxv4i32i32f64:
+; CHECK-COMMON: // %bb.0: // %entry
+; CHECK-COMMON-NEXT: str d8, [sp, #-16]! // 8-byte Folded Spill
+; CHECK-COMMON-NEXT: str x29, [sp, #8] // 8-byte Folded Spill
+; CHECK-COMMON-NEXT: sub sp, sp, #16
+; CHECK-COMMON-NEXT: addvl sp, sp, #-1
+; CHECK-COMMON-NEXT: .cfi_escape 0x0f, 0x0c, 0x8f, 0x00, 0x11, 0x20, 0x22, 0x11, 0x08, 0x92, 0x2e, 0x00, 0x1e, 0x22 // sp + 32 + 8 * VG
+; CHECK-COMMON-NEXT: .cfi_offset w29, -8
+; CHECK-COMMON-NEXT: .cfi_offset b8, -16
+; CHECK-COMMON-NEXT: mov z1.s, #0 // =0x0
+; CHECK-COMMON-NEXT: add x8, sp, #16
+; CHECK-COMMON-NEXT: mov w0, wzr
+; CHECK-COMMON-NEXT: //APP
+; CHECK-COMMON-NEXT: //NO_APP
+; CHECK-COMMON-NEXT: str wzr, [sp, #12]
+; CHECK-COMMON-NEXT: str d0, [sp]
+; CHECK-COMMON-NEXT: str z1, [x8]
+; CHECK-COMMON-NEXT: addvl sp, sp, #1
+; CHECK-COMMON-NEXT: add sp, sp, #16
+; CHECK-COMMON-NEXT: ldr x29, [sp, #8] // 8-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr d8, [sp], #16 // 8-byte Folded Reload
+; CHECK-COMMON-NEXT: ret
entry:
%a = alloca <vscale x 4 x i32>
%b = alloca i32
@@ -52,31 +53,31 @@ entry:
; CHECK-FRAMELAYOUT-NEXT: Offset: [SP-40-16 x vscale], Type: Variable, Align: 8, Size: 8
define i32 @csr_d8_allocnxv4i32i32f64_fp(double %d) "aarch64_pstate_sm_compatible" "frame-pointer"="all" {
-; CHECK-LABEL: csr_d8_allocnxv4i32i32f64_fp:
-; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: str d8, [sp, #-32]! // 8-byte Folded Spill
-; CHECK-NEXT: stp x29, x30, [sp, #16] // 16-byte Folded Spill
-; CHECK-NEXT: add x29, sp, #16
-; CHECK-NEXT: sub sp, sp, #16
-; CHECK-NEXT: addvl sp, sp, #-1
-; CHECK-NEXT: .cfi_def_cfa w29, 16
-; CHECK-NEXT: .cfi_offset w30, -8
-; CHECK-NEXT: .cfi_offset w29, -16
-; CHECK-NEXT: .cfi_offset b8, -32
-; CHECK-NEXT: mov z1.s, #0 // =0x0
-; CHECK-NEXT: addvl x8, sp, #1
-; CHECK-NEXT: //APP
-; CHECK-NEXT: //NO_APP
-; CHECK-NEXT: str wzr, [x8, #28]
-; CHECK-NEXT: sub x8, x29, #16
-; CHECK-NEXT: mov w0, wzr
-; CHECK-NEXT: str d0, [sp, #8]
-; CHECK-NEXT: str z1, [x8, #-1, mul vl]
-; CHECK-NEXT: addvl sp, sp, #1
-; CHECK-NEXT: add sp, sp, #16
-; CHECK-NEXT: ldp x29, x30, [sp, #16] // 16-byte Folded Reload
-; CHECK-NEXT: ldr d8, [sp], #32 // 8-byte Folded Reload
-; CHECK-NEXT: ret
+; CHECK-COMMON-LABEL: csr_d8_allocnxv4i32i32f64_fp:
+; CHECK-COMMON: // %bb.0: // %entry
+; CHECK-COMMON-NEXT: str d8, [sp, #-32]! // 8-byte Folded Spill
+; CHECK-COMMON-NEXT: stp x29, x30, [sp, #16] // 16-byte Folded Spill
+; CHECK-COMMON-NEXT: add x29, sp, #16
+; CHECK-COMMON-NEXT: sub sp, sp, #16
+; CHECK-COMMON-NEXT: addvl sp, sp, #-1
+; CHECK-COMMON-NEXT: .cfi_def_cfa w29, 16
+; CHECK-COMMON-NEXT: .cfi_offset w30, -8
+; CHECK-COMMON-NEXT: .cfi_offset w29, -16
+; CHECK-COMMON-NEXT: .cfi_offset b8, -32
+; CHECK-COMMON-NEXT: mov z1.s, #0 // =0x0
+; CHECK-COMMON-NEXT: addvl x8, sp, #1
+; CHECK-COMMON-NEXT: //APP
+; CHECK-COMMON-NEXT: //NO_APP
+; CHECK-COMMON-NEXT: str wzr, [x8, #28]
+; CHECK-COMMON-NEXT: sub x8, x29, #16
+; CHECK-COMMON-NEXT: mov w0, wzr
+; CHECK-COMMON-NEXT: str d0, [sp, #8]
+; CHECK-COMMON-NEXT: str z1, [x8, #-1, mul vl]
+; CHECK-COMMON-NEXT: addvl sp, sp, #1
+; CHECK-COMMON-NEXT: add sp, sp, #16
+; CHECK-COMMON-NEXT: ldp x29, x30, [sp, #16] // 16-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr d8, [sp], #32 // 8-byte Folded Reload
+; CHECK-COMMON-NEXT: ret
entry:
%a = alloca <vscale x 4 x i32>
%b = alloca i32
@@ -102,30 +103,30 @@ entry:
; CHECK-FRAMELAYOUT-NEXT: Offset: [SP-128-16 x vscale], Type: Variable, Align: 128, Size: 4
define i32 @csr_d8_allocnxv4i32i32f64_dynamicrealign(double %d) "aarch64_pstate_sm_compatible" {
-; CHECK-LABEL: csr_d8_allocnxv4i32i32f64_dynamicrealign:
-; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: str d8, [sp, #-32]! // 8-byte Folded Spill
-; CHECK-NEXT: sub x9, sp, #96
-; CHECK-NEXT: stp x29, x30, [sp, #16] // 16-byte Folded Spill
-; CHECK-NEXT: add x29, sp, #16
-; CHECK-NEXT: addvl x9, x9, #-1
-; CHECK-NEXT: and sp, x9, #0xffffffffffffff80
-; CHECK-NEXT: .cfi_def_cfa w29, 16
-; CHECK-NEXT: .cfi_offset w30, -8
-; CHECK-NEXT: .cfi_offset w29, -16
-; CHECK-NEXT: .cfi_offset b8, -32
-; CHECK-NEXT: mov z1.s, #0 // =0x0
-; CHECK-NEXT: sub x8, x29, #16
-; CHECK-NEXT: mov w0, wzr
-; CHECK-NEXT: //APP
-; CHECK-NEXT: //NO_APP
-; CHECK-NEXT: str wzr, [sp]
-; CHECK-NEXT: stur d0, [x29, #-8]
-; CHECK-NEXT: str z1, [x8, #-1, mul vl]
-; CHECK-NEXT: sub sp, x29, #16
-; CHECK-NEXT: ldp x29, x30, [sp, #16] // 16-byte Folded Reload
-; CHECK-NEXT: ldr d8, [sp], #32 // 8-byte Folded Reload
-; CHECK-NEXT: ret
+; CHECK-COMMON-LABEL: csr_d8_allocnxv4i32i32f64_dynamicrealign:
+; CHECK-COMMON: // %bb.0: // %entry
+; CHECK-COMMON-NEXT: str d8, [sp, #-32]! // 8-byte Folded Spill
+; CHECK-COMMON-NEXT: sub x9, sp, #96
+; CHECK-COMMON-NEXT: stp x29, x30, [sp, #16] // 16-byte Folded Spill
+; CHECK-COMMON-NEXT: add x29, sp, #16
+; CHECK-COMMON-NEXT: addvl x9, x9, #-1
+; CHECK-COMMON-NEXT: and sp, x9, #0xffffffffffffff80
+; CHECK-COMMON-NEXT: .cfi_def_cfa w29, 16
+; CHECK-COMMON-NEXT: .cfi_offset w30, -8
+; CHECK-COMMON-NEXT: .cfi_offset w29, -16
+; CHECK-COMMON-NEXT: .cfi_offset b8, -32
+; CHECK-COMMON-NEXT: mov z1.s, #0 // =0x0
+; CHECK-COMMON-NEXT: sub x8, x29, #16
+; CHECK-COMMON-NEXT: mov w0, wzr
+; CHECK-COMMON-NEXT: //APP
+; CHECK-COMMON-NEXT: //NO_APP
+; CHECK-COMMON-NEXT: str wzr, [sp]
+; CHECK-COMMON-NEXT: stur d0, [x29, #-8]
+; CHECK-COMMON-NEXT: str z1, [x8, #-1, mul vl]
+; CHECK-COMMON-NEXT: sub sp, x29, #16
+; CHECK-COMMON-NEXT: ldp x29, x30, [sp, #16] // 16-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr d8, [sp], #32 // 8-byte Folded Reload
+; CHECK-COMMON-NEXT: ret
entry:
%a = alloca <vscale x 4 x i32>
%b = alloca i32, align 128
@@ -151,44 +152,44 @@ entry:
; CHECK-FRAMELAYOUT-NEXT: Offset: [SP-48-16 x vscale], Type: VariableSized, Align: 1, Size: 0
define i32 @csr_d8_allocnxv4i32i32f64_vla(double %d, i32 %i) "aarch64_pstate_sm_compatible" {
-; CHECK-LABEL: csr_d8_allocnxv4i32i32f64_vla:
-; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: str d8, [sp, #-32]! // 8-byte Folded Spill
-; CHECK-NEXT: stp x29, x30, [sp, #8] // 16-byte Folded Spill
-; CHECK-NEXT: add x29, sp, #8
-; CHECK-NEXT: str x19, [sp, #24] // 8-byte Folded Spill
-; CHECK-NEXT: sub sp, sp, #16
-; CHECK-NEXT: addvl sp, sp, #-1
-; CHECK-NEXT: mov x19, sp
-; CHECK-NEXT: .cfi_def_cfa w29, 24
-; CHECK-NEXT: .cfi_offset w19, -8
-; CHECK-NEXT: .cfi_offset w30, -16
-; CHECK-NEXT: .cfi_offset w29, -24
-; CHECK-NEXT: .cfi_offset b8, -32
-; CHECK-NEXT: // kill: def $w0 killed $w0 def $x0
-; CHECK-NEXT: ubfiz x8, x0, #2, #32
-; CHECK-NEXT: mov x9, sp
-; CHECK-NEXT: add x8, x8, #15
-; CHECK-NEXT: and x8, x8, #0x7fffffff0
-; CHECK-NEXT: sub x9, x9, x8
-; CHECK-NEXT: mov sp, x9
-; CHECK-NEXT: mov x10, sp
-; CHECK-NEXT: sub x8, x10, x8
-; CHECK-NEXT: mov sp, x8
-; CHECK-NEXT: mov z1.s, #0 // =0x0
-; CHECK-NEXT: //APP
-; CHECK-NEXT: //NO_APP
-; CHECK-NEXT: str wzr, [x8]
-; CHECK-NEXT: sub x8, x29, #8
-; CHECK-NEXT: mov w0, wzr
-; CHECK-NEXT: str wzr, [x9]
-; CHECK-NEXT: str d0, [x19, #8]
-; CHECK-NEXT: str z1, [x8, #-1, mul vl]
-; CHECK-NEXT: sub sp, x29, #8
-; CHECK-NEXT: ldp x29, x30, [sp, #8] // 16-byte Folded Reload
-; CHECK-NEXT: ldr x19, [sp, #24] // 8-byte Folded Reload
-; CHECK-NEXT: ldr d8, [sp], #32 // 8-byte Folded Reload
-; CHECK-NEXT: ret
+; CHECK-COMMON-LABEL: csr_d8_allocnxv4i32i32f64_vla:
+; CHECK-COMMON: // %bb.0: // %entry
+; CHECK-COMMON-NEXT: str d8, [sp, #-32]! // 8-byte Folded Spill
+; CHECK-COMMON-NEXT: stp x29, x30, [sp, #8] // 16-byte Folded Spill
+; CHECK-COMMON-NEXT: add x29, sp, #8
+; CHECK-COMMON-NEXT: str x19, [sp, #24] // 8-byte Folded Spill
+; CHECK-COMMON-NEXT: sub sp, sp, #16
+; CHECK-COMMON-NEXT: addvl sp, sp, #-1
+; CHECK-COMMON-NEXT: mov x19, sp
+; CHECK-COMMON-NEXT: .cfi_def_cfa w29, 24
+; CHECK-COMMON-NEXT: .cfi_offset w19, -8
+; CHECK-COMMON-NEXT: .cfi_offset w30, -16
+; CHECK-COMMON-NEXT: .cfi_offset w29, -24
+; CHECK-COMMON-NEXT: .cfi_offset b8, -32
+; CHECK-COMMON-NEXT: // kill: def $w0 killed $w0 def $x0
+; CHECK-COMMON-NEXT: ubfiz x8, x0, #2, #32
+; CHECK-COMMON-NEXT: mov x9, sp
+; CHECK-COMMON-NEXT: add x8, x8, #15
+; CHECK-COMMON-NEXT: and x8, x8, #0x7fffffff0
+; CHECK-COMMON-NEXT: sub x9, x9, x8
+; CHECK-COMMON-NEXT: mov sp, x9
+; CHECK-COMMON-NEXT: mov x10, sp
+; CHECK-COMMON-NEXT: sub x8, x10, x8
+; CHECK-COMMON-NEXT: mov sp, x8
+; CHECK-COMMON-NEXT: mov z1.s, #0 // =0x0
+; CHECK-COMMON-NEXT: //APP
+; CHECK-COMMON-NEXT: //NO_APP
+; CHECK-COMMON-NEXT: str wzr, [x8]
+; CHECK-COMMON-NEXT: sub x8, x29, #8
+; CHECK-COMMON-NEXT: mov w0, wzr
+; CHECK-COMMON-NEXT: str wzr, [x9]
+; CHECK-COMMON-NEXT: str d0, [x19, #8]
+; CHECK-COMMON-NEXT: str z1, [x8, #-1, mul vl]
+; CHECK-COMMON-NEXT: sub sp, x29, #8
+; CHECK-COMMON-NEXT: ldp x29, x30, [sp, #8] // 16-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr x19, [sp, #24] // 8-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr d8, [sp], #32 // 8-byte Folded Reload
+; CHECK-COMMON-NEXT: ret
entry:
%a = alloca <vscale x 4 x i32>
%0 = zext i32 %i to i64
@@ -213,28 +214,28 @@ entry:
; CHECK-FRAMELAYOUT-NEXT: Offset: [SP-32-16 x vscale], Type: Variable, Align: 8, Size: 8
define i32 @csr_d8_allocnxv4i32i32f64_stackargsi32f64(double %d0, double %d1, double %d2, double %d3, double %d4, double %d5, double %d6, double %d7, double %d8, i32 %i0, i32 %i1, i32 %i2, i32 %i3, i32 %i4, i32 %i5, i32 %i6, i32 %i7, i32 %i8) "aarch64_pstate_sm_compatible" {
-; CHECK-LABEL: csr_d8_allocnxv4i32i32f64_stackargsi32f64:
-; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: str d8, [sp, #-16]! // 8-byte Folded Spill
-; CHECK-NEXT: str x29, [sp, #8] // 8-byte Folded Spill
-; CHECK-NEXT: sub sp, sp, #16
-; CHECK-NEXT: addvl sp, sp, #-1
-; CHECK-NEXT: .cfi_escape 0x0f, 0x0c, 0x8f, 0x00, 0x11, 0x20, 0x22, 0x11, 0x08, 0x92, 0x2e, 0x00, 0x1e, 0x22 // sp + 32 + 8 * VG
-; CHECK-NEXT: .cfi_offset w29, -8
-; CHECK-NEXT: .cfi_offset b8, -16
-; CHECK-NEXT: mov z1.s, #0 // =0x0
-; CHECK-NEXT: add x8, sp, #16
-; CHECK-NEXT: mov w0, wzr
-; CHECK-NEXT: //APP
-; CHECK-NEXT: //NO_APP
-; CHECK-NEXT: str wzr, [sp, #12]
-; CHECK-NEXT: str d0, [sp]
-; CHECK-NEXT: str z1, [x8]
-; CHECK-NEXT: addvl sp, sp, #1
-; CHECK-NEXT: add sp, sp, #16
-; CHECK-NEXT: ldr x29, [sp, #8] // 8-byte Folded Reload
-; CHECK-NEXT: ldr d8, [sp], #16 // 8-byte Folded Reload
-; CHECK-NEXT: ret
+; CHECK-COMMON-LABEL: csr_d8_allocnxv4i32i32f64_stackargsi32f64:
+; CHECK-COMMON: // %bb.0: // %entry
+; CHECK-COMMON-NEXT: str d8, [sp, #-16]! // 8-byte Folded Spill
+; CHECK-COMMON-NEXT: str x29, [sp, #8] // 8-byte Folded Spill
+; CHECK-COMMON-NEXT: sub sp, sp, #16
+; CHECK-COMMON-NEXT: addvl sp, sp, #-1
+; CHECK-COMMON-NEXT: .cfi_escape 0x0f, 0x0c, 0x8f, 0x00, 0x11, 0x20, 0x22, 0x11, 0x08, 0x92, 0x2e, 0x00, 0x1e, 0x22 // sp + 32 + 8 * VG
+; CHECK-COMMON-NEXT: .cfi_offset w29, -8
+; CHECK-COMMON-NEXT: .cfi_offset b8, -16
+; CHECK-COMMON-NEXT: mov z1.s, #0 // =0x0
+; CHECK-COMMON-NEXT: add x8, sp, #16
+; CHECK-COMMON-NEXT: mov w0, wzr
+; CHECK-COMMON-NEXT: //APP
+; CHECK-COMMON-NEXT: //NO_APP
+; CHECK-COMMON-NEXT: str wzr, [sp, #12]
+; CHECK-COMMON-NEXT: str d0, [sp]
+; CHECK-COMMON-NEXT: str z1, [x8]
+; CHECK-COMMON-NEXT: addvl sp, sp, #1
+; CHECK-COMMON-NEXT: add sp, sp, #16
+; CHECK-COMMON-NEXT: ldr x29, [sp, #8] // 8-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr d8, [sp], #16 // 8-byte Folded Reload
+; CHECK-COMMON-NEXT: ret
entry:
%a = alloca <vscale x 4 x i32>
%b = alloca i32
@@ -255,29 +256,29 @@ entry:
; CHECK-FRAMELAYOUT-NEXT: Offset: [SP-32-32 x vscale], Type: Variable, Align: 8, Size: 8
define i32 @svecc_z8_allocnxv4i32i32f64_fp(double %d, <vscale x 4 x i32> %v) "aarch64_pstate_sm_compatible" "frame-pointer"="all" {
-; CHECK-LABEL: svecc_z8_allocnxv4i32i32f64_fp:
-; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
-; CHECK-NEXT: mov x29, sp
-; CHECK-NEXT: addvl sp, sp, #-1
-; CHECK-NEXT: str z8, [sp] // 16-byte Folded Spill
-; CHECK-NEXT: sub sp, sp, #16
-; CHECK-NEXT: addvl sp, sp, #-1
-; CHECK-NEXT: .cfi_def_cfa w29, 16
-; CHECK-NEXT: .cfi_offset w30, -8
-; CHECK-NEXT: .cfi_offset w29, -16
-; CHECK-NEXT: .cfi_escape 0x10, 0x48, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x78, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d8 @ cfa - 16 - 8 * VG
-; CHECK-NEXT: mov w0, wzr
-; CHECK-NEXT: //APP
-; CHECK-NEXT: //NO_APP
-; CHECK-NEXT: str wzr, [sp, #12]
-; CHECK-NEXT: str z1, [x29, #-2, mul vl]
-; CHECK-NEXT: str d0, [sp], #16
-; CHECK-NEXT: addvl sp, sp, #1
-; CHECK-NEXT: ldr z8, [sp] // 16-byte Folded Reload
-; CHECK-NEXT: addvl sp, sp, #1
-; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
-; CHECK-NEXT: ret
+; CHECK-COMMON-LABEL: svecc_z8_allocnxv4i32i32f64_fp:
+; CHECK-COMMON: // %bb.0: // %entry
+; CHECK-COMMON-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-COMMON-NEXT: mov x29, sp
+; CHECK-COMMON-NEXT: addvl sp, sp, #-1
+; CHECK-COMMON-NEXT: str z8, [sp] // 16-byte Folded Spill
+; CHECK-COMMON-NEXT: sub sp, sp, #16
+; CHECK-COMMON-NEXT: addvl sp, sp, #-1
+; CHECK-COMMON-NEXT: .cfi_def_cfa w29, 16
+; CHECK-COMMON-NEXT: .cfi_offset w30, -8
+; CHECK-COMMON-NEXT: .cfi_offset w29, -16
+; CHECK-COMMON-NEXT: .cfi_escape 0x10, 0x48, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x78, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d8 @ cfa - 16 - 8 * VG
+; CHECK-COMMON-NEXT: mov w0, wzr
+; CHECK-COMMON-NEXT: //APP
+; CHECK-COMMON-NEXT: //NO_APP
+; CHECK-COMMON-NEXT: str wzr, [sp, #12]
+; CHECK-COMMON-NEXT: str z1, [x29, #-2, mul vl]
+; CHECK-COMMON-NEXT: str d0, [sp], #16
+; CHECK-COMMON-NEXT: addvl sp, sp, #1
+; CHECK-COMMON-NEXT: ldr z8, [sp] // 16-byte Folded Reload
+; CHECK-COMMON-NEXT: addvl sp, sp, #1
+; CHECK-COMMON-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-COMMON-NEXT: ret
entry:
%a = alloca <vscale x 4 x i32>
%b = alloca i32
@@ -299,29 +300,29 @@ entry:
; CHECK-FRAMELAYOUT-NEXT: Offset: [SP-32-32 x vscale], Type: Variable, Align: 8, Size: 8
define i32 @svecc_z8_allocnxv4i32i32f64_stackargsi32_fp(double %d, i32 %i0, i32 %i1, i32 %i2, i32 %i3, i32 %i4, i32 %i5, i32 %i6, i32 %i7, i32 %i8, <vscale x 4 x i32> %v) "aarch64_pstate_sm_compatible" "frame-pointer"="all"{
-; CHECK-LABEL: svecc_z8_allocnxv4i32i32f64_stackargsi32_fp:
-; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
-; CHECK-NEXT: mov x29, sp
-; CHECK-NEXT: addvl sp, sp, #-1
-; CHECK-NEXT: str z8, [sp] // 16-byte Folded Spill
-; CHECK-NEXT: sub sp, sp, #16
-; CHECK-NEXT: addvl sp, sp, #-1
-; CHECK-NEXT: .cfi_def_cfa w29, 16
-; CHECK-NEXT: .cfi_offset w30, -8
-; CHECK-NEXT: .cfi_offset w29, -16
-; CHECK-NEXT: .cfi_escape 0x10, 0x48, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x78, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d8 @ cfa - 16 - 8 * VG
-; CHECK-NEXT: mov w0, wzr
-; CHECK-NEXT: //APP
-; CHECK-NEXT: //NO_APP
-; CHECK-NEXT: str wzr, [sp, #12]
-; CHECK-NEXT: str z1, [x29, #-2, mul vl]
-; CHECK-NEXT: str d0, [sp], #16
-; CHECK-NEXT: addvl sp, sp, #1
-; CHECK-NEXT: ldr z8, [sp] // 16-byte Folded Reload
-; CHECK-NEXT: addvl sp, sp, #1
-; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
-; CHECK-NEXT: ret
+; CHECK-COMMON-LABEL: svecc_z8_allocnxv4i32i32f64_stackargsi32_fp:
+; CHECK-COMMON: // %bb.0: // %entry
+; CHECK-COMMON-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-COMMON-NEXT: mov x29, sp
+; CHECK-COMMON-NEXT: addvl sp, sp, #-1
+; CHECK-COMMON-NEXT: str z8, [sp] // 16-byte Folded Spill
+; CHECK-COMMON-NEXT: sub sp, sp, #16
+; CHECK-COMMON-NEXT: addvl sp, sp, #-1
+; CHECK-COMMON-NEXT: .cfi_def_cfa w29, 16
+; CHECK-COMMON-NEXT: .cfi_offset w30, -8
+; CHECK-COMMON-NEXT: .cfi_offset w29, -16
+; CHECK-COMMON-NEXT: .cfi_escape 0x10, 0x48, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x78, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d8 @ cfa - 16 - 8 * VG
+; CHECK-COMMON-NEXT: mov w0, wzr
+; CHECK-COMMON-NEXT: //APP
+; CHECK-COMMON-NEXT: //NO_APP
+; CHECK-COMMON-NEXT: str wzr, [sp, #12]
+; CHECK-COMMON-NEXT: str z1, [x29, #-2, mul vl]
+; CHECK-COMMON-NEXT: str d0, [sp], #16
+; CHECK-COMMON-NEXT: addvl sp, sp, #1
+; CHECK-COMMON-NEXT: ldr z8, [sp] // 16-byte Folded Reload
+; CHECK-COMMON-NEXT: addvl sp, sp, #1
+; CHECK-COMMON-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-COMMON-NEXT: ret
entry:
%a = alloca <vscale x 4 x i32>
%b = alloca i32
@@ -370,125 +371,125 @@ entry:
; CHECK-FRAMELAYOUT-NEXT: Offset: [SP-48-280 x vscale], Type: Spill, Align: 2, Size: vscale x 2
define i32 @svecc_call(<4 x i16> %P0, ptr %P1, i32 %P2, <vscale x 16 x i8> %P3, i16 %P4) "aarch64_pstate_sm_compatible" {
-; CHECK-LABEL: svecc_call:
-; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: stp x29, x30, [sp, #-48]! // 16-byte Folded Spill
-; CHECK-NEXT: .cfi_def_cfa_offset 48
-; CHECK-NEXT: cntd x9
-; CHECK-NEXT: stp x9, x28, [sp, #16] // 16-byte Folded Spill
-; CHECK-NEXT: stp x27, x19, [sp, #32] // 16-byte Folded Spill
-; CHECK-NEXT: .cfi_offset w19, -8
-; CHECK-NEXT: .cfi_offset w27, -16
-; CHECK-NEXT: .cfi_offset w28, -24
-; CHECK-NEXT: .cfi_offset w30, -40
-; CHECK-NEXT: .cfi_offset w29, -48
-; CHECK-NEXT: addvl sp, sp, #-18
-; CHECK-NEXT: .cfi_escape 0x0f, 0x0d, 0x8f, 0x00, 0x11, 0x30, 0x22, 0x11, 0x90, 0x01, 0x92, 0x2e, 0x00, 0x1e, 0x22 // sp + 48 + 144 * VG
-; CHECK-NEXT: str p15, [sp, #4, mul vl] // 2-byte Folded Spill
-; CHECK-NEXT: str p14, [sp, #5, mul vl] // 2-byte Folded Spill
-; CHECK-NEXT: str p13, [sp, #6, mul vl] // 2-byte Folded Spill
-; CHECK-NEXT: str p12, [sp, #7, mul vl] // 2-byte Folded Spill
-; CHECK-NEXT: str p11, [sp, #8, mul vl] // 2-byte Folded Spill
-; CHECK-NEXT: str p10, [sp, #9, mul vl] // 2-byte Folded Spill
-; CHECK-NEXT: str p9, [sp, #10, mul vl] // 2-byte Folded Spill
-; CHECK-NEXT: str p8, [sp, #11, mul vl] // 2-byte Folded Spill
-; CHECK-NEXT: str p7, [sp, #12, mul vl] // 2-byte Folded Spill
-; CHECK-NEXT: str p6, [sp, #13, mul vl] // 2-byte Folded Spill
-; CHECK-NEXT: str p5, [sp, #14, mul vl] // 2-byte Folded Spill
-; CHECK-NEXT: str p4, [sp, #15, mul vl] // 2-byte Folded Spill
-; CHECK-NEXT: str z23, [sp, #2, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: str z22, [sp, #3, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: str z21, [sp, #4, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: str z20, [sp, #5, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: str z19, [sp, #6, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: str z18, [sp, #7, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: str z17, [sp, #8, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: str z16, [sp, #9, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: str z15, [sp, #10, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: str z14, [sp, #11, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: str z13, [sp, #12, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: str z12, [sp, #13, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: str z11, [sp, #14, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: str z10, [sp, #15, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: str z9, [sp, #16, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: str z8, [sp, #17, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: .cfi_escape 0x10, 0x48, 0x0a, 0x11, 0x50, 0x22, 0x11, 0x78, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d8 @ cfa - 48 - 8 * VG
-; CHECK-NEXT: .cfi_escape 0x10, 0x49, 0x0a, 0x11, 0x50, 0x22, 0x11, 0x70, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d9 @ cfa - 48 - 16 * VG
-; CHECK-NEXT: .cfi_escape 0x10, 0x4a, 0x0a, 0x11, 0x50, 0x22, 0x11, 0x68, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d10 @ cfa - 48 - 24 * VG
-; CHECK-NEXT: .cfi_escape 0x10, 0x4b, 0x0a, 0x11, 0x50, 0x22, 0x11, 0x60, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d11 @ cfa - 48 - 32 * VG
-; CHECK-NEXT: .cfi_escape 0x10, 0x4c, 0x0a, 0x11, 0x50, 0x22, 0x11, 0x58, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d12 @ cfa - 48 - 40 * VG
-; CHECK-NEXT: .cfi_escape 0x10, 0x4d, 0x0a, 0x11, 0x50, 0x22, 0x11, 0x50, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d13 @ cfa - 48 - 48 * VG
-; CHECK-NEXT: .cfi_escape 0x10, 0x4e, 0x0a, 0x11, 0x50, 0x22, 0x11, 0x48, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d14 @ cfa - 48 - 56 * VG
-; CHECK-NEXT: .cfi_escape 0x10, 0x4f, 0x0a, 0x11, 0x50, 0x22, 0x11, 0x40, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d15 @ cfa - 48 - 64 * VG
-; CHECK-NEXT: mov x8, x0
-; CHECK-NEXT: //APP
-; CHECK-NEXT: //NO_APP
-; CHECK-NEXT: bl __arm_sme_state
-; CHECK-NEXT: and x19, x0, #0x1
-; CHECK-NEXT: .cfi_offset vg, -32
-; CHECK-NEXT: tbz w19, #0, .LBB7_2
-; CHECK-NEXT: // %bb.1: // %entry
-; CHECK-NEXT: smstop sm
-; CHECK-NEXT: .LBB7_2: // %entry
-; CHECK-NEXT: mov x0, x8
-; CHECK-NEXT: mov w1, #45 // =0x2d
-; CHECK-NEXT: mov w2, #37 // =0x25
-; CHECK-NEXT: bl memset
-; CHECK-NEXT: tbz w19, #0, .LBB7_4
-; CHECK-NEXT: // %bb.3: // %entry
-; CHECK-NEXT: smstart sm
-; CHECK-NEXT: .LBB7_4: // %entry
-; CHECK-NEXT: mov w0, #22647 // =0x5877
-; CHECK-NEXT: movk w0, #59491, lsl #16
-; CHECK-NEXT: .cfi_restore vg
-; CHECK-NEXT: ldr z23, [sp, #2, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z22, [sp, #3, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z21, [sp, #4, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z20, [sp, #5, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z19, [sp, #6, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z18, [sp, #7, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z17, [sp, #8, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z16, [sp, #9, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z15, [sp, #10, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z14, [sp, #11, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z13, [sp, #12, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z12, [sp, #13, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z11, [sp, #14, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z10, [sp, #15, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z9, [sp, #16, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z8, [sp, #17, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr p15, [sp, #4, mul vl] // 2-byte Folded Reload
-; CHECK-NEXT: ldr p14, [sp, #5, mul vl] // 2-byte Folded Reload
-; CHECK-NEXT: ldr p13, [sp, #6, mul vl] // 2-byte Folded Reload
-; CHECK-NEXT: ldr p12, [sp, #7, mul vl] // 2-byte Folded Reload
-; CHECK-NEXT: ldr p11, [sp, #8, mul vl] // 2-byte Folded Reload
-; CHECK-NEXT: ldr p10, [sp, #9, mul vl] // 2-byte Folded Reload
-; CHECK-NEXT: ldr p9, [sp, #10, mul vl] // 2-byte Folded Reload
-; CHECK-NEXT: ldr p8, [sp, #11, mul vl] // 2-byte Folded Reload
-; CHECK-NEXT: ldr p7, [sp, #12, mul vl] // 2-byte Folded Reload
-; CHECK-NEXT: ldr p6, [sp, #13, mul vl] // 2-byte Folded Reload
-; CHECK-NEXT: ldr p5, [sp, #14, mul vl] // 2-byte Folded Reload
-; CHECK-NEXT: ldr p4, [sp, #15, mul vl] // 2-byte Folded Reload
-; CHECK-NEXT: addvl sp, sp, #18
-; CHECK-NEXT: .cfi_def_cfa wsp, 48
-; CHECK-NEXT: .cfi_restore z8
-; CHECK-NEXT: .cfi_restore z9
-; CHECK-NEXT: .cfi_restore z10
-; CHECK-NEXT: .cfi_restore z11
-; CHECK-NEXT: .cfi_restore z12
-; CHECK-NEXT: .cfi_restore z13
-; CHECK-NEXT: .cfi_restore z14
-; CHECK-NEXT: .cfi_restore z15
-; CHECK-NEXT: ldp x27, x19, [sp, #32] // 16-byte Folded Reload
-; CHECK-NEXT: ldr x28, [sp, #24] // 8-byte Folded Reload
-; CHECK-NEXT: ldp x29, x30, [sp], #48 // 16-byte Folded Reload
-; CHECK-NEXT: .cfi_def_cfa_offset 0
-; CHECK-NEXT: .cfi_restore w19
-; CHECK-NEXT: .cfi_restore w27
-; CHECK-NEXT: .cfi_restore w28
-; CHECK-NEXT: .cfi_restore w30
-; CHECK-NEXT: .cfi_restore w29
-; CHECK-NEXT: ret
+; CHECK-COMMON-LABEL: svecc_call:
+; CHECK-COMMON: // %bb.0: // %entry
+; CHECK-COMMON-NEXT: stp x29, x30, [sp, #-48]! // 16-byte Folded Spill
+; CHECK-COMMON-NEXT: .cfi_def_cfa_offset 48
+; CHECK-COMMON-NEXT: cntd x9
+; CHECK-COMMON-NEXT: stp x9, x28, [sp, #16] // 16-byte Folded Spill
+; CHECK-COMMON-NEXT: stp x27, x19, [sp, #32] // 16-byte Folded Spill
+; CHECK-COMMON-NEXT: .cfi_offset w19, -8
+; CHECK-COMMON-NEXT: .cfi_offset w27, -16
+; CHECK-COMMON-NEXT: .cfi_offset w28, -24
+; CHECK-COMMON-NEXT: .cfi_offset w30, -40
+; CHECK-COMMON-NEXT: .cfi_offset w29, -48
+; CHECK-COMMON-NEXT: addvl sp, sp, #-18
+; CHECK-COMMON-NEXT: .cfi_escape 0x0f, 0x0d, 0x8f, 0x00, 0x11, 0x30, 0x22, 0x11, 0x90, 0x01, 0x92, 0x2e, 0x00, 0x1e, 0x22 // sp + 48 + 144 * VG
+; CHECK-COMMON-NEXT: str p15, [sp, #4, mul vl] // 2-byte Folded Spill
+; CHECK-COMMON-NEXT: str p14, [sp, #5, mul vl] // 2-byte Folded Spill
+; CHECK-COMMON-NEXT: str p13, [sp, #6, mul vl] // 2-byte Folded Spill
+; CHECK-COMMON-NEXT: str p12, [sp, #7, mul vl] // 2-byte Folded Spill
+; CHECK-COMMON-NEXT: str p11, [sp, #8, mul vl] // 2-byte Folded Spill
+; CHECK-COMMON-NEXT: str p10, [sp, #9, mul vl] // 2-byte Folded Spill
+; CHECK-COMMON-NEXT: str p9, [sp, #10, mul vl] // 2-byte Folded Spill
+; CHECK-COMMON-NEXT: str p8, [sp, #11, mul vl] // 2-byte Folded Spill
+; CHECK-COMMON-NEXT: str p7, [sp, #12, mul vl] // 2-byte Folded Spill
+; CHECK-COMMON-NEXT: str p6, [sp, #13, mul vl] // 2-byte Folded Spill
+; CHECK-COMMON-NEXT: str p5, [sp, #14, mul vl] // 2-byte Folded Spill
+; CHECK-COMMON-NEXT: str p4, [sp, #15, mul vl] // 2-byte Folded Spill
+; CHECK-COMMON-NEXT: str z23, [sp, #2, mul vl] // 16-byte Folded Spill
+; CHECK-COMMON-NEXT: str z22, [sp, #3, mul vl] // 16-byte Folded Spill
+; CHECK-COMMON-NEXT: str z21, [sp, #4, mul vl] // 16-byte Folded Spill
+; CHECK-COMMON-NEXT: str z20, [sp, #5, mul vl] // 16-byte Folded Spill
+; CHECK-COMMON-NEXT: str z19, [sp, #6, mul vl] // 16-byte Folded Spill
+; CHECK-COMMON-NEXT: str z18, [sp, #7, mul vl] // 16-byte Folded Spill
+; CHECK-COMMON-NEXT: str z17, [sp, #8, mul vl] // 16-byte Folded Spill
+; CHECK-COMMON-NEXT: str z16, [sp, #9, mul vl] // 16-byte Folded Spill
+; CHECK-COMMON-NEXT: str z15, [sp, #10, mul vl] // 16-byte Folded Spill
+; CHECK-COMMON-NEXT: str z14, [sp, #11, mul vl] // 16-byte Folded Spill
+; CHECK-COMMON-NEXT: str z13, [sp, #12, mul vl] // 16-byte Folded Spill
+; CHECK-COMMON-NEXT: str z12, [sp, #13, mul vl] // 16-byte Folded Spill
+; CHECK-COMMON-NEXT: str z11, [sp, #14, mul vl] // 16-byte Folded Spill
+; CHECK-COMMON-NEXT: str z10, [sp, #15, mul vl] // 16-byte Folded Spill
+; CHECK-COMMON-NEXT: str z9, [sp, #16, mul vl] // 16-byte Folded Spill
+; CHECK-COMMON-NEXT: str z8, [sp, #17, mul vl] // 16-byte Folded Spill
+; CHECK-COMMON-NEXT: .cfi_escape 0x10, 0x48, 0x0a, 0x11, 0x50, 0x22, 0x11, 0x78, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d8 @ cfa - 48 - 8 * VG
+; CHECK-COMMON-NEXT: .cfi_escape 0x10, 0x49, 0x0a, 0x11, 0x50, 0x22, 0x11, 0x70, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d9 @ cfa - 48 - 16 * VG
+; CHECK-COMMON-NEXT: .cfi_escape 0x10, 0x4a, 0x0a, 0x11, 0x50, 0x22, 0x11, 0x68, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d10 @ cfa - 48 - 24 * VG
+; CHECK-COMMON-NEXT: .cfi_escape 0x10, 0x4b, 0x0a, 0x11, 0x50, 0x22, 0x11, 0x60, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d11 @ cfa - 48 - 32 * VG
+; CHECK-COMMON-NEXT: .cfi_escape 0x10, 0x4c, 0x0a, 0x11, 0x50, 0x22, 0x11, 0x58, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d12 @ cfa - 48 - 40 * VG
+; CHECK-COMMON-NEXT: .cfi_escape 0x10, 0x4d, 0x0a, 0x11, 0x50, 0x22, 0x11, 0x50, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d13 @ cfa - 48 - 48 * VG
+; CHECK-COMMON-NEXT: .cfi_escape 0x10, 0x4e, 0x0a, 0x11, 0x50, 0x22, 0x11, 0x48, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d14 @ cfa - 48 - 56 * VG
+; CHECK-COMMON-NEXT: .cfi_escape 0x10, 0x4f, 0x0a, 0x11, 0x50, 0x22, 0x11, 0x40, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d15 @ cfa - 48 - 64 * VG
+; CHECK-COMMON-NEXT: mov x8, x0
+; CHECK-COMMON-NEXT: //APP
+; CHECK-COMMON-NEXT: //NO_APP
+; CHECK-COMMON-NEXT: bl __arm_sme_state
+; CHECK-COMMON-NEXT: and x19, x0, #0x1
+; CHECK-COMMON-NEXT: .cfi_offset vg, -32
+; CHECK-COMMON-NEXT: tbz w19, #0, .LBB7_2
+; CHECK-COMMON-NEXT: // %bb.1: // %entry
+; CHECK-COMMON-NEXT: smstop sm
+; CHECK-COMMON-NEXT: .LBB7_2: // %entry
+; CHECK-COMMON-NEXT: mov x0, x8
+; CHECK-COMMON-NEXT: mov w1, #45 // =0x2d
+; CHECK-COMMON-NEXT: mov w2, #37 // =0x25
+; CHECK-COMMON-NEXT: bl memset
+; CHECK-COMMON-NEXT: tbz w19, #0, .LBB7_4
+; CHECK-COMMON-NEXT: // %bb.3: // %entry
+; CHECK-COMMON-NEXT: smstart sm
+; CHECK-COMMON-NEXT: .LBB7_4: // %entry
+; CHECK-COMMON-NEXT: mov w0, #22647 // =0x5877
+; CHECK-COMMON-NEXT: movk w0, #59491, lsl #16
+; CHECK-COMMON-NEXT: .cfi_restore vg
+; CHECK-COMMON-NEXT: ldr z23, [sp, #2, mul vl] // 16-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr z22, [sp, #3, mul vl] // 16-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr z21, [sp, #4, mul vl] // 16-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr z20, [sp, #5, mul vl] // 16-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr z19, [sp, #6, mul vl] // 16-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr z18, [sp, #7, mul vl] // 16-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr z17, [sp, #8, mul vl] // 16-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr z16, [sp, #9, mul vl] // 16-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr z15, [sp, #10, mul vl] // 16-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr z14, [sp, #11, mul vl] // 16-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr z13, [sp, #12, mul vl] // 16-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr z12, [sp, #13, mul vl] // 16-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr z11, [sp, #14, mul vl] // 16-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr z10, [sp, #15, mul vl] // 16-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr z9, [sp, #16, mul vl] // 16-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr z8, [sp, #17, mul vl] // 16-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr p15, [sp, #4, mul vl] // 2-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr p14, [sp, #5, mul vl] // 2-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr p13, [sp, #6, mul vl] // 2-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr p12, [sp, #7, mul vl] // 2-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr p11, [sp, #8, mul vl] // 2-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr p10, [sp, #9, mul vl] // 2-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr p9, [sp, #10, mul vl] // 2-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr p8, [sp, #11, mul vl] // 2-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr p7, [sp, #12, mul vl] // 2-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr p6, [sp, #13, mul vl] // 2-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr p5, [sp, #14, mul vl] // 2-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr p4, [sp, #15, mul vl] // 2-byte Folded Reload
+; CHECK-COMMON-NEXT: addvl sp, sp, #18
+; CHECK-COMMON-NEXT: .cfi_def_cfa wsp, 48
+; CHECK-COMMON-NEXT: .cfi_restore z8
+; CHECK-COMMON-NEXT: .cfi_restore z9
+; CHECK-COMMON-NEXT: .cfi_restore z10
+; CHECK-COMMON-NEXT: .cfi_restore z11
+; CHECK-COMMON-NEXT: .cfi_restore z12
+; CHECK-COMMON-NEXT: .cfi_restore z13
+; CHECK-COMMON-NEXT: .cfi_restore z14
+; CHECK-COMMON-NEXT: .cfi_restore z15
+; CHECK-COMMON-NEXT: ldp x27, x19, [sp, #32] // 16-byte Folded Reload
+; CHECK-COMMON-NEXT: ldr x28, [sp, #24] // 8-byte Folded Reload
+; CHECK-COMMON-NEXT: ldp x29, x30, [sp], #48 // 16-byte Folded Reload
+; CHECK-COMMON-NEXT: .cfi_def_cfa_offset 0
+; CHECK-COMMON-NEXT: .cfi_restore w19
+; CHECK-COMMON-NEXT: .cfi_restore w27
+; CHECK-COMMON-NEXT: .cfi_restore w28
+; CHECK-COMMON-NEXT: .cfi_restore w30
+; CHECK-COMMON-NEXT: .cfi_restore w29
+; CHECK-COMMON-NEXT: ret
entry:
tail call void asm sideeffect "", "~{x0},~{x28},~{x27},~{x3}"() #2
%call = call ptr @memset(ptr noundef nonnull %P1, i32 noundef 45, i32 noundef 37)
@@ -590,6 +591,79 @@ define i32 @vastate(i32 %x) "aarch64_inout_za" "aarch64_pstate_sm_enabled" "targ
; CHECK-NEXT: .cfi_restore b14
; CHECK-NEXT: .cfi_restore b15
; CHECK-NEXT: ret
+;
+; CHECK-NEWLOWERING-LABEL: vastate:
+; CHECK-NEWLOWERING: // %bb.0: // %entry
+; CHECK-NEWLOWERING-NEXT: stp d15, d14, [sp, #-112]! // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: .cfi_def_cfa_offset 112
+; CHECK-NEWLOWERING-NEXT: cntd x9
+; CHECK-NEWLOWERING-NEXT: stp d13, d12, [sp, #16] // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: stp d11, d10, [sp, #32] // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: stp d9, d8, [sp, #48] // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: stp x29, x30, [sp, #64] // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: str x9, [sp, #80] // 8-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: stp x20, x19, [sp, #96] // 16-byte Folded Spill
+; CHECK-NEWLOWERING-NEXT: add x29, sp, #64
+; CHECK-NEWLOWERING-NEXT: .cfi_def_cfa w29, 48
+; CHECK-NEWLOWERING-NEXT: .cfi_offset w19, -8
+; CHECK-NEWLOWERING-NEXT: .cfi_offset w20, -16
+; CHECK-NEWLOWERING-NEXT: .cfi_offset w30, -40
+; CHECK-NEWLOWERING-NEXT: .cfi_offset w29, -48
+; CHECK-NEWLOWERING-NEXT: .cfi_offset b8, -56
+; CHECK-NEWLOWERING-NEXT: .cfi_offset b9, -64
+; CHECK-NEWLOWERING-NEXT: .cfi_offset b10, -72
+; CHECK-NEWLOWERING-NEXT: .cfi_offset b11, -80
+; CHECK-NEWLOWERING-NEXT: .cfi_offset b12, -88
+; CHECK-NEWLOWERING-NEXT: .cfi_offset b13, -96
+; CHECK-NEWLOWERING-NEXT: .cfi_offset b14, -104
+; CHECK-NEWLOWERING-NEXT: .cfi_offset b15, -112
+; CHECK-NEWLOWERING-NEXT: sub sp, sp, #16
+; CHECK-NEWLOWERING-NEXT: rdsvl x8, #1
+; CHECK-NEWLOWERING-NEXT: mov x9, sp
+; CHECK-NEWLOWERING-NEXT: msub x9, x8, x8, x9
+; CHECK-NEWLOWERING-NEXT: mov sp, x9
+; CHECK-NEWLOWERING-NEXT: sub x10, x29, #80
+; CHECK-NEWLOWERING-NEXT: mov w20, w0
+; CHECK-NEWLOWERING-NEXT: stp x9, x8, [x29, #-80]
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, x10
+; CHECK-NEWLOWERING-NEXT: .cfi_offset vg, -32
+; CHECK-NEWLOWERING-NEXT: smstop sm
+; CHECK-NEWLOWERING-NEXT: bl other
+; CHECK-NEWLOWERING-NEXT: smstart sm
+; CHECK-NEWLOWERING-NEXT: mov w0, w20
+; CHECK-NEWLOWERING-NEXT: mov w8, w0
+; CHECK-NEWLOWERING-NEXT: .cfi_restore vg
+; CHECK-NEWLOWERING-NEXT: smstart za
+; CHECK-NEWLOWERING-NEXT: mrs x9, TPIDR2_EL0
+; CHECK-NEWLOWERING-NEXT: sub x0, x29, #80
+; CHECK-NEWLOWERING-NEXT: cbnz x9, .LBB8_2
+; CHECK-NEWLOWERING-NEXT: // %bb.1: // %entry
+; CHECK-NEWLOWERING-NEXT: bl __arm_tpidr2_restore
+; CHECK-NEWLOWERING-NEXT: .LBB8_2: // %entry
+; CHECK-NEWLOWERING-NEXT: mov w0, w8
+; CHECK-NEWLOWERING-NEXT: msr TPIDR2_EL0, xzr
+; CHECK-NEWLOWERING-NEXT: sub sp, x29, #64
+; CHECK-NEWLOWERING-NEXT: .cfi_def_cfa wsp, 112
+; CHECK-NEWLOWERING-NEXT: ldp x20, x19, [sp, #96] // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ldp x29, x30, [sp, #64] // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ldp d9, d8, [sp, #48] // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ldp d11, d10, [sp, #32] // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ldp d13, d12, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: ldp d15, d14, [sp], #112 // 16-byte Folded Reload
+; CHECK-NEWLOWERING-NEXT: .cfi_def_cfa_offset 0
+; CHECK-NEWLOWERING-NEXT: .cfi_restore w19
+; CHECK-NEWLOWERING-NEXT: .cfi_restore w20
+; CHECK-NEWLOWERING-NEXT: .cfi_restore w30
+; CHECK-NEWLOWERING-NEXT: .cfi_restore w29
+; CHECK-NEWLOWERING-NEXT: .cfi_restore b8
+; CHECK-NEWLOWERING-NEXT: .cfi_restore b9
+; CHECK-NEWLOWERING-NEXT: .cfi_restore b10
+; CHECK-NEWLOWERING-NEXT: .cfi_restore b11
+; CHECK-NEWLOWERING-NEXT: .cfi_restore b12
+; CHECK-NEWLOWERING-NEXT: .cfi_restore b13
+; CHECK-NEWLOWERING-NEXT: .cfi_restore b14
+; CHECK-NEWLOWERING-NEXT: .cfi_restore b15
+; CHECK-NEWLOWERING-NEXT: ret
entry:
tail call void @other()
ret i32 %x
diff --git a/llvm/unittests/Target/AArch64/SMEAttributesTest.cpp b/llvm/unittests/Target/AArch64/SMEAttributesTest.cpp
index f13252f3a4c28..93ad6874c15c0 100644
--- a/llvm/unittests/Target/AArch64/SMEAttributesTest.cpp
+++ b/llvm/unittests/Target/AArch64/SMEAttributesTest.cpp
@@ -309,7 +309,7 @@ TEST(SMEAttributes, Transitions) {
// Shared ZA -> Private ZA Interface
ASSERT_FALSE(CA(ZA_Shared, Private_ZA).requiresDisablingZABeforeCall());
- ASSERT_TRUE(CA(ZA_Shared, Private_ZA).requiresEnablingZAAfterCall());
+ ASSERT_FALSE(CA(ZA_Shared, Private_ZA).requiresEnablingZAAfterCall());
// Shared ZT0 -> Private ZA Interface
ASSERT_TRUE(CA(ZT0_Shared, Private_ZA).requiresDisablingZABeforeCall());
@@ -328,7 +328,7 @@ TEST(SMEAttributes, Transitions) {
// Shared ZA & ZT0 -> Private ZA Interface
ASSERT_FALSE(CA(ZA_ZT0_Shared, Private_ZA).requiresDisablingZABeforeCall());
ASSERT_TRUE(CA(ZA_ZT0_Shared, Private_ZA).requiresPreservingZT0());
- ASSERT_TRUE(CA(ZA_ZT0_Shared, Private_ZA).requiresEnablingZAAfterCall());
+ ASSERT_FALSE(CA(ZA_ZT0_Shared, Private_ZA).requiresEnablingZAAfterCall());
// Shared ZA -> Shared ZA Interface
ASSERT_FALSE(CA(ZA_Shared, ZT0_Shared).requiresDisablingZABeforeCall());
More information about the llvm-commits
mailing list