[llvm] [AArch64][RegAlloc] Enable callee-saved register optimization for AArch64 (PR #214773)
Shreeyash Pandey via llvm-commits
llvm-commits at lists.llvm.org
Fri Aug 7 08:21:00 PDT 2026
https://github.com/bojle created https://github.com/llvm/llvm-project/pull/214773
This patch enables the AArch64 backend to use the new CSR (callee-saved register) cost model introduced in
https://github.com/llvm/llvm-project/pull/177226, which changed the computation of CSRCost.
This is a sister patch to https://github.com/llvm/llvm-project/pull/188609.
Fixes https://github.com/llvm/llvm-project/issues/214748.
I swept across different scale values to find the optimal one for this arch. The table below shows the % improvement in SPECrate vs. the trunk baseline (positive = faster). Values are median of 3 iterations.
```
bench 30 40 50 60 70 80 90* 100* 110 120
----------- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
perlbench_r +0.5 +1.2 +0.5 +1.7 +0.7 +1.0 +0.9 +2.1 +1.7 +2.1
gcc_r +0.4 +0.8 +1.0 +1.2 +1.3 +1.2 +0.9 +1.3 +1.5 +1.3
omnetpp_r +0.2 -0.7 0.0 -0.3 -1.2 -0.8 +0.2 -0.2 0.0 -0.8
mcf_r -0.7 -0.7 -0.7 -0.3 -0.8 -0.8 -0.7 -0.1 -0.3 -0.1
x264_r +0.6 +1.2 +1.2 +1.2 +1.2 +1.2 +1.2 +1.2 +1.2 +1.2
deepsjeng_r -0.5 -0.7 -0.5 -0.5 -0.5 -0.5 -0.7 -0.9 -1.3 -1.3
xz_r -1.6 -0.2 -0.9 -1.2 -0.7 -1.2 -0.5 -1.6 -1.4 -1.6
```
At scale 100, the wins are concentrated in the call-heavy benchmarks: perlbench_r (+2.1%), gcc_r (+1.3%), and x264_r (+1.2%), while omnetpp_r and mcf_r are essentially flat. The regressions at this scale are deepsjeng_r (−0.9%) and xz_r (−1.6%); both quite small, closer to run-to-run noise. CSR Value 100 looks like the most favourable scale value for this AArch64.
This change updates 14 AArch64 CodeGen tests, whose diffs are register-allocation churn from the new cost model.
At the moment, this change in the csr-cost-scale is applied globally to all targets (via the cl::init default), so CodeGen tests on other targets may also see similar churn. It was suggested in #188609 that the existing cost calculation be refactored so that each target can be enabled incrementally. Once that refactor lands, I'll rebase this to make the change AArch64-specific.
>From f2d4d0e2fc023db3b984331cbddb8c4425076dc3 Mon Sep 17 00:00:00 2001
From: Shreeyash Pandey <shrpand at qti.qualcomm.com>
Date: Wed, 5 Aug 2026 08:53:04 +0000
Subject: [PATCH] [AArch64][RegAlloc] Enable callee-saved register optimization
for AArch64
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
This patch enables the AArch64 backend to use the new CSR (callee-saved
register) cost model introduced in
https://github.com/llvm/llvm-project/pull/177226, which changed the
computation of CSRCost.
This is a sister patch to https://github.com/llvm/llvm-project/pull/188609.
Fixes https://github.com/llvm/llvm-project/issues/214748.
I swept across different scale values to find the optimal one for this
arch. The table below shows the % improvement in SPECrate vs. the trunk
baseline (positive = faster). Values are median of 3 iterations.
bench 30 40 50 60 70 80 90* 100* 110 120
----------- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
perlbench_r +0.5 +1.2 +0.5 +1.7 +0.7 +1.0 +0.9 +2.1 +1.7 +2.1
gcc_r +0.4 +0.8 +1.0 +1.2 +1.3 +1.2 +0.9 +1.3 +1.5 +1.3
omnetpp_r +0.2 -0.7 0.0 -0.3 -1.2 -0.8 +0.2 -0.2 0.0 -0.8
mcf_r -0.7 -0.7 -0.7 -0.3 -0.8 -0.8 -0.7 -0.1 -0.3 -0.1
x264_r +0.6 +1.2 +1.2 +1.2 +1.2 +1.2 +1.2 +1.2 +1.2 +1.2
deepsjeng_r -0.5 -0.7 -0.5 -0.5 -0.5 -0.5 -0.7 -0.9 -1.3 -1.3
xz_r -1.6 -0.2 -0.9 -1.2 -0.7 -1.2 -0.5 -1.6 -1.4 -1.6
At scale 100, the wins are concentrated in the call-heavy benchmarks:
perlbench_r (+2.1%), gcc_r (+1.3%), and x264_r (+1.2%), while omnetpp_r
and mcf_r are essentially flat. The regressions at this scale are
deepsjeng_r (−0.9%) and xz_r (−1.6%); both quite small, closer to
run-to-run noise. CSR Value 100 looks like the most favourable scale
value for this AArch64.
This change updates 14 AArch64 CodeGen tests, whose diffs are
register-allocation churn from the new cost model.
At the moment, this change in the csr-cost-scale is applied globally to
all targets (via the cl::init default), so CodeGen tests on other
targets may also see similar churn. It was suggested in #188609 that the
existing cost calculation be refactored so that each target can be
enabled incrementally. Once that refactor lands, I'll rebase this to
make the change AArch64-specific.
Signed-off-by: Shreeyash Pandey <shrpand at qti.qualcomm.com>
---
llvm/lib/CodeGen/RegAllocGreedy.cpp | 2 +-
llvm/lib/Target/AArch64/AArch64RegisterInfo.h | 6 -
llvm/test/CodeGen/AArch64/cgp-usubo.ll | 21 +-
llvm/test/CodeGen/AArch64/cmpxchg-idioms.ll | 77 +--
.../AArch64/combine-comparisons-by-cse.ll | 37 +-
llvm/test/CodeGen/AArch64/csr-split.ll | 60 +-
llvm/test/CodeGen/AArch64/div-i256.ll | 122 ++--
llvm/test/CodeGen/AArch64/pr166870.ll | 5 +-
llvm/test/CodeGen/AArch64/pr51516.mir | 200 +++++-
.../AArch64/sme-callee-save-restore-pairs.ll | 22 +-
.../test/CodeGen/AArch64/sme-peephole-opts.ll | 35 +-
.../CodeGen/AArch64/sme-streaming-checkvl.ll | 32 +-
.../sme-streaming-compatible-interface.ll | 34 +-
.../CodeGen/AArch64/spill-reload-remarks.ll | 2 +-
.../AArch64/statepoint-call-lowering.ll | 23 +-
.../sve-breakdown-scalable-vectortype.ll | 567 ++++++++----------
16 files changed, 693 insertions(+), 552 deletions(-)
diff --git a/llvm/lib/CodeGen/RegAllocGreedy.cpp b/llvm/lib/CodeGen/RegAllocGreedy.cpp
index ef23468b1e752..41ff45bfe1e87 100644
--- a/llvm/lib/CodeGen/RegAllocGreedy.cpp
+++ b/llvm/lib/CodeGen/RegAllocGreedy.cpp
@@ -118,7 +118,7 @@ CSRFirstTimeCost("regalloc-csr-first-time-cost",
static cl::opt<unsigned> CSRCostScale(
"regalloc-csr-cost-scale",
cl::desc("Scale for the callee-saved register cost, in percentage."),
- cl::init(80), cl::Hidden);
+ cl::init(100), cl::Hidden);
static cl::opt<unsigned long> GrowRegionComplexityBudget(
"grow-region-complexity-budget",
diff --git a/llvm/lib/Target/AArch64/AArch64RegisterInfo.h b/llvm/lib/Target/AArch64/AArch64RegisterInfo.h
index b31104aac6551..60af8f6da5fbf 100644
--- a/llvm/lib/Target/AArch64/AArch64RegisterInfo.h
+++ b/llvm/lib/Target/AArch64/AArch64RegisterInfo.h
@@ -54,12 +54,6 @@ class AArch64RegisterInfo final : public AArch64GenRegisterInfo {
const uint32_t *getDarwinCallPreservedMask(const MachineFunction &MF,
CallingConv::ID) const;
- unsigned getCSRCost() const override {
- // The cost will be compared against BlockFrequency where entry has the
- // value of 1 << 14. A value of 5 will choose to spill or split really
- // cold path instead of using a callee-saved register.
- return 5;
- }
unsigned getCSRFirstUseCost() const override {
// The cost of 2 means push and pop for each CSR.
return 2;
diff --git a/llvm/test/CodeGen/AArch64/cgp-usubo.ll b/llvm/test/CodeGen/AArch64/cgp-usubo.ll
index 71a873014a61f..8273013a5db2c 100644
--- a/llvm/test/CodeGen/AArch64/cgp-usubo.ll
+++ b/llvm/test/CodeGen/AArch64/cgp-usubo.ll
@@ -279,29 +279,30 @@ end:
define i1 @usubo_ult_cmp_dominates_i64(i64 %x, i64 %y, ptr %p, i1 %cond) nounwind {
; CHECK-LABEL: usubo_ult_cmp_dominates_i64:
; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: stp x30, x23, [sp, #-48]! // 16-byte Folded Spill
+; CHECK-NEXT: str x30, [sp, #-48]! // 8-byte Folded Spill
; CHECK-NEXT: stp x20, x19, [sp, #32] // 16-byte Folded Spill
; CHECK-NEXT: mov w19, w3
; CHECK-NEXT: stp x22, x21, [sp, #16] // 16-byte Folded Spill
; CHECK-NEXT: tbz w3, #0, .LBB15_3
; CHECK-NEXT: // %bb.1: // %t
; CHECK-NEXT: cmp x0, x1
-; CHECK-NEXT: mov x22, x0
-; CHECK-NEXT: mov x20, x2
-; CHECK-NEXT: cset w21, lo
-; CHECK-NEXT: mov x23, x1
-; CHECK-NEXT: mov w0, w21
+; CHECK-NEXT: mov x21, x0
+; CHECK-NEXT: str x2, [sp, #8] // 8-byte Spill
+; CHECK-NEXT: cset w20, lo
+; CHECK-NEXT: mov x22, x1
+; CHECK-NEXT: mov w0, w20
; CHECK-NEXT: bl call
-; CHECK-NEXT: subs x8, x22, x23
+; CHECK-NEXT: subs x8, x21, x22
; CHECK-NEXT: b.hs .LBB15_3
; CHECK-NEXT: // %bb.2: // %end
-; CHECK-NEXT: mov w19, w21
-; CHECK-NEXT: str x8, [x20]
+; CHECK-NEXT: ldr x9, [sp, #8] // 8-byte Reload
+; CHECK-NEXT: mov w19, w20
+; CHECK-NEXT: str x8, [x9]
; CHECK-NEXT: .LBB15_3: // %common.ret
; CHECK-NEXT: and w0, w19, #0x1
; CHECK-NEXT: ldp x20, x19, [sp, #32] // 16-byte Folded Reload
; CHECK-NEXT: ldp x22, x21, [sp, #16] // 16-byte Folded Reload
-; CHECK-NEXT: ldp x30, x23, [sp], #48 // 16-byte Folded Reload
+; CHECK-NEXT: ldr x30, [sp], #48 // 8-byte Folded Reload
; CHECK-NEXT: ret
entry:
br i1 %cond, label %t, label %f
diff --git a/llvm/test/CodeGen/AArch64/cmpxchg-idioms.ll b/llvm/test/CodeGen/AArch64/cmpxchg-idioms.ll
index 3f4dd116d91f8..51d00704ff1d2 100644
--- a/llvm/test/CodeGen/AArch64/cmpxchg-idioms.ll
+++ b/llvm/test/CodeGen/AArch64/cmpxchg-idioms.ll
@@ -164,53 +164,50 @@ declare void @baz()
define i1 @test_conditional2(i32 %a, i32 %b, ptr %c) {
; CHECK-LABEL: test_conditional2:
; CHECK: ; %bb.0: ; %entry
-; CHECK-NEXT: stp x22, x21, [sp, #-48]! ; 16-byte Folded Spill
-; CHECK-NEXT: stp x20, x19, [sp, #16] ; 16-byte Folded Spill
+; CHECK-NEXT: sub sp, sp, #48
; CHECK-NEXT: stp x29, x30, [sp, #32] ; 16-byte Folded Spill
; CHECK-NEXT: .cfi_def_cfa_offset 48
; CHECK-NEXT: .cfi_offset w30, -8
; CHECK-NEXT: .cfi_offset w29, -16
-; CHECK-NEXT: .cfi_offset w19, -24
-; CHECK-NEXT: .cfi_offset w20, -32
-; CHECK-NEXT: .cfi_offset w21, -40
-; CHECK-NEXT: .cfi_offset w22, -48
-; CHECK-NEXT: mov x19, x2
-; CHECK-NEXT: mov w20, w1
-; CHECK-NEXT: mov w21, w0
; CHECK-NEXT: LBB3_1: ; %cmpxchg.start
; CHECK-NEXT: ; =>This Inner Loop Header: Depth=1
-; CHECK-NEXT: ldaxr w8, [x19]
-; CHECK-NEXT: cmp w8, w21
+; CHECK-NEXT: ldaxr w8, [x2]
+; CHECK-NEXT: cmp w8, w0
; CHECK-NEXT: b.ne LBB3_9
; CHECK-NEXT: ; %bb.2: ; %cmpxchg.trystore
; CHECK-NEXT: ; in Loop: Header=BB3_1 Depth=1
-; CHECK-NEXT: stlxr w8, w20, [x19]
+; CHECK-NEXT: stlxr w8, w1, [x2]
; CHECK-NEXT: cbnz w8, LBB3_1
; CHECK-NEXT: ; %bb.3:
; CHECK-NEXT: mov w8, #1 ; =0x1
; CHECK-NEXT: LBB3_4: ; %for.cond.preheader
-; CHECK-NEXT: mov w22, #2 ; =0x2
+; CHECK-NEXT: mov w9, #2 ; =0x2
; CHECK-NEXT: LBB3_5: ; %for.cond
; CHECK-NEXT: ; =>This Inner Loop Header: Depth=1
-; CHECK-NEXT: cbz w22, LBB3_8
+; CHECK-NEXT: cbz w9, LBB3_8
; CHECK-NEXT: ; %bb.6: ; %for.body
; CHECK-NEXT: ; in Loop: Header=BB3_5 Depth=1
-; CHECK-NEXT: sub w22, w22, #1
-; CHECK-NEXT: orr w9, w21, w20
-; CHECK-NEXT: ldr w10, [x19, w22, sxtw #2]
-; CHECK-NEXT: cmp w9, w10
+; CHECK-NEXT: sub w9, w9, #1
+; CHECK-NEXT: orr w10, w0, w1
+; CHECK-NEXT: ldr w11, [x2, w9, sxtw #2]
+; CHECK-NEXT: cmp w10, w11
; CHECK-NEXT: b.eq LBB3_5
; CHECK-NEXT: ; %bb.7: ; %if.then
; CHECK-NEXT: ; in Loop: Header=BB3_5 Depth=1
-; CHECK-NEXT: str w9, [x19, w22, sxtw #2]
+; CHECK-NEXT: str w10, [x2, w9, sxtw #2]
+; CHECK-NEXT: str x2, [sp, #24] ; 8-byte Spill
+; CHECK-NEXT: stp w1, w0, [sp, #16] ; 8-byte Folded Spill
+; CHECK-NEXT: str w9, [sp, #12] ; 4-byte Spill
; CHECK-NEXT: bl _foo
+; CHECK-NEXT: ldp w9, w1, [sp, #12] ; 8-byte Folded Reload
+; CHECK-NEXT: ldr w0, [sp, #20] ; 4-byte Reload
+; CHECK-NEXT: ldr x2, [sp, #24] ; 8-byte Reload
; CHECK-NEXT: mov w8, wzr
; CHECK-NEXT: b LBB3_5
; CHECK-NEXT: LBB3_8: ; %for.cond.cleanup
; CHECK-NEXT: ldp x29, x30, [sp, #32] ; 16-byte Folded Reload
; CHECK-NEXT: and w0, w8, #0x1
-; CHECK-NEXT: ldp x20, x19, [sp, #16] ; 16-byte Folded Reload
-; CHECK-NEXT: ldp x22, x21, [sp], #48 ; 16-byte Folded Reload
+; CHECK-NEXT: add sp, sp, #48
; CHECK-NEXT: ret
; CHECK-NEXT: LBB3_9: ; %cmpxchg.nostore
; CHECK-NEXT: mov w8, wzr
@@ -219,10 +216,11 @@ define i1 @test_conditional2(i32 %a, i32 %b, ptr %c) {
;
; OUTLINE-ATOMICS-LABEL: test_conditional2:
; OUTLINE-ATOMICS: ; %bb.0: ; %entry
-; OUTLINE-ATOMICS-NEXT: stp x22, x21, [sp, #-48]! ; 16-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT: stp x20, x19, [sp, #16] ; 16-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT: stp x29, x30, [sp, #32] ; 16-byte Folded Spill
-; OUTLINE-ATOMICS-NEXT: .cfi_def_cfa_offset 48
+; OUTLINE-ATOMICS-NEXT: sub sp, sp, #64
+; OUTLINE-ATOMICS-NEXT: stp x22, x21, [sp, #16] ; 16-byte Folded Spill
+; OUTLINE-ATOMICS-NEXT: stp x20, x19, [sp, #32] ; 16-byte Folded Spill
+; OUTLINE-ATOMICS-NEXT: stp x29, x30, [sp, #48] ; 16-byte Folded Spill
+; OUTLINE-ATOMICS-NEXT: .cfi_def_cfa_offset 64
; OUTLINE-ATOMICS-NEXT: .cfi_offset w30, -8
; OUTLINE-ATOMICS-NEXT: .cfi_offset w29, -16
; OUTLINE-ATOMICS-NEXT: .cfi_offset w19, -24
@@ -234,29 +232,32 @@ define i1 @test_conditional2(i32 %a, i32 %b, ptr %c) {
; OUTLINE-ATOMICS-NEXT: mov w21, w0
; OUTLINE-ATOMICS-NEXT: bl ___aarch64_cas4_acq_rel
; OUTLINE-ATOMICS-NEXT: cmp w0, w21
-; OUTLINE-ATOMICS-NEXT: mov w22, #2 ; =0x2
-; OUTLINE-ATOMICS-NEXT: cset w8, eq
+; OUTLINE-ATOMICS-NEXT: mov w8, #2 ; =0x2
+; OUTLINE-ATOMICS-NEXT: cset w9, eq
; OUTLINE-ATOMICS-NEXT: LBB3_1: ; %for.cond
; OUTLINE-ATOMICS-NEXT: ; =>This Inner Loop Header: Depth=1
-; OUTLINE-ATOMICS-NEXT: cbz w22, LBB3_4
+; OUTLINE-ATOMICS-NEXT: cbz w8, LBB3_4
; OUTLINE-ATOMICS-NEXT: ; %bb.2: ; %for.body
; OUTLINE-ATOMICS-NEXT: ; in Loop: Header=BB3_1 Depth=1
-; OUTLINE-ATOMICS-NEXT: sub w22, w22, #1
-; OUTLINE-ATOMICS-NEXT: orr w9, w21, w20
-; OUTLINE-ATOMICS-NEXT: ldr w10, [x19, w22, sxtw #2]
-; OUTLINE-ATOMICS-NEXT: cmp w9, w10
+; OUTLINE-ATOMICS-NEXT: sub w8, w8, #1
+; OUTLINE-ATOMICS-NEXT: orr w10, w21, w20
+; OUTLINE-ATOMICS-NEXT: ldr w11, [x19, w8, sxtw #2]
+; OUTLINE-ATOMICS-NEXT: cmp w10, w11
; OUTLINE-ATOMICS-NEXT: b.eq LBB3_1
; OUTLINE-ATOMICS-NEXT: ; %bb.3: ; %if.then
; OUTLINE-ATOMICS-NEXT: ; in Loop: Header=BB3_1 Depth=1
-; OUTLINE-ATOMICS-NEXT: str w9, [x19, w22, sxtw #2]
+; OUTLINE-ATOMICS-NEXT: str w10, [x19, w8, sxtw #2]
+; OUTLINE-ATOMICS-NEXT: str w8, [sp, #12] ; 4-byte Spill
; OUTLINE-ATOMICS-NEXT: bl _foo
-; OUTLINE-ATOMICS-NEXT: mov w8, wzr
+; OUTLINE-ATOMICS-NEXT: ldr w8, [sp, #12] ; 4-byte Reload
+; OUTLINE-ATOMICS-NEXT: mov w9, wzr
; OUTLINE-ATOMICS-NEXT: b LBB3_1
; OUTLINE-ATOMICS-NEXT: LBB3_4: ; %for.cond.cleanup
-; OUTLINE-ATOMICS-NEXT: ldp x29, x30, [sp, #32] ; 16-byte Folded Reload
-; OUTLINE-ATOMICS-NEXT: and w0, w8, #0x1
-; OUTLINE-ATOMICS-NEXT: ldp x20, x19, [sp, #16] ; 16-byte Folded Reload
-; OUTLINE-ATOMICS-NEXT: ldp x22, x21, [sp], #48 ; 16-byte Folded Reload
+; OUTLINE-ATOMICS-NEXT: ldp x29, x30, [sp, #48] ; 16-byte Folded Reload
+; OUTLINE-ATOMICS-NEXT: and w0, w9, #0x1
+; OUTLINE-ATOMICS-NEXT: ldp x20, x19, [sp, #32] ; 16-byte Folded Reload
+; OUTLINE-ATOMICS-NEXT: ldp x22, x21, [sp, #16] ; 16-byte Folded Reload
+; OUTLINE-ATOMICS-NEXT: add sp, sp, #64
; OUTLINE-ATOMICS-NEXT: ret
entry:
%pair = cmpxchg ptr %c, i32 %a, i32 %b seq_cst seq_cst
diff --git a/llvm/test/CodeGen/AArch64/combine-comparisons-by-cse.ll b/llvm/test/CodeGen/AArch64/combine-comparisons-by-cse.ll
index 09e80ee936738..3bb09723381fa 100644
--- a/llvm/test/CodeGen/AArch64/combine-comparisons-by-cse.ll
+++ b/llvm/test/CodeGen/AArch64/combine-comparisons-by-cse.ll
@@ -582,26 +582,30 @@ declare void @do_something() #1
define i32 @do_nothing_if_resultant_opcodes_would_differ() #0 {
; CHECK-LABEL: do_nothing_if_resultant_opcodes_would_differ:
; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: str x30, [sp, #-32]! // 8-byte Folded Spill
-; CHECK-NEXT: .cfi_def_cfa_offset 32
-; CHECK-NEXT: stp x20, x19, [sp, #16] // 16-byte Folded Spill
-; CHECK-NEXT: .cfi_offset w19, -8
-; CHECK-NEXT: .cfi_offset w20, -16
-; CHECK-NEXT: .cfi_offset w30, -32
-; CHECK-NEXT: adrp x19, :got:a
-; CHECK-NEXT: ldr x19, [x19, :got_lo12:a]
-; CHECK-NEXT: ldr w8, [x19]
+; CHECK-NEXT: adrp x8, :got:a
+; CHECK-NEXT: ldr x8, [x8, :got_lo12:a]
+; CHECK-NEXT: ldr w8, [x8]
; CHECK-NEXT: cmn w8, #2
; CHECK-NEXT: b.gt .LBB10_4
; CHECK-NEXT: // %bb.1: // %while.body.preheader
-; CHECK-NEXT: sub w20, w8, #1
+; CHECK-NEXT: stp x30, x19, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT: .cfi_def_cfa_offset 16
+; CHECK-NEXT: .cfi_offset w19, -8
+; CHECK-NEXT: .cfi_offset w30, -16
+; CHECK-NEXT: sub w19, w8, #1
; CHECK-NEXT: .LBB10_2: // %while.body
; CHECK-NEXT: // =>This Inner Loop Header: Depth=1
; CHECK-NEXT: bl do_something
-; CHECK-NEXT: adds w20, w20, #1
+; CHECK-NEXT: adds w19, w19, #1
; CHECK-NEXT: b.mi .LBB10_2
; CHECK-NEXT: // %bb.3: // %while.cond.while.end_crit_edge
-; CHECK-NEXT: ldr w8, [x19]
+; CHECK-NEXT: adrp x8, :got:a
+; CHECK-NEXT: ldr x8, [x8, :got_lo12:a]
+; CHECK-NEXT: ldr w8, [x8]
+; CHECK-NEXT: ldp x30, x19, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT: .cfi_def_cfa_offset 0
+; CHECK-NEXT: .cfi_restore w19
+; CHECK-NEXT: .cfi_restore w30
; CHECK-NEXT: .LBB10_4: // %while.end
; CHECK-NEXT: cmp w8, #1
; CHECK-NEXT: b.gt .LBB10_7
@@ -616,16 +620,9 @@ define i32 @do_nothing_if_resultant_opcodes_would_differ() #0 {
; CHECK-NEXT: b.ne .LBB10_7
; CHECK-NEXT: // %bb.6:
; CHECK-NEXT: mov w0, #123 // =0x7b
-; CHECK-NEXT: b .LBB10_8
+; CHECK-NEXT: ret
; CHECK-NEXT: .LBB10_7: // %if.end
; CHECK-NEXT: mov w0, wzr
-; CHECK-NEXT: .LBB10_8: // %return
-; CHECK-NEXT: ldp x20, x19, [sp, #16] // 16-byte Folded Reload
-; CHECK-NEXT: ldr x30, [sp], #32 // 8-byte Folded Reload
-; CHECK-NEXT: .cfi_def_cfa_offset 0
-; CHECK-NEXT: .cfi_restore w19
-; CHECK-NEXT: .cfi_restore w20
-; CHECK-NEXT: .cfi_restore w30
; CHECK-NEXT: ret
entry:
%0 = load i32, ptr @a, align 4
diff --git a/llvm/test/CodeGen/AArch64/csr-split.ll b/llvm/test/CodeGen/AArch64/csr-split.ll
index 7b092b00b9655..e99d4f30beb80 100644
--- a/llvm/test/CodeGen/AArch64/csr-split.ll
+++ b/llvm/test/CodeGen/AArch64/csr-split.ll
@@ -164,54 +164,58 @@ return: ; preds = %if.end, %entry, %if
define dso_local ptr @test3(ptr nocapture %p1, i8 zeroext %p2) local_unnamed_addr uwtable {
; CHECK-LABEL: test3:
; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: str x30, [sp, #-32]! // 8-byte Folded Spill
+; CHECK-NEXT: mov x8, x0
+; CHECK-NEXT: ldr x0, [x0]
+; CHECK-NEXT: cbz x0, .LBB2_2
+; CHECK-NEXT: // %bb.1: // %land.rhs
+; CHECK-NEXT: sub sp, sp, #32
; CHECK-NEXT: .cfi_def_cfa_offset 32
-; CHECK-NEXT: stp x20, x19, [sp, #16] // 16-byte Folded Spill
+; CHECK-NEXT: stp x30, x19, [sp, #16] // 16-byte Folded Spill
; CHECK-NEXT: .cfi_offset w19, -8
-; CHECK-NEXT: .cfi_offset w20, -16
-; CHECK-NEXT: .cfi_offset w30, -32
-; CHECK-NEXT: ldr x19, [x0]
-; CHECK-NEXT: cbz x19, .LBB2_2
-; CHECK-NEXT: // %bb.1: // %land.rhs
-; CHECK-NEXT: mov x20, x0
-; CHECK-NEXT: mov x0, x19
+; CHECK-NEXT: .cfi_offset w30, -16
+; CHECK-NEXT: str x0, [sp, #8] // 8-byte Spill
+; CHECK-NEXT: mov x19, x8
; CHECK-NEXT: bl bar
-; CHECK-NEXT: str x0, [x20]
-; CHECK-NEXT: .LBB2_2: // %land.end
-; CHECK-NEXT: mov x0, x19
-; CHECK-NEXT: ldp x20, x19, [sp, #16] // 16-byte Folded Reload
-; CHECK-NEXT: ldr x30, [sp], #32 // 8-byte Folded Reload
+; CHECK-NEXT: mov x8, x0
+; CHECK-NEXT: ldr x0, [sp, #8] // 8-byte Reload
+; CHECK-NEXT: str x8, [x19]
+; CHECK-NEXT: ldp x30, x19, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEXT: add sp, sp, #32
; CHECK-NEXT: .cfi_def_cfa_offset 0
; CHECK-NEXT: .cfi_restore w19
-; CHECK-NEXT: .cfi_restore w20
; CHECK-NEXT: .cfi_restore w30
+; CHECK-NEXT: .LBB2_2: // %land.end
; CHECK-NEXT: ret
;
; CHECK-APPLE-LABEL: test3:
; CHECK-APPLE: ; %bb.0: ; %entry
-; CHECK-APPLE-NEXT: stp x20, x19, [sp, #-32]! ; 16-byte Folded Spill
-; CHECK-APPLE-NEXT: .cfi_def_cfa_offset 32
-; CHECK-APPLE-NEXT: stp x29, x30, [sp, #16] ; 16-byte Folded Spill
+; CHECK-APPLE-NEXT: mov x8, x0
+; CHECK-APPLE-NEXT: ldr x0, [x0]
+; CHECK-APPLE-NEXT: cbz x0, LBB2_2
+; CHECK-APPLE-NEXT: ; %bb.1: ; %land.rhs
+; CHECK-APPLE-NEXT: sub sp, sp, #48
+; CHECK-APPLE-NEXT: .cfi_def_cfa_offset 48
+; CHECK-APPLE-NEXT: stp x20, x19, [sp, #16] ; 16-byte Folded Spill
+; CHECK-APPLE-NEXT: stp x29, x30, [sp, #32] ; 16-byte Folded Spill
; CHECK-APPLE-NEXT: .cfi_offset w30, -8
; CHECK-APPLE-NEXT: .cfi_offset w29, -16
; CHECK-APPLE-NEXT: .cfi_offset w19, -24
; CHECK-APPLE-NEXT: .cfi_offset w20, -32
-; CHECK-APPLE-NEXT: ldr x19, [x0]
-; CHECK-APPLE-NEXT: cbz x19, LBB2_2
-; CHECK-APPLE-NEXT: ; %bb.1: ; %land.rhs
-; CHECK-APPLE-NEXT: mov x20, x0
-; CHECK-APPLE-NEXT: mov x0, x19
+; CHECK-APPLE-NEXT: str x0, [sp, #8] ; 8-byte Spill
+; CHECK-APPLE-NEXT: mov x19, x8
; CHECK-APPLE-NEXT: bl _bar
-; CHECK-APPLE-NEXT: str x0, [x20]
-; CHECK-APPLE-NEXT: LBB2_2: ; %land.end
-; CHECK-APPLE-NEXT: ldp x29, x30, [sp, #16] ; 16-byte Folded Reload
-; CHECK-APPLE-NEXT: mov x0, x19
-; CHECK-APPLE-NEXT: ldp x20, x19, [sp], #32 ; 16-byte Folded Reload
+; CHECK-APPLE-NEXT: mov x8, x0
+; CHECK-APPLE-NEXT: ldp x29, x30, [sp, #32] ; 16-byte Folded Reload
+; CHECK-APPLE-NEXT: str x8, [x19]
+; CHECK-APPLE-NEXT: ldp x20, x19, [sp, #16] ; 16-byte Folded Reload
+; CHECK-APPLE-NEXT: ldr x0, [sp, #8] ; 8-byte Reload
+; CHECK-APPLE-NEXT: add sp, sp, #48
; CHECK-APPLE-NEXT: .cfi_def_cfa_offset 0
; CHECK-APPLE-NEXT: .cfi_restore w30
; CHECK-APPLE-NEXT: .cfi_restore w29
; CHECK-APPLE-NEXT: .cfi_restore w19
; CHECK-APPLE-NEXT: .cfi_restore w20
+; CHECK-APPLE-NEXT: LBB2_2: ; %land.end
; CHECK-APPLE-NEXT: ret
entry:
%0 = load ptr, ptr %p1, align 8, !tbaa !6
diff --git a/llvm/test/CodeGen/AArch64/div-i256.ll b/llvm/test/CodeGen/AArch64/div-i256.ll
index 5f8d362c85bb2..02a65ec9f3207 100644
--- a/llvm/test/CodeGen/AArch64/div-i256.ll
+++ b/llvm/test/CodeGen/AArch64/div-i256.ll
@@ -110,36 +110,36 @@ define i256 @udiv256(i256 %a, i256 %b) nounwind {
; CHECK-NEXT: // %bb.3: // %udiv-preheader
; CHECK-NEXT: lsr x20, x8, #3
; CHECK-NEXT: stp x0, x1, [sp]
-; CHECK-NEXT: mov x1, sp
+; CHECK-NEXT: mov x19, sp
; CHECK-NEXT: stp q0, q0, [sp, #32]
; CHECK-NEXT: mov x18, xzr
-; CHECK-NEXT: mov x19, xzr
-; CHECK-NEXT: and x0, x20, #0x18
+; CHECK-NEXT: mov x17, xzr
+; CHECK-NEXT: and x1, x20, #0x18
; CHECK-NEXT: stp x2, x3, [sp, #16]
-; CHECK-NEXT: and x3, x8, #0x3f
-; CHECK-NEXT: add x0, x1, x0
; CHECK-NEXT: mvn w20, w8
-; CHECK-NEXT: eor x3, x3, #0x3f
-; CHECK-NEXT: ldp x1, x2, [x0, #16]
-; CHECK-NEXT: mov x17, xzr
-; CHECK-NEXT: ldp x23, x21, [x0]
-; CHECK-NEXT: lsl x0, x1, #1
-; CHECK-NEXT: lsl x22, x2, #1
-; CHECK-NEXT: lsr x24, x1, x8
-; CHECK-NEXT: lsl x1, x21, #1
+; CHECK-NEXT: add x1, x19, x1
+; CHECK-NEXT: and x19, x8, #0x3f
+; CHECK-NEXT: mov x0, xzr
+; CHECK-NEXT: ldp x2, x3, [x1, #16]
+; CHECK-NEXT: eor x19, x19, #0x3f
+; CHECK-NEXT: ldp x23, x21, [x1]
+; CHECK-NEXT: lsl x1, x2, #1
+; CHECK-NEXT: lsl x22, x3, #1
+; CHECK-NEXT: lsr x24, x2, x8
+; CHECK-NEXT: lsl x2, x21, #1
; CHECK-NEXT: lsr x26, x21, x8
; CHECK-NEXT: lsr x27, x23, x8
-; CHECK-NEXT: lsl x25, x0, x20
-; CHECK-NEXT: subs x0, x4, #1
-; CHECK-NEXT: lsl x22, x22, x3
-; CHECK-NEXT: lsl x3, x1, x3
-; CHECK-NEXT: sbcs x1, x5, xzr
-; CHECK-NEXT: lsr x21, x2, x8
-; CHECK-NEXT: sbcs x2, x6, xzr
+; CHECK-NEXT: lsl x25, x1, x20
+; CHECK-NEXT: subs x1, x4, #1
+; CHECK-NEXT: lsl x22, x22, x19
+; CHECK-NEXT: lsl x19, x2, x19
+; CHECK-NEXT: sbcs x2, x5, xzr
+; CHECK-NEXT: lsr x21, x3, x8
+; CHECK-NEXT: sbcs x3, x6, xzr
; CHECK-NEXT: orr x20, x22, x24
; CHECK-NEXT: orr x23, x26, x25
-; CHECK-NEXT: orr x22, x3, x27
-; CHECK-NEXT: sbc x3, x7, xzr
+; CHECK-NEXT: orr x22, x19, x27
+; CHECK-NEXT: sbc x19, x7, xzr
; CHECK-NEXT: .LBB0_4: // %udiv-do-while
; CHECK-NEXT: // =>This Inner Loop Header: Depth=1
; CHECK-NEXT: extr x24, x22, x15, #63
@@ -148,13 +148,13 @@ define i256 @udiv256(i256 %a, i256 %b) nounwind {
; CHECK-NEXT: extr x21, x21, x20, #63
; CHECK-NEXT: extr x15, x15, x13, #63
; CHECK-NEXT: extr x13, x13, x12, #63
-; CHECK-NEXT: cmp x0, x24
-; CHECK-NEXT: sbcs xzr, x1, x25
-; CHECK-NEXT: orr x13, x19, x13
+; CHECK-NEXT: cmp x1, x24
+; CHECK-NEXT: sbcs xzr, x2, x25
+; CHECK-NEXT: orr x13, x0, x13
; CHECK-NEXT: orr x15, x17, x15
-; CHECK-NEXT: sbcs xzr, x2, x26
+; CHECK-NEXT: sbcs xzr, x3, x26
; CHECK-NEXT: mov x17, xzr
-; CHECK-NEXT: sbc x20, x3, x21
+; CHECK-NEXT: sbc x20, x19, x21
; CHECK-NEXT: asr x27, x20, #63
; CHECK-NEXT: and x20, x27, x4
; CHECK-NEXT: subs x22, x24, x20
@@ -172,11 +172,11 @@ define i256 @udiv256(i256 %a, i256 %b) nounwind {
; CHECK-NEXT: sbcs x10, x10, xzr
; CHECK-NEXT: orr x12, x18, x24
; CHECK-NEXT: sbc x14, x14, xzr
-; CHECK-NEXT: orr x19, x8, x10
+; CHECK-NEXT: orr x0, x8, x10
; CHECK-NEXT: orr x18, x9, x14
-; CHECK-NEXT: orr x24, x19, x18
+; CHECK-NEXT: orr x24, x0, x18
; CHECK-NEXT: mov x18, xzr
-; CHECK-NEXT: mov x19, xzr
+; CHECK-NEXT: mov x0, xzr
; CHECK-NEXT: cbnz x24, .LBB0_4
; CHECK-NEXT: .LBB0_5: // %udiv-loop-exit
; CHECK-NEXT: ldp x20, x19, [sp, #192] // 16-byte Folded Reload
@@ -327,36 +327,36 @@ define i256 @sdiv256(i256 %a, i256 %b) nounwind {
; CHECK-NEXT: // %bb.3: // %udiv-preheader
; CHECK-NEXT: lsr x21, x13, #3
; CHECK-NEXT: stp x14, x15, [sp]
-; CHECK-NEXT: mov x15, sp
+; CHECK-NEXT: mov x20, sp
; CHECK-NEXT: stp q0, q0, [sp, #32]
; CHECK-NEXT: mov x19, xzr
-; CHECK-NEXT: mov x20, xzr
-; CHECK-NEXT: and x14, x21, #0x18
+; CHECK-NEXT: mov x7, xzr
+; CHECK-NEXT: and x15, x21, #0x18
; CHECK-NEXT: stp x18, x0, [sp, #16]
-; CHECK-NEXT: and x0, x13, #0x3f
-; CHECK-NEXT: add x14, x15, x14
; CHECK-NEXT: mvn w21, w13
-; CHECK-NEXT: eor x0, x0, #0x3f
-; CHECK-NEXT: ldp x15, x18, [x14, #16]
-; CHECK-NEXT: mov x7, xzr
-; CHECK-NEXT: ldp x24, x22, [x14]
-; CHECK-NEXT: lsl x14, x15, #1
-; CHECK-NEXT: lsl x23, x18, #1
-; CHECK-NEXT: lsr x25, x15, x13
-; CHECK-NEXT: lsl x15, x22, #1
+; CHECK-NEXT: add x15, x20, x15
+; CHECK-NEXT: and x20, x13, #0x3f
+; CHECK-NEXT: mov x14, xzr
+; CHECK-NEXT: ldp x18, x0, [x15, #16]
+; CHECK-NEXT: eor x20, x20, #0x3f
+; CHECK-NEXT: ldp x24, x22, [x15]
+; CHECK-NEXT: lsl x15, x18, #1
+; CHECK-NEXT: lsl x23, x0, #1
+; CHECK-NEXT: lsr x25, x18, x13
+; CHECK-NEXT: lsl x18, x22, #1
; CHECK-NEXT: lsr x27, x22, x13
; CHECK-NEXT: lsr x28, x24, x13
-; CHECK-NEXT: lsl x26, x14, x21
-; CHECK-NEXT: subs x14, x8, #1
-; CHECK-NEXT: lsl x23, x23, x0
-; CHECK-NEXT: lsl x0, x15, x0
-; CHECK-NEXT: sbcs x15, x9, xzr
-; CHECK-NEXT: lsr x22, x18, x13
-; CHECK-NEXT: sbcs x18, x10, xzr
+; CHECK-NEXT: lsl x26, x15, x21
+; CHECK-NEXT: subs x15, x8, #1
+; CHECK-NEXT: lsl x23, x23, x20
+; CHECK-NEXT: lsl x20, x18, x20
+; CHECK-NEXT: sbcs x18, x9, xzr
+; CHECK-NEXT: lsr x22, x0, x13
+; CHECK-NEXT: sbcs x0, x10, xzr
; CHECK-NEXT: orr x21, x23, x25
; CHECK-NEXT: orr x24, x27, x26
-; CHECK-NEXT: orr x23, x0, x28
-; CHECK-NEXT: sbc x0, x11, xzr
+; CHECK-NEXT: orr x23, x20, x28
+; CHECK-NEXT: sbc x20, x11, xzr
; CHECK-NEXT: .LBB1_4: // %udiv-do-while
; CHECK-NEXT: // =>This Inner Loop Header: Depth=1
; CHECK-NEXT: extr x25, x23, x5, #63
@@ -365,13 +365,13 @@ define i256 @sdiv256(i256 %a, i256 %b) nounwind {
; CHECK-NEXT: extr x22, x22, x21, #63
; CHECK-NEXT: extr x5, x5, x3, #63
; CHECK-NEXT: extr x3, x3, x2, #63
-; CHECK-NEXT: cmp x14, x25
-; CHECK-NEXT: sbcs xzr, x15, x26
-; CHECK-NEXT: orr x3, x20, x3
+; CHECK-NEXT: cmp x15, x25
+; CHECK-NEXT: sbcs xzr, x18, x26
+; CHECK-NEXT: orr x3, x14, x3
; CHECK-NEXT: orr x5, x7, x5
-; CHECK-NEXT: sbcs xzr, x18, x27
+; CHECK-NEXT: sbcs xzr, x0, x27
; CHECK-NEXT: mov x7, xzr
-; CHECK-NEXT: sbc x21, x0, x22
+; CHECK-NEXT: sbc x21, x20, x22
; CHECK-NEXT: asr x28, x21, #63
; CHECK-NEXT: and x21, x28, x8
; CHECK-NEXT: subs x23, x25, x21
@@ -389,11 +389,11 @@ define i256 @sdiv256(i256 %a, i256 %b) nounwind {
; CHECK-NEXT: sbcs x17, x17, xzr
; CHECK-NEXT: orr x2, x19, x25
; CHECK-NEXT: sbc x4, x4, xzr
-; CHECK-NEXT: orr x20, x13, x17
-; CHECK-NEXT: orr x19, x16, x4
-; CHECK-NEXT: orr x25, x20, x19
+; CHECK-NEXT: orr x19, x13, x17
+; CHECK-NEXT: orr x14, x16, x4
+; CHECK-NEXT: orr x25, x19, x14
; CHECK-NEXT: mov x19, xzr
-; CHECK-NEXT: mov x20, xzr
+; CHECK-NEXT: mov x14, xzr
; CHECK-NEXT: cbnz x25, .LBB1_4
; CHECK-NEXT: .LBB1_5: // %udiv-loop-exit
; CHECK-NEXT: ldp x20, x19, [sp, #192] // 16-byte Folded Reload
diff --git a/llvm/test/CodeGen/AArch64/pr166870.ll b/llvm/test/CodeGen/AArch64/pr166870.ll
index fd597dbec4a90..7cec08a4bf0e7 100644
--- a/llvm/test/CodeGen/AArch64/pr166870.ll
+++ b/llvm/test/CodeGen/AArch64/pr166870.ll
@@ -23,7 +23,7 @@ define i32 @widget(i32 %arg, i32 %arg1, i1 %arg2, ptr %arg3, i1 %arg4) #0 nounwi
; CHECK-NEXT: stp x20, x19, [sp, #32] // 16-byte Folded Spill
; CHECK-NEXT: mov x19, x3
; CHECK-NEXT: mov x21, x0
-; CHECK-NEXT: mov x22, x1
+; CHECK-NEXT: str w1, [sp, #12] // 4-byte Spill
; CHECK-NEXT: bl baz
; CHECK-NEXT: mov w8, #1 // =0x1
; CHECK-NEXT: cbnz w8, .LBB0_8
@@ -31,7 +31,8 @@ define i32 @widget(i32 %arg, i32 %arg1, i1 %arg2, ptr %arg3, i1 %arg4) #0 nounwi
; CHECK-NEXT: mov w20, #0 // =0x0
; CHECK-NEXT: mov w8, w21
; CHECK-NEXT: mov x21, x8
-; CHECK-NEXT: mov w8, w22
+; CHECK-NEXT: ldr w8, [sp, #12] // 4-byte Reload
+; CHECK-NEXT: mov w8, w8
; CHECK-NEXT: mov x22, x8
; CHECK-NEXT: .LBB0_6: // %bb10
; CHECK-NEXT: // =>This Inner Loop Header: Depth=1
diff --git a/llvm/test/CodeGen/AArch64/pr51516.mir b/llvm/test/CodeGen/AArch64/pr51516.mir
index ae54ad0d5cef4..42c501d31f385 100644
--- a/llvm/test/CodeGen/AArch64/pr51516.mir
+++ b/llvm/test/CodeGen/AArch64/pr51516.mir
@@ -1,14 +1,10 @@
+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py UTC_ARGS: --version 6
# RUN: llc -mtriple=aarch64-unknown-fuchsia -run-pass=greedy -verify-machineinstrs -o - %s | FileCheck %s
# RUN: llc -mtriple=aarch64-unknown-fuchsia -passes=greedy -verify-machineinstrs -o - %s | FileCheck %s
# Check that we spill %31 and do not rematerialize it since the use operand
# of ADDXri is killed by the STRXui in this block.
-# CHECK-LABEL: name: test
-# CHECK: bb.17:
-# CHECK: STRXui
-# CHECK: LDRXui
-# CHECK: bb.18:
---
name: test
@@ -17,6 +13,200 @@ stack:
- { id: 0, type: variable-sized, offset: 0, alignment: 32,
stack-id: default }
body: |
+ ; CHECK-LABEL: name: test
+ ; CHECK: bb.0.entry:
+ ; CHECK-NEXT: successors: %bb.2(0x40000000), %bb.1(0x40000000)
+ ; CHECK-NEXT: liveins: $x0
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64common = COPY $x0
+ ; CHECK-NEXT: CBZW $wzr, %bb.2
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.1:
+ ; CHECK-NEXT: successors: %bb.3(0x80000000)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr64 = COPY $xzr
+ ; CHECK-NEXT: B %bb.3
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.2:
+ ; CHECK-NEXT: successors: %bb.3(0x80000000)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: $x0 = IMPLICIT_DEF
+ ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr64 = COPY $x0
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.3:
+ ; CHECK-NEXT: successors: %bb.4(0x30000000), %bb.5(0x50000000)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: [[COPY2:%[0-9]+]]:gpr64common = COPY [[COPY1]]
+ ; CHECK-NEXT: CBNZX [[COPY1]], %bb.5
+ ; CHECK-NEXT: B %bb.4
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.4:
+ ; CHECK-NEXT: successors: %bb.5(0x80000000)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: [[SUBXri:%[0-9]+]]:gpr64common = SUBXri $sp, 288, 0
+ ; CHECK-NEXT: $sp = ANDXri [[SUBXri]], 7930
+ ; CHECK-NEXT: [[COPY2:%[0-9]+]]:gpr64common = COPY $xzr
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.5:
+ ; CHECK-NEXT: successors: %bb.6(0x80000000)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: [[COPY3:%[0-9]+]]:gpr64common = COPY $xzr
+ ; CHECK-NEXT: [[ADDXri:%[0-9]+]]:gpr64sp = ADDXri [[COPY2]], 32, 0
+ ; CHECK-NEXT: [[UBFMXri:%[0-9]+]]:gpr64common = UBFMXri [[COPY2]], 3, 63
+ ; CHECK-NEXT: [[MOVi64imm:%[0-9]+]]:gpr64 = MOVi64imm -506381209866536712
+ ; CHECK-NEXT: [[MOVi32imm:%[0-9]+]]:gpr32 = MOVi32imm -202116109
+ ; CHECK-NEXT: [[ADDXri1:%[0-9]+]]:gpr64common = nuw ADDXri [[COPY]], 836, 0
+ ; CHECK-NEXT: [[ADDXri2:%[0-9]+]]:gpr64common = nuw ADDXri [[COPY]], 648, 0
+ ; CHECK-NEXT: STRXui [[MOVi64imm]], [[UBFMXri]], 1
+ ; CHECK-NEXT: STRXui [[UBFMXri]], %stack.2, 0 :: (store (s64) into %stack.2)
+ ; CHECK-NEXT: STRWui [[MOVi32imm]], [[UBFMXri]], 8
+ ; CHECK-NEXT: [[UBFMXri1:%[0-9]+]]:gpr64common = UBFMXri [[ADDXri1]], 3, 63
+ ; CHECK-NEXT: [[UBFMXri2:%[0-9]+]]:gpr64common = UBFMXri [[ADDXri2]], 3, 63
+ ; CHECK-NEXT: dead [[MOVi64imm1:%[0-9]+]]:gpr64 = MOVi64imm 0
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.6:
+ ; CHECK-NEXT: successors: %bb.8(0x30000000), %bb.7(0x50000000)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: [[ADDXrr:%[0-9]+]]:gpr64common = ADDXrr [[COPY]], [[COPY3]]
+ ; CHECK-NEXT: [[ADDXri3:%[0-9]+]]:gpr64common = ADDXri [[ADDXrr]], 648, 0
+ ; CHECK-NEXT: [[ANDSXri:%[0-9]+]]:gpr64common = ANDSXri [[ADDXri3]], 4098, implicit-def $nzcv
+ ; CHECK-NEXT: Bcc 0, %bb.8, implicit killed $nzcv
+ ; CHECK-NEXT: B %bb.7
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.7:
+ ; CHECK-NEXT: successors: %bb.8(0x80000000)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: BL 0, csr_aarch64_aapcs, implicit-def dead $lr
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.8:
+ ; CHECK-NEXT: successors: %bb.9(0x04000000), %bb.23(0x7c000000)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: CBNZW $wzr, %bb.23
+ ; CHECK-NEXT: B %bb.9
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.9:
+ ; CHECK-NEXT: successors: %bb.10(0x7ffff800), %bb.11(0x00000800)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: [[ADDXrr1:%[0-9]+]]:gpr64common = ADDXrr [[COPY]], [[COPY3]]
+ ; CHECK-NEXT: [[ADDXri4:%[0-9]+]]:gpr64common = ADDXri [[ADDXrr1]], 648, 0
+ ; CHECK-NEXT: STRXui [[ADDXri4]], %stack.1, 0 :: (store (s64) into %stack.1)
+ ; CHECK-NEXT: CBZX [[ANDSXri]], %bb.10
+ ; CHECK-NEXT: B %bb.11
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.10:
+ ; CHECK-NEXT: successors: %bb.11(0x80000000)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: $x0 = ADDXri [[ADDXrr1]], 648, 0
+ ; CHECK-NEXT: $x2 = IMPLICIT_DEF
+ ; CHECK-NEXT: $w1 = COPY $wzr
+ ; CHECK-NEXT: $x1 = nuw ADDXri [[ADDXri]], 32, 0
+ ; CHECK-NEXT: BL 0, csr_aarch64_aapcs, implicit-def dead $lr
+ ; CHECK-NEXT: [[ADDXrr1:%[0-9]+]]:gpr64common = ADDXrr [[COPY]], [[COPY3]]
+ ; CHECK-NEXT: [[ADDXri5:%[0-9]+]]:gpr64sp = nuw ADDXri [[ADDXri]], 48, 0
+ ; CHECK-NEXT: $x0 = ADDXri [[ADDXrr1]], 696, 0
+ ; CHECK-NEXT: $x2 = IMPLICIT_DEF
+ ; CHECK-NEXT: $x1 = COPY [[ADDXri5]]
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.11:
+ ; CHECK-NEXT: successors: %bb.15(0x7ffff800), %bb.12(0x00000800)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: CBZX [[ANDSXri]], %bb.15
+ ; CHECK-NEXT: B %bb.12
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.12:
+ ; CHECK-NEXT: successors: %bb.13(0x00000000), %bb.14(0x80000000)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: CBNZW $wzr, %bb.14
+ ; CHECK-NEXT: B %bb.13
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.13:
+ ; CHECK-NEXT: successors:
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.14:
+ ; CHECK-NEXT: successors: %bb.18(0x80000000)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: $x1 = LDRXui %stack.1, 0 :: (load (s64) from %stack.1)
+ ; CHECK-NEXT: B %bb.18
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.15:
+ ; CHECK-NEXT: successors: %bb.16(0x00000000), %bb.17(0x80000000)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: [[LDRBBui:%[0-9]+]]:gpr32 = LDRBBui [[UBFMXri2]], 0
+ ; CHECK-NEXT: CBZW [[LDRBBui]], %bb.17
+ ; CHECK-NEXT: B %bb.16
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.16:
+ ; CHECK-NEXT: successors:
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: [[ADDXrr2:%[0-9]+]]:gpr64common = ADDXrr [[COPY]], [[COPY3]]
+ ; CHECK-NEXT: [[ADDXri6:%[0-9]+]]:gpr64sp = ADDXri [[ADDXrr2]], 648, 0
+ ; CHECK-NEXT: $x0 = COPY [[ADDXri6]]
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.17:
+ ; CHECK-NEXT: successors: %bb.18(0x80000000)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: STRXui [[COPY]], [[ADDXrr1]], 81
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.18:
+ ; CHECK-NEXT: successors: %bb.19(0x80000000), %bb.20(0x00000000)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: [[LDRBBui1:%[0-9]+]]:gpr32 = LDRBBui [[UBFMXri1]], 0
+ ; CHECK-NEXT: CBNZW [[LDRBBui1]], %bb.20
+ ; CHECK-NEXT: B %bb.19
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.19:
+ ; CHECK-NEXT: successors: %bb.21(0x80000000), %bb.20(0x00000000)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: [[MOVi32imm1:%[0-9]+]]:gpr32 = MOVi32imm 1
+ ; CHECK-NEXT: CBNZW [[MOVi32imm1]], %bb.21
+ ; CHECK-NEXT: B %bb.20
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.20:
+ ; CHECK-NEXT: successors:
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: [[ADDXri7:%[0-9]+]]:gpr64sp = ADDXri [[COPY]], 836, 0
+ ; CHECK-NEXT: $x0 = COPY [[ADDXri7]]
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.21:
+ ; CHECK-NEXT: successors: %bb.23(0x00000000), %bb.22(0x80000000)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: CBZW $wzr, %bb.23
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.22:
+ ; CHECK-NEXT: successors: %bb.25(0x80000000)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: [[LDRXui:%[0-9]+]]:gpr64common = LDRXui %stack.1, 0 :: (load (s64) from %stack.1)
+ ; CHECK-NEXT: B %bb.25
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.23:
+ ; CHECK-NEXT: successors: %bb.24(0x04000000), %bb.6(0x7c000000)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: [[UBFMXri1:%[0-9]+]]:gpr64common = ADDXri [[UBFMXri1]], 24, 0
+ ; CHECK-NEXT: [[UBFMXri2:%[0-9]+]]:gpr64common = ADDXri [[UBFMXri2]], 24, 0
+ ; CHECK-NEXT: CBNZW $wzr, %bb.6
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.24:
+ ; CHECK-NEXT: successors: %bb.25(0x80000000)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: [[LDRXui:%[0-9]+]]:gpr64common = COPY $xzr
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.25:
+ ; CHECK-NEXT: successors: %bb.27(0x50000000), %bb.26(0x30000000)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: undef [[MOVi32imm2:%[0-9]+]].sub_32:gpr64 = MOVi32imm 1172321806
+ ; CHECK-NEXT: STRXui [[MOVi32imm2]], [[COPY2]], 0
+ ; CHECK-NEXT: [[LDRXui1:%[0-9]+]]:gpr64common = LDRXui %stack.2, 0 :: (load (s64) from %stack.2)
+ ; CHECK-NEXT: CBNZX [[COPY1]], %bb.27
+ ; CHECK-NEXT: B %bb.26
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.26:
+ ; CHECK-NEXT: successors: %bb.27(0x80000000)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: STRXui $xzr, [[LDRXui1]], 0
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.27:
+ ; CHECK-NEXT: $x0 = COPY [[LDRXui]]
+ ; CHECK-NEXT: RET_ReallyLR implicit $x0
bb.0.entry:
successors: %bb.2(0x40000000), %bb.1(0x40000000)
liveins: $x0
diff --git a/llvm/test/CodeGen/AArch64/sme-callee-save-restore-pairs.ll b/llvm/test/CodeGen/AArch64/sme-callee-save-restore-pairs.ll
index b9a542b330c0f..03ebba5a3e308 100644
--- a/llvm/test/CodeGen/AArch64/sme-callee-save-restore-pairs.ll
+++ b/llvm/test/CodeGen/AArch64/sme-callee-save-restore-pairs.ll
@@ -42,18 +42,19 @@ define void @fbyte(<vscale x 16 x i8> %v) #0{
; NOPAIR-NEXT: str z8, [sp, #17, mul vl] // 16-byte Folded Spill
; NOPAIR-NEXT: addvl sp, sp, #-1
; NOPAIR-NEXT: str z0, [sp] // 16-byte Folded Spill
-; NOPAIR-NEXT: mrs x19, SVCR
-; NOPAIR-NEXT: tbz w19, #0, .LBB0_2
+; NOPAIR-NEXT: mrs x8, SVCR
+; NOPAIR-NEXT: tbz w8, #0, .LBB0_2
; NOPAIR-NEXT: // %bb.1:
; NOPAIR-NEXT: smstop sm
; NOPAIR-NEXT: .LBB0_2:
-; NOPAIR-NEXT: rdvl x8, #1
-; NOPAIR-NEXT: addsvl x8, x8, #-1
-; NOPAIR-NEXT: cbz x8, .LBB0_4
+; NOPAIR-NEXT: rdvl x9, #1
+; NOPAIR-NEXT: addsvl x9, x9, #-1
+; NOPAIR-NEXT: cbz x9, .LBB0_4
; NOPAIR-NEXT: // %bb.3:
; NOPAIR-NEXT: brk #0x1
; NOPAIR-NEXT: .LBB0_4:
; NOPAIR-NEXT: ldr z0, [sp] // 16-byte Folded Reload
+; NOPAIR-NEXT: mov x19, x8
; NOPAIR-NEXT: bl my_func2
; NOPAIR-NEXT: tbz w19, #0, .LBB0_6
; NOPAIR-NEXT: // %bb.5:
@@ -128,18 +129,19 @@ define void @fbyte(<vscale x 16 x i8> %v) #0{
; PAIR-NEXT: str z8, [sp, #17, mul vl] // 16-byte Folded Spill
; PAIR-NEXT: addvl sp, sp, #-1
; PAIR-NEXT: str z0, [sp] // 16-byte Folded Spill
-; PAIR-NEXT: mrs x19, SVCR
-; PAIR-NEXT: tbz w19, #0, .LBB0_2
+; PAIR-NEXT: mrs x8, SVCR
+; PAIR-NEXT: tbz w8, #0, .LBB0_2
; PAIR-NEXT: // %bb.1:
; PAIR-NEXT: smstop sm
; PAIR-NEXT: .LBB0_2:
-; PAIR-NEXT: rdvl x8, #1
-; PAIR-NEXT: addsvl x8, x8, #-1
-; PAIR-NEXT: cbz x8, .LBB0_4
+; PAIR-NEXT: rdvl x9, #1
+; PAIR-NEXT: addsvl x9, x9, #-1
+; PAIR-NEXT: cbz x9, .LBB0_4
; PAIR-NEXT: // %bb.3:
; PAIR-NEXT: brk #0x1
; PAIR-NEXT: .LBB0_4:
; PAIR-NEXT: ldr z0, [sp] // 16-byte Folded Reload
+; PAIR-NEXT: mov x19, x8
; PAIR-NEXT: bl my_func2
; PAIR-NEXT: tbz w19, #0, .LBB0_6
; PAIR-NEXT: // %bb.5:
diff --git a/llvm/test/CodeGen/AArch64/sme-peephole-opts.ll b/llvm/test/CodeGen/AArch64/sme-peephole-opts.ll
index 36539d94338a0..850aa7a63e016 100644
--- a/llvm/test/CodeGen/AArch64/sme-peephole-opts.ll
+++ b/llvm/test/CodeGen/AArch64/sme-peephole-opts.ll
@@ -514,24 +514,27 @@ define void @test12() "aarch64_pstate_sm_body" {
define void @test13(ptr %ptr) nounwind "aarch64_pstate_sm_enabled" {
; CHECK-LABEL: test13:
; CHECK: // %bb.0:
-; CHECK-NEXT: stp d15, d14, [sp, #-96]! // 16-byte Folded Spill
+; CHECK-NEXT: stp d15, d14, [sp, #-80]! // 16-byte Folded Spill
; CHECK-NEXT: stp d13, d12, [sp, #16] // 16-byte Folded Spill
; CHECK-NEXT: stp d11, d10, [sp, #32] // 16-byte Folded Spill
; CHECK-NEXT: stp d9, d8, [sp, #48] // 16-byte Folded Spill
-; CHECK-NEXT: str x29, [sp, #64] // 8-byte Spill
-; CHECK-NEXT: stp x30, x19, [sp, #80] // 16-byte Folded Spill
+; CHECK-NEXT: stp x29, x30, [sp, #64] // 16-byte Folded Spill
+; CHECK-NEXT: sub sp, sp, #16
; CHECK-NEXT: addvl sp, sp, #-1
; CHECK-NEXT: mov z0.s, #0 // =0x0
-; CHECK-NEXT: str z0, [sp] // 16-byte Folded Spill
+; CHECK-NEXT: add x8, sp, #16
+; CHECK-NEXT: str z0, [x8] // 16-byte Folded Spill
; CHECK-NEXT: smstop sm
; CHECK-NEXT: rdvl x8, #1
; CHECK-NEXT: addsvl x8, x8, #-1
; CHECK-NEXT: cbnz x8, .LBB14_2
; CHECK-NEXT: // %bb.1:
-; CHECK-NEXT: ldr z0, [sp] // 16-byte Folded Reload
-; CHECK-NEXT: mov x19, x0
+; CHECK-NEXT: add x8, sp, #16
+; CHECK-NEXT: str x0, [sp, #8] // 8-byte Spill
+; CHECK-NEXT: ldr z0, [x8] // 16-byte Folded Reload
; CHECK-NEXT: bl callee_farg_fret
-; CHECK-NEXT: str z0, [sp] // 16-byte Folded Spill
+; CHECK-NEXT: add x8, sp, #16
+; CHECK-NEXT: str z0, [x8] // 16-byte Folded Spill
; CHECK-NEXT: smstart sm
; CHECK-NEXT: smstop sm
; CHECK-NEXT: rdvl x8, #1
@@ -540,19 +543,23 @@ define void @test13(ptr %ptr) nounwind "aarch64_pstate_sm_enabled" {
; CHECK-NEXT: .LBB14_2:
; CHECK-NEXT: brk #0x1
; CHECK-NEXT: .LBB14_3:
-; CHECK-NEXT: ldr z0, [sp] // 16-byte Folded Reload
+; CHECK-NEXT: add x8, sp, #16
+; CHECK-NEXT: ldr z0, [x8] // 16-byte Folded Reload
; CHECK-NEXT: bl callee_farg_fret
-; CHECK-NEXT: str z0, [sp] // 16-byte Folded Spill
+; CHECK-NEXT: add x8, sp, #16
+; CHECK-NEXT: str z0, [x8] // 16-byte Folded Spill
; CHECK-NEXT: smstart sm
-; CHECK-NEXT: ldr z0, [sp] // 16-byte Folded Reload
-; CHECK-NEXT: str z0, [x19]
+; CHECK-NEXT: add x9, sp, #16
+; CHECK-NEXT: ldr x8, [sp, #8] // 8-byte Reload
+; CHECK-NEXT: ldr z0, [x9] // 16-byte Folded Reload
+; CHECK-NEXT: str z0, [x8]
; CHECK-NEXT: addvl sp, sp, #1
-; CHECK-NEXT: ldp x30, x19, [sp, #80] // 16-byte Folded Reload
-; CHECK-NEXT: ldr x29, [sp, #64] // 8-byte Reload
+; CHECK-NEXT: add sp, sp, #16
+; CHECK-NEXT: ldp x29, x30, [sp, #64] // 16-byte Folded Reload
; CHECK-NEXT: ldp d9, d8, [sp, #48] // 16-byte Folded Reload
; CHECK-NEXT: ldp d11, d10, [sp, #32] // 16-byte Folded Reload
; CHECK-NEXT: ldp d13, d12, [sp, #16] // 16-byte Folded Reload
-; CHECK-NEXT: ldp d15, d14, [sp], #96 // 16-byte Folded Reload
+; CHECK-NEXT: ldp d15, d14, [sp], #80 // 16-byte Folded Reload
; CHECK-NEXT: ret
%res0 = call <vscale x 4 x float> @callee_farg_fret(<vscale x 4 x float> zeroinitializer)
%res1 = call <vscale x 4 x float> @callee_farg_fret(<vscale x 4 x float> %res0)
diff --git a/llvm/test/CodeGen/AArch64/sme-streaming-checkvl.ll b/llvm/test/CodeGen/AArch64/sme-streaming-checkvl.ll
index 475cb2879da70..0562097936c99 100644
--- a/llvm/test/CodeGen/AArch64/sme-streaming-checkvl.ll
+++ b/llvm/test/CodeGen/AArch64/sme-streaming-checkvl.ll
@@ -109,21 +109,22 @@ define void @foo_streaming_compatible_pass_arg(ptr %arg) #1 {
; CHECK-NEXT: .cfi_offset b15, -1136
; CHECK-NEXT: sub sp, sp, #1024
; CHECK-NEXT: addvl sp, sp, #-1
-; CHECK-NEXT: mrs x19, SVCR
+; CHECK-NEXT: mrs x8, SVCR
; CHECK-NEXT: ldr z0, [x0]
-; CHECK-NEXT: rdvl x8, #1
-; CHECK-NEXT: addsvl x8, x8, #-1
-; CHECK-NEXT: cbz x8, .LBB1_2
+; CHECK-NEXT: rdvl x9, #1
+; CHECK-NEXT: addsvl x9, x9, #-1
+; CHECK-NEXT: cbz x9, .LBB1_2
; CHECK-NEXT: // %bb.1: // %entry
; CHECK-NEXT: brk #0x1
; CHECK-NEXT: .LBB1_2: // %entry
-; CHECK-NEXT: sub x8, x29, #1088
-; CHECK-NEXT: str z0, [x8, #-1, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: tbnz w19, #0, .LBB1_4
+; CHECK-NEXT: sub x9, x29, #1088
+; CHECK-NEXT: str z0, [x9, #-1, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: tbnz w8, #0, .LBB1_4
; CHECK-NEXT: // %bb.3: // %entry
; CHECK-NEXT: smstart sm
; CHECK-NEXT: .LBB1_4: // %entry
-; CHECK-NEXT: ldr z0, [x8, #-1, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z0, [x9, #-1, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: mov x19, x8
; CHECK-NEXT: bl bar_enabled
; CHECK-NEXT: tbnz w19, #0, .LBB1_6
; CHECK-NEXT: // %bb.5: // %entry
@@ -273,8 +274,8 @@ define void @foo_non_streaming_retval(ptr %ptr) {
; CHECK-NEXT: // %bb.1: // %entry
; CHECK-NEXT: brk #0x1
; CHECK-NEXT: .LBB3_2: // %entry
-; CHECK-NEXT: mov x19, x0
; CHECK-NEXT: smstart sm
+; CHECK-NEXT: mov x19, x0
; CHECK-NEXT: bl bar_retv_enabled
; CHECK-NEXT: sub x8, x29, #64
; CHECK-NEXT: str z0, [x8, #-1, mul vl] // 16-byte Folded Spill
@@ -345,18 +346,19 @@ define void @foo_streaming_compatible_retval(ptr %ptr) #1 {
; CHECK-NEXT: .cfi_offset b15, -1136
; CHECK-NEXT: sub sp, sp, #1024
; CHECK-NEXT: addvl sp, sp, #-1
-; CHECK-NEXT: rdvl x8, #1
-; CHECK-NEXT: mrs x20, SVCR
-; CHECK-NEXT: addsvl x8, x8, #-1
-; CHECK-NEXT: cbz x8, .LBB4_2
+; CHECK-NEXT: rdvl x9, #1
+; CHECK-NEXT: mrs x8, SVCR
+; CHECK-NEXT: addsvl x9, x9, #-1
+; CHECK-NEXT: cbz x9, .LBB4_2
; CHECK-NEXT: // %bb.1: // %entry
; CHECK-NEXT: brk #0x1
; CHECK-NEXT: .LBB4_2: // %entry
-; CHECK-NEXT: mov x19, x0
-; CHECK-NEXT: tbnz w20, #0, .LBB4_4
+; CHECK-NEXT: tbnz w8, #0, .LBB4_4
; CHECK-NEXT: // %bb.3: // %entry
; CHECK-NEXT: smstart sm
; CHECK-NEXT: .LBB4_4: // %entry
+; CHECK-NEXT: mov x19, x0
+; CHECK-NEXT: mov x20, x8
; CHECK-NEXT: bl bar_retv_enabled
; CHECK-NEXT: sub x8, x29, #1088
; CHECK-NEXT: str z0, [x8, #-1, mul vl] // 16-byte Folded Spill
diff --git a/llvm/test/CodeGen/AArch64/sme-streaming-compatible-interface.ll b/llvm/test/CodeGen/AArch64/sme-streaming-compatible-interface.ll
index 944c131260c6e..249b5643eff81 100644
--- a/llvm/test/CodeGen/AArch64/sme-streaming-compatible-interface.ll
+++ b/llvm/test/CodeGen/AArch64/sme-streaming-compatible-interface.ll
@@ -203,18 +203,19 @@ define <vscale x 2 x double> @streaming_compatible_with_scalable_vectors(<vscale
; CHECK-NEXT: str z8, [sp, #17, mul vl] // 16-byte Folded Spill
; CHECK-NEXT: addvl sp, sp, #-2
; CHECK-NEXT: str z0, [sp, #1, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mrs x19, SVCR
-; CHECK-NEXT: tbz w19, #0, .LBB5_2
+; CHECK-NEXT: mrs x8, SVCR
+; CHECK-NEXT: tbz w8, #0, .LBB5_2
; CHECK-NEXT: // %bb.1:
; CHECK-NEXT: smstop sm
; CHECK-NEXT: .LBB5_2:
-; CHECK-NEXT: rdvl x8, #1
-; CHECK-NEXT: addsvl x8, x8, #-1
-; CHECK-NEXT: cbz x8, .LBB5_4
+; CHECK-NEXT: rdvl x9, #1
+; CHECK-NEXT: addsvl x9, x9, #-1
+; CHECK-NEXT: cbz x9, .LBB5_4
; CHECK-NEXT: // %bb.3:
; CHECK-NEXT: brk #0x1
; CHECK-NEXT: .LBB5_4:
; CHECK-NEXT: ldr z0, [sp, #1, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: mov x19, x8
; CHECK-NEXT: bl normal_callee_scalable_vec_arg
; CHECK-NEXT: str z0, [sp] // 16-byte Folded Spill
; CHECK-NEXT: tbz w19, #0, .LBB5_6
@@ -300,18 +301,19 @@ define <vscale x 2 x i1> @streaming_compatible_with_predicate_vectors(<vscale x
; CHECK-NEXT: str z8, [sp, #17, mul vl] // 16-byte Folded Spill
; CHECK-NEXT: addvl sp, sp, #-1
; CHECK-NEXT: str p0, [sp, #7, mul vl] // 2-byte Spill
-; CHECK-NEXT: mrs x19, SVCR
-; CHECK-NEXT: tbz w19, #0, .LBB6_2
+; CHECK-NEXT: mrs x8, SVCR
+; CHECK-NEXT: tbz w8, #0, .LBB6_2
; CHECK-NEXT: // %bb.1:
; CHECK-NEXT: smstop sm
; CHECK-NEXT: .LBB6_2:
-; CHECK-NEXT: rdvl x8, #1
-; CHECK-NEXT: addsvl x8, x8, #-1
-; CHECK-NEXT: cbz x8, .LBB6_4
+; CHECK-NEXT: rdvl x9, #1
+; CHECK-NEXT: addsvl x9, x9, #-1
+; CHECK-NEXT: cbz x9, .LBB6_4
; CHECK-NEXT: // %bb.3:
; CHECK-NEXT: brk #0x1
; CHECK-NEXT: .LBB6_4:
; CHECK-NEXT: ldr p0, [sp, #7, mul vl] // 2-byte Reload
+; CHECK-NEXT: mov x19, x8
; CHECK-NEXT: bl normal_callee_predicate_vec_arg
; CHECK-NEXT: str p0, [sp, #6, mul vl] // 2-byte Spill
; CHECK-NEXT: tbz w19, #0, .LBB6_6
@@ -382,28 +384,30 @@ define i32 @conditional_smstart_unreachable_block() "aarch64_pstate_sm_compatibl
define void @conditional_smstart_no_successor_block(i1 %p) "aarch64_pstate_sm_compatible" nounwind {
; CHECK-LABEL: conditional_smstart_no_successor_block:
; CHECK: // %bb.0:
+; CHECK-NEXT: mrs x8, SVCR
+; CHECK-NEXT: tbz w0, #0, .LBB8_6
+; CHECK-NEXT: // %bb.1: // %if.then
; CHECK-NEXT: stp d15, d14, [sp, #-80]! // 16-byte Folded Spill
; CHECK-NEXT: stp d13, d12, [sp, #16] // 16-byte Folded Spill
; CHECK-NEXT: stp d11, d10, [sp, #32] // 16-byte Folded Spill
; CHECK-NEXT: stp d9, d8, [sp, #48] // 16-byte Folded Spill
; CHECK-NEXT: stp x30, x19, [sp, #64] // 16-byte Folded Spill
-; CHECK-NEXT: mrs x19, SVCR
-; CHECK-NEXT: tbz w0, #0, .LBB8_5
-; CHECK-NEXT: // %bb.1: // %if.then
-; CHECK-NEXT: tbnz w19, #0, .LBB8_3
+; CHECK-NEXT: tbnz w8, #0, .LBB8_3
; CHECK-NEXT: // %bb.2: // %if.then
; CHECK-NEXT: smstart sm
; CHECK-NEXT: .LBB8_3: // %if.then
+; CHECK-NEXT: mov x19, x8
; CHECK-NEXT: bl streaming_callee
; CHECK-NEXT: tbnz w19, #0, .LBB8_5
; CHECK-NEXT: // %bb.4: // %if.then
; CHECK-NEXT: smstop sm
-; CHECK-NEXT: .LBB8_5: // %exit
+; CHECK-NEXT: .LBB8_5: // %if.then
; CHECK-NEXT: ldp x30, x19, [sp, #64] // 16-byte Folded Reload
; CHECK-NEXT: ldp d9, d8, [sp, #48] // 16-byte Folded Reload
; CHECK-NEXT: ldp d11, d10, [sp, #32] // 16-byte Folded Reload
; CHECK-NEXT: ldp d13, d12, [sp, #16] // 16-byte Folded Reload
; CHECK-NEXT: ldp d15, d14, [sp], #80 // 16-byte Folded Reload
+; CHECK-NEXT: .LBB8_6: // %exit
; CHECK-NEXT: ret
br i1 %p, label %if.then, label %exit
diff --git a/llvm/test/CodeGen/AArch64/spill-reload-remarks.ll b/llvm/test/CodeGen/AArch64/spill-reload-remarks.ll
index 33a4ecd56e35b..8a12822aac985 100644
--- a/llvm/test/CodeGen/AArch64/spill-reload-remarks.ll
+++ b/llvm/test/CodeGen/AArch64/spill-reload-remarks.ll
@@ -2,7 +2,7 @@
; We should have both spill and reload for %arg.
-; CHECK: remark: <unknown>:0:0: 2 spills 1.500000e+00 total spills cost 3 reloads 1.500000e+00 total reloads cost generated in function
+; CHECK: remark: <unknown>:0:0: 2 spills 1.500000e+00 total spills cost 3 reloads 1.500000e+00 total reloads cost 1 virtual registers copies 5.000000e-01 total copies cost generated in function
define <vscale x 2 x i1> @streaming_compatible_with_predicate_vectors(<vscale x 2 x i1> %arg) "aarch64_pstate_sm_compatible" nounwind #0 {
%res = call <vscale x 2 x i1> @normal_callee_predicate_vec_arg(<vscale x 2 x i1> %arg)
%and = and <vscale x 2 x i1> %res, %arg
diff --git a/llvm/test/CodeGen/AArch64/statepoint-call-lowering.ll b/llvm/test/CodeGen/AArch64/statepoint-call-lowering.ll
index 167c6659cd2f6..d00dd7c84b227 100644
--- a/llvm/test/CodeGen/AArch64/statepoint-call-lowering.ll
+++ b/llvm/test/CodeGen/AArch64/statepoint-call-lowering.ll
@@ -157,28 +157,29 @@ declare void @consume(ptr addrspace(1) %obj)
define i1 @test_cross_bb(ptr addrspace(1) %a, i1 %external_cond) gc "statepoint-example" {
; CHECK-LABEL: test_cross_bb:
; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: str x30, [sp, #-32]! // 8-byte Folded Spill
-; CHECK-NEXT: stp x20, x19, [sp, #16] // 16-byte Folded Spill
+; CHECK-NEXT: sub sp, sp, #32
+; CHECK-NEXT: stp x30, x19, [sp, #16] // 16-byte Folded Spill
; CHECK-NEXT: .cfi_def_cfa_offset 32
; CHECK-NEXT: .cfi_offset w19, -8
-; CHECK-NEXT: .cfi_offset w20, -16
-; CHECK-NEXT: .cfi_offset w30, -32
-; CHECK-NEXT: mov w20, w1
+; CHECK-NEXT: .cfi_offset w30, -16
+; CHECK-NEXT: mov w19, w1
; CHECK-NEXT: str x0, [sp, #8]
; CHECK-NEXT: bl return_i1
; CHECK-NEXT: .Ltmp8:
-; CHECK-NEXT: tbz w20, #0, .LBB8_2
+; CHECK-NEXT: tbz w19, #0, .LBB8_2
; CHECK-NEXT: // %bb.1: // %left
+; CHECK-NEXT: ldr x8, [sp, #8]
; CHECK-NEXT: mov w19, w0
-; CHECK-NEXT: ldr x0, [sp, #8]
+; CHECK-NEXT: mov x0, x8
; CHECK-NEXT: bl consume
+; CHECK-NEXT: mov w8, w19
; CHECK-NEXT: b .LBB8_3
; CHECK-NEXT: .LBB8_2:
-; CHECK-NEXT: mov w19, #1 // =0x1
+; CHECK-NEXT: mov w8, #1 // =0x1
; CHECK-NEXT: .LBB8_3: // %common.ret
-; CHECK-NEXT: and w0, w19, #0x1
-; CHECK-NEXT: ldp x20, x19, [sp, #16] // 16-byte Folded Reload
-; CHECK-NEXT: ldr x30, [sp], #32 // 8-byte Folded Reload
+; CHECK-NEXT: ldp x30, x19, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEXT: and w0, w8, #0x1
+; CHECK-NEXT: add sp, sp, #32
; CHECK-NEXT: ret
entry:
%safepoint_token = tail call token (i64, i32, ptr, i32, i32, ...) @llvm.experimental.gc.statepoint.p0(i64 0, i32 0, ptr elementtype(i1 ()) @return_i1, i32 0, i32 0, i32 0, i32 0) ["gc-live" (ptr addrspace(1) %a)]
diff --git a/llvm/test/CodeGen/AArch64/sve-breakdown-scalable-vectortype.ll b/llvm/test/CodeGen/AArch64/sve-breakdown-scalable-vectortype.ll
index 3645af3ccaee0..8224c4938c4b9 100644
--- a/llvm/test/CodeGen/AArch64/sve-breakdown-scalable-vectortype.ll
+++ b/llvm/test/CodeGen/AArch64/sve-breakdown-scalable-vectortype.ll
@@ -13,22 +13,20 @@ declare aarch64_sve_vector_pcs void @bar()
define <vscale x 32 x i8> @wide_32i8(i1 %b, <vscale x 16 x i8> %legal, <vscale x 32 x i8> %illegal) nounwind {
; CHECK-LABEL: wide_32i8:
; CHECK: // %bb.0:
-; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
-; CHECK-NEXT: addvl sp, sp, #-2
-; CHECK-NEXT: str z9, [sp] // 16-byte Folded Spill
-; CHECK-NEXT: mov z9.d, z1.d
-; CHECK-NEXT: str z8, [sp, #1, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z8.d, z2.d
+; CHECK-NEXT: mov z0.d, z1.d
; CHECK-NEXT: tbz w0, #0, .LBB0_2
; CHECK-NEXT: // %bb.1: // %L1
+; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT: addvl sp, sp, #-2
+; CHECK-NEXT: str z2, [sp, #1, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z0, [sp] // 16-byte Folded Spill
; CHECK-NEXT: bl bar
-; CHECK-NEXT: .LBB0_2: // %common.ret
-; CHECK-NEXT: mov z0.d, z9.d
-; CHECK-NEXT: mov z1.d, z8.d
-; CHECK-NEXT: ldr z9, [sp] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z8, [sp, #1, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z0, [sp] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z2, [sp, #1, mul vl] // 16-byte Folded Reload
; CHECK-NEXT: addvl sp, sp, #2
; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT: .LBB0_2: // %common.ret
+; CHECK-NEXT: mov z1.d, z2.d
; CHECK-NEXT: ret
br i1 %b, label %L1, label %L2
L1:
@@ -41,22 +39,20 @@ L2:
define <vscale x 16 x i16> @wide_16i16(i1 %b, <vscale x 16 x i8> %legal, <vscale x 16 x i16> %illegal) nounwind {
; CHECK-LABEL: wide_16i16:
; CHECK: // %bb.0:
-; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
-; CHECK-NEXT: addvl sp, sp, #-2
-; CHECK-NEXT: str z9, [sp] // 16-byte Folded Spill
-; CHECK-NEXT: mov z9.d, z1.d
-; CHECK-NEXT: str z8, [sp, #1, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z8.d, z2.d
+; CHECK-NEXT: mov z0.d, z1.d
; CHECK-NEXT: tbz w0, #0, .LBB1_2
; CHECK-NEXT: // %bb.1: // %L1
+; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT: addvl sp, sp, #-2
+; CHECK-NEXT: str z2, [sp, #1, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z0, [sp] // 16-byte Folded Spill
; CHECK-NEXT: bl bar
-; CHECK-NEXT: .LBB1_2: // %common.ret
-; CHECK-NEXT: mov z0.d, z9.d
-; CHECK-NEXT: mov z1.d, z8.d
-; CHECK-NEXT: ldr z9, [sp] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z8, [sp, #1, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z0, [sp] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z2, [sp, #1, mul vl] // 16-byte Folded Reload
; CHECK-NEXT: addvl sp, sp, #2
; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT: .LBB1_2: // %common.ret
+; CHECK-NEXT: mov z1.d, z2.d
; CHECK-NEXT: ret
br i1 %b, label %L1, label %L2
L1:
@@ -69,22 +65,20 @@ L2:
define <vscale x 8 x i32> @wide_8i32(i1 %b, <vscale x 16 x i8> %legal, <vscale x 8 x i32> %illegal) nounwind {
; CHECK-LABEL: wide_8i32:
; CHECK: // %bb.0:
-; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
-; CHECK-NEXT: addvl sp, sp, #-2
-; CHECK-NEXT: str z9, [sp] // 16-byte Folded Spill
-; CHECK-NEXT: mov z9.d, z1.d
-; CHECK-NEXT: str z8, [sp, #1, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z8.d, z2.d
+; CHECK-NEXT: mov z0.d, z1.d
; CHECK-NEXT: tbz w0, #0, .LBB2_2
; CHECK-NEXT: // %bb.1: // %L1
+; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT: addvl sp, sp, #-2
+; CHECK-NEXT: str z2, [sp, #1, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z0, [sp] // 16-byte Folded Spill
; CHECK-NEXT: bl bar
-; CHECK-NEXT: .LBB2_2: // %common.ret
-; CHECK-NEXT: mov z0.d, z9.d
-; CHECK-NEXT: mov z1.d, z8.d
-; CHECK-NEXT: ldr z9, [sp] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z8, [sp, #1, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z0, [sp] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z2, [sp, #1, mul vl] // 16-byte Folded Reload
; CHECK-NEXT: addvl sp, sp, #2
; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT: .LBB2_2: // %common.ret
+; CHECK-NEXT: mov z1.d, z2.d
; CHECK-NEXT: ret
br i1 %b, label %L1, label %L2
L1:
@@ -97,22 +91,20 @@ L2:
define <vscale x 4 x i64> @wide_4i64(i1 %b, <vscale x 16 x i8> %legal, <vscale x 4 x i64> %illegal) nounwind {
; CHECK-LABEL: wide_4i64:
; CHECK: // %bb.0:
-; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
-; CHECK-NEXT: addvl sp, sp, #-2
-; CHECK-NEXT: str z9, [sp] // 16-byte Folded Spill
-; CHECK-NEXT: mov z9.d, z1.d
-; CHECK-NEXT: str z8, [sp, #1, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z8.d, z2.d
+; CHECK-NEXT: mov z0.d, z1.d
; CHECK-NEXT: tbz w0, #0, .LBB3_2
; CHECK-NEXT: // %bb.1: // %L1
+; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT: addvl sp, sp, #-2
+; CHECK-NEXT: str z2, [sp, #1, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z0, [sp] // 16-byte Folded Spill
; CHECK-NEXT: bl bar
-; CHECK-NEXT: .LBB3_2: // %common.ret
-; CHECK-NEXT: mov z0.d, z9.d
-; CHECK-NEXT: mov z1.d, z8.d
-; CHECK-NEXT: ldr z9, [sp] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z8, [sp, #1, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z0, [sp] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z2, [sp, #1, mul vl] // 16-byte Folded Reload
; CHECK-NEXT: addvl sp, sp, #2
; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT: .LBB3_2: // %common.ret
+; CHECK-NEXT: mov z1.d, z2.d
; CHECK-NEXT: ret
br i1 %b, label %L1, label %L2
L1:
@@ -125,22 +117,20 @@ L2:
define <vscale x 16 x half> @wide_16f16(i1 %b, <vscale x 16 x i8> %legal, <vscale x 16 x half> %illegal) nounwind {
; CHECK-LABEL: wide_16f16:
; CHECK: // %bb.0:
-; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
-; CHECK-NEXT: addvl sp, sp, #-2
-; CHECK-NEXT: str z9, [sp] // 16-byte Folded Spill
-; CHECK-NEXT: mov z9.d, z1.d
-; CHECK-NEXT: str z8, [sp, #1, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z8.d, z2.d
+; CHECK-NEXT: mov z0.d, z1.d
; CHECK-NEXT: tbz w0, #0, .LBB4_2
; CHECK-NEXT: // %bb.1: // %L1
+; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT: addvl sp, sp, #-2
+; CHECK-NEXT: str z2, [sp, #1, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z0, [sp] // 16-byte Folded Spill
; CHECK-NEXT: bl bar
-; CHECK-NEXT: .LBB4_2: // %common.ret
-; CHECK-NEXT: mov z0.d, z9.d
-; CHECK-NEXT: mov z1.d, z8.d
-; CHECK-NEXT: ldr z9, [sp] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z8, [sp, #1, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z0, [sp] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z2, [sp, #1, mul vl] // 16-byte Folded Reload
; CHECK-NEXT: addvl sp, sp, #2
; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT: .LBB4_2: // %common.ret
+; CHECK-NEXT: mov z1.d, z2.d
; CHECK-NEXT: ret
br i1 %b, label %L1, label %L2
L1:
@@ -153,22 +143,20 @@ L2:
define <vscale x 8 x float> @wide_8f32(i1 %b, <vscale x 16 x i8> %legal, <vscale x 8 x float> %illegal) nounwind {
; CHECK-LABEL: wide_8f32:
; CHECK: // %bb.0:
-; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
-; CHECK-NEXT: addvl sp, sp, #-2
-; CHECK-NEXT: str z9, [sp] // 16-byte Folded Spill
-; CHECK-NEXT: mov z9.d, z1.d
-; CHECK-NEXT: str z8, [sp, #1, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z8.d, z2.d
+; CHECK-NEXT: mov z0.d, z1.d
; CHECK-NEXT: tbz w0, #0, .LBB5_2
; CHECK-NEXT: // %bb.1: // %L1
+; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT: addvl sp, sp, #-2
+; CHECK-NEXT: str z2, [sp, #1, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z0, [sp] // 16-byte Folded Spill
; CHECK-NEXT: bl bar
-; CHECK-NEXT: .LBB5_2: // %common.ret
-; CHECK-NEXT: mov z0.d, z9.d
-; CHECK-NEXT: mov z1.d, z8.d
-; CHECK-NEXT: ldr z9, [sp] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z8, [sp, #1, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z0, [sp] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z2, [sp, #1, mul vl] // 16-byte Folded Reload
; CHECK-NEXT: addvl sp, sp, #2
; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT: .LBB5_2: // %common.ret
+; CHECK-NEXT: mov z1.d, z2.d
; CHECK-NEXT: ret
br i1 %b, label %L1, label %L2
L1:
@@ -181,22 +169,20 @@ L2:
define <vscale x 4 x double> @wide_4f64(i1 %b, <vscale x 16 x i8> %legal, <vscale x 4 x double> %illegal) nounwind {
; CHECK-LABEL: wide_4f64:
; CHECK: // %bb.0:
-; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
-; CHECK-NEXT: addvl sp, sp, #-2
-; CHECK-NEXT: str z9, [sp] // 16-byte Folded Spill
-; CHECK-NEXT: mov z9.d, z1.d
-; CHECK-NEXT: str z8, [sp, #1, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z8.d, z2.d
+; CHECK-NEXT: mov z0.d, z1.d
; CHECK-NEXT: tbz w0, #0, .LBB6_2
; CHECK-NEXT: // %bb.1: // %L1
+; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT: addvl sp, sp, #-2
+; CHECK-NEXT: str z2, [sp, #1, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z0, [sp] // 16-byte Folded Spill
; CHECK-NEXT: bl bar
-; CHECK-NEXT: .LBB6_2: // %common.ret
-; CHECK-NEXT: mov z0.d, z9.d
-; CHECK-NEXT: mov z1.d, z8.d
-; CHECK-NEXT: ldr z9, [sp] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z8, [sp, #1, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z0, [sp] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z2, [sp, #1, mul vl] // 16-byte Folded Reload
; CHECK-NEXT: addvl sp, sp, #2
; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT: .LBB6_2: // %common.ret
+; CHECK-NEXT: mov z1.d, z2.d
; CHECK-NEXT: ret
br i1 %b, label %L1, label %L2
L1:
@@ -213,26 +199,23 @@ L2:
define <vscale x 48 x i8> @wide_48i8(i1 %b, <vscale x 16 x i8> %legal, <vscale x 48 x i8> %illegal) nounwind {
; CHECK-LABEL: wide_48i8:
; CHECK: // %bb.0:
-; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
-; CHECK-NEXT: addvl sp, sp, #-3
-; CHECK-NEXT: str z10, [sp] // 16-byte Folded Spill
-; CHECK-NEXT: mov z10.d, z1.d
-; CHECK-NEXT: str z9, [sp, #1, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z9.d, z2.d
-; CHECK-NEXT: str z8, [sp, #2, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z8.d, z3.d
+; CHECK-NEXT: mov z0.d, z1.d
; CHECK-NEXT: tbz w0, #0, .LBB7_2
; CHECK-NEXT: // %bb.1: // %L1
+; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT: addvl sp, sp, #-3
+; CHECK-NEXT: str z3, [sp, #2, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z2, [sp] // 16-byte Folded Spill
+; CHECK-NEXT: str z0, [sp, #1, mul vl] // 16-byte Folded Spill
; CHECK-NEXT: bl bar
-; CHECK-NEXT: .LBB7_2: // %common.ret
-; CHECK-NEXT: mov z0.d, z10.d
-; CHECK-NEXT: mov z1.d, z9.d
-; CHECK-NEXT: ldr z10, [sp] // 16-byte Folded Reload
-; CHECK-NEXT: mov z2.d, z8.d
-; CHECK-NEXT: ldr z9, [sp, #1, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z8, [sp, #2, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z0, [sp, #1, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z2, [sp] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z3, [sp, #2, mul vl] // 16-byte Folded Reload
; CHECK-NEXT: addvl sp, sp, #3
; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT: .LBB7_2: // %common.ret
+; CHECK-NEXT: mov z1.d, z2.d
+; CHECK-NEXT: mov z2.d, z3.d
; CHECK-NEXT: ret
br i1 %b, label %L1, label %L2
L1:
@@ -245,26 +228,23 @@ L2:
define <vscale x 24 x i16> @wide_24i16(i1 %b, <vscale x 16 x i8> %legal, <vscale x 24 x i16> %illegal) nounwind {
; CHECK-LABEL: wide_24i16:
; CHECK: // %bb.0:
-; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
-; CHECK-NEXT: addvl sp, sp, #-3
-; CHECK-NEXT: str z10, [sp] // 16-byte Folded Spill
-; CHECK-NEXT: mov z10.d, z1.d
-; CHECK-NEXT: str z9, [sp, #1, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z9.d, z2.d
-; CHECK-NEXT: str z8, [sp, #2, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z8.d, z3.d
+; CHECK-NEXT: mov z0.d, z1.d
; CHECK-NEXT: tbz w0, #0, .LBB8_2
; CHECK-NEXT: // %bb.1: // %L1
+; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT: addvl sp, sp, #-3
+; CHECK-NEXT: str z3, [sp, #2, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z2, [sp] // 16-byte Folded Spill
+; CHECK-NEXT: str z0, [sp, #1, mul vl] // 16-byte Folded Spill
; CHECK-NEXT: bl bar
-; CHECK-NEXT: .LBB8_2: // %common.ret
-; CHECK-NEXT: mov z0.d, z10.d
-; CHECK-NEXT: mov z1.d, z9.d
-; CHECK-NEXT: ldr z10, [sp] // 16-byte Folded Reload
-; CHECK-NEXT: mov z2.d, z8.d
-; CHECK-NEXT: ldr z9, [sp, #1, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z8, [sp, #2, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z0, [sp, #1, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z2, [sp] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z3, [sp, #2, mul vl] // 16-byte Folded Reload
; CHECK-NEXT: addvl sp, sp, #3
; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT: .LBB8_2: // %common.ret
+; CHECK-NEXT: mov z1.d, z2.d
+; CHECK-NEXT: mov z2.d, z3.d
; CHECK-NEXT: ret
br i1 %b, label %L1, label %L2
L1:
@@ -277,26 +257,23 @@ L2:
define <vscale x 12 x i32> @wide_12i32(i1 %b, <vscale x 16 x i8> %legal, <vscale x 12 x i32> %illegal) nounwind {
; CHECK-LABEL: wide_12i32:
; CHECK: // %bb.0:
-; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
-; CHECK-NEXT: addvl sp, sp, #-3
-; CHECK-NEXT: str z10, [sp] // 16-byte Folded Spill
-; CHECK-NEXT: mov z10.d, z1.d
-; CHECK-NEXT: str z9, [sp, #1, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z9.d, z2.d
-; CHECK-NEXT: str z8, [sp, #2, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z8.d, z3.d
+; CHECK-NEXT: mov z0.d, z1.d
; CHECK-NEXT: tbz w0, #0, .LBB9_2
; CHECK-NEXT: // %bb.1: // %L1
+; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT: addvl sp, sp, #-3
+; CHECK-NEXT: str z3, [sp, #2, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z2, [sp] // 16-byte Folded Spill
+; CHECK-NEXT: str z0, [sp, #1, mul vl] // 16-byte Folded Spill
; CHECK-NEXT: bl bar
-; CHECK-NEXT: .LBB9_2: // %common.ret
-; CHECK-NEXT: mov z0.d, z10.d
-; CHECK-NEXT: mov z1.d, z9.d
-; CHECK-NEXT: ldr z10, [sp] // 16-byte Folded Reload
-; CHECK-NEXT: mov z2.d, z8.d
-; CHECK-NEXT: ldr z9, [sp, #1, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z8, [sp, #2, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z0, [sp, #1, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z2, [sp] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z3, [sp, #2, mul vl] // 16-byte Folded Reload
; CHECK-NEXT: addvl sp, sp, #3
; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT: .LBB9_2: // %common.ret
+; CHECK-NEXT: mov z1.d, z2.d
+; CHECK-NEXT: mov z2.d, z3.d
; CHECK-NEXT: ret
br i1 %b, label %L1, label %L2
L1:
@@ -309,26 +286,23 @@ L2:
define <vscale x 6 x i64> @wide_6i64(i1 %b, <vscale x 16 x i8> %legal, <vscale x 6 x i64> %illegal) nounwind {
; CHECK-LABEL: wide_6i64:
; CHECK: // %bb.0:
-; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
-; CHECK-NEXT: addvl sp, sp, #-3
-; CHECK-NEXT: str z10, [sp] // 16-byte Folded Spill
-; CHECK-NEXT: mov z10.d, z1.d
-; CHECK-NEXT: str z9, [sp, #1, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z9.d, z2.d
-; CHECK-NEXT: str z8, [sp, #2, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z8.d, z3.d
+; CHECK-NEXT: mov z0.d, z1.d
; CHECK-NEXT: tbz w0, #0, .LBB10_2
; CHECK-NEXT: // %bb.1: // %L1
+; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT: addvl sp, sp, #-3
+; CHECK-NEXT: str z3, [sp, #2, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z2, [sp] // 16-byte Folded Spill
+; CHECK-NEXT: str z0, [sp, #1, mul vl] // 16-byte Folded Spill
; CHECK-NEXT: bl bar
-; CHECK-NEXT: .LBB10_2: // %common.ret
-; CHECK-NEXT: mov z0.d, z10.d
-; CHECK-NEXT: mov z1.d, z9.d
-; CHECK-NEXT: ldr z10, [sp] // 16-byte Folded Reload
-; CHECK-NEXT: mov z2.d, z8.d
-; CHECK-NEXT: ldr z9, [sp, #1, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z8, [sp, #2, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z0, [sp, #1, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z2, [sp] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z3, [sp, #2, mul vl] // 16-byte Folded Reload
; CHECK-NEXT: addvl sp, sp, #3
; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT: .LBB10_2: // %common.ret
+; CHECK-NEXT: mov z1.d, z2.d
+; CHECK-NEXT: mov z2.d, z3.d
; CHECK-NEXT: ret
br i1 %b, label %L1, label %L2
L1:
@@ -341,26 +315,23 @@ L2:
define <vscale x 24 x half> @wide_24f16(i1 %b, <vscale x 16 x i8> %legal, <vscale x 24 x half> %illegal) nounwind {
; CHECK-LABEL: wide_24f16:
; CHECK: // %bb.0:
-; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
-; CHECK-NEXT: addvl sp, sp, #-3
-; CHECK-NEXT: str z10, [sp] // 16-byte Folded Spill
-; CHECK-NEXT: mov z10.d, z1.d
-; CHECK-NEXT: str z9, [sp, #1, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z9.d, z2.d
-; CHECK-NEXT: str z8, [sp, #2, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z8.d, z3.d
+; CHECK-NEXT: mov z0.d, z1.d
; CHECK-NEXT: tbz w0, #0, .LBB11_2
; CHECK-NEXT: // %bb.1: // %L1
+; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT: addvl sp, sp, #-3
+; CHECK-NEXT: str z3, [sp, #2, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z2, [sp] // 16-byte Folded Spill
+; CHECK-NEXT: str z0, [sp, #1, mul vl] // 16-byte Folded Spill
; CHECK-NEXT: bl bar
-; CHECK-NEXT: .LBB11_2: // %common.ret
-; CHECK-NEXT: mov z0.d, z10.d
-; CHECK-NEXT: mov z1.d, z9.d
-; CHECK-NEXT: ldr z10, [sp] // 16-byte Folded Reload
-; CHECK-NEXT: mov z2.d, z8.d
-; CHECK-NEXT: ldr z9, [sp, #1, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z8, [sp, #2, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z0, [sp, #1, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z2, [sp] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z3, [sp, #2, mul vl] // 16-byte Folded Reload
; CHECK-NEXT: addvl sp, sp, #3
; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT: .LBB11_2: // %common.ret
+; CHECK-NEXT: mov z1.d, z2.d
+; CHECK-NEXT: mov z2.d, z3.d
; CHECK-NEXT: ret
br i1 %b, label %L1, label %L2
L1:
@@ -373,26 +344,23 @@ L2:
define <vscale x 12 x float> @wide_12f32(i1 %b, <vscale x 16 x i8> %legal, <vscale x 12 x float> %illegal) nounwind {
; CHECK-LABEL: wide_12f32:
; CHECK: // %bb.0:
-; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
-; CHECK-NEXT: addvl sp, sp, #-3
-; CHECK-NEXT: str z10, [sp] // 16-byte Folded Spill
-; CHECK-NEXT: mov z10.d, z1.d
-; CHECK-NEXT: str z9, [sp, #1, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z9.d, z2.d
-; CHECK-NEXT: str z8, [sp, #2, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z8.d, z3.d
+; CHECK-NEXT: mov z0.d, z1.d
; CHECK-NEXT: tbz w0, #0, .LBB12_2
; CHECK-NEXT: // %bb.1: // %L1
+; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT: addvl sp, sp, #-3
+; CHECK-NEXT: str z3, [sp, #2, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z2, [sp] // 16-byte Folded Spill
+; CHECK-NEXT: str z0, [sp, #1, mul vl] // 16-byte Folded Spill
; CHECK-NEXT: bl bar
-; CHECK-NEXT: .LBB12_2: // %common.ret
-; CHECK-NEXT: mov z0.d, z10.d
-; CHECK-NEXT: mov z1.d, z9.d
-; CHECK-NEXT: ldr z10, [sp] // 16-byte Folded Reload
-; CHECK-NEXT: mov z2.d, z8.d
-; CHECK-NEXT: ldr z9, [sp, #1, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z8, [sp, #2, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z0, [sp, #1, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z2, [sp] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z3, [sp, #2, mul vl] // 16-byte Folded Reload
; CHECK-NEXT: addvl sp, sp, #3
; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT: .LBB12_2: // %common.ret
+; CHECK-NEXT: mov z1.d, z2.d
+; CHECK-NEXT: mov z2.d, z3.d
; CHECK-NEXT: ret
br i1 %b, label %L1, label %L2
L1:
@@ -405,26 +373,23 @@ L2:
define <vscale x 6 x double> @wide_6f64(i1 %b, <vscale x 16 x i8> %legal, <vscale x 6 x double> %illegal) nounwind {
; CHECK-LABEL: wide_6f64:
; CHECK: // %bb.0:
-; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
-; CHECK-NEXT: addvl sp, sp, #-3
-; CHECK-NEXT: str z10, [sp] // 16-byte Folded Spill
-; CHECK-NEXT: mov z10.d, z1.d
-; CHECK-NEXT: str z9, [sp, #1, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z9.d, z2.d
-; CHECK-NEXT: str z8, [sp, #2, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z8.d, z3.d
+; CHECK-NEXT: mov z0.d, z1.d
; CHECK-NEXT: tbz w0, #0, .LBB13_2
; CHECK-NEXT: // %bb.1: // %L1
+; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT: addvl sp, sp, #-3
+; CHECK-NEXT: str z3, [sp, #2, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z2, [sp] // 16-byte Folded Spill
+; CHECK-NEXT: str z0, [sp, #1, mul vl] // 16-byte Folded Spill
; CHECK-NEXT: bl bar
-; CHECK-NEXT: .LBB13_2: // %common.ret
-; CHECK-NEXT: mov z0.d, z10.d
-; CHECK-NEXT: mov z1.d, z9.d
-; CHECK-NEXT: ldr z10, [sp] // 16-byte Folded Reload
-; CHECK-NEXT: mov z2.d, z8.d
-; CHECK-NEXT: ldr z9, [sp, #1, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z8, [sp, #2, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z0, [sp, #1, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z2, [sp] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z3, [sp, #2, mul vl] // 16-byte Folded Reload
; CHECK-NEXT: addvl sp, sp, #3
; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT: .LBB13_2: // %common.ret
+; CHECK-NEXT: mov z1.d, z2.d
+; CHECK-NEXT: mov z2.d, z3.d
; CHECK-NEXT: ret
br i1 %b, label %L1, label %L2
L1:
@@ -441,30 +406,26 @@ L2:
define <vscale x 64 x i8> @wide_64i8(i1 %b, <vscale x 16 x i8> %legal, <vscale x 64 x i8> %illegal) nounwind {
; CHECK-LABEL: wide_64i8:
; CHECK: // %bb.0:
-; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
-; CHECK-NEXT: addvl sp, sp, #-4
-; CHECK-NEXT: str z11, [sp] // 16-byte Folded Spill
-; CHECK-NEXT: mov z11.d, z1.d
-; CHECK-NEXT: str z10, [sp, #1, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z10.d, z2.d
-; CHECK-NEXT: str z9, [sp, #2, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z9.d, z3.d
-; CHECK-NEXT: str z8, [sp, #3, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z8.d, z4.d
+; CHECK-NEXT: mov z0.d, z1.d
; CHECK-NEXT: tbz w0, #0, .LBB14_2
; CHECK-NEXT: // %bb.1: // %L1
+; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT: addvl sp, sp, #-4
+; CHECK-NEXT: str z4, [sp, #3, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z3, [sp, #1, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z2, [sp, #2, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z0, [sp] // 16-byte Folded Spill
; CHECK-NEXT: bl bar
-; CHECK-NEXT: .LBB14_2: // %common.ret
-; CHECK-NEXT: mov z0.d, z11.d
-; CHECK-NEXT: mov z1.d, z10.d
-; CHECK-NEXT: ldr z11, [sp] // 16-byte Folded Reload
-; CHECK-NEXT: mov z2.d, z9.d
-; CHECK-NEXT: mov z3.d, z8.d
-; CHECK-NEXT: ldr z10, [sp, #1, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z9, [sp, #2, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z8, [sp, #3, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z0, [sp] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z2, [sp, #2, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z3, [sp, #1, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z4, [sp, #3, mul vl] // 16-byte Folded Reload
; CHECK-NEXT: addvl sp, sp, #4
; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT: .LBB14_2: // %common.ret
+; CHECK-NEXT: mov z1.d, z2.d
+; CHECK-NEXT: mov z2.d, z3.d
+; CHECK-NEXT: mov z3.d, z4.d
; CHECK-NEXT: ret
br i1 %b, label %L1, label %L2
L1:
@@ -477,30 +438,26 @@ L2:
define <vscale x 32 x i16> @wide_32i16(i1 %b, <vscale x 16 x i8> %legal, <vscale x 32 x i16> %illegal) nounwind {
; CHECK-LABEL: wide_32i16:
; CHECK: // %bb.0:
-; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
-; CHECK-NEXT: addvl sp, sp, #-4
-; CHECK-NEXT: str z11, [sp] // 16-byte Folded Spill
-; CHECK-NEXT: mov z11.d, z1.d
-; CHECK-NEXT: str z10, [sp, #1, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z10.d, z2.d
-; CHECK-NEXT: str z9, [sp, #2, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z9.d, z3.d
-; CHECK-NEXT: str z8, [sp, #3, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z8.d, z4.d
+; CHECK-NEXT: mov z0.d, z1.d
; CHECK-NEXT: tbz w0, #0, .LBB15_2
; CHECK-NEXT: // %bb.1: // %L1
+; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT: addvl sp, sp, #-4
+; CHECK-NEXT: str z4, [sp, #3, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z3, [sp, #1, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z2, [sp, #2, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z0, [sp] // 16-byte Folded Spill
; CHECK-NEXT: bl bar
-; CHECK-NEXT: .LBB15_2: // %common.ret
-; CHECK-NEXT: mov z0.d, z11.d
-; CHECK-NEXT: mov z1.d, z10.d
-; CHECK-NEXT: ldr z11, [sp] // 16-byte Folded Reload
-; CHECK-NEXT: mov z2.d, z9.d
-; CHECK-NEXT: mov z3.d, z8.d
-; CHECK-NEXT: ldr z10, [sp, #1, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z9, [sp, #2, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z8, [sp, #3, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z0, [sp] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z2, [sp, #2, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z3, [sp, #1, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z4, [sp, #3, mul vl] // 16-byte Folded Reload
; CHECK-NEXT: addvl sp, sp, #4
; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT: .LBB15_2: // %common.ret
+; CHECK-NEXT: mov z1.d, z2.d
+; CHECK-NEXT: mov z2.d, z3.d
+; CHECK-NEXT: mov z3.d, z4.d
; CHECK-NEXT: ret
br i1 %b, label %L1, label %L2
L1:
@@ -513,30 +470,26 @@ L2:
define <vscale x 16 x i32> @wide_16i32(i1 %b, <vscale x 16 x i8> %legal, <vscale x 16 x i32> %illegal) nounwind {
; CHECK-LABEL: wide_16i32:
; CHECK: // %bb.0:
-; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
-; CHECK-NEXT: addvl sp, sp, #-4
-; CHECK-NEXT: str z11, [sp] // 16-byte Folded Spill
-; CHECK-NEXT: mov z11.d, z1.d
-; CHECK-NEXT: str z10, [sp, #1, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z10.d, z2.d
-; CHECK-NEXT: str z9, [sp, #2, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z9.d, z3.d
-; CHECK-NEXT: str z8, [sp, #3, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z8.d, z4.d
+; CHECK-NEXT: mov z0.d, z1.d
; CHECK-NEXT: tbz w0, #0, .LBB16_2
; CHECK-NEXT: // %bb.1: // %L1
+; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT: addvl sp, sp, #-4
+; CHECK-NEXT: str z4, [sp, #3, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z3, [sp, #1, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z2, [sp, #2, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z0, [sp] // 16-byte Folded Spill
; CHECK-NEXT: bl bar
-; CHECK-NEXT: .LBB16_2: // %common.ret
-; CHECK-NEXT: mov z0.d, z11.d
-; CHECK-NEXT: mov z1.d, z10.d
-; CHECK-NEXT: ldr z11, [sp] // 16-byte Folded Reload
-; CHECK-NEXT: mov z2.d, z9.d
-; CHECK-NEXT: mov z3.d, z8.d
-; CHECK-NEXT: ldr z10, [sp, #1, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z9, [sp, #2, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z8, [sp, #3, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z0, [sp] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z2, [sp, #2, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z3, [sp, #1, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z4, [sp, #3, mul vl] // 16-byte Folded Reload
; CHECK-NEXT: addvl sp, sp, #4
; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT: .LBB16_2: // %common.ret
+; CHECK-NEXT: mov z1.d, z2.d
+; CHECK-NEXT: mov z2.d, z3.d
+; CHECK-NEXT: mov z3.d, z4.d
; CHECK-NEXT: ret
br i1 %b, label %L1, label %L2
L1:
@@ -549,30 +502,26 @@ L2:
define <vscale x 8 x i64> @wide_8i64(i1 %b, <vscale x 16 x i8> %legal, <vscale x 8 x i64> %illegal) nounwind {
; CHECK-LABEL: wide_8i64:
; CHECK: // %bb.0:
-; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
-; CHECK-NEXT: addvl sp, sp, #-4
-; CHECK-NEXT: str z11, [sp] // 16-byte Folded Spill
-; CHECK-NEXT: mov z11.d, z1.d
-; CHECK-NEXT: str z10, [sp, #1, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z10.d, z2.d
-; CHECK-NEXT: str z9, [sp, #2, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z9.d, z3.d
-; CHECK-NEXT: str z8, [sp, #3, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z8.d, z4.d
+; CHECK-NEXT: mov z0.d, z1.d
; CHECK-NEXT: tbz w0, #0, .LBB17_2
; CHECK-NEXT: // %bb.1: // %L1
+; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT: addvl sp, sp, #-4
+; CHECK-NEXT: str z4, [sp, #3, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z3, [sp, #1, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z2, [sp, #2, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z0, [sp] // 16-byte Folded Spill
; CHECK-NEXT: bl bar
-; CHECK-NEXT: .LBB17_2: // %common.ret
-; CHECK-NEXT: mov z0.d, z11.d
-; CHECK-NEXT: mov z1.d, z10.d
-; CHECK-NEXT: ldr z11, [sp] // 16-byte Folded Reload
-; CHECK-NEXT: mov z2.d, z9.d
-; CHECK-NEXT: mov z3.d, z8.d
-; CHECK-NEXT: ldr z10, [sp, #1, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z9, [sp, #2, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z8, [sp, #3, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z0, [sp] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z2, [sp, #2, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z3, [sp, #1, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z4, [sp, #3, mul vl] // 16-byte Folded Reload
; CHECK-NEXT: addvl sp, sp, #4
; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT: .LBB17_2: // %common.ret
+; CHECK-NEXT: mov z1.d, z2.d
+; CHECK-NEXT: mov z2.d, z3.d
+; CHECK-NEXT: mov z3.d, z4.d
; CHECK-NEXT: ret
br i1 %b, label %L1, label %L2
L1:
@@ -585,30 +534,26 @@ L2:
define <vscale x 32 x half> @wide_32f16(i1 %b, <vscale x 16 x i8> %legal, <vscale x 32 x half> %illegal) nounwind {
; CHECK-LABEL: wide_32f16:
; CHECK: // %bb.0:
-; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
-; CHECK-NEXT: addvl sp, sp, #-4
-; CHECK-NEXT: str z11, [sp] // 16-byte Folded Spill
-; CHECK-NEXT: mov z11.d, z1.d
-; CHECK-NEXT: str z10, [sp, #1, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z10.d, z2.d
-; CHECK-NEXT: str z9, [sp, #2, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z9.d, z3.d
-; CHECK-NEXT: str z8, [sp, #3, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z8.d, z4.d
+; CHECK-NEXT: mov z0.d, z1.d
; CHECK-NEXT: tbz w0, #0, .LBB18_2
; CHECK-NEXT: // %bb.1: // %L1
+; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT: addvl sp, sp, #-4
+; CHECK-NEXT: str z4, [sp, #3, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z3, [sp, #1, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z2, [sp, #2, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z0, [sp] // 16-byte Folded Spill
; CHECK-NEXT: bl bar
-; CHECK-NEXT: .LBB18_2: // %common.ret
-; CHECK-NEXT: mov z0.d, z11.d
-; CHECK-NEXT: mov z1.d, z10.d
-; CHECK-NEXT: ldr z11, [sp] // 16-byte Folded Reload
-; CHECK-NEXT: mov z2.d, z9.d
-; CHECK-NEXT: mov z3.d, z8.d
-; CHECK-NEXT: ldr z10, [sp, #1, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z9, [sp, #2, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z8, [sp, #3, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z0, [sp] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z2, [sp, #2, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z3, [sp, #1, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z4, [sp, #3, mul vl] // 16-byte Folded Reload
; CHECK-NEXT: addvl sp, sp, #4
; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT: .LBB18_2: // %common.ret
+; CHECK-NEXT: mov z1.d, z2.d
+; CHECK-NEXT: mov z2.d, z3.d
+; CHECK-NEXT: mov z3.d, z4.d
; CHECK-NEXT: ret
br i1 %b, label %L1, label %L2
L1:
@@ -621,30 +566,26 @@ L2:
define <vscale x 16 x float> @wide_16f32(i1 %b, <vscale x 16 x i8> %legal, <vscale x 16 x float> %illegal) nounwind {
; CHECK-LABEL: wide_16f32:
; CHECK: // %bb.0:
-; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
-; CHECK-NEXT: addvl sp, sp, #-4
-; CHECK-NEXT: str z11, [sp] // 16-byte Folded Spill
-; CHECK-NEXT: mov z11.d, z1.d
-; CHECK-NEXT: str z10, [sp, #1, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z10.d, z2.d
-; CHECK-NEXT: str z9, [sp, #2, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z9.d, z3.d
-; CHECK-NEXT: str z8, [sp, #3, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z8.d, z4.d
+; CHECK-NEXT: mov z0.d, z1.d
; CHECK-NEXT: tbz w0, #0, .LBB19_2
; CHECK-NEXT: // %bb.1: // %L1
+; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT: addvl sp, sp, #-4
+; CHECK-NEXT: str z4, [sp, #3, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z3, [sp, #1, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z2, [sp, #2, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z0, [sp] // 16-byte Folded Spill
; CHECK-NEXT: bl bar
-; CHECK-NEXT: .LBB19_2: // %common.ret
-; CHECK-NEXT: mov z0.d, z11.d
-; CHECK-NEXT: mov z1.d, z10.d
-; CHECK-NEXT: ldr z11, [sp] // 16-byte Folded Reload
-; CHECK-NEXT: mov z2.d, z9.d
-; CHECK-NEXT: mov z3.d, z8.d
-; CHECK-NEXT: ldr z10, [sp, #1, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z9, [sp, #2, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z8, [sp, #3, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z0, [sp] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z2, [sp, #2, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z3, [sp, #1, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z4, [sp, #3, mul vl] // 16-byte Folded Reload
; CHECK-NEXT: addvl sp, sp, #4
; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT: .LBB19_2: // %common.ret
+; CHECK-NEXT: mov z1.d, z2.d
+; CHECK-NEXT: mov z2.d, z3.d
+; CHECK-NEXT: mov z3.d, z4.d
; CHECK-NEXT: ret
br i1 %b, label %L1, label %L2
L1:
@@ -657,30 +598,26 @@ L2:
define <vscale x 8 x double> @wide_8f64(i1 %b, <vscale x 16 x i8> %legal, <vscale x 8 x double> %illegal) nounwind {
; CHECK-LABEL: wide_8f64:
; CHECK: // %bb.0:
-; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
-; CHECK-NEXT: addvl sp, sp, #-4
-; CHECK-NEXT: str z11, [sp] // 16-byte Folded Spill
-; CHECK-NEXT: mov z11.d, z1.d
-; CHECK-NEXT: str z10, [sp, #1, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z10.d, z2.d
-; CHECK-NEXT: str z9, [sp, #2, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z9.d, z3.d
-; CHECK-NEXT: str z8, [sp, #3, mul vl] // 16-byte Folded Spill
-; CHECK-NEXT: mov z8.d, z4.d
+; CHECK-NEXT: mov z0.d, z1.d
; CHECK-NEXT: tbz w0, #0, .LBB20_2
; CHECK-NEXT: // %bb.1: // %L1
+; CHECK-NEXT: stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT: addvl sp, sp, #-4
+; CHECK-NEXT: str z4, [sp, #3, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z3, [sp, #1, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z2, [sp, #2, mul vl] // 16-byte Folded Spill
+; CHECK-NEXT: str z0, [sp] // 16-byte Folded Spill
; CHECK-NEXT: bl bar
-; CHECK-NEXT: .LBB20_2: // %common.ret
-; CHECK-NEXT: mov z0.d, z11.d
-; CHECK-NEXT: mov z1.d, z10.d
-; CHECK-NEXT: ldr z11, [sp] // 16-byte Folded Reload
-; CHECK-NEXT: mov z2.d, z9.d
-; CHECK-NEXT: mov z3.d, z8.d
-; CHECK-NEXT: ldr z10, [sp, #1, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z9, [sp, #2, mul vl] // 16-byte Folded Reload
-; CHECK-NEXT: ldr z8, [sp, #3, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z0, [sp] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z2, [sp, #2, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z3, [sp, #1, mul vl] // 16-byte Folded Reload
+; CHECK-NEXT: ldr z4, [sp, #3, mul vl] // 16-byte Folded Reload
; CHECK-NEXT: addvl sp, sp, #4
; CHECK-NEXT: ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT: .LBB20_2: // %common.ret
+; CHECK-NEXT: mov z1.d, z2.d
+; CHECK-NEXT: mov z2.d, z3.d
+; CHECK-NEXT: mov z3.d, z4.d
; CHECK-NEXT: ret
br i1 %b, label %L1, label %L2
L1:
More information about the llvm-commits
mailing list