[llvm] [CodeGen] Don't inflate the register pressure set limit by live-thru pressure (PR #214957)
Nazar Mokrynskyi via llvm-commits
llvm-commits at lists.llvm.org
Sat Aug 8 04:25:05 PDT 2026
https://github.com/nazar-pc created https://github.com/llvm/llvm-project/pull/214957
I was originally looking into https://github.com/llvm/llvm-project/issues/212375 with LLM's help. Along the way I ran into what looks like a straightforward bug in how the machine scheduler decides whether a region is over its register budget, and that is what this PR fixes.
I'm not an LLVM developer, but I cross-checked and believe this is a legitimate issue worth fixing.
<details>
<summary>LLM-generated description and additional details</summary>
## The problem
Before the register allocator runs, the machine scheduler reorders instructions. To avoid reordering into something the allocator cannot handle, it tracks *register pressure* — how many values are live at once — and compares it against a limit, which is essentially "how many registers of this kind does this CPU actually have".
Some values are live across the *entire* region being scheduled — for example loop-invariant constants that are computed before the loop and used on every iteration. LLVM calls this **live-thru** pressure and describes it in `initLiveThru()` as pressure the tracker "can never drop below".
Both places that compute excess pressure were *adding* live-thru pressure to the limit:
```c++
unsigned Limit = RCI->getRegPressureSetLimit(PSetID);
if (!LiveThruPressure.empty())
Limit += LiveThruPressure[PSetID];
```
That is backwards. Live-thru values sit in real registers just like anything else, and they are already included in the pressure counters (`CurrSetPressure`, `MaxSetPressure`) that get compared against this limit. Adding them to the limit as well hides exactly that much genuine excess pressure, so the scheduler's "am I over budget?" check goes quiet precisely in the regions that are most starved of registers.
`ScheduleDAGMILive::initRegPressure()` already compares against the *unadjusted* limit when it builds `RegionCriticalPSets`, so the two computations also disagreed with each other.
## Concrete example
For the inner loop of the BLAKE3 AVX-512 kernel at `-mcpu=znver4`, `llc -debug-only=machine-scheduler` reports:
```
Live Thru: FR16X=7 ... GR16=41
FR16X Limit 32 Actual 40
GR16 Limit 30 Actual 44
```
(`FR16X` is the pressure set covering the `xmm`/`ymm`/`zmm` register file — pressure sets are named after a representative register class, so the name is misleading; `GR16` likewise covers the general-purpose registers.)
So with the old code the scheduler's effective budgets were:
* vector: `32 + 7 = 39`, on a machine with **32** `zmm` registers
* general purpose: `30 + 41 = 71`, on a machine with **30** units of GPR
The GPR case is the clearest: measured pressure was 44 against a real limit of 30, and the scheduler was told the budget was 71, so it saw no problem at all.
## Effect on generated code
Reduced BLAKE3 AVX-512 kernel from #212375, `-mcpu=znver4`:
| | before | after |
| --- | --- | --- |
| `zmm` spill stores | 20 | **16** |
| `zmm` reloads | 24 | **18** |
| stack frame | 1224 B | **968 B** |
On BLAKE3's actual `blake3_avx512.c` at `-mcpu=znver4`, spill/reload references drop by 28.
Compiling every test in `llvm/test/CodeGen/X86` with `llc -O3` and counting spill/reload instructions:
| configuration | files | spill/reloads before → after | files better | files worse |
| --- | --- | --- | --- | --- |
| default arch | 4465 | 137846 → 137824 | 5 | 1 |
| `-mcpu=znver4` | 4759 | 33812 → 33808 | 1 | 0 |
Small, but net negative in both, and no configuration I measured came out worse in aggregate.
Caveat on the `-mcpu=znver4` row: fewer files move than in the default-arch row because most X86 tests pin their own `-mtriple`/`-mattr` or carry `target-features` function attributes that override `-mcpu`, so the forced CPU only reaches the subset that does not.
## Other targets
This file is generic `llvm/lib/CodeGen` code, not X86-specific. The two functions changed are used by `MachineScheduler` (every backend) and by `MachinePipeliner` (the software pipeliner), so X86-only testing would not have been enough.
I built with `LLVM_TARGETS_TO_BUILD=all` plus `llvm-test-depends`, and ran the **entire `llvm/test` suite (64,849 tests)** twice — once with the patch and once with only `RegisterPressure.cpp` reverted to its pre-patch state, everything else identical:
* **with the patch: 0 failures**
* without the patch: 14 failures — precisely the 14 tests whose check lines this PR regenerates, and nothing else
So no test fails as a result of this change, on any target, and exactly 14 tests move. **9 on X86 and 5 elsewhere:**
| test | spill/reloads | instructions | |
| --- | --- | --- | --- |
| PowerPC/more-dq-form-prepare.ll | 117 → 110 | 291 → 282 | better (frame 592 → 576) |
| PowerPC/common-chain.ll | 68 → 65 | 417 → 415 | better |
| RISCV/rvv/pr165232.ll | 16 → 15 | 164 → 163 | better |
| AArch64/ragreedy-local-interval-cost.ll | 22 → 22 | 151 → 155 | worse |
| Thumb2/LowOverheadLoops/spillingmove.ll | 8 → 16 | 158 → 167 | **worse** |
Net across those five: −3 spill/reloads, +1 instruction.
**AMDGPU is completely unaffected** — 0 of 4964 tests changed. That is the result I was most worried about, since AMDGPU is the only target that overrides `getRegPressureSetLimit()` (occupancy-dependent) and forces `ShouldTrackPressure = true`, and register usage there controls occupancy rather than just spill counts.
The Thumb2 MVE case is a genuine regression and I want to flag it rather than bury it: vector spills double, 8 → 16. The low-overhead loop structure survives (`dls`/`le`/`letp` counts are unchanged), so nothing functional is lost, but that test is specifically about spilling in MVE loops. It is the same phenomenon described below — once the scheduler can see the real excess pressure it starts reordering, and the greedy heuristic reacting to that information sometimes makes things worse. I do not think that argues for keeping a wrong limit, but a reviewer may disagree.
Backends beyond those five are unaffected. That includes the `MachinePipeliner` users (Hexagon, SystemZ), which call the same code and which I expected to move — they did not.
## Test changes
All 14 affected tests are autogenerated; their check lines were regenerated with `update_llc_test_checks.py` / `update_mir_test_checks.py`. No hand-written test asserting a specific optimisation had to be touched.
The diffs look large because those scripts rewrite whole function bodies, but the content barely moves. On X86, summed over all 25 `llc` RUN lines in the affected tests, the change is **−7 spill/reloads and −2 instructions**, with no single RUN line moving by more than 4 spills or 4 instructions. 8 of the 9 affected X86 RUN lines are 32-bit `i686`, which is what you would expect: with only ~6 allocatable general-purpose registers, the inflated limit was masking the most excess there.
</details>
>From 3f58ae06ca2fe5fb60dea64a41cd57ed0f7f5f45 Mon Sep 17 00:00:00 2001
From: Nazar Mokrynskyi <nazar at mokrynskyi.com>
Date: Sat, 8 Aug 2026 14:17:46 +0300
Subject: [PATCH] [CodeGen] Don't inflate the register pressure set limit by
live-thru pressure
computeExcessPressureDelta() and RegPressureTracker::getUpwardPressureDelta()
raised the pressure set limit by the region's live-thru pressure before
deciding whether a scheduling candidate pushes a set beyond its limit:
unsigned Limit = RCI->getRegPressureSetLimit(PSetID);
if (!LiveThruPressure.empty())
Limit += LiveThruPressure[PSetID];
Live-thru pressure is, per initLiveThru(), the pressure present at every point
of the scheduling region -- values the tracker "can never drop below". Those
values occupy physical registers like any other live value, and they are
already counted in CurrSetPressure and MaxSetPressure. Adding them to the
limit therefore hides exactly that much real excess pressure, so the
RegExcess heuristic stops firing in the regions that are most register
starved.
RegionCriticalPSets in ScheduleDAGMILive::initRegPressure() already compares
against the uninflated getRegPressureSetLimit(), so the two computations were
inconsistent with each other as well.
With the live-thru adjustment gone, computeExcessPressureDelta()'s
LiveThruPressureVec parameter is unused and is dropped.
---
llvm/lib/CodeGen/RegisterPressure.cpp | 13 +-
.../AArch64/ragreedy-local-interval-cost.ll | 114 ++---
llvm/test/CodeGen/PowerPC/common-chain.ll | 214 +++++----
.../CodeGen/PowerPC/more-dq-form-prepare.ll | 439 +++++++++---------
llvm/test/CodeGen/RISCV/rvv/pr165232.ll | 83 ++--
.../Thumb2/LowOverheadLoops/spillingmove.ll | 53 ++-
.../X86/AMX/amx-greedy-ra-spill-shape.ll | 131 +++---
.../X86/div-rem-pair-recomposition-signed.ll | 22 +-
llvm/test/CodeGen/X86/i128-mul.ll | 95 ++--
llvm/test/CodeGen/X86/i128-udiv.ll | 165 +++----
llvm/test/CodeGen/X86/mmx-arith.ll | 9 +-
llvm/test/CodeGen/X86/pr38539.ll | 7 +-
llvm/test/CodeGen/X86/sdiv_fix.ll | 48 +-
llvm/test/CodeGen/X86/sdiv_fix_sat.ll | 98 ++--
.../X86/vector-shift-by-select-loop.ll | 98 ++--
15 files changed, 796 insertions(+), 793 deletions(-)
diff --git a/llvm/lib/CodeGen/RegisterPressure.cpp b/llvm/lib/CodeGen/RegisterPressure.cpp
index dd7468a1309de..b00cf0b14b9c8 100644
--- a/llvm/lib/CodeGen/RegisterPressure.cpp
+++ b/llvm/lib/CodeGen/RegisterPressure.cpp
@@ -948,8 +948,7 @@ void RegPressureTracker::advance() {
static void computeExcessPressureDelta(ArrayRef<unsigned> OldPressureVec,
ArrayRef<unsigned> NewPressureVec,
RegPressureDelta &Delta,
- const RegisterClassInfo *RCI,
- ArrayRef<unsigned> LiveThruPressureVec) {
+ const RegisterClassInfo *RCI) {
Delta.Excess = PressureChange();
for (unsigned i = 0, e = OldPressureVec.size(); i < e; ++i) {
unsigned POld = OldPressureVec[i];
@@ -959,8 +958,6 @@ static void computeExcessPressureDelta(ArrayRef<unsigned> OldPressureVec,
continue;
// Only consider change beyond the limit.
unsigned Limit = RCI->getRegPressureSetLimit(i);
- if (!LiveThruPressureVec.empty())
- Limit += LiveThruPressureVec[i];
if (Limit > POld) {
if (Limit > PNew)
@@ -1093,8 +1090,7 @@ getMaxUpwardPressureDelta(const MachineInstr *MI, PressureDiff *PDiff,
bumpUpwardPressure(MI);
- computeExcessPressureDelta(SavedPressure, CurrSetPressure, Delta, RCI,
- LiveThruPressure);
+ computeExcessPressureDelta(SavedPressure, CurrSetPressure, Delta, RCI);
computeMaxPressureDelta(SavedMaxPressure, P.MaxSetPressure, CriticalPSets,
MaxPressureLimit, Delta);
assert(Delta.CriticalMax.getUnitInc() >= 0 &&
@@ -1160,8 +1156,6 @@ getUpwardPressureDelta(const MachineInstr *MI, /*const*/ PressureDiff &PDiff,
unsigned PSetID = PDiffI->getPSet();
unsigned Limit = RCI->getRegPressureSetLimit(PSetID);
- if (!LiveThruPressure.empty())
- Limit += LiveThruPressure[PSetID];
unsigned POld = CurrSetPressure[PSetID];
unsigned MOld = P.MaxSetPressure[PSetID];
@@ -1342,8 +1336,7 @@ getMaxDownwardPressureDelta(const MachineInstr *MI, RegPressureDelta &Delta,
bumpDownwardPressure(MI);
- computeExcessPressureDelta(SavedPressure, CurrSetPressure, Delta, RCI,
- LiveThruPressure);
+ computeExcessPressureDelta(SavedPressure, CurrSetPressure, Delta, RCI);
computeMaxPressureDelta(SavedMaxPressure, P.MaxSetPressure, CriticalPSets,
MaxPressureLimit, Delta);
assert(Delta.CriticalMax.getUnitInc() >= 0 &&
diff --git a/llvm/test/CodeGen/AArch64/ragreedy-local-interval-cost.ll b/llvm/test/CodeGen/AArch64/ragreedy-local-interval-cost.ll
index 9f513a940015e..8ee47f36bdf5c 100644
--- a/llvm/test/CodeGen/AArch64/ragreedy-local-interval-cost.ll
+++ b/llvm/test/CodeGen/AArch64/ragreedy-local-interval-cost.ll
@@ -25,7 +25,7 @@ define dso_local void @run_test() local_unnamed_addr uwtable {
; CHECK-NEXT: .cfi_offset b13, -64
; CHECK-NEXT: .cfi_offset b14, -72
; CHECK-NEXT: .cfi_offset b15, -80
-; CHECK-NEXT: movi v7.2d, #0000000000000000
+; CHECK-NEXT: movi v1.2d, #0000000000000000
; CHECK-NEXT: adrp x14, B+48
; CHECK-NEXT: add x14, x14, :lo12:B+48
; CHECK-NEXT: // implicit-def: $q18
@@ -43,15 +43,15 @@ define dso_local void @run_test() local_unnamed_addr uwtable {
; CHECK-NEXT: // implicit-def: $q3
; CHECK-NEXT: // implicit-def: $q4
; CHECK-NEXT: // implicit-def: $q5
-; CHECK-NEXT: // implicit-def: $q1
+; CHECK-NEXT: // implicit-def: $q6
; CHECK-NEXT: // implicit-def: $q16
; CHECK-NEXT: // implicit-def: $q17
-; CHECK-NEXT: // implicit-def: $q6
+; CHECK-NEXT: // implicit-def: $q7
; CHECK-NEXT: // implicit-def: $q19
; CHECK-NEXT: // implicit-def: $q20
; CHECK-NEXT: // implicit-def: $q21
; CHECK-NEXT: // implicit-def: $q22
-; CHECK-NEXT: // implicit-def: $q12
+; CHECK-NEXT: // implicit-def: $q24
; CHECK-NEXT: // implicit-def: $q23
; CHECK-NEXT: // implicit-def: $q25
; CHECK-NEXT: // implicit-def: $q26
@@ -59,7 +59,7 @@ define dso_local void @run_test() local_unnamed_addr uwtable {
; CHECK-NEXT: // implicit-def: $q30
; CHECK-NEXT: // implicit-def: $q8
; CHECK-NEXT: // implicit-def: $q11
-; CHECK-NEXT: // implicit-def: $q28
+; CHECK-NEXT: // implicit-def: $q12
; CHECK-NEXT: // implicit-def: $q29
; CHECK-NEXT: // implicit-def: $q13
; CHECK-NEXT: // implicit-def: $q10
@@ -76,100 +76,104 @@ define dso_local void @run_test() local_unnamed_addr uwtable {
; CHECK-NEXT: stp q15, q4, [sp] // 32-byte Folded Spill
; CHECK-NEXT: add x5, x10, x11
; CHECK-NEXT: mul x1, x18, x15
-; CHECK-NEXT: ldr x2, [x13], #64
-; CHECK-NEXT: stp q6, q23, [sp, #32] // 32-byte Folded Spill
-; CHECK-NEXT: ldr q23, [sp, #80] // 16-byte Reload
; CHECK-NEXT: ldr x14, [x14, #8]
-; CHECK-NEXT: mul x0, x18, x18
; CHECK-NEXT: ldr x5, [x5, #128]
+; CHECK-NEXT: ldr x2, [x13], #64
+; CHECK-NEXT: stp q7, q23, [sp, #32] // 32-byte Folded Spill
+; CHECK-NEXT: mul x0, x18, x18
+; CHECK-NEXT: ldr q23, [sp, #80] // 16-byte Reload
; CHECK-NEXT: mov v9.16b, v30.16b
; CHECK-NEXT: mov v30.16b, v25.16b
; CHECK-NEXT: mov v25.16b, v20.16b
-; CHECK-NEXT: mov v20.16b, v1.16b
+; CHECK-NEXT: mov v20.16b, v6.16b
; CHECK-NEXT: mul x17, x16, x18
+; CHECK-NEXT: mov v6.16b, v1.16b
+; CHECK-NEXT: mov v28.16b, v24.16b
+; CHECK-NEXT: fmov d14, x1
+; CHECK-NEXT: mov v24.16b, v19.16b
+; CHECK-NEXT: mov v19.16b, v5.16b
+; CHECK-NEXT: mul x3, x14, x18
; CHECK-NEXT: mov v31.16b, v26.16b
; CHECK-NEXT: mov v26.16b, v21.16b
-; CHECK-NEXT: fmov d14, x1
+; CHECK-NEXT: fmov d15, x0
; CHECK-NEXT: mov v21.16b, v16.16b
; CHECK-NEXT: mov v16.16b, v2.16b
+; CHECK-NEXT: mov v0.16b, v14.16b
; CHECK-NEXT: mul x4, x2, x18
-; CHECK-NEXT: mov v6.16b, v10.16b
+; CHECK-NEXT: mov v7.16b, v10.16b
; CHECK-NEXT: mov v10.16b, v17.16b
-; CHECK-NEXT: fmov d15, x0
; CHECK-NEXT: mov v17.16b, v3.16b
-; CHECK-NEXT: mov v24.16b, v19.16b
-; CHECK-NEXT: mov v0.16b, v14.16b
-; CHECK-NEXT: mul x3, x14, x18
-; CHECK-NEXT: mov v19.16b, v5.16b
; CHECK-NEXT: add x11, x11, #8
-; CHECK-NEXT: add x12, x12, #1
; CHECK-NEXT: mov v15.d[1], x17
-; CHECK-NEXT: mul x6, x15, x15
+; CHECK-NEXT: mul x7, x15, x5
; CHECK-NEXT: cmp x11, #64
; CHECK-NEXT: mov v0.d[1], x1
+; CHECK-NEXT: add x12, x12, #1
+; CHECK-NEXT: mul x19, x16, x5
; CHECK-NEXT: fmov d1, x4
-; CHECK-NEXT: mul x7, x15, x5
; CHECK-NEXT: mul x18, x18, x5
-; CHECK-NEXT: mov v1.d[1], x3
; CHECK-NEXT: add v23.2d, v23.2d, v0.2d
; CHECK-NEXT: ldr q0, [sp, #64] // 16-byte Reload
-; CHECK-NEXT: fmov d4, x6
-; CHECK-NEXT: mul x20, x2, x5
-; CHECK-NEXT: add v0.2d, v0.2d, v15.2d
; CHECK-NEXT: fmov d3, x7
-; CHECK-NEXT: mul x19, x16, x5
-; CHECK-NEXT: mov v4.d[1], x6
+; CHECK-NEXT: mul x20, x14, x5
+; CHECK-NEXT: mov v1.d[1], x3
+; CHECK-NEXT: add v0.2d, v0.2d, v15.2d
+; CHECK-NEXT: mul x5, x2, x5
+; CHECK-NEXT: mov v3.d[1], x7
; CHECK-NEXT: fmov d2, x18
-; CHECK-NEXT: mul x0, x14, x5
+; CHECK-NEXT: mul x6, x15, x15
; CHECK-NEXT: stp q0, q23, [sp, #64] // 32-byte Folded Spill
; CHECK-NEXT: ldr q0, [sp, #96] // 16-byte Reload
-; CHECK-NEXT: fmov d5, x20
-; CHECK-NEXT: mov v3.d[1], x7
; CHECK-NEXT: ldr q23, [sp, #48] // 16-byte Reload
; CHECK-NEXT: mul x17, x2, x15
; CHECK-NEXT: add v0.2d, v0.2d, v15.2d
; CHECK-NEXT: ldr q15, [sp] // 16-byte Reload
+; CHECK-NEXT: fmov d5, x5
; CHECK-NEXT: mov v2.d[1], x19
-; CHECK-NEXT: add v13.2d, v13.2d, v4.2d
-; CHECK-NEXT: add v12.2d, v12.2d, v4.2d
+; CHECK-NEXT: add v11.2d, v11.2d, v3.2d
; CHECK-NEXT: mul x16, x16, x15
; CHECK-NEXT: add v15.2d, v15.2d, v1.2d
-; CHECK-NEXT: mov v1.16b, v20.16b
-; CHECK-NEXT: mov v5.d[1], x0
+; CHECK-NEXT: mov v1.16b, v6.16b
+; CHECK-NEXT: fmov d4, x6
; CHECK-NEXT: str q0, [sp, #96] // 16-byte Spill
-; CHECK-NEXT: mov v20.16b, v25.16b
+; CHECK-NEXT: mov v6.16b, v20.16b
+; CHECK-NEXT: mov v5.d[1], x20
; CHECK-NEXT: mul x14, x14, x15
-; CHECK-NEXT: mov v25.16b, v30.16b
-; CHECK-NEXT: add v11.2d, v11.2d, v3.2d
+; CHECK-NEXT: mov v20.16b, v25.16b
; CHECK-NEXT: fmov d0, x17
+; CHECK-NEXT: mov v25.16b, v30.16b
; CHECK-NEXT: mov v3.16b, v17.16b
+; CHECK-NEXT: mov v4.d[1], x6
; CHECK-NEXT: mov v17.16b, v10.16b
-; CHECK-NEXT: mov v10.16b, v6.16b
+; CHECK-NEXT: mov v10.16b, v7.16b
+; CHECK-NEXT: mov v14.d[1], x16
; CHECK-NEXT: add v8.2d, v8.2d, v2.2d
; CHECK-NEXT: mov v2.16b, v16.16b
-; CHECK-NEXT: mov v14.d[1], x16
-; CHECK-NEXT: mov v16.16b, v21.16b
-; CHECK-NEXT: mov v21.16b, v26.16b
; CHECK-NEXT: add v30.2d, v9.2d, v5.2d
; CHECK-NEXT: mov v5.16b, v19.16b
-; CHECK-NEXT: add v26.2d, v31.2d, v4.2d
+; CHECK-NEXT: mov v19.16b, v24.16b
; CHECK-NEXT: mov v0.d[1], x14
-; CHECK-NEXT: add v19.2d, v24.2d, v4.2d
+; CHECK-NEXT: mov v16.16b, v21.16b
+; CHECK-NEXT: mov v21.16b, v26.16b
+; CHECK-NEXT: add v13.2d, v13.2d, v4.2d
+; CHECK-NEXT: add v26.2d, v31.2d, v4.2d
+; CHECK-NEXT: add v24.2d, v28.2d, v4.2d
+; CHECK-NEXT: add v19.2d, v19.2d, v4.2d
+; CHECK-NEXT: add v6.2d, v6.2d, v4.2d
; CHECK-NEXT: add v1.2d, v1.2d, v4.2d
-; CHECK-NEXT: add v7.2d, v7.2d, v4.2d
-; CHECK-NEXT: ldp q4, q6, [sp, #16] // 32-byte Folded Reload
+; CHECK-NEXT: ldp q4, q7, [sp, #16] // 32-byte Folded Reload
; CHECK-NEXT: add v10.2d, v10.2d, v14.2d
; CHECK-NEXT: add v29.2d, v29.2d, v14.2d
; CHECK-NEXT: add v27.2d, v27.2d, v14.2d
; CHECK-NEXT: add v23.2d, v23.2d, v14.2d
; CHECK-NEXT: add v22.2d, v22.2d, v14.2d
; CHECK-NEXT: add v20.2d, v20.2d, v14.2d
-; CHECK-NEXT: add v6.2d, v6.2d, v14.2d
; CHECK-NEXT: add v16.2d, v16.2d, v14.2d
+; CHECK-NEXT: add v7.2d, v7.2d, v14.2d
; CHECK-NEXT: add v5.2d, v5.2d, v14.2d
; CHECK-NEXT: add v3.2d, v3.2d, v14.2d
; CHECK-NEXT: add v2.2d, v2.2d, v14.2d
-; CHECK-NEXT: add v28.2d, v28.2d, v0.2d
+; CHECK-NEXT: add v12.2d, v12.2d, v0.2d
; CHECK-NEXT: add v25.2d, v25.2d, v0.2d
; CHECK-NEXT: add v21.2d, v21.2d, v0.2d
; CHECK-NEXT: add v17.2d, v17.2d, v0.2d
@@ -178,30 +182,30 @@ define dso_local void @run_test() local_unnamed_addr uwtable {
; CHECK-NEXT: mov x14, x13
; CHECK-NEXT: b.ne .LBB0_1
; CHECK-NEXT: // %bb.2: // %for.cond.cleanup
-; CHECK-NEXT: ldp q24, q18, [sp, #64] // 32-byte Folded Reload
+; CHECK-NEXT: ldp q28, q18, [sp, #64] // 32-byte Folded Reload
; CHECK-NEXT: adrp x8, C
; CHECK-NEXT: add x8, x8, :lo12:C
; CHECK-NEXT: ldp x20, x19, [sp, #176] // 16-byte Folded Reload
; CHECK-NEXT: stp q10, q13, [x8, #64]
-; CHECK-NEXT: stp q24, q18, [x8]
+; CHECK-NEXT: stp q28, q18, [x8]
; CHECK-NEXT: ldr q18, [sp, #96] // 16-byte Reload
-; CHECK-NEXT: stp q29, q28, [x8, #96]
+; CHECK-NEXT: stp q29, q12, [x8, #96]
+; CHECK-NEXT: ldp d13, d12, [sp, #128] // 16-byte Folded Reload
; CHECK-NEXT: stp q18, q15, [x8, #32]
; CHECK-NEXT: ldp d15, d14, [sp, #112] // 16-byte Folded Reload
; CHECK-NEXT: stp q11, q8, [x8, #144]
; CHECK-NEXT: ldp d9, d8, [sp, #160] // 16-byte Folded Reload
-; CHECK-NEXT: stp q12, q22, [x8, #272]
-; CHECK-NEXT: ldp d11, d10, [sp, #144] // 16-byte Folded Reload
-; CHECK-NEXT: ldp d13, d12, [sp, #128] // 16-byte Folded Reload
; CHECK-NEXT: stp q30, q27, [x8, #176]
+; CHECK-NEXT: ldp d11, d10, [sp, #144] // 16-byte Folded Reload
; CHECK-NEXT: str q26, [x8, #208]
; CHECK-NEXT: stp q25, q23, [x8, #240]
+; CHECK-NEXT: stp q24, q22, [x8, #272]
; CHECK-NEXT: stp q21, q20, [x8, #304]
-; CHECK-NEXT: stp q19, q6, [x8, #336]
+; CHECK-NEXT: stp q19, q7, [x8, #336]
; CHECK-NEXT: stp q17, q16, [x8, #368]
-; CHECK-NEXT: stp q1, q5, [x8, #400]
+; CHECK-NEXT: stp q6, q5, [x8, #400]
; CHECK-NEXT: stp q4, q3, [x8, #432]
-; CHECK-NEXT: stp q7, q2, [x8, #464]
+; CHECK-NEXT: stp q1, q2, [x8, #464]
; CHECK-NEXT: str q0, [x8, #496]
; CHECK-NEXT: add sp, sp, #192
; CHECK-NEXT: .cfi_def_cfa_offset 0
diff --git a/llvm/test/CodeGen/PowerPC/common-chain.ll b/llvm/test/CodeGen/PowerPC/common-chain.ll
index 8283e7bac3457..c7c80637a0f87 100644
--- a/llvm/test/CodeGen/PowerPC/common-chain.ll
+++ b/llvm/test/CodeGen/PowerPC/common-chain.ll
@@ -753,176 +753,174 @@ define signext i32 @spill_reduce_succ(ptr %input1, ptr %input2, ptr %output, i64
; CHECK-NEXT: blt cr0, .LBB7_5
; CHECK-NEXT: # %bb.2: # %for.body.preheader.new
; CHECK-NEXT: rldicl r11, r11, 62, 2
-; CHECK-NEXT: sldi r20, r8, 3
-; CHECK-NEXT: mr r14, r7
-; CHECK-NEXT: sldi r7, r7, 3
-; CHECK-NEXT: sldi r21, r9, 3
-; CHECK-NEXT: std r3, -160(r1) # 8-byte Folded Spill
-; CHECK-NEXT: std r9, -208(r1) # 8-byte Folded Spill
-; CHECK-NEXT: std r8, -184(r1) # 8-byte Folded Spill
-; CHECK-NEXT: std r5, -200(r1) # 8-byte Folded Spill
-; CHECK-NEXT: std r4, -168(r1) # 8-byte Folded Spill
-; CHECK-NEXT: std r11, -192(r1) # 8-byte Folded Spill
+; CHECK-NEXT: sldi r31, r8, 3
+; CHECK-NEXT: mr r28, r4
+; CHECK-NEXT: add r14, r10, r7
+; CHECK-NEXT: add r16, r10, r9
+; CHECK-NEXT: add r15, r10, r8
+; CHECK-NEXT: mulli r24, r10, 24
+; CHECK-NEXT: add r25, r24, r31
+; CHECK-NEXT: std r11, -168(r1) # 8-byte Folded Spill
; CHECK-NEXT: sldi r11, r10, 5
-; CHECK-NEXT: add r0, r11, r20
-; CHECK-NEXT: add r12, r11, r21
+; CHECK-NEXT: sldi r2, r9, 3
+; CHECK-NEXT: add r0, r11, r31
+; CHECK-NEXT: add r12, r11, r2
+; CHECK-NEXT: sldi r19, r10, 4
+; CHECK-NEXT: std r3, -160(r1) # 8-byte Folded Spill
+; CHECK-NEXT: add r26, r24, r2
+; CHECK-NEXT: add r21, r19, r2
+; CHECK-NEXT: add r20, r19, r31
+; CHECK-NEXT: std r7, -176(r1) # 8-byte Folded Spill
+; CHECK-NEXT: add r21, r5, r21
+; CHECK-NEXT: add r20, r5, r20
; CHECK-NEXT: add r30, r5, r0
-; CHECK-NEXT: add r0, r11, r7
-; CHECK-NEXT: std r21, -216(r1) # 8-byte Folded Spill
-; CHECK-NEXT: std r20, -224(r1) # 8-byte Folded Spill
-; CHECK-NEXT: add r12, r5, r12
-; CHECK-NEXT: add r29, r5, r0
-; CHECK-NEXT: add r28, r4, r0
-; CHECK-NEXT: add r27, r3, r0
-; CHECK-NEXT: mulli r0, r10, 24
-; CHECK-NEXT: std r14, -176(r1) # 8-byte Folded Spill
-; CHECK-NEXT: add r26, r0, r21
-; CHECK-NEXT: add r25, r0, r20
-; CHECK-NEXT: add r0, r0, r7
-; CHECK-NEXT: add r24, r5, r0
-; CHECK-NEXT: add r23, r4, r0
-; CHECK-NEXT: add r22, r3, r0
-; CHECK-NEXT: sldi r0, r10, 4
+; CHECK-NEXT: sldi r0, r7, 3
+; CHECK-NEXT: add r4, r5, r12
+; CHECK-NEXT: mr r12, r28
; CHECK-NEXT: add r26, r5, r26
+; CHECK-NEXT: std r9, -208(r1) # 8-byte Folded Spill
; CHECK-NEXT: add r25, r5, r25
-; CHECK-NEXT: add r21, r0, r21
-; CHECK-NEXT: add r20, r0, r20
-; CHECK-NEXT: add r0, r0, r7
-; CHECK-NEXT: add r19, r5, r0
-; CHECK-NEXT: add r18, r4, r0
-; CHECK-NEXT: add r17, r3, r0
-; CHECK-NEXT: add r0, r10, r9
-; CHECK-NEXT: add r21, r5, r21
-; CHECK-NEXT: add r20, r5, r20
-; CHECK-NEXT: sldi r0, r0, 3
-; CHECK-NEXT: add r16, r5, r0
-; CHECK-NEXT: add r0, r10, r8
-; CHECK-NEXT: sldi r0, r0, 3
-; CHECK-NEXT: add r15, r5, r0
-; CHECK-NEXT: add r0, r10, r14
-; CHECK-NEXT: sldi r0, r0, 3
-; CHECK-NEXT: add r2, r3, r0
-; CHECK-NEXT: ld r3, -224(r1) # 8-byte Folded Reload
-; CHECK-NEXT: add r14, r5, r0
-; CHECK-NEXT: add r31, r4, r0
-; CHECK-NEXT: sub r0, r3, r7
-; CHECK-NEXT: ld r3, -192(r1) # 8-byte Folded Reload
-; CHECK-NEXT: rldicl r9, r3, 2, 1
-; CHECK-NEXT: ld r3, -216(r1) # 8-byte Folded Reload
-; CHECK-NEXT: addi r8, r9, -4
-; CHECK-NEXT: rldicl r8, r8, 62, 2
-; CHECK-NEXT: sub r7, r3, r7
-; CHECK-NEXT: ori r3, r9, 1
-; CHECK-NEXT: addi r8, r8, 1
+; CHECK-NEXT: add r27, r11, r0
+; CHECK-NEXT: add r22, r24, r0
+; CHECK-NEXT: add r17, r19, r0
+; CHECK-NEXT: sldi r16, r16, 3
+; CHECK-NEXT: add r16, r5, r16
+; CHECK-NEXT: sub r31, r31, r0
+; CHECK-NEXT: sub r2, r2, r0
+; CHECK-NEXT: li r0, 0
+; CHECK-NEXT: add r29, r5, r27
+; CHECK-NEXT: add r28, r28, r27
+; CHECK-NEXT: add r27, r3, r27
+; CHECK-NEXT: add r24, r5, r22
+; CHECK-NEXT: add r23, r12, r22
+; CHECK-NEXT: add r22, r3, r22
+; CHECK-NEXT: add r19, r5, r17
+; CHECK-NEXT: add r18, r12, r17
+; CHECK-NEXT: add r17, r3, r17
+; CHECK-NEXT: ld r3, -168(r1) # 8-byte Folded Reload
+; CHECK-NEXT: std r8, -184(r1) # 8-byte Folded Spill
+; CHECK-NEXT: sldi r15, r15, 3
+; CHECK-NEXT: add r15, r5, r15
+; CHECK-NEXT: sldi r8, r14, 3
+; CHECK-NEXT: add r14, r5, r8
+; CHECK-NEXT: std r5, -192(r1) # 8-byte Folded Spill
+; CHECK-NEXT: rldicl r3, r3, 2, 1
+; CHECK-NEXT: addi r7, r3, -4
+; CHECK-NEXT: ori r3, r3, 1
+; CHECK-NEXT: rldicl r7, r7, 62, 2
; CHECK-NEXT: mulld r3, r10, r3
-; CHECK-NEXT: mtctr r8
-; CHECK-NEXT: li r8, 0
-; CHECK-NEXT: std r10, -192(r1) # 8-byte Folded Spill
+; CHECK-NEXT: addi r7, r7, 1
+; CHECK-NEXT: mtctr r7
+; CHECK-NEXT: std r12, -168(r1) # 8-byte Folded Spill
+; CHECK-NEXT: add r12, r12, r8
+; CHECK-NEXT: std r10, -200(r1) # 8-byte Folded Spill
+; CHECK-NEXT: ld r5, -160(r1) # 8-byte Folded Reload
; CHECK-NEXT: std r3, -216(r1) # 8-byte Folded Spill
+; CHECK-NEXT: add r7, r5, r8
; CHECK-NEXT: .p2align 4
; CHECK-NEXT: .LBB7_3: # %for.body
; CHECK-NEXT: #
-; CHECK-NEXT: lfd f0, 0(r2)
-; CHECK-NEXT: lfd f1, 0(r31)
+; CHECK-NEXT: lfd f0, 0(r7)
+; CHECK-NEXT: lfd f1, 0(r12)
; CHECK-NEXT: xsmuldp f0, f0, f1
; CHECK-NEXT: lfd f1, 0(r14)
; CHECK-NEXT: xsadddp f0, f1, f0
; CHECK-NEXT: stfd f0, 0(r14)
; CHECK-NEXT: add r14, r14, r11
-; CHECK-NEXT: lfdx f0, r2, r0
-; CHECK-NEXT: lfdx f1, r31, r0
+; CHECK-NEXT: lfdx f0, r7, r31
+; CHECK-NEXT: lfdx f1, r12, r31
; CHECK-NEXT: xsmuldp f0, f0, f1
-; CHECK-NEXT: lfdx f1, r15, r8
+; CHECK-NEXT: lfdx f1, r15, r0
; CHECK-NEXT: xsadddp f0, f1, f0
-; CHECK-NEXT: stfdx f0, r15, r8
-; CHECK-NEXT: lfdx f0, r2, r7
-; CHECK-NEXT: lfdx f1, r31, r7
-; CHECK-NEXT: add r2, r2, r11
-; CHECK-NEXT: add r31, r31, r11
+; CHECK-NEXT: stfdx f0, r15, r0
+; CHECK-NEXT: lfdx f0, r7, r2
+; CHECK-NEXT: lfdx f1, r12, r2
+; CHECK-NEXT: add r7, r7, r11
+; CHECK-NEXT: add r12, r12, r11
; CHECK-NEXT: xsmuldp f0, f0, f1
-; CHECK-NEXT: lfdx f1, r16, r8
+; CHECK-NEXT: lfdx f1, r16, r0
; CHECK-NEXT: xsadddp f0, f1, f0
-; CHECK-NEXT: stfdx f0, r16, r8
+; CHECK-NEXT: stfdx f0, r16, r0
; CHECK-NEXT: lfd f0, 0(r17)
; CHECK-NEXT: lfd f1, 0(r18)
; CHECK-NEXT: xsmuldp f0, f0, f1
-; CHECK-NEXT: lfdx f1, r19, r8
+; CHECK-NEXT: lfdx f1, r19, r0
; CHECK-NEXT: xsadddp f0, f1, f0
-; CHECK-NEXT: stfdx f0, r19, r8
-; CHECK-NEXT: lfdx f0, r17, r0
-; CHECK-NEXT: lfdx f1, r18, r0
+; CHECK-NEXT: stfdx f0, r19, r0
+; CHECK-NEXT: lfdx f0, r17, r31
+; CHECK-NEXT: lfdx f1, r18, r31
; CHECK-NEXT: xsmuldp f0, f0, f1
-; CHECK-NEXT: lfdx f1, r20, r8
+; CHECK-NEXT: lfdx f1, r20, r0
; CHECK-NEXT: xsadddp f0, f1, f0
-; CHECK-NEXT: stfdx f0, r20, r8
-; CHECK-NEXT: lfdx f0, r17, r7
-; CHECK-NEXT: lfdx f1, r18, r7
+; CHECK-NEXT: stfdx f0, r20, r0
+; CHECK-NEXT: lfdx f0, r17, r2
+; CHECK-NEXT: lfdx f1, r18, r2
; CHECK-NEXT: add r17, r17, r11
; CHECK-NEXT: add r18, r18, r11
; CHECK-NEXT: xsmuldp f0, f0, f1
-; CHECK-NEXT: lfdx f1, r21, r8
+; CHECK-NEXT: lfdx f1, r21, r0
; CHECK-NEXT: xsadddp f0, f1, f0
-; CHECK-NEXT: stfdx f0, r21, r8
+; CHECK-NEXT: stfdx f0, r21, r0
; CHECK-NEXT: lfd f0, 0(r22)
; CHECK-NEXT: lfd f1, 0(r23)
; CHECK-NEXT: xsmuldp f0, f0, f1
-; CHECK-NEXT: lfdx f1, r24, r8
+; CHECK-NEXT: lfdx f1, r24, r0
; CHECK-NEXT: xsadddp f0, f1, f0
-; CHECK-NEXT: stfdx f0, r24, r8
-; CHECK-NEXT: lfdx f0, r22, r0
-; CHECK-NEXT: lfdx f1, r23, r0
+; CHECK-NEXT: stfdx f0, r24, r0
+; CHECK-NEXT: lfdx f0, r22, r31
+; CHECK-NEXT: lfdx f1, r23, r31
; CHECK-NEXT: xsmuldp f0, f0, f1
-; CHECK-NEXT: lfdx f1, r25, r8
+; CHECK-NEXT: lfdx f1, r25, r0
; CHECK-NEXT: xsadddp f0, f1, f0
-; CHECK-NEXT: stfdx f0, r25, r8
-; CHECK-NEXT: lfdx f0, r22, r7
-; CHECK-NEXT: lfdx f1, r23, r7
+; CHECK-NEXT: stfdx f0, r25, r0
+; CHECK-NEXT: lfdx f0, r22, r2
+; CHECK-NEXT: lfdx f1, r23, r2
; CHECK-NEXT: add r22, r22, r11
; CHECK-NEXT: add r23, r23, r11
; CHECK-NEXT: xsmuldp f0, f0, f1
-; CHECK-NEXT: lfdx f1, r26, r8
+; CHECK-NEXT: lfdx f1, r26, r0
; CHECK-NEXT: xsadddp f0, f1, f0
-; CHECK-NEXT: stfdx f0, r26, r8
+; CHECK-NEXT: stfdx f0, r26, r0
; CHECK-NEXT: lfd f0, 0(r27)
; CHECK-NEXT: lfd f1, 0(r28)
; CHECK-NEXT: xsmuldp f0, f0, f1
-; CHECK-NEXT: lfdx f1, r29, r8
+; CHECK-NEXT: lfdx f1, r29, r0
; CHECK-NEXT: xsadddp f0, f1, f0
-; CHECK-NEXT: stfdx f0, r29, r8
-; CHECK-NEXT: lfdx f0, r27, r0
-; CHECK-NEXT: lfdx f1, r28, r0
+; CHECK-NEXT: stfdx f0, r29, r0
+; CHECK-NEXT: lfdx f0, r27, r31
+; CHECK-NEXT: lfdx f1, r28, r31
; CHECK-NEXT: xsmuldp f0, f0, f1
-; CHECK-NEXT: lfdx f1, r30, r8
+; CHECK-NEXT: lfdx f1, r30, r0
; CHECK-NEXT: xsadddp f0, f1, f0
-; CHECK-NEXT: stfdx f0, r30, r8
-; CHECK-NEXT: lfdx f0, r27, r7
-; CHECK-NEXT: lfdx f1, r28, r7
+; CHECK-NEXT: stfdx f0, r30, r0
+; CHECK-NEXT: lfdx f0, r27, r2
+; CHECK-NEXT: lfdx f1, r28, r2
; CHECK-NEXT: add r27, r27, r11
; CHECK-NEXT: add r28, r28, r11
; CHECK-NEXT: xsmuldp f0, f0, f1
-; CHECK-NEXT: lfdx f1, r12, r8
+; CHECK-NEXT: lfdx f1, r4, r0
; CHECK-NEXT: xsadddp f0, f1, f0
-; CHECK-NEXT: stfdx f0, r12, r8
-; CHECK-NEXT: add r8, r8, r11
+; CHECK-NEXT: stfdx f0, r4, r0
+; CHECK-NEXT: add r0, r0, r11
; CHECK-NEXT: bdnz .LBB7_3
; CHECK-NEXT: # %bb.4:
; CHECK-NEXT: ld r3, -160(r1) # 8-byte Folded Reload
; CHECK-NEXT: ld r4, -168(r1) # 8-byte Folded Reload
; CHECK-NEXT: ld r7, -176(r1) # 8-byte Folded Reload
; CHECK-NEXT: ld r8, -184(r1) # 8-byte Folded Reload
-; CHECK-NEXT: ld r10, -192(r1) # 8-byte Folded Reload
-; CHECK-NEXT: ld r5, -200(r1) # 8-byte Folded Reload
+; CHECK-NEXT: ld r5, -192(r1) # 8-byte Folded Reload
+; CHECK-NEXT: ld r10, -200(r1) # 8-byte Folded Reload
; CHECK-NEXT: ld r9, -208(r1) # 8-byte Folded Reload
; CHECK-NEXT: ld r26, -216(r1) # 8-byte Folded Reload
; CHECK-NEXT: .LBB7_5: # %for.cond.cleanup.loopexit.unr-lcssa
; CHECK-NEXT: cmpldi r6, 0
; CHECK-NEXT: beq cr0, .LBB7_8
; CHECK-NEXT: # %bb.6: # %for.body.epil.preheader
-; CHECK-NEXT: add r11, r26, r9
+; CHECK-NEXT: add r9, r26, r9
; CHECK-NEXT: add r12, r26, r8
-; CHECK-NEXT: add r9, r26, r7
; CHECK-NEXT: sldi r27, r10, 3
-; CHECK-NEXT: sldi r11, r11, 3
+; CHECK-NEXT: sldi r11, r9, 3
+; CHECK-NEXT: add r9, r26, r7
; CHECK-NEXT: sldi r0, r12, 3
; CHECK-NEXT: sldi r9, r9, 3
; CHECK-NEXT: add r28, r5, r11
diff --git a/llvm/test/CodeGen/PowerPC/more-dq-form-prepare.ll b/llvm/test/CodeGen/PowerPC/more-dq-form-prepare.ll
index af0942e99182d..a85b4f51ae621 100644
--- a/llvm/test/CodeGen/PowerPC/more-dq-form-prepare.ll
+++ b/llvm/test/CodeGen/PowerPC/more-dq-form-prepare.ll
@@ -18,8 +18,8 @@ define void @foo(ptr %.m, ptr %.n, ptr %.a, ptr %.x, ptr %.l, ptr %.vy01, ptr %.
; CHECK-NEXT: cmpwi 3, 1
; CHECK-NEXT: bltlr 0
; CHECK-NEXT: # %bb.2: # %_loop_1_do_.preheader
-; CHECK-NEXT: stdu 1, -592(1)
-; CHECK-NEXT: .cfi_def_cfa_offset 592
+; CHECK-NEXT: stdu 1, -576(1)
+; CHECK-NEXT: .cfi_def_cfa_offset 576
; CHECK-NEXT: .cfi_offset r14, -192
; CHECK-NEXT: .cfi_offset r15, -184
; CHECK-NEXT: .cfi_offset r16, -176
@@ -56,152 +56,146 @@ define void @foo(ptr %.m, ptr %.n, ptr %.a, ptr %.x, ptr %.l, ptr %.vy01, ptr %.
; CHECK-NEXT: .cfi_offset v29, -240
; CHECK-NEXT: .cfi_offset v30, -224
; CHECK-NEXT: .cfi_offset v31, -208
-; CHECK-NEXT: std 14, 400(1) # 8-byte Folded Spill
-; CHECK-NEXT: std 15, 408(1) # 8-byte Folded Spill
-; CHECK-NEXT: ld 2, 728(1)
-; CHECK-NEXT: ld 14, 688(1)
-; CHECK-NEXT: ld 11, 704(1)
-; CHECK-NEXT: std 20, 448(1) # 8-byte Folded Spill
-; CHECK-NEXT: std 21, 456(1) # 8-byte Folded Spill
+; CHECK-NEXT: std 20, 432(1) # 8-byte Folded Spill
+; CHECK-NEXT: std 21, 440(1) # 8-byte Folded Spill
; CHECK-NEXT: mr 21, 5
; CHECK-NEXT: lwa 5, 0(7)
-; CHECK-NEXT: ld 7, 720(1)
-; CHECK-NEXT: std 22, 464(1) # 8-byte Folded Spill
-; CHECK-NEXT: std 23, 472(1) # 8-byte Folded Spill
+; CHECK-NEXT: ld 7, 696(1)
+; CHECK-NEXT: ld 11, 688(1)
+; CHECK-NEXT: std 16, 400(1) # 8-byte Folded Spill
+; CHECK-NEXT: std 17, 408(1) # 8-byte Folded Spill
+; CHECK-NEXT: ld 16, 672(1)
+; CHECK-NEXT: ld 12, 680(1)
+; CHECK-NEXT: std 22, 448(1) # 8-byte Folded Spill
+; CHECK-NEXT: std 23, 456(1) # 8-byte Folded Spill
; CHECK-NEXT: mr 22, 6
-; CHECK-NEXT: ld 6, 848(1)
+; CHECK-NEXT: ld 6, 704(1)
; CHECK-NEXT: addi 3, 3, 1
-; CHECK-NEXT: ld 15, 736(1)
-; CHECK-NEXT: std 18, 432(1) # 8-byte Folded Spill
-; CHECK-NEXT: std 19, 440(1) # 8-byte Folded Spill
-; CHECK-NEXT: ld 19, 768(1)
-; CHECK-NEXT: ld 18, 760(1)
-; CHECK-NEXT: std 30, 528(1) # 8-byte Folded Spill
-; CHECK-NEXT: std 31, 536(1) # 8-byte Folded Spill
-; CHECK-NEXT: ld 12, 696(1)
-; CHECK-NEXT: lxv 0, 0(9)
-; CHECK-NEXT: std 9, 64(1) # 8-byte Folded Spill
-; CHECK-NEXT: std 10, 72(1) # 8-byte Folded Spill
-; CHECK-NEXT: lxv 1, 0(8)
+; CHECK-NEXT: ld 17, 712(1)
+; CHECK-NEXT: std 14, 384(1) # 8-byte Folded Spill
+; CHECK-NEXT: std 15, 392(1) # 8-byte Folded Spill
+; CHECK-NEXT: ld 14, 728(1)
; CHECK-NEXT: cmpldi 3, 9
-; CHECK-NEXT: ld 30, 824(1)
-; CHECK-NEXT: std 28, 512(1) # 8-byte Folded Spill
-; CHECK-NEXT: std 29, 520(1) # 8-byte Folded Spill
-; CHECK-NEXT: ld 29, 840(1)
-; CHECK-NEXT: ld 28, 832(1)
-; CHECK-NEXT: std 16, 416(1) # 8-byte Folded Spill
-; CHECK-NEXT: std 17, 424(1) # 8-byte Folded Spill
-; CHECK-NEXT: ld 23, 784(1)
-; CHECK-NEXT: ld 20, 776(1)
-; CHECK-NEXT: std 24, 480(1) # 8-byte Folded Spill
-; CHECK-NEXT: std 25, 488(1) # 8-byte Folded Spill
-; CHECK-NEXT: ld 25, 800(1)
-; CHECK-NEXT: ld 24, 792(1)
-; CHECK-NEXT: std 26, 496(1) # 8-byte Folded Spill
-; CHECK-NEXT: std 27, 504(1) # 8-byte Folded Spill
-; CHECK-NEXT: ld 27, 816(1)
-; CHECK-NEXT: ld 26, 808(1)
-; CHECK-NEXT: stfd 26, 544(1) # 8-byte Folded Spill
-; CHECK-NEXT: stfd 27, 552(1) # 8-byte Folded Spill
-; CHECK-NEXT: ld 17, 752(1)
-; CHECK-NEXT: extswsli 9, 5, 3
-; CHECK-NEXT: lxv 4, 0(14)
-; CHECK-NEXT: std 14, 32(1) # 8-byte Folded Spill
-; CHECK-NEXT: std 12, 40(1) # 8-byte Folded Spill
+; CHECK-NEXT: ld 2, 816(1)
+; CHECK-NEXT: std 18, 416(1) # 8-byte Folded Spill
+; CHECK-NEXT: std 19, 424(1) # 8-byte Folded Spill
+; CHECK-NEXT: ld 18, 720(1)
+; CHECK-NEXT: ld 23, 752(1)
+; CHECK-NEXT: std 24, 464(1) # 8-byte Folded Spill
+; CHECK-NEXT: std 25, 472(1) # 8-byte Folded Spill
+; CHECK-NEXT: ld 25, 768(1)
+; CHECK-NEXT: ld 24, 760(1)
+; CHECK-NEXT: std 26, 480(1) # 8-byte Folded Spill
+; CHECK-NEXT: std 27, 488(1) # 8-byte Folded Spill
+; CHECK-NEXT: ld 27, 784(1)
+; CHECK-NEXT: ld 26, 776(1)
+; CHECK-NEXT: std 28, 496(1) # 8-byte Folded Spill
+; CHECK-NEXT: std 29, 504(1) # 8-byte Folded Spill
+; CHECK-NEXT: ld 29, 800(1)
+; CHECK-NEXT: ld 28, 792(1)
+; CHECK-NEXT: std 30, 512(1) # 8-byte Folded Spill
+; CHECK-NEXT: std 31, 520(1) # 8-byte Folded Spill
+; CHECK-NEXT: ld 30, 808(1)
+; CHECK-NEXT: ld 20, 744(1)
+; CHECK-NEXT: stfd 26, 528(1) # 8-byte Folded Spill
+; CHECK-NEXT: stfd 27, 536(1) # 8-byte Folded Spill
+; CHECK-NEXT: ld 19, 736(1)
+; CHECK-NEXT: lxv 7, 0(8)
+; CHECK-NEXT: stfd 28, 544(1) # 8-byte Folded Spill
+; CHECK-NEXT: stfd 29, 552(1) # 8-byte Folded Spill
+; CHECK-NEXT: lxv 6, 0(9)
+; CHECK-NEXT: mulli 15, 5, 24
+; CHECK-NEXT: lxv 4, 0(16)
+; CHECK-NEXT: std 16, 40(1) # 8-byte Folded Spill
+; CHECK-NEXT: std 12, 48(1) # 8-byte Folded Spill
+; CHECK-NEXT: lxv 1, 0(7)
+; CHECK-NEXT: sldi 16, 5, 5
+; CHECK-NEXT: lxv 0, 0(6)
+; CHECK-NEXT: std 11, 56(1) # 8-byte Folded Spill
+; CHECK-NEXT: std 7, 64(1) # 8-byte Folded Spill
+; CHECK-NEXT: li 7, 9
+; CHECK-NEXT: lxv 3, 0(12)
; CHECK-NEXT: mulli 0, 5, 40
-; CHECK-NEXT: sldi 14, 5, 5
-; CHECK-NEXT: mulli 31, 5, 24
-; CHECK-NEXT: lxv 38, 0(2)
-; CHECK-NEXT: lxv 2, 0(11)
-; CHECK-NEXT: std 2, 80(1) # 8-byte Folded Spill
-; CHECK-NEXT: std 15, 88(1) # 8-byte Folded Spill
-; CHECK-NEXT: mulli 2, 5, 48
-; CHECK-NEXT: sldi 5, 5, 4
-; CHECK-NEXT: ld 16, 744(1)
+; CHECK-NEXT: lxv 39, 0(14)
+; CHECK-NEXT: std 6, 72(1) # 8-byte Folded Spill
+; CHECK-NEXT: std 17, 80(1) # 8-byte Folded Spill
; CHECK-NEXT: lxv 5, 0(10)
-; CHECK-NEXT: std 6, 200(1) # 8-byte Folded Spill
-; CHECK-NEXT: std 29, 192(1) # 8-byte Folded Spill
-; CHECK-NEXT: ld 6, 712(1)
-; CHECK-NEXT: mr 10, 7
-; CHECK-NEXT: add 7, 14, 21
-; CHECK-NEXT: lxv 13, 0(19)
-; CHECK-NEXT: std 8, 48(1) # 8-byte Folded Spill
-; CHECK-NEXT: std 6, 56(1) # 8-byte Folded Spill
-; CHECK-NEXT: mr 8, 11
-; CHECK-NEXT: li 11, 9
-; CHECK-NEXT: iselgt 3, 3, 11
-; CHECK-NEXT: addi 3, 3, -2
-; CHECK-NEXT: rldicl 11, 3, 61, 3
-; CHECK-NEXT: lxv 3, 0(12)
-; CHECK-NEXT: lxv 40, 0(6)
-; CHECK-NEXT: std 18, 112(1) # 8-byte Folded Spill
-; CHECK-NEXT: std 19, 120(1) # 8-byte Folded Spill
-; CHECK-NEXT: add 19, 21, 5
-; CHECK-NEXT: ld 5, 200(1) # 8-byte Folded Reload
-; CHECK-NEXT: lxv 39, 0(10)
-; CHECK-NEXT: addi 3, 7, 32
-; CHECK-NEXT: add 12, 31, 21
-; CHECK-NEXT: std 20, 128(1) # 8-byte Folded Spill
-; CHECK-NEXT: std 23, 136(1) # 8-byte Folded Spill
-; CHECK-NEXT: lxv 33, 0(15)
-; CHECK-NEXT: lxv 32, 0(16)
-; CHECK-NEXT: std 26, 160(1) # 8-byte Folded Spill
-; CHECK-NEXT: std 27, 168(1) # 8-byte Folded Spill
-; CHECK-NEXT: lxv 37, 0(17)
-; CHECK-NEXT: lxv 36, 0(18)
+; CHECK-NEXT: add 6, 16, 21
+; CHECK-NEXT: iselgt 3, 3, 7
+; CHECK-NEXT: lxv 2, 0(11)
+; CHECK-NEXT: std 18, 88(1) # 8-byte Folded Spill
+; CHECK-NEXT: std 14, 96(1) # 8-byte Folded Spill
+; CHECK-NEXT: ld 14, 824(1)
+; CHECK-NEXT: lxv 41, 0(17)
+; CHECK-NEXT: std 29, 168(1) # 8-byte Folded Spill
; CHECK-NEXT: std 30, 176(1) # 8-byte Folded Spill
-; CHECK-NEXT: std 28, 184(1) # 8-byte Folded Spill
-; CHECK-NEXT: lxv 12, 0(20)
-; CHECK-NEXT: lxv 11, 0(23)
-; CHECK-NEXT: add 20, 21, 9
-; CHECK-NEXT: stfd 28, 560(1) # 8-byte Folded Spill
-; CHECK-NEXT: stfd 29, 568(1) # 8-byte Folded Spill
-; CHECK-NEXT: lxv 10, 0(24)
-; CHECK-NEXT: lxv 9, 0(25)
-; CHECK-NEXT: stfd 30, 576(1) # 8-byte Folded Spill
-; CHECK-NEXT: stfd 31, 584(1) # 8-byte Folded Spill
-; CHECK-NEXT: lxv 8, 0(26)
-; CHECK-NEXT: lxv 7, 0(27)
-; CHECK-NEXT: addi 12, 12, 32
+; CHECK-NEXT: lxv 40, 0(18)
+; CHECK-NEXT: lxv 38, 0(19)
+; CHECK-NEXT: std 19, 104(1) # 8-byte Folded Spill
+; CHECK-NEXT: std 20, 112(1) # 8-byte Folded Spill
+; CHECK-NEXT: lxv 33, 0(20)
+; CHECK-NEXT: lxv 32, 0(23)
+; CHECK-NEXT: std 25, 136(1) # 8-byte Folded Spill
+; CHECK-NEXT: std 26, 144(1) # 8-byte Folded Spill
+; CHECK-NEXT: addi 3, 3, -2
+; CHECK-NEXT: addi 31, 6, 32
+; CHECK-NEXT: lxv 37, 0(24)
+; CHECK-NEXT: lxv 36, 0(25)
+; CHECK-NEXT: std 27, 152(1) # 8-byte Folded Spill
+; CHECK-NEXT: std 28, 160(1) # 8-byte Folded Spill
+; CHECK-NEXT: lxv 13, 0(26)
+; CHECK-NEXT: add 7, 15, 21
+; CHECK-NEXT: add 6, 0, 21
+; CHECK-NEXT: lxv 12, 0(27)
; CHECK-NEXT: li 27, 0
; CHECK-NEXT: mr 26, 21
-; CHECK-NEXT: stxv 52, 208(1) # 16-byte Folded Spill
-; CHECK-NEXT: stxv 53, 224(1) # 16-byte Folded Spill
-; CHECK-NEXT: lxv 6, 0(30)
-; CHECK-NEXT: lxv 41, 0(28)
-; CHECK-NEXT: addi 7, 11, 1
-; CHECK-NEXT: add 11, 0, 21
+; CHECK-NEXT: stfd 30, 560(1) # 8-byte Folded Spill
+; CHECK-NEXT: stfd 31, 568(1) # 8-byte Folded Spill
+; CHECK-NEXT: lxv 11, 0(28)
+; CHECK-NEXT: lxv 10, 0(29)
+; CHECK-NEXT: rldicl 3, 3, 61, 3
; CHECK-NEXT: li 28, 1
-; CHECK-NEXT: stxv 54, 240(1) # 16-byte Folded Spill
-; CHECK-NEXT: stxv 55, 256(1) # 16-byte Folded Spill
-; CHECK-NEXT: lxv 43, 0(29)
-; CHECK-NEXT: lxv 42, 0(5)
-; CHECK-NEXT: stxv 56, 272(1) # 16-byte Folded Spill
-; CHECK-NEXT: stxv 57, 288(1) # 16-byte Folded Spill
-; CHECK-NEXT: addi 11, 11, 32
-; CHECK-NEXT: stxv 58, 304(1) # 16-byte Folded Spill
-; CHECK-NEXT: stxv 59, 320(1) # 16-byte Folded Spill
-; CHECK-NEXT: stxv 60, 336(1) # 16-byte Folded Spill
-; CHECK-NEXT: stxv 61, 352(1) # 16-byte Folded Spill
-; CHECK-NEXT: stxv 62, 368(1) # 16-byte Folded Spill
-; CHECK-NEXT: stxv 63, 384(1) # 16-byte Folded Spill
-; CHECK-NEXT: std 16, 96(1) # 8-byte Folded Spill
-; CHECK-NEXT: std 17, 104(1) # 8-byte Folded Spill
-; CHECK-NEXT: std 24, 144(1) # 8-byte Folded Spill
-; CHECK-NEXT: std 25, 152(1) # 8-byte Folded Spill
+; CHECK-NEXT: stxv 52, 192(1) # 16-byte Folded Spill
+; CHECK-NEXT: stxv 53, 208(1) # 16-byte Folded Spill
+; CHECK-NEXT: lxv 9, 0(30)
+; CHECK-NEXT: lxv 8, 0(2)
+; CHECK-NEXT: mulli 30, 5, 48
+; CHECK-NEXT: addi 12, 7, 32
+; CHECK-NEXT: addi 11, 6, 32
+; CHECK-NEXT: extswsli 6, 5, 3
+; CHECK-NEXT: sldi 5, 5, 4
+; CHECK-NEXT: stxv 54, 224(1) # 16-byte Folded Spill
+; CHECK-NEXT: stxv 55, 240(1) # 16-byte Folded Spill
+; CHECK-NEXT: ld 7, 832(1)
+; CHECK-NEXT: lxv 42, 0(14)
+; CHECK-NEXT: addi 3, 3, 1
+; CHECK-NEXT: stxv 56, 256(1) # 16-byte Folded Spill
+; CHECK-NEXT: stxv 57, 272(1) # 16-byte Folded Spill
+; CHECK-NEXT: add 20, 21, 6
+; CHECK-NEXT: add 19, 21, 5
+; CHECK-NEXT: lxv 43, 0(7)
+; CHECK-NEXT: stxv 58, 288(1) # 16-byte Folded Spill
+; CHECK-NEXT: stxv 59, 304(1) # 16-byte Folded Spill
+; CHECK-NEXT: stxv 60, 320(1) # 16-byte Folded Spill
+; CHECK-NEXT: stxv 61, 336(1) # 16-byte Folded Spill
+; CHECK-NEXT: stxv 62, 352(1) # 16-byte Folded Spill
+; CHECK-NEXT: stxv 63, 368(1) # 16-byte Folded Spill
+; CHECK-NEXT: std 23, 120(1) # 8-byte Folded Spill
+; CHECK-NEXT: std 24, 128(1) # 8-byte Folded Spill
+; CHECK-NEXT: std 2, 184(1) # 8-byte Folded Spill
; CHECK-NEXT: .p2align 4
; CHECK-NEXT: .LBB0_3: # %_loop_2_do_.lr.ph
; CHECK-NEXT: # =>This Loop Header: Depth=1
; CHECK-NEXT: # Child Loop BB0_4 Depth 2
-; CHECK-NEXT: maddld 5, 2, 27, 0
+; CHECK-NEXT: maddld 5, 30, 27, 0
; CHECK-NEXT: mr 6, 22
-; CHECK-NEXT: mr 30, 20
+; CHECK-NEXT: mr 2, 20
; CHECK-NEXT: mr 29, 19
-; CHECK-NEXT: mtctr 7
+; CHECK-NEXT: mtctr 3
; CHECK-NEXT: add 25, 21, 5
-; CHECK-NEXT: maddld 5, 2, 27, 14
+; CHECK-NEXT: maddld 5, 30, 27, 16
; CHECK-NEXT: add 24, 21, 5
-; CHECK-NEXT: maddld 5, 2, 27, 31
+; CHECK-NEXT: maddld 5, 30, 27, 15
; CHECK-NEXT: add 23, 21, 5
; CHECK-NEXT: mr 5, 26
; CHECK-NEXT: .p2align 5
@@ -210,16 +204,16 @@ define void @foo(ptr %.m, ptr %.n, ptr %.a, ptr %.x, ptr %.l, ptr %.vy01, ptr %.
; CHECK-NEXT: # => This Inner Loop Header: Depth=2
; CHECK-NEXT: lxvp 34, 0(6)
; CHECK-NEXT: lxvp 44, 0(5)
-; CHECK-NEXT: xvmaddadp 1, 45, 35
-; CHECK-NEXT: lxvp 46, 0(30)
-; CHECK-NEXT: xvmaddadp 0, 47, 35
+; CHECK-NEXT: xvmaddadp 7, 45, 35
+; CHECK-NEXT: lxvp 46, 0(2)
+; CHECK-NEXT: xvmaddadp 6, 47, 35
; CHECK-NEXT: lxvp 48, 0(29)
; CHECK-NEXT: lxvp 50, 0(23)
; CHECK-NEXT: lxvp 62, 0(24)
; CHECK-NEXT: lxvp 60, 0(25)
; CHECK-NEXT: lxvp 58, 32(6)
; CHECK-NEXT: lxvp 56, 32(5)
-; CHECK-NEXT: lxvp 54, 32(30)
+; CHECK-NEXT: lxvp 54, 32(2)
; CHECK-NEXT: lxvp 52, 32(29)
; CHECK-NEXT: lxvp 30, 32(23)
; CHECK-NEXT: lxvp 28, 32(24)
@@ -228,27 +222,27 @@ define void @foo(ptr %.m, ptr %.n, ptr %.a, ptr %.x, ptr %.l, ptr %.vy01, ptr %.
; CHECK-NEXT: xvmaddadp 4, 51, 35
; CHECK-NEXT: xvmaddadp 3, 63, 35
; CHECK-NEXT: xvmaddadp 2, 61, 35
-; CHECK-NEXT: xvmaddadp 40, 44, 34
-; CHECK-NEXT: xvmaddadp 39, 46, 34
-; CHECK-NEXT: xvmaddadp 38, 48, 34
-; CHECK-NEXT: xvmaddadp 33, 50, 34
-; CHECK-NEXT: xvmaddadp 32, 62, 34
-; CHECK-NEXT: xvmaddadp 37, 60, 34
-; CHECK-NEXT: xvmaddadp 36, 57, 59
-; CHECK-NEXT: xvmaddadp 13, 55, 59
-; CHECK-NEXT: xvmaddadp 12, 53, 59
-; CHECK-NEXT: xvmaddadp 11, 31, 59
-; CHECK-NEXT: xvmaddadp 10, 29, 59
-; CHECK-NEXT: xvmaddadp 9, 27, 59
-; CHECK-NEXT: xvmaddadp 8, 56, 58
-; CHECK-NEXT: xvmaddadp 7, 54, 58
-; CHECK-NEXT: xvmaddadp 6, 52, 58
-; CHECK-NEXT: xvmaddadp 41, 30, 58
-; CHECK-NEXT: xvmaddadp 43, 28, 58
-; CHECK-NEXT: xvmaddadp 42, 26, 58
+; CHECK-NEXT: xvmaddadp 1, 44, 34
+; CHECK-NEXT: xvmaddadp 0, 46, 34
+; CHECK-NEXT: xvmaddadp 41, 48, 34
+; CHECK-NEXT: xvmaddadp 40, 50, 34
+; CHECK-NEXT: xvmaddadp 39, 62, 34
+; CHECK-NEXT: xvmaddadp 38, 60, 34
+; CHECK-NEXT: xvmaddadp 33, 57, 59
+; CHECK-NEXT: xvmaddadp 32, 55, 59
+; CHECK-NEXT: xvmaddadp 37, 53, 59
+; CHECK-NEXT: xvmaddadp 36, 31, 59
+; CHECK-NEXT: xvmaddadp 13, 29, 59
+; CHECK-NEXT: xvmaddadp 12, 27, 59
+; CHECK-NEXT: xvmaddadp 11, 56, 58
+; CHECK-NEXT: xvmaddadp 10, 54, 58
+; CHECK-NEXT: xvmaddadp 9, 52, 58
+; CHECK-NEXT: xvmaddadp 8, 30, 58
+; CHECK-NEXT: xvmaddadp 42, 28, 58
+; CHECK-NEXT: xvmaddadp 43, 26, 58
; CHECK-NEXT: addi 6, 6, 64
; CHECK-NEXT: addi 5, 5, 64
-; CHECK-NEXT: addi 30, 30, 64
+; CHECK-NEXT: addi 2, 2, 64
; CHECK-NEXT: addi 29, 29, 64
; CHECK-NEXT: addi 23, 23, 64
; CHECK-NEXT: addi 24, 24, 64
@@ -257,99 +251,96 @@ define void @foo(ptr %.m, ptr %.n, ptr %.a, ptr %.x, ptr %.l, ptr %.vy01, ptr %.
; CHECK-NEXT: # %bb.5: # %_loop_2_endl_
; CHECK-NEXT: #
; CHECK-NEXT: addi 28, 28, 6
-; CHECK-NEXT: add 26, 26, 2
-; CHECK-NEXT: add 20, 20, 2
-; CHECK-NEXT: add 11, 11, 2
-; CHECK-NEXT: add 19, 19, 2
-; CHECK-NEXT: add 3, 3, 2
-; CHECK-NEXT: add 12, 12, 2
+; CHECK-NEXT: add 26, 26, 30
+; CHECK-NEXT: add 20, 20, 30
+; CHECK-NEXT: add 11, 11, 30
+; CHECK-NEXT: add 19, 19, 30
+; CHECK-NEXT: add 31, 31, 30
+; CHECK-NEXT: add 12, 12, 30
; CHECK-NEXT: addi 27, 27, 1
; CHECK-NEXT: cmpld 28, 4
; CHECK-NEXT: ble 0, .LBB0_3
; CHECK-NEXT: # %bb.6: # %_loop_1_loopHeader_._return_bb_crit_edge.loopexit
-; CHECK-NEXT: ld 3, 48(1) # 8-byte Folded Reload
-; CHECK-NEXT: lxv 63, 384(1) # 16-byte Folded Reload
-; CHECK-NEXT: stxv 1, 0(3)
-; CHECK-NEXT: ld 3, 64(1) # 8-byte Folded Reload
-; CHECK-NEXT: lxv 62, 368(1) # 16-byte Folded Reload
-; CHECK-NEXT: lxv 61, 352(1) # 16-byte Folded Reload
-; CHECK-NEXT: lxv 60, 336(1) # 16-byte Folded Reload
-; CHECK-NEXT: lxv 59, 320(1) # 16-byte Folded Reload
-; CHECK-NEXT: lxv 58, 304(1) # 16-byte Folded Reload
-; CHECK-NEXT: lxv 57, 288(1) # 16-byte Folded Reload
-; CHECK-NEXT: lxv 56, 272(1) # 16-byte Folded Reload
-; CHECK-NEXT: lxv 55, 256(1) # 16-byte Folded Reload
-; CHECK-NEXT: lxv 54, 240(1) # 16-byte Folded Reload
-; CHECK-NEXT: lxv 53, 224(1) # 16-byte Folded Reload
-; CHECK-NEXT: lxv 52, 208(1) # 16-byte Folded Reload
-; CHECK-NEXT: stxv 0, 0(3)
-; CHECK-NEXT: ld 3, 72(1) # 8-byte Folded Reload
-; CHECK-NEXT: lfd 31, 584(1) # 8-byte Folded Reload
-; CHECK-NEXT: lfd 30, 576(1) # 8-byte Folded Reload
-; CHECK-NEXT: lfd 29, 568(1) # 8-byte Folded Reload
-; CHECK-NEXT: lfd 28, 560(1) # 8-byte Folded Reload
-; CHECK-NEXT: lfd 27, 552(1) # 8-byte Folded Reload
-; CHECK-NEXT: lfd 26, 544(1) # 8-byte Folded Reload
-; CHECK-NEXT: ld 31, 536(1) # 8-byte Folded Reload
-; CHECK-NEXT: ld 30, 528(1) # 8-byte Folded Reload
-; CHECK-NEXT: ld 29, 520(1) # 8-byte Folded Reload
-; CHECK-NEXT: ld 28, 512(1) # 8-byte Folded Reload
-; CHECK-NEXT: ld 27, 504(1) # 8-byte Folded Reload
-; CHECK-NEXT: stxv 5, 0(3)
-; CHECK-NEXT: ld 3, 32(1) # 8-byte Folded Reload
-; CHECK-NEXT: ld 26, 496(1) # 8-byte Folded Reload
-; CHECK-NEXT: ld 25, 488(1) # 8-byte Folded Reload
-; CHECK-NEXT: ld 24, 480(1) # 8-byte Folded Reload
-; CHECK-NEXT: ld 23, 472(1) # 8-byte Folded Reload
-; CHECK-NEXT: ld 22, 464(1) # 8-byte Folded Reload
-; CHECK-NEXT: ld 21, 456(1) # 8-byte Folded Reload
-; CHECK-NEXT: ld 20, 448(1) # 8-byte Folded Reload
-; CHECK-NEXT: ld 19, 440(1) # 8-byte Folded Reload
-; CHECK-NEXT: ld 18, 432(1) # 8-byte Folded Reload
-; CHECK-NEXT: ld 17, 424(1) # 8-byte Folded Reload
-; CHECK-NEXT: ld 16, 416(1) # 8-byte Folded Reload
-; CHECK-NEXT: stxv 4, 0(3)
; CHECK-NEXT: ld 3, 40(1) # 8-byte Folded Reload
-; CHECK-NEXT: ld 15, 408(1) # 8-byte Folded Reload
-; CHECK-NEXT: ld 14, 400(1) # 8-byte Folded Reload
+; CHECK-NEXT: stxv 7, 0(8)
+; CHECK-NEXT: stxv 6, 0(9)
+; CHECK-NEXT: lxv 63, 368(1) # 16-byte Folded Reload
+; CHECK-NEXT: lxv 62, 352(1) # 16-byte Folded Reload
+; CHECK-NEXT: stxv 5, 0(10)
+; CHECK-NEXT: stxv 4, 0(3)
+; CHECK-NEXT: ld 3, 48(1) # 8-byte Folded Reload
+; CHECK-NEXT: lxv 61, 336(1) # 16-byte Folded Reload
+; CHECK-NEXT: lxv 60, 320(1) # 16-byte Folded Reload
+; CHECK-NEXT: lxv 59, 304(1) # 16-byte Folded Reload
+; CHECK-NEXT: lxv 58, 288(1) # 16-byte Folded Reload
+; CHECK-NEXT: lxv 57, 272(1) # 16-byte Folded Reload
+; CHECK-NEXT: lxv 56, 256(1) # 16-byte Folded Reload
+; CHECK-NEXT: lxv 55, 240(1) # 16-byte Folded Reload
+; CHECK-NEXT: lxv 54, 224(1) # 16-byte Folded Reload
+; CHECK-NEXT: lxv 53, 208(1) # 16-byte Folded Reload
+; CHECK-NEXT: lxv 52, 192(1) # 16-byte Folded Reload
; CHECK-NEXT: stxv 3, 0(3)
; CHECK-NEXT: ld 3, 56(1) # 8-byte Folded Reload
-; CHECK-NEXT: stxv 2, 0(8)
-; CHECK-NEXT: stxv 40, 0(3)
+; CHECK-NEXT: lfd 31, 568(1) # 8-byte Folded Reload
+; CHECK-NEXT: lfd 30, 560(1) # 8-byte Folded Reload
+; CHECK-NEXT: lfd 29, 552(1) # 8-byte Folded Reload
+; CHECK-NEXT: lfd 28, 544(1) # 8-byte Folded Reload
+; CHECK-NEXT: lfd 27, 536(1) # 8-byte Folded Reload
+; CHECK-NEXT: lfd 26, 528(1) # 8-byte Folded Reload
+; CHECK-NEXT: ld 31, 520(1) # 8-byte Folded Reload
+; CHECK-NEXT: ld 30, 512(1) # 8-byte Folded Reload
+; CHECK-NEXT: ld 29, 504(1) # 8-byte Folded Reload
+; CHECK-NEXT: ld 28, 496(1) # 8-byte Folded Reload
+; CHECK-NEXT: ld 27, 488(1) # 8-byte Folded Reload
+; CHECK-NEXT: stxv 2, 0(3)
+; CHECK-NEXT: ld 3, 64(1) # 8-byte Folded Reload
+; CHECK-NEXT: ld 26, 480(1) # 8-byte Folded Reload
+; CHECK-NEXT: ld 25, 472(1) # 8-byte Folded Reload
+; CHECK-NEXT: ld 24, 464(1) # 8-byte Folded Reload
+; CHECK-NEXT: ld 23, 456(1) # 8-byte Folded Reload
+; CHECK-NEXT: ld 22, 448(1) # 8-byte Folded Reload
+; CHECK-NEXT: ld 21, 440(1) # 8-byte Folded Reload
+; CHECK-NEXT: ld 20, 432(1) # 8-byte Folded Reload
+; CHECK-NEXT: ld 19, 424(1) # 8-byte Folded Reload
+; CHECK-NEXT: ld 18, 416(1) # 8-byte Folded Reload
+; CHECK-NEXT: ld 17, 408(1) # 8-byte Folded Reload
+; CHECK-NEXT: ld 16, 400(1) # 8-byte Folded Reload
+; CHECK-NEXT: stxv 1, 0(3)
+; CHECK-NEXT: ld 3, 72(1) # 8-byte Folded Reload
+; CHECK-NEXT: ld 15, 392(1) # 8-byte Folded Reload
+; CHECK-NEXT: stxv 0, 0(3)
; CHECK-NEXT: ld 3, 80(1) # 8-byte Folded Reload
-; CHECK-NEXT: stxv 39, 0(10)
-; CHECK-NEXT: stxv 38, 0(3)
+; CHECK-NEXT: stxv 41, 0(3)
; CHECK-NEXT: ld 3, 88(1) # 8-byte Folded Reload
-; CHECK-NEXT: stxv 33, 0(3)
+; CHECK-NEXT: stxv 40, 0(3)
; CHECK-NEXT: ld 3, 96(1) # 8-byte Folded Reload
-; CHECK-NEXT: stxv 32, 0(3)
+; CHECK-NEXT: stxv 39, 0(3)
; CHECK-NEXT: ld 3, 104(1) # 8-byte Folded Reload
-; CHECK-NEXT: stxv 37, 0(3)
+; CHECK-NEXT: stxv 38, 0(3)
; CHECK-NEXT: ld 3, 112(1) # 8-byte Folded Reload
-; CHECK-NEXT: stxv 36, 0(3)
+; CHECK-NEXT: stxv 33, 0(3)
; CHECK-NEXT: ld 3, 120(1) # 8-byte Folded Reload
-; CHECK-NEXT: stxv 13, 0(3)
+; CHECK-NEXT: stxv 32, 0(3)
; CHECK-NEXT: ld 3, 128(1) # 8-byte Folded Reload
-; CHECK-NEXT: stxv 12, 0(3)
+; CHECK-NEXT: stxv 37, 0(3)
; CHECK-NEXT: ld 3, 136(1) # 8-byte Folded Reload
-; CHECK-NEXT: stxv 11, 0(3)
+; CHECK-NEXT: stxv 36, 0(3)
; CHECK-NEXT: ld 3, 144(1) # 8-byte Folded Reload
-; CHECK-NEXT: stxv 10, 0(3)
+; CHECK-NEXT: stxv 13, 0(3)
; CHECK-NEXT: ld 3, 152(1) # 8-byte Folded Reload
-; CHECK-NEXT: stxv 9, 0(3)
+; CHECK-NEXT: stxv 12, 0(3)
; CHECK-NEXT: ld 3, 160(1) # 8-byte Folded Reload
-; CHECK-NEXT: stxv 8, 0(3)
+; CHECK-NEXT: stxv 11, 0(3)
; CHECK-NEXT: ld 3, 168(1) # 8-byte Folded Reload
-; CHECK-NEXT: stxv 7, 0(3)
+; CHECK-NEXT: stxv 10, 0(3)
; CHECK-NEXT: ld 3, 176(1) # 8-byte Folded Reload
-; CHECK-NEXT: stxv 6, 0(3)
+; CHECK-NEXT: stxv 9, 0(3)
; CHECK-NEXT: ld 3, 184(1) # 8-byte Folded Reload
-; CHECK-NEXT: stxv 41, 0(3)
-; CHECK-NEXT: ld 3, 192(1) # 8-byte Folded Reload
-; CHECK-NEXT: stxv 43, 0(3)
-; CHECK-NEXT: ld 3, 200(1) # 8-byte Folded Reload
-; CHECK-NEXT: stxv 42, 0(3)
-; CHECK-NEXT: addi 1, 1, 592
+; CHECK-NEXT: stxv 8, 0(3)
+; CHECK-NEXT: stxv 42, 0(14)
+; CHECK-NEXT: ld 14, 384(1) # 8-byte Folded Reload
+; CHECK-NEXT: stxv 43, 0(7)
+; CHECK-NEXT: addi 1, 1, 576
; CHECK-NEXT: blr
entry:
%_val_l_ = load i32, ptr %.l, align 4
diff --git a/llvm/test/CodeGen/RISCV/rvv/pr165232.ll b/llvm/test/CodeGen/RISCV/rvv/pr165232.ll
index 3a12ea323c7cf..b66765d3c4e0e 100644
--- a/llvm/test/CodeGen/RISCV/rvv/pr165232.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/pr165232.ll
@@ -32,16 +32,16 @@ define i1 @main(ptr %var_117, ptr %arrayinit.element3045, ptr %arrayinit.element
; CHECK-NEXT: addi a0, a0, 16
; CHECK-NEXT: vs4r.v v8, (a0) # vscale x 32-byte Folded Spill
; CHECK-NEXT: vsetvli t0, zero, e8, m1, ta, ma
+; CHECK-NEXT: vmv.v.i v11, 0
+; CHECK-NEXT: # kill: def $v12 killed $v11 killed $vtype
; CHECK-NEXT: vmv.v.i v13, 0
-; CHECK-NEXT: # kill: def $v14 killed $v13 killed $vtype
+; CHECK-NEXT: vmv.v.i v14, 0
; CHECK-NEXT: vmv.v.i v15, 0
-; CHECK-NEXT: vmv.v.i v16, 0
-; CHECK-NEXT: vmv.v.i v17, 0
; CHECK-NEXT: vsetvli t0, zero, e8, m2, ta, ma
; CHECK-NEXT: vmv.v.i v20, 0
-; CHECK-NEXT: vmv1r.v v18, v13
+; CHECK-NEXT: vmv1r.v v16, v11
; CHECK-NEXT: vmv.v.i v22, 0
-; CHECK-NEXT: vmv1r.v v19, v13
+; CHECK-NEXT: vmv1r.v v17, v11
; CHECK-NEXT: csrr a0, vlenb
; CHECK-NEXT: slli t0, a0, 4
; CHECK-NEXT: add a0, t0, a0
@@ -68,8 +68,8 @@ define i1 @main(ptr %var_117, ptr %arrayinit.element3045, ptr %arrayinit.element
; CHECK-NEXT: add a0, sp, a0
; CHECK-NEXT: ld t4, 16(a0)
; CHECK-NEXT: vmv.v.i v24, 0
-; CHECK-NEXT: vmv1r.v v9, v13
-; CHECK-NEXT: vmv1r.v v10, v13
+; CHECK-NEXT: vmv1r.v v26, v11
+; CHECK-NEXT: vmv1r.v v27, v11
; CHECK-NEXT: vsetivli zero, 0, e32, m1, ta, ma
; CHECK-NEXT: vmv.v.i v8, 0
; CHECK-NEXT: csrr a0, vlenb
@@ -77,19 +77,17 @@ define i1 @main(ptr %var_117, ptr %arrayinit.element3045, ptr %arrayinit.element
; CHECK-NEXT: add a0, t5, a0
; CHECK-NEXT: add a0, sp, a0
; CHECK-NEXT: ld t5, 24(a0)
-; CHECK-NEXT: vmv1r.v v11, v13
-; CHECK-NEXT: vmv1r.v v12, v13
+; CHECK-NEXT: vmv1r.v v28, v11
+; CHECK-NEXT: vmv1r.v v29, v11
; CHECK-NEXT: csrr t6, vlenb
; CHECK-NEXT: add t6, sp, t6
; CHECK-NEXT: addi t6, t6, 16
-; CHECK-NEXT: vs1r.v v9, (t6) # vscale x 8-byte Folded Spill
+; CHECK-NEXT: vs2r.v v26, (t6) # vscale x 16-byte Folded Spill
; CHECK-NEXT: csrr a0, vlenb
-; CHECK-NEXT: add t6, t6, a0
-; CHECK-NEXT: vs2r.v v10, (t6) # vscale x 16-byte Folded Spill
; CHECK-NEXT: slli a0, a0, 1
; CHECK-NEXT: add t6, t6, a0
; CHECK-NEXT: ld a0, 8(sp) # 8-byte Folded Reload
-; CHECK-NEXT: vs1r.v v12, (t6) # vscale x 8-byte Folded Spill
+; CHECK-NEXT: vs2r.v v28, (t6) # vscale x 16-byte Folded Spill
; CHECK-NEXT: vmclr.m v9
; CHECK-NEXT: addi t6, sp, 16
; CHECK-NEXT: vs1r.v v9, (t6) # vscale x 8-byte Folded Spill
@@ -101,28 +99,28 @@ define i1 @main(ptr %var_117, ptr %arrayinit.element3045, ptr %arrayinit.element
; CHECK-NEXT: .LBB0_1: # %for.body
; CHECK-NEXT: # =>This Inner Loop Header: Depth=1
; CHECK-NEXT: vsetvli zero, zero, e64, m2, ta, ma
-; CHECK-NEXT: vmv.v.i v10, 0
+; CHECK-NEXT: vmv.v.i v18, 0
; CHECK-NEXT: vmv.v.i v30, 0
-; CHECK-NEXT: vmv1r.v v12, v8
; CHECK-NEXT: vsetvli zero, zero, e64, m2, tu, ma
; CHECK-NEXT: vle64.v v30, (a4)
+; CHECK-NEXT: vmv1r.v v0, v8
; CHECK-NEXT: vmv1r.v v28, v8
-; CHECK-NEXT: vle32.v v12, (t4)
-; CHECK-NEXT: vmv1r.v v29, v8
-; CHECK-NEXT: vle32.v v28, (t1)
-; CHECK-NEXT: vmv1r.v v9, v8
-; CHECK-NEXT: vle32.v v29, (t2)
+; CHECK-NEXT: vle32.v v0, (t1)
+; CHECK-NEXT: vmv1r.v v0, v8
+; CHECK-NEXT: vle32.v v28, (t2)
+; CHECK-NEXT: vmv1r.v v28, v8
+; CHECK-NEXT: vle32.v v0, (t3)
+; CHECK-NEXT: vmv1r.v v0, v8
+; CHECK-NEXT: vle32.v v28, (a6)
+; CHECK-NEXT: vmv1r.v v28, v8
+; CHECK-NEXT: vle32.v v0, (a5)
; CHECK-NEXT: vmv1r.v v10, v8
-; CHECK-NEXT: vle32.v v9, (t3)
+; CHECK-NEXT: vle32.v v28, (t4)
+; CHECK-NEXT: vmv1r.v v29, v8
+; CHECK-NEXT: vle32.v v10, (a7)
; CHECK-NEXT: vmv1r.v v9, v8
-; CHECK-NEXT: vle32.v v10, (a6)
-; CHECK-NEXT: vmv1r.v v10, v8
-; CHECK-NEXT: vle32.v v9, (a7)
-; CHECK-NEXT: vmv1r.v v11, v8
-; CHECK-NEXT: vle32.v v10, (a5)
-; CHECK-NEXT: vmv1r.v v10, v8
-; CHECK-NEXT: vle32.v v11, (a2)
-; CHECK-NEXT: vle32.v v10, (a3)
+; CHECK-NEXT: vle32.v v29, (a2)
+; CHECK-NEXT: vle32.v v9, (a3)
; CHECK-NEXT: csrr t5, vlenb
; CHECK-NEXT: add t5, sp, t5
; CHECK-NEXT: addi t5, t5, 16
@@ -135,15 +133,17 @@ define i1 @main(ptr %var_117, ptr %arrayinit.element3045, ptr %arrayinit.element
; CHECK-NEXT: vl1r.v v4, (t5) # vscale x 8-byte Folded Reload
; CHECK-NEXT: vsetvli zero, zero, e32, m1, ta, ma
; CHECK-NEXT: vsseg4e32.v v1, (zero)
-; CHECK-NEXT: vmv4r.v v0, v12
-; CHECK-NEXT: vmv4r.v v4, v16
-; CHECK-NEXT: vmv1r.v v2, v9
-; CHECK-NEXT: vmv1r.v v0, v12
+; CHECK-NEXT: vmv2r.v v0, v10
+; CHECK-NEXT: vmv2r.v v2, v12
+; CHECK-NEXT: vmv2r.v v4, v14
+; CHECK-NEXT: vmv2r.v v6, v16
+; CHECK-NEXT: vmv1r.v v2, v10
+; CHECK-NEXT: vmv1r.v v0, v28
; CHECK-NEXT: vsseg8e32.v v0, (a1)
; CHECK-NEXT: addi t5, sp, 16
-; CHECK-NEXT: vl1r.v v12, (t5) # vscale x 8-byte Folded Reload
-; CHECK-NEXT: vmv1r.v v9, v12
-; CHECK-NEXT: vmv1r.v v0, v12
+; CHECK-NEXT: vl1r.v v28, (t5) # vscale x 8-byte Folded Reload
+; CHECK-NEXT: vmv1r.v v10, v28
+; CHECK-NEXT: vmv1r.v v0, v28
; CHECK-NEXT: csrr t5, vlenb
; CHECK-NEXT: slli t6, t5, 2
; CHECK-NEXT: add t5, t6, t5
@@ -157,16 +157,15 @@ define i1 @main(ptr %var_117, ptr %arrayinit.element3045, ptr %arrayinit.element
; CHECK-NEXT: vsetvli zero, t0, e64, m2, ta, ma
; CHECK-NEXT: vsseg2e64.v v2, (zero)
; CHECK-NEXT: vsetivli zero, 0, e64, m2, ta, mu
-; CHECK-NEXT: vmv.v.i v28, 0
-; CHECK-NEXT: vmflt.vv v9, v30, v28, v0.t
-; CHECK-NEXT: vmv1r.v v0, v9
+; CHECK-NEXT: vmflt.vv v10, v30, v18, v0.t
+; CHECK-NEXT: vmv1r.v v0, v10
; CHECK-NEXT: vsetvli zero, zero, e32, m1, tu, mu
-; CHECK-NEXT: vssub.vv v10, v8, v11, v0.t
-; CHECK-NEXT: vmv1r.v v0, v12
+; CHECK-NEXT: vssub.vv v9, v8, v29, v0.t
+; CHECK-NEXT: vmv1r.v v0, v28
; CHECK-NEXT: vsetvli zero, zero, e64, m2, ta, ma
; CHECK-NEXT: vsseg4e64.v v20, (zero), v0.t
; CHECK-NEXT: vsetvli zero, zero, e32, m1, ta, ma
-; CHECK-NEXT: vsseg8e32.v v10, (a0)
+; CHECK-NEXT: vsseg8e32.v v9, (a0)
; CHECK-NEXT: csrr t5, vlenb
; CHECK-NEXT: slli t6, t5, 3
; CHECK-NEXT: add t5, t6, t5
diff --git a/llvm/test/CodeGen/Thumb2/LowOverheadLoops/spillingmove.ll b/llvm/test/CodeGen/Thumb2/LowOverheadLoops/spillingmove.ll
index fe06601e77bb3..a9ece9f157bf6 100644
--- a/llvm/test/CodeGen/Thumb2/LowOverheadLoops/spillingmove.ll
+++ b/llvm/test/CodeGen/Thumb2/LowOverheadLoops/spillingmove.ll
@@ -185,7 +185,7 @@ define void @__arm_2d_impl_rgb16_colour_filling_with_alpha_sched(ptr noalias noc
; CHECK: @ %bb.0: @ %entry
; CHECK-NEXT: ldrsh.w r12, [r2, #2]
; CHECK-NEXT: cmp.w r12, #1
-; CHECK-NEXT: blt .LBB1_7
+; CHECK-NEXT: blt.w .LBB1_7
; CHECK-NEXT: @ %bb.1: @ %for.cond3.preheader.lr.ph
; CHECK-NEXT: ldrsh.w r2, [r2]
; CHECK-NEXT: cmp r2, #1
@@ -195,30 +195,38 @@ define void @__arm_2d_impl_rgb16_colour_filling_with_alpha_sched(ptr noalias noc
; CHECK-NEXT: push {r4, r5, r6, r7, lr}
; CHECK-NEXT: sub sp, #4
; CHECK-NEXT: vpush {d8, d9, d10, d11, d12, d13, d14, d15}
-; CHECK-NEXT: ldr r7, [sp, #88]
+; CHECK-NEXT: sub sp, #64
+; CHECK-NEXT: ldr r7, [sp, #152]
; CHECK-NEXT: movs r5, #120
; CHECK-NEXT: lsls r6, r3, #3
; CHECK-NEXT: movs r4, #252
; CHECK-NEXT: and.w r5, r5, r3, lsr #9
; CHECK-NEXT: uxtb r6, r6
; CHECK-NEXT: and.w r3, r4, r3, lsr #3
-; CHECK-NEXT: adds r4, r2, #7
; CHECK-NEXT: muls r6, r7, r6
-; CHECK-NEXT: bic r4, r4, #7
; CHECK-NEXT: mul lr, r3, r7
; CHECK-NEXT: muls r5, r7, r5
+; CHECK-NEXT: vdup.16 q0, r6
+; CHECK-NEXT: adds r4, r2, #7
+; CHECK-NEXT: mov.w r6, #2016
+; CHECK-NEXT: vstrw.32 q0, [sp, #48] @ 16-byte Spill
+; CHECK-NEXT: vdup.16 q0, lr
+; CHECK-NEXT: bic r4, r4, #7
+; CHECK-NEXT: vstrw.32 q0, [sp, #32] @ 16-byte Spill
+; CHECK-NEXT: vdup.16 q0, r5
; CHECK-NEXT: rsb.w r3, r7, #256
; CHECK-NEXT: lsls r7, r1, #1
; CHECK-NEXT: sub.w r1, r4, #8
; CHECK-NEXT: movs r4, #1
+; CHECK-NEXT: vstrw.32 q0, [sp, #16] @ 16-byte Spill
+; CHECK-NEXT: vdup.16 q0, r6
; CHECK-NEXT: vmov.i16 q2, #0xf8
; CHECK-NEXT: add.w r1, r4, r1, lsr #3
-; CHECK-NEXT: vdup.16 q6, r6
-; CHECK-NEXT: mov.w r6, #2016
+; CHECK-NEXT: vstrw.32 q0, [sp] @ 16-byte Spill
; CHECK-NEXT: movs r4, #0
-; CHECK-NEXT: vdup.16 q3, lr
-; CHECK-NEXT: vdup.16 q5, r5
-; CHECK-NEXT: vdup.16 q7, r6
+; CHECK-NEXT: vldrw.u32 q5, [sp, #32] @ 16-byte Reload
+; CHECK-NEXT: vldrw.u32 q7, [sp, #16] @ 16-byte Reload
+; CHECK-NEXT: vldrw.u32 q3, [sp] @ 16-byte Reload
; CHECK-NEXT: .p2align 2
; CHECK-NEXT: .LBB1_3: @ %vector.ph
; CHECK-NEXT: @ =>This Loop Header: Depth=1
@@ -234,27 +242,27 @@ define void @__arm_2d_impl_rgb16_colour_filling_with_alpha_sched(ptr noalias noc
; CHECK-NEXT: vpst
; CHECK-NEXT: vldrht.u16 q0, [r5]
; CHECK-NEXT: vshl.i16 q1, q0, #3
-; CHECK-NEXT: subs r6, #8
+; CHECK-NEXT: vmov.i16 q6, #0xf800
; CHECK-NEXT: vand q1, q1, q2
-; CHECK-NEXT: vmov.i16 q2, #0x78
-; CHECK-NEXT: vshr.u16 q4, q0, #9
-; CHECK-NEXT: vand q4, q4, q2
-; CHECK-NEXT: vmov q2, q6
+; CHECK-NEXT: vldrw.u32 q2, [sp, #48] @ 16-byte Reload
; CHECK-NEXT: vmla.i16 q2, q1, r3
-; CHECK-NEXT: vshr.u16 q0, q0, #3
+; CHECK-NEXT: vshr.u16 q4, q0, #9
; CHECK-NEXT: vmov.i16 q1, #0xfc
+; CHECK-NEXT: vshr.u16 q0, q0, #3
+; CHECK-NEXT: subs r6, #8
; CHECK-NEXT: vand q0, q0, q1
-; CHECK-NEXT: vmov q1, q3
+; CHECK-NEXT: vmov q1, q5
; CHECK-NEXT: vmla.i16 q1, q0, r3
; CHECK-NEXT: vshr.u16 q0, q2, #11
-; CHECK-NEXT: vmov q2, q5
-; CHECK-NEXT: vmla.i16 q2, q4, r3
+; CHECK-NEXT: vmov.i16 q2, #0x78
; CHECK-NEXT: vshr.u16 q1, q1, #5
-; CHECK-NEXT: vmov.i16 q4, #0xf800
-; CHECK-NEXT: vand q1, q1, q7
-; CHECK-NEXT: vorr q0, q1, q0
-; CHECK-NEXT: vand q1, q2, q4
+; CHECK-NEXT: vand q2, q4, q2
+; CHECK-NEXT: vmov q4, q7
+; CHECK-NEXT: vmla.i16 q4, q2, r3
+; CHECK-NEXT: vand q1, q1, q3
; CHECK-NEXT: vmov.i16 q2, #0xf8
+; CHECK-NEXT: vorr q0, q1, q0
+; CHECK-NEXT: vand q1, q4, q6
; CHECK-NEXT: vorr q0, q0, q1
; CHECK-NEXT: vpst
; CHECK-NEXT: vstrht.16 q0, [r5], #16
@@ -266,6 +274,7 @@ define void @__arm_2d_impl_rgb16_colour_filling_with_alpha_sched(ptr noalias noc
; CHECK-NEXT: cmp r4, r12
; CHECK-NEXT: bne .LBB1_3
; CHECK-NEXT: @ %bb.6:
+; CHECK-NEXT: add sp, #64
; CHECK-NEXT: vpop {d8, d9, d10, d11, d12, d13, d14, d15}
; CHECK-NEXT: add sp, #4
; CHECK-NEXT: pop.w {r4, r5, r6, r7, lr}
diff --git a/llvm/test/CodeGen/X86/AMX/amx-greedy-ra-spill-shape.ll b/llvm/test/CodeGen/X86/AMX/amx-greedy-ra-spill-shape.ll
index 9a93b52f2d74c..ad1b6b3a2eaf5 100644
--- a/llvm/test/CodeGen/X86/AMX/amx-greedy-ra-spill-shape.ll
+++ b/llvm/test/CodeGen/X86/AMX/amx-greedy-ra-spill-shape.ll
@@ -47,6 +47,7 @@ define void @foo(i32 %M, i32 %N, i32 %K, ptr %A, ptr %B_rcr4, ptr %C, i32 %c_row
; CHECK-NEXT: MOV16mr %stack.0, 1, $noreg, 20, $noreg, [[LEA64_32r1]].sub_16bit :: (store (s512) into %stack.0 + 20, align 4)
; CHECK-NEXT: PLDTILECFGV %stack.0, 1, $noreg, 0, $noreg, implicit-def dead $tmm0, implicit-def dead $tmm1, implicit-def dead $tmm2, implicit-def dead $tmm3, implicit-def dead $tmm4, implicit-def dead $tmm5, implicit-def dead $tmm6, implicit-def dead $tmm7 :: (load (s512) from %stack.0, align 4)
; CHECK-NEXT: [[MOVSX64rr32_:%[0-9]+]]:gr64_nosp = MOVSX64rr32 [[COPY2]].sub_32bit
+ ; CHECK-NEXT: MOV32mr %stack.14, 1, $noreg, 0, $noreg, [[COPY4]] :: (store (s32) into %stack.14)
; CHECK-NEXT: [[COPY2:%[0-9]+]].sub_32bit:gr64_with_sub_8bit = SUB32rr [[COPY2]].sub_32bit, [[COPY4]], implicit-def dead $eflags
; CHECK-NEXT: undef [[MOVZX32rr16_1:%[0-9]+]].sub_32bit:gr64_with_sub_8bit = MOVZX32rr16 [[COPY2]].sub_16bit
; CHECK-NEXT: ADD64mr %stack.1, 1, $noreg, 0, $noreg, [[MOVZX32rr16_1]], implicit-def dead $eflags :: (store (s64) into %stack.1)
@@ -57,54 +58,57 @@ define void @foo(i32 %M, i32 %N, i32 %K, ptr %A, ptr %B_rcr4, ptr %C, i32 %c_row
; CHECK-NEXT: MOV64mr %stack.3, 1, $noreg, 0, $noreg, [[MOVSX64rr32_1]] :: (store (s64) into %stack.3)
; CHECK-NEXT: [[MOVSX64rr32_2:%[0-9]+]]:gr64_nosp = MOVSX64rr32 [[COPY3]].sub_32bit
; CHECK-NEXT: [[MOVSX64rm32_:%[0-9]+]]:gr64_nosp = MOVSX64rm32 %fixed-stack.2, 1, $noreg, 0, $noreg :: (load (s32) from %fixed-stack.2, align 8)
- ; CHECK-NEXT: [[MOVSX64rr32_3:%[0-9]+]]:gr64_nosp = MOVSX64rr32 [[MOV32rm2]].sub_32bit
- ; CHECK-NEXT: [[MOVSX64rm32_1:%[0-9]+]]:gr64 = MOVSX64rm32 %fixed-stack.1, 1, $noreg, 0, $noreg :: (load (s32) from %fixed-stack.1, align 16)
- ; CHECK-NEXT: [[MOVSX64rr32_4:%[0-9]+]]:gr64 = MOVSX64rr32 [[MOV32rm1]]
- ; CHECK-NEXT: [[MOVSX64rr32_5:%[0-9]+]]:gr64 = MOVSX64rr32 [[MOV32rm3]]
- ; CHECK-NEXT: [[MOVSX64rr32_6:%[0-9]+]]:gr64 = MOVSX64rr32 [[MOV32rm]]
- ; CHECK-NEXT: MOV64mr %stack.8, 1, $noreg, 0, $noreg, [[MOVSX64rr32_6]] :: (store (s64) into %stack.8)
- ; CHECK-NEXT: MOV64mr %stack.5, 1, $noreg, 0, $noreg, [[COPY1]] :: (store (s64) into %stack.5)
- ; CHECK-NEXT: MOV64mr %stack.6, 1, $noreg, 0, $noreg, [[MOVSX64rr32_4]] :: (store (s64) into %stack.6)
- ; CHECK-NEXT: [[COPY6:%[0-9]+]]:gr64_nosp = COPY [[MOVSX64rr32_4]]
+ ; CHECK-NEXT: [[MOVSX64rr32_3:%[0-9]+]]:gr64 = MOVSX64rr32 [[MOV32rm1]]
+ ; CHECK-NEXT: MOV64mr %stack.5, 1, $noreg, 0, $noreg, [[MOVSX64rr32_3]] :: (store (s64) into %stack.5)
+ ; CHECK-NEXT: [[COPY6:%[0-9]+]]:gr64_nosp = COPY [[MOVSX64rr32_3]]
; CHECK-NEXT: [[COPY6:%[0-9]+]]:gr64_nosp = IMUL64rr [[COPY6]], [[MOVSX64rr32_2]], implicit-def dead $eflags
; CHECK-NEXT: [[COPY6:%[0-9]+]]:gr64_nosp = ADD64rr [[COPY6]], [[MOVSX64rm32_]], implicit-def dead $eflags
; CHECK-NEXT: [[LEA64r:%[0-9]+]]:gr64 = LEA64r [[COPY]], 4, [[COPY6]], 0, $noreg
- ; CHECK-NEXT: MOV64mr %stack.9, 1, $noreg, 0, $noreg, [[LEA64r]] :: (store (s64) into %stack.9)
- ; CHECK-NEXT: [[COPY7:%[0-9]+]]:gr64_with_sub_8bit = lr-split COPY [[MOV32rm2]]
- ; CHECK-NEXT: MOV64mr %stack.7, 1, $noreg, 0, $noreg, [[MOVSX64rr32_5]] :: (store (s64) into %stack.7)
- ; CHECK-NEXT: [[COPY8:%[0-9]+]]:gr64 = COPY [[MOVSX64rr32_5]]
- ; CHECK-NEXT: [[COPY8:%[0-9]+]]:gr64 = IMUL64rr [[COPY8]], [[MOVSX64rr32_2]], implicit-def dead $eflags
- ; CHECK-NEXT: [[COPY8:%[0-9]+]]:gr64 = SHL64ri [[COPY8]], 2, implicit-def dead $eflags
- ; CHECK-NEXT: MOV64mr %stack.10, 1, $noreg, 0, $noreg, [[COPY8]] :: (store (s64) into %stack.10)
- ; CHECK-NEXT: [[LEA64r1:%[0-9]+]]:gr64 = LEA64r $noreg, 4, [[MOVSX64rr32_3]], 0, $noreg
- ; CHECK-NEXT: MOV64mr %stack.11, 1, $noreg, 0, $noreg, [[LEA64r1]] :: (store (s64) into %stack.11)
+ ; CHECK-NEXT: MOV64mr %stack.6, 1, $noreg, 0, $noreg, [[LEA64r]] :: (store (s64) into %stack.6)
+ ; CHECK-NEXT: [[MOVSX64rr32_4:%[0-9]+]]:gr64 = MOVSX64rr32 [[MOV32rm3]]
+ ; CHECK-NEXT: [[MOVSX64rr32_5:%[0-9]+]]:gr64 = MOVSX64rr32 [[MOV32rm]]
+ ; CHECK-NEXT: MOV64mr %stack.8, 1, $noreg, 0, $noreg, [[MOVSX64rr32_5]] :: (store (s64) into %stack.8)
+ ; CHECK-NEXT: MOV64mr %stack.7, 1, $noreg, 0, $noreg, [[MOVSX64rr32_4]] :: (store (s64) into %stack.7)
+ ; CHECK-NEXT: [[COPY7:%[0-9]+]]:gr64 = COPY [[MOVSX64rr32_4]]
+ ; CHECK-NEXT: [[COPY7:%[0-9]+]]:gr64 = IMUL64rr [[COPY7]], [[MOVSX64rr32_2]], implicit-def dead $eflags
+ ; CHECK-NEXT: [[COPY7:%[0-9]+]]:gr64 = SHL64ri [[COPY7]], 2, implicit-def dead $eflags
+ ; CHECK-NEXT: MOV64mr %stack.9, 1, $noreg, 0, $noreg, [[COPY7]] :: (store (s64) into %stack.9)
; CHECK-NEXT: MOV64mr %stack.4, 1, $noreg, 0, $noreg, [[MOVSX64rm32_]] :: (store (s64) into %stack.4)
; CHECK-NEXT: [[LEA64_32r3:%[0-9]+]]:gr32 = LEA64_32r [[COPY5]], 4, [[MOVSX64rm32_]], 0, $noreg
- ; CHECK-NEXT: MOV32mr %stack.12, 1, $noreg, 0, $noreg, [[LEA64_32r3]] :: (store (s32) into %stack.12)
+ ; CHECK-NEXT: MOV32mr %stack.10, 1, $noreg, 0, $noreg, [[LEA64_32r3]] :: (store (s32) into %stack.10)
+ ; CHECK-NEXT: [[MOVSX64rr32_6:%[0-9]+]]:gr64_nosp = MOVSX64rr32 [[MOV32rm2]].sub_32bit
+ ; CHECK-NEXT: [[MOVSX64rm32_1:%[0-9]+]]:gr64 = MOVSX64rm32 %fixed-stack.1, 1, $noreg, 0, $noreg :: (load (s32) from %fixed-stack.1, align 16)
+ ; CHECK-NEXT: MOV64mr %stack.12, 1, $noreg, 0, $noreg, [[MOVSX64rm32_1]] :: (store (s64) into %stack.12)
+ ; CHECK-NEXT: MOV64mr %stack.11, 1, $noreg, 0, $noreg, [[MOVSX64rr32_6]] :: (store (s64) into %stack.11)
+ ; CHECK-NEXT: [[LEA64r1:%[0-9]+]]:gr64 = LEA64r $noreg, 4, [[MOVSX64rr32_6]], 0, $noreg
+ ; CHECK-NEXT: MOV64mr %stack.13, 1, $noreg, 0, $noreg, [[LEA64r1]] :: (store (s64) into %stack.13)
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: bb.2.for.cond14.preheader:
; CHECK-NEXT: successors: %bb.3(0x40000000), %bb.5(0x40000000)
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: [[MOV32rm4:%[0-9]+]]:gr32 = MOV32rm %fixed-stack.2, 1, $noreg, 0, $noreg :: (load (s32) from %fixed-stack.2, align 8)
; CHECK-NEXT: CMP32rm [[MOV32rm4]], %fixed-stack.1, 1, $noreg, 0, $noreg, implicit-def $eflags :: (load (s32) from %fixed-stack.1, align 16)
- ; CHECK-NEXT: [[MOV32rm5:%[0-9]+]]:gr32 = MOV32rm %stack.2, 1, $noreg, 0, $noreg :: (load (s32) from %stack.2)
- ; CHECK-NEXT: [[MOV64rm:%[0-9]+]]:gr64_nosp = MOV64rm %stack.3, 1, $noreg, 0, $noreg :: (load (s64) from %stack.3)
+ ; CHECK-NEXT: [[MOV64rm:%[0-9]+]]:gr64 = MOV64rm %stack.3, 1, $noreg, 0, $noreg :: (load (s64) from %stack.3)
; CHECK-NEXT: [[MOV64rm1:%[0-9]+]]:gr64 = MOV64rm %stack.11, 1, $noreg, 0, $noreg :: (load (s64) from %stack.11)
- ; CHECK-NEXT: [[MOV32rm6:%[0-9]+]]:gr32 = MOV32rm %fixed-stack.3, 1, $noreg, 0, $noreg :: (load (s32) from %fixed-stack.3, align 16)
- ; CHECK-NEXT: [[COPY9:%[0-9]+]]:gr32 = lr-split COPY [[MOV32rm6]]
+ ; CHECK-NEXT: [[MOV64rm2:%[0-9]+]]:gr64 = MOV64rm %stack.12, 1, $noreg, 0, $noreg :: (load (s64) from %stack.12)
+ ; CHECK-NEXT: [[MOV32rm5:%[0-9]+]]:gr32 = MOV32rm %fixed-stack.3, 1, $noreg, 0, $noreg :: (load (s32) from %fixed-stack.3, align 16)
+ ; CHECK-NEXT: [[COPY8:%[0-9]+]]:gr32 = lr-split COPY [[MOV32rm5]]
+ ; CHECK-NEXT: [[MOV64rm3:%[0-9]+]]:gr64 = MOV64rm %stack.13, 1, $noreg, 0, $noreg :: (load (s64) from %stack.13)
+ ; CHECK-NEXT: [[MOV32rm6:%[0-9]+]]:gr32 = MOV32rm %stack.14, 1, $noreg, 0, $noreg :: (load (s32) from %stack.14)
; CHECK-NEXT: JCC_1 %bb.5, 13, implicit $eflags
; CHECK-NEXT: JMP_1 %bb.3
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: bb.3.for.body17.lr.ph:
; CHECK-NEXT: successors: %bb.6(0x80000000)
; CHECK-NEXT: {{ $}}
- ; CHECK-NEXT: [[MOV64rm2:%[0-9]+]]:gr64 = MOV64rm %stack.6, 1, $noreg, 0, $noreg :: (load (s64) from %stack.6)
- ; CHECK-NEXT: [[MOV64rm2:%[0-9]+]]:gr64 = nsw IMUL64rr [[MOV64rm2]], [[MOVSX64rr32_]], implicit-def dead $eflags
- ; CHECK-NEXT: [[MOV64rm2:%[0-9]+]]:gr64 = ADD64rm [[MOV64rm2]], %stack.1, 1, $noreg, 0, $noreg, implicit-def dead $eflags :: (load (s64) from %stack.1)
- ; CHECK-NEXT: [[MOV32rm7:%[0-9]+]]:gr32 = MOV32rm %stack.12, 1, $noreg, 0, $noreg :: (load (s32) from %stack.12)
- ; CHECK-NEXT: undef [[COPY10:%[0-9]+]].sub_32bit:gr64_nosp = COPY [[MOV32rm7]]
- ; CHECK-NEXT: [[MOV64rm3:%[0-9]+]]:gr64 = MOV64rm %stack.9, 1, $noreg, 0, $noreg :: (load (s64) from %stack.9)
- ; CHECK-NEXT: [[MOV64rm4:%[0-9]+]]:gr64 = MOV64rm %stack.4, 1, $noreg, 0, $noreg :: (load (s64) from %stack.4)
+ ; CHECK-NEXT: [[MOV64rm4:%[0-9]+]]:gr64 = MOV64rm %stack.5, 1, $noreg, 0, $noreg :: (load (s64) from %stack.5)
+ ; CHECK-NEXT: [[MOV64rm4:%[0-9]+]]:gr64 = nsw IMUL64rr [[MOV64rm4]], [[MOVSX64rr32_]], implicit-def dead $eflags
+ ; CHECK-NEXT: [[MOV64rm4:%[0-9]+]]:gr64 = ADD64rm [[MOV64rm4]], %stack.1, 1, $noreg, 0, $noreg, implicit-def dead $eflags :: (load (s64) from %stack.1)
+ ; CHECK-NEXT: MOV64mr %stack.15, 1, $noreg, 0, $noreg, [[MOV64rm4]] :: (store (s64) into %stack.15)
+ ; CHECK-NEXT: [[MOV32rm7:%[0-9]+]]:gr32 = MOV32rm %stack.10, 1, $noreg, 0, $noreg :: (load (s32) from %stack.10)
+ ; CHECK-NEXT: undef [[COPY9:%[0-9]+]].sub_32bit:gr64_nosp = COPY [[MOV32rm7]]
+ ; CHECK-NEXT: [[MOV64rm5:%[0-9]+]]:gr64 = MOV64rm %stack.6, 1, $noreg, 0, $noreg :: (load (s64) from %stack.6)
+ ; CHECK-NEXT: [[MOV64rm6:%[0-9]+]]:gr64 = MOV64rm %stack.4, 1, $noreg, 0, $noreg :: (load (s64) from %stack.4)
; CHECK-NEXT: JMP_1 %bb.6
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: bb.4.for.cond.cleanup:
@@ -113,42 +117,49 @@ define void @foo(i32 %M, i32 %N, i32 %K, ptr %A, ptr %B_rcr4, ptr %C, i32 %c_row
; CHECK-NEXT: bb.5.for.cond.cleanup16:
; CHECK-NEXT: successors: %bb.2(0x7c000000), %bb.4(0x04000000)
; CHECK-NEXT: {{ $}}
- ; CHECK-NEXT: [[MOV64rm5:%[0-9]+]]:gr64 = MOV64rm %stack.6, 1, $noreg, 0, $noreg :: (load (s64) from %stack.6)
- ; CHECK-NEXT: [[MOV64rm5:%[0-9]+]]:gr64 = ADD64rm [[MOV64rm5]], %stack.7, 1, $noreg, 0, $noreg, implicit-def dead $eflags :: (load (s64) from %stack.7)
- ; CHECK-NEXT: [[MOV64rm6:%[0-9]+]]:gr64 = MOV64rm %stack.10, 1, $noreg, 0, $noreg :: (load (s64) from %stack.10)
- ; CHECK-NEXT: ADD64mr %stack.9, 1, $noreg, 0, $noreg, [[MOV64rm6]], implicit-def dead $eflags :: (store (s64) into %stack.9)
- ; CHECK-NEXT: MOV64mr %stack.6, 1, $noreg, 0, $noreg, [[MOV64rm5]] :: (store (s64) into %stack.6)
- ; CHECK-NEXT: CMP64rm [[MOV64rm5]], %stack.8, 1, $noreg, 0, $noreg, implicit-def $eflags :: (load (s64) from %stack.8)
+ ; CHECK-NEXT: [[MOV64rm7:%[0-9]+]]:gr64 = MOV64rm %stack.5, 1, $noreg, 0, $noreg :: (load (s64) from %stack.5)
+ ; CHECK-NEXT: [[MOV64rm7:%[0-9]+]]:gr64 = ADD64rm [[MOV64rm7]], %stack.7, 1, $noreg, 0, $noreg, implicit-def dead $eflags :: (load (s64) from %stack.7)
+ ; CHECK-NEXT: [[MOV64rm8:%[0-9]+]]:gr64 = MOV64rm %stack.9, 1, $noreg, 0, $noreg :: (load (s64) from %stack.9)
+ ; CHECK-NEXT: ADD64mr %stack.6, 1, $noreg, 0, $noreg, [[MOV64rm8]], implicit-def dead $eflags :: (store (s64) into %stack.6)
+ ; CHECK-NEXT: MOV64mr %stack.5, 1, $noreg, 0, $noreg, [[MOV64rm7]] :: (store (s64) into %stack.5)
+ ; CHECK-NEXT: CMP64rm [[MOV64rm7]], %stack.8, 1, $noreg, 0, $noreg, implicit-def $eflags :: (load (s64) from %stack.8)
; CHECK-NEXT: JCC_1 %bb.2, 12, implicit $eflags
; CHECK-NEXT: JMP_1 %bb.4
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: bb.6.for.body17:
; CHECK-NEXT: successors: %bb.6(0x7c000000), %bb.5(0x04000000)
; CHECK-NEXT: {{ $}}
- ; CHECK-NEXT: [[PTILEZEROV:%[0-9]+]]:tile = PTILEZEROV [[COPY9]].sub_16bit, [[COPY7]].sub_16bit
- ; CHECK-NEXT: [[PTILELOADDV:%[0-9]+]]:tile = PTILELOADDV [[COPY9]].sub_16bit, [[COPY4]].sub_16bit, [[MOV64rm2]], 1, [[MOVSX64rr32_]], 0, $noreg
- ; CHECK-NEXT: [[COPY10:%[0-9]+]]:gr64_nosp = MOVSX64rr32 [[COPY10]].sub_32bit
- ; CHECK-NEXT: [[COPY11:%[0-9]+]]:gr64 = lr-split COPY [[MOVSX64rm32_1]]
- ; CHECK-NEXT: [[COPY12:%[0-9]+]]:gr64 = lr-split COPY [[MOVSX64rr32_3]]
- ; CHECK-NEXT: [[COPY13:%[0-9]+]]:gr64 = lr-split COPY [[MOVSX64rr32_2]]
- ; CHECK-NEXT: [[COPY14:%[0-9]+]]:gr64 = lr-split COPY [[MOVSX64rr32_]]
- ; CHECK-NEXT: [[COPY15:%[0-9]+]]:gr64 = lr-split COPY [[COPY7]]
- ; CHECK-NEXT: [[MOV64rm7:%[0-9]+]]:gr64 = MOV64rm %stack.5, 1, $noreg, 0, $noreg :: (load (s64) from %stack.5)
- ; CHECK-NEXT: [[LEA64r2:%[0-9]+]]:gr64 = LEA64r [[MOV64rm7]], 1, [[COPY10]], 0, $noreg
- ; CHECK-NEXT: [[PTILELOADDV1:%[0-9]+]]:tile = PTILELOADDV [[MOV32rm5]].sub_16bit, [[LEA64_32r1]].sub_16bit, [[LEA64r2]], 1, [[MOV64rm]], 0, $noreg
- ; CHECK-NEXT: [[COPY7:%[0-9]+]]:gr64_with_sub_8bit = lr-split COPY [[COPY15]]
- ; CHECK-NEXT: [[MOVSX64rr32_:%[0-9]+]]:gr64_nosp = lr-split COPY [[COPY14]]
- ; CHECK-NEXT: [[MOVSX64rr32_2:%[0-9]+]]:gr64_nosp = lr-split COPY [[COPY13]]
- ; CHECK-NEXT: [[MOVSX64rr32_3:%[0-9]+]]:gr64_nosp = lr-split COPY [[COPY12]]
- ; CHECK-NEXT: [[MOVSX64rm32_1:%[0-9]+]]:gr64 = lr-split COPY [[COPY11]]
- ; CHECK-NEXT: [[MOV32rm8:%[0-9]+]]:gr32 = MOV32rm %fixed-stack.3, 1, $noreg, 0, $noreg :: (load (s32) from %fixed-stack.3, align 16)
- ; CHECK-NEXT: [[COPY9:%[0-9]+]]:gr32 = lr-split COPY [[MOV32rm8]]
- ; CHECK-NEXT: [[PTILEZEROV:%[0-9]+]]:tile = PTDPBSSDV [[COPY9]].sub_16bit, [[LEA64_32r1]].sub_16bit, [[COPY4]].sub_16bit, [[PTILEZEROV]], [[PTILELOADDV]], [[PTILELOADDV1]]
- ; CHECK-NEXT: PTILESTOREDV [[COPY9]].sub_16bit, [[COPY7]].sub_16bit, [[MOV64rm3]], 1, [[MOVSX64rr32_2]], 0, $noreg, [[PTILEZEROV]]
- ; CHECK-NEXT: [[MOV64rm4:%[0-9]+]]:gr64 = ADD64rr [[MOV64rm4]], [[MOVSX64rr32_3]], implicit-def dead $eflags
- ; CHECK-NEXT: [[MOV64rm3:%[0-9]+]]:gr64 = ADD64rr [[MOV64rm3]], [[MOV64rm1]], implicit-def dead $eflags
- ; CHECK-NEXT: [[COPY10:%[0-9]+]].sub_32bit:gr64_nosp = ADD32rr [[COPY10]].sub_32bit, [[LEA64_32r1]], implicit-def dead $eflags
- ; CHECK-NEXT: CMP64rr [[MOV64rm4]], [[MOVSX64rm32_1]], implicit-def $eflags
+ ; CHECK-NEXT: [[PTILEZEROV:%[0-9]+]]:tile = PTILEZEROV [[COPY8]].sub_16bit, [[MOV32rm2]].sub_16bit
+ ; CHECK-NEXT: [[MOV64rm9:%[0-9]+]]:gr64 = MOV64rm %stack.15, 1, $noreg, 0, $noreg :: (load (s64) from %stack.15)
+ ; CHECK-NEXT: [[PTILELOADDV:%[0-9]+]]:tile = PTILELOADDV [[COPY8]].sub_16bit, [[MOV32rm6]].sub_16bit, [[MOV64rm9]], 1, [[MOVSX64rr32_]], 0, $noreg
+ ; CHECK-NEXT: [[COPY9:%[0-9]+]]:gr64_nosp = MOVSX64rr32 [[COPY9]].sub_32bit
+ ; CHECK-NEXT: [[COPY10:%[0-9]+]]:gr64 = lr-split COPY [[MOVSX64rr32_2]]
+ ; CHECK-NEXT: [[COPY11:%[0-9]+]]:gr64 = lr-split COPY [[MOVSX64rr32_]]
+ ; CHECK-NEXT: [[COPY12:%[0-9]+]]:gr64 = lr-split COPY [[MOV32rm2]]
+ ; CHECK-NEXT: [[COPY13:%[0-9]+]]:gr64 = lr-split COPY [[COPY1]]
+ ; CHECK-NEXT: [[LEA64r2:%[0-9]+]]:gr64 = LEA64r [[COPY13]], 1, [[COPY9]], 0, $noreg
+ ; CHECK-NEXT: [[COPY14:%[0-9]+]]:gr64 = lr-split COPY [[MOV64rm3]]
+ ; CHECK-NEXT: [[COPY15:%[0-9]+]]:gr64 = lr-split COPY [[MOV64rm2]]
+ ; CHECK-NEXT: [[COPY16:%[0-9]+]]:gr64 = lr-split COPY [[MOV64rm1]]
+ ; CHECK-NEXT: [[COPY17:%[0-9]+]]:gr64_nosp = lr-split COPY [[MOV64rm]]
+ ; CHECK-NEXT: [[MOV32rm8:%[0-9]+]]:gr32 = MOV32rm %stack.2, 1, $noreg, 0, $noreg :: (load (s32) from %stack.2)
+ ; CHECK-NEXT: [[PTILELOADDV1:%[0-9]+]]:tile = PTILELOADDV [[MOV32rm8]].sub_16bit, [[LEA64_32r1]].sub_16bit, [[LEA64r2]], 1, [[COPY17]], 0, $noreg
+ ; CHECK-NEXT: [[MOV64rm:%[0-9]+]]:gr64 = lr-split COPY [[COPY17]]
+ ; CHECK-NEXT: [[MOV64rm1:%[0-9]+]]:gr64 = lr-split COPY [[COPY16]]
+ ; CHECK-NEXT: [[MOV64rm2:%[0-9]+]]:gr64 = lr-split COPY [[COPY15]]
+ ; CHECK-NEXT: [[MOV64rm3:%[0-9]+]]:gr64 = lr-split COPY [[COPY14]]
+ ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gr64 = lr-split COPY [[COPY13]]
+ ; CHECK-NEXT: [[MOV32rm2:%[0-9]+]]:gr64_nosp = lr-split COPY [[COPY12]]
+ ; CHECK-NEXT: [[MOVSX64rr32_:%[0-9]+]]:gr64_nosp = lr-split COPY [[COPY11]]
+ ; CHECK-NEXT: [[MOVSX64rr32_2:%[0-9]+]]:gr64_nosp = lr-split COPY [[COPY10]]
+ ; CHECK-NEXT: [[MOV32rm9:%[0-9]+]]:gr32 = MOV32rm %fixed-stack.3, 1, $noreg, 0, $noreg :: (load (s32) from %fixed-stack.3, align 16)
+ ; CHECK-NEXT: [[COPY8:%[0-9]+]]:gr32 = lr-split COPY [[MOV32rm9]]
+ ; CHECK-NEXT: [[PTILEZEROV:%[0-9]+]]:tile = PTDPBSSDV [[COPY8]].sub_16bit, [[LEA64_32r1]].sub_16bit, [[MOV32rm6]].sub_16bit, [[PTILEZEROV]], [[PTILELOADDV]], [[PTILELOADDV1]]
+ ; CHECK-NEXT: PTILESTOREDV [[COPY8]].sub_16bit, [[MOV32rm2]].sub_16bit, [[MOV64rm5]], 1, [[MOVSX64rr32_2]], 0, $noreg, [[PTILEZEROV]]
+ ; CHECK-NEXT: [[MOV64rm6:%[0-9]+]]:gr64 = ADD64rr [[MOV64rm6]], [[MOV64rm1]], implicit-def dead $eflags
+ ; CHECK-NEXT: [[MOV64rm5:%[0-9]+]]:gr64 = ADD64rr [[MOV64rm5]], [[MOV64rm3]], implicit-def dead $eflags
+ ; CHECK-NEXT: [[COPY9:%[0-9]+]].sub_32bit:gr64_nosp = ADD32rr [[COPY9]].sub_32bit, [[LEA64_32r1]], implicit-def dead $eflags
+ ; CHECK-NEXT: CMP64rr [[MOV64rm6]], [[MOV64rm2]], implicit-def $eflags
; CHECK-NEXT: JCC_1 %bb.6, 12, implicit $eflags
; CHECK-NEXT: JMP_1 %bb.5
entry:
diff --git a/llvm/test/CodeGen/X86/div-rem-pair-recomposition-signed.ll b/llvm/test/CodeGen/X86/div-rem-pair-recomposition-signed.ll
index 5cfdbb7af38ad..a7fdac0dbcc6e 100644
--- a/llvm/test/CodeGen/X86/div-rem-pair-recomposition-signed.ll
+++ b/llvm/test/CodeGen/X86/div-rem-pair-recomposition-signed.ll
@@ -276,12 +276,14 @@ define i128 @scalar_i128(i128 %x, i128 %y, ptr %divdst) nounwind {
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
; X86-NEXT: cmovnel %ebx, %eax
; X86-NEXT: movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NEXT: movl $0, %edi
-; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %ebx # 4-byte Reload
-; X86-NEXT: cmovnel %edi, %ebx
+; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %edi # 4-byte Reload
+; X86-NEXT: movl $0, %eax
+; X86-NEXT: movl %edi, %ebx
+; X86-NEXT: cmovnel %eax, %ebx
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %esi # 4-byte Reload
-; X86-NEXT: cmovnel %edi, %esi
-; X86-NEXT: cmovel {{[-0-9]+}}(%e{{[sb]}}p), %edi # 4-byte Folded Reload
+; X86-NEXT: cmovnel %eax, %esi
+; X86-NEXT: cmovel {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Folded Reload
+; X86-NEXT: movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
; X86-NEXT: jne .LBB4_4
; X86-NEXT: # %bb.10: # %select.end
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
@@ -294,8 +296,7 @@ define i128 @scalar_i128(i128 %x, i128 %y, ptr %divdst) nounwind {
; X86-NEXT: je .LBB4_11
; X86-NEXT: # %bb.8: # %udiv-bb1
; X86-NEXT: movl %edx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
-; X86-NEXT: movl %eax, {{[0-9]+}}(%esp)
+; X86-NEXT: movl %edi, {{[0-9]+}}(%esp)
; X86-NEXT: xorps %xmm0, %xmm0
; X86-NEXT: movaps %xmm0, {{[0-9]+}}(%esp)
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
@@ -331,7 +332,6 @@ define i128 @scalar_i128(i128 %x, i128 %y, ptr %divdst) nounwind {
; X86-NEXT: jb .LBB4_9
; X86-NEXT: # %bb.5: # %udiv-preheader
; X86-NEXT: movl %ebx, %ecx
-; X86-NEXT: movaps %xmm0, {{[0-9]+}}(%esp)
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
; X86-NEXT: movl %eax, {{[0-9]+}}(%esp)
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
@@ -339,6 +339,7 @@ define i128 @scalar_i128(i128 %x, i128 %y, ptr %divdst) nounwind {
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
; X86-NEXT: movl %eax, {{[0-9]+}}(%esp)
; X86-NEXT: movl %edi, {{[0-9]+}}(%esp)
+; X86-NEXT: movaps %xmm0, {{[0-9]+}}(%esp)
; X86-NEXT: movl %ecx, %eax
; X86-NEXT: shrb $3, %al
; X86-NEXT: andb $12, %al
@@ -448,16 +449,15 @@ define i128 @scalar_i128(i128 %x, i128 %y, ptr %divdst) nounwind {
; X86-NEXT: jne .LBB4_6
; X86-NEXT: .LBB4_7: # %udiv-loop-exit
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %esi # 4-byte Reload
-; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
-; X86-NEXT: shldl $1, %esi, %eax
+; X86-NEXT: shldl $1, %esi, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Folded Spill
; X86-NEXT: shldl $1, %ecx, %esi
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %edx # 4-byte Reload
; X86-NEXT: shldl $1, %edx, %ecx
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %edi # 4-byte Reload
; X86-NEXT: leal (%edi,%edx,2), %ebx
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %edx # 4-byte Reload
-; X86-NEXT: movl %eax, %edi
; X86-NEXT: .LBB4_11: # %udiv-end
+; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %edi # 4-byte Reload
; X86-NEXT: xorl %edx, %edi
; X86-NEXT: xorl %edx, %esi
; X86-NEXT: xorl %edx, %ecx
diff --git a/llvm/test/CodeGen/X86/i128-mul.ll b/llvm/test/CodeGen/X86/i128-mul.ll
index cffd88c55bb0a..dd94f2b4f29e0 100644
--- a/llvm/test/CodeGen/X86/i128-mul.ll
+++ b/llvm/test/CodeGen/X86/i128-mul.ll
@@ -113,60 +113,59 @@ define i64 @mul1(i64 %n, ptr nocapture %z, ptr nocapture %x, i64 %y) nounwind {
; X86-NOBMI-NEXT: # %bb.1: # %for.body.preheader
; X86-NOBMI-NEXT: xorl %eax, %eax
; X86-NOBMI-NEXT: xorl %edx, %edx
-; X86-NOBMI-NEXT: xorl %ecx, %ecx
+; X86-NOBMI-NEXT: xorl %edi, %edi
; X86-NOBMI-NEXT: movl $0, (%esp) # 4-byte Folded Spill
; X86-NOBMI-NEXT: .p2align 4
; X86-NOBMI-NEXT: .LBB1_2: # %for.body
; X86-NOBMI-NEXT: # =>This Inner Loop Header: Depth=1
; X86-NOBMI-NEXT: movl %edx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
; X86-NOBMI-NEXT: movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NOBMI-NEXT: movl {{[0-9]+}}(%esp), %eax
-; X86-NOBMI-NEXT: movl (%eax,%ecx,8), %edi
-; X86-NOBMI-NEXT: movl 4(%eax,%ecx,8), %ebx
-; X86-NOBMI-NEXT: movl %ebx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NOBMI-NEXT: movl %edi, %eax
-; X86-NOBMI-NEXT: movl {{[0-9]+}}(%esp), %esi
-; X86-NOBMI-NEXT: mull %esi
+; X86-NOBMI-NEXT: movl {{[0-9]+}}(%esp), %ebx
+; X86-NOBMI-NEXT: movl (%ebx,%edi,8), %esi
+; X86-NOBMI-NEXT: movl %esi, %eax
+; X86-NOBMI-NEXT: movl {{[0-9]+}}(%esp), %ecx
+; X86-NOBMI-NEXT: mull %ecx
+; X86-NOBMI-NEXT: movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
; X86-NOBMI-NEXT: movl %edx, %ebp
+; X86-NOBMI-NEXT: movl 4(%ebx,%edi,8), %eax
; X86-NOBMI-NEXT: movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NOBMI-NEXT: movl %ebx, %eax
-; X86-NOBMI-NEXT: mull %esi
+; X86-NOBMI-NEXT: mull %ecx
; X86-NOBMI-NEXT: movl %edx, %ebx
-; X86-NOBMI-NEXT: movl %eax, %esi
-; X86-NOBMI-NEXT: addl %ebp, %esi
+; X86-NOBMI-NEXT: movl %eax, %ecx
+; X86-NOBMI-NEXT: addl %ebp, %ecx
; X86-NOBMI-NEXT: adcl $0, %ebx
-; X86-NOBMI-NEXT: movl %edi, %eax
+; X86-NOBMI-NEXT: movl %esi, %eax
; X86-NOBMI-NEXT: movl {{[0-9]+}}(%esp), %edx
; X86-NOBMI-NEXT: mull %edx
; X86-NOBMI-NEXT: movl %edx, %ebp
-; X86-NOBMI-NEXT: movl %eax, %edi
-; X86-NOBMI-NEXT: addl %esi, %edi
+; X86-NOBMI-NEXT: movl %eax, %esi
+; X86-NOBMI-NEXT: addl %ecx, %esi
; X86-NOBMI-NEXT: adcl %ebx, %ebp
-; X86-NOBMI-NEXT: setb %bl
+; X86-NOBMI-NEXT: setb %cl
; X86-NOBMI-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
; X86-NOBMI-NEXT: mull {{[0-9]+}}(%esp)
; X86-NOBMI-NEXT: addl %ebp, %eax
-; X86-NOBMI-NEXT: movzbl %bl, %esi
; X86-NOBMI-NEXT: movl {{[0-9]+}}(%esp), %ebp
-; X86-NOBMI-NEXT: adcl %esi, %edx
-; X86-NOBMI-NEXT: movl %ecx, %ebx
-; X86-NOBMI-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Reload
-; X86-NOBMI-NEXT: addl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Folded Reload
-; X86-NOBMI-NEXT: adcl {{[-0-9]+}}(%e{{[sb]}}p), %edi # 4-byte Folded Reload
+; X86-NOBMI-NEXT: movzbl %cl, %ecx
+; X86-NOBMI-NEXT: adcl %ecx, %edx
+; X86-NOBMI-NEXT: movl %edi, %ebx
+; X86-NOBMI-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %edi # 4-byte Reload
+; X86-NOBMI-NEXT: addl {{[-0-9]+}}(%e{{[sb]}}p), %edi # 4-byte Folded Reload
+; X86-NOBMI-NEXT: adcl {{[-0-9]+}}(%e{{[sb]}}p), %esi # 4-byte Folded Reload
+; X86-NOBMI-NEXT: movl {{[0-9]+}}(%esp), %ecx
+; X86-NOBMI-NEXT: movl %edi, (%ecx,%ebx,8)
+; X86-NOBMI-NEXT: movl %ebx, %edi
+; X86-NOBMI-NEXT: movl %esi, 4(%ecx,%ebx,8)
; X86-NOBMI-NEXT: adcl $0, %eax
; X86-NOBMI-NEXT: adcl $0, %edx
-; X86-NOBMI-NEXT: movl {{[0-9]+}}(%esp), %esi
-; X86-NOBMI-NEXT: movl %ecx, (%esi,%ebx,8)
-; X86-NOBMI-NEXT: movl %ebx, %ecx
-; X86-NOBMI-NEXT: movl %edi, 4(%esi,%ebx,8)
-; X86-NOBMI-NEXT: addl $1, %ecx
-; X86-NOBMI-NEXT: movl (%esp), %edi # 4-byte Reload
-; X86-NOBMI-NEXT: adcl $0, %edi
-; X86-NOBMI-NEXT: movl %ecx, %esi
+; X86-NOBMI-NEXT: addl $1, %edi
+; X86-NOBMI-NEXT: movl (%esp), %esi # 4-byte Reload
+; X86-NOBMI-NEXT: adcl $0, %esi
+; X86-NOBMI-NEXT: movl %edi, %ecx
+; X86-NOBMI-NEXT: xorl %ebp, %ecx
+; X86-NOBMI-NEXT: movl %esi, (%esp) # 4-byte Spill
; X86-NOBMI-NEXT: xorl {{[0-9]+}}(%esp), %esi
-; X86-NOBMI-NEXT: movl %edi, (%esp) # 4-byte Spill
-; X86-NOBMI-NEXT: xorl %ebp, %edi
-; X86-NOBMI-NEXT: orl %esi, %edi
+; X86-NOBMI-NEXT: orl %ecx, %esi
; X86-NOBMI-NEXT: jne .LBB1_2
; X86-NOBMI-NEXT: .LBB1_3: # %for.end
; X86-NOBMI-NEXT: xorl %eax, %eax
@@ -198,17 +197,16 @@ define i64 @mul1(i64 %n, ptr nocapture %z, ptr nocapture %x, i64 %y) nounwind {
; X86-BMI-NEXT: .LBB1_2: # %for.body
; X86-BMI-NEXT: # =>This Inner Loop Header: Depth=1
; X86-BMI-NEXT: movl %ecx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-BMI-NEXT: movl %eax, (%esp) # 4-byte Spill
+; X86-BMI-NEXT: movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
; X86-BMI-NEXT: movl %ebp, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-BMI-NEXT: movl {{[0-9]+}}(%esp), %eax
-; X86-BMI-NEXT: movl (%eax,%ebx,8), %ecx
-; X86-BMI-NEXT: movl 4(%eax,%ebx,8), %esi
-; X86-BMI-NEXT: movl %esi, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-BMI-NEXT: movl {{[0-9]+}}(%esp), %esi
+; X86-BMI-NEXT: movl (%esi,%ebx,8), %ecx
; X86-BMI-NEXT: movl %ecx, %edx
; X86-BMI-NEXT: movl {{[0-9]+}}(%esp), %eax
; X86-BMI-NEXT: mulxl %eax, %edx, %edi
; X86-BMI-NEXT: movl %edx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-BMI-NEXT: movl %esi, %edx
+; X86-BMI-NEXT: movl 4(%esi,%ebx,8), %edx
+; X86-BMI-NEXT: movl %edx, (%esp) # 4-byte Spill
; X86-BMI-NEXT: mulxl %eax, %esi, %eax
; X86-BMI-NEXT: addl %edi, %esi
; X86-BMI-NEXT: adcl $0, %eax
@@ -217,33 +215,30 @@ define i64 @mul1(i64 %n, ptr nocapture %z, ptr nocapture %x, i64 %y) nounwind {
; X86-BMI-NEXT: mulxl %ecx, %edi, %ebp
; X86-BMI-NEXT: addl %esi, %edi
; X86-BMI-NEXT: adcl %eax, %ebp
-; X86-BMI-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %edx # 4-byte Reload
-; X86-BMI-NEXT: mulxl %ecx, %ecx, %eax
+; X86-BMI-NEXT: movl (%esp), %edx # 4-byte Reload
+; X86-BMI-NEXT: mulxl %ecx, %ecx, %esi
; X86-BMI-NEXT: setb %dl
; X86-BMI-NEXT: addl %ebp, %ecx
; X86-BMI-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %ebp # 4-byte Reload
-; X86-BMI-NEXT: movl {{[0-9]+}}(%esp), %esi
; X86-BMI-NEXT: movzbl %dl, %edx
-; X86-BMI-NEXT: adcl %edx, %eax
-; X86-BMI-NEXT: movl %eax, %edx
+; X86-BMI-NEXT: adcl %edx, %esi
; X86-BMI-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
; X86-BMI-NEXT: addl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Folded Reload
-; X86-BMI-NEXT: adcl (%esp), %edi # 4-byte Folded Reload
-; X86-BMI-NEXT: adcl $0, %ecx
-; X86-BMI-NEXT: adcl $0, %edx
-; X86-BMI-NEXT: movl %edx, (%esp) # 4-byte Spill
+; X86-BMI-NEXT: adcl {{[-0-9]+}}(%e{{[sb]}}p), %edi # 4-byte Folded Reload
; X86-BMI-NEXT: movl {{[0-9]+}}(%esp), %edx
; X86-BMI-NEXT: movl %eax, (%edx,%ebx,8)
; X86-BMI-NEXT: movl %edi, 4(%edx,%ebx,8)
; X86-BMI-NEXT: movl {{[0-9]+}}(%esp), %edi
+; X86-BMI-NEXT: adcl $0, %ecx
+; X86-BMI-NEXT: movl %esi, %eax
+; X86-BMI-NEXT: adcl $0, %eax
; X86-BMI-NEXT: addl $1, %ebx
; X86-BMI-NEXT: adcl $0, %ebp
; X86-BMI-NEXT: movl %ebx, %edx
-; X86-BMI-NEXT: xorl %esi, %edx
+; X86-BMI-NEXT: xorl {{[0-9]+}}(%esp), %edx
; X86-BMI-NEXT: movl %ebp, %esi
; X86-BMI-NEXT: xorl %edi, %esi
; X86-BMI-NEXT: orl %edx, %esi
-; X86-BMI-NEXT: movl (%esp), %eax # 4-byte Reload
; X86-BMI-NEXT: jne .LBB1_2
; X86-BMI-NEXT: .LBB1_3: # %for.end
; X86-BMI-NEXT: xorl %eax, %eax
diff --git a/llvm/test/CodeGen/X86/i128-udiv.ll b/llvm/test/CodeGen/X86/i128-udiv.ll
index e11725e426602..685aaad516f14 100644
--- a/llvm/test/CodeGen/X86/i128-udiv.ll
+++ b/llvm/test/CodeGen/X86/i128-udiv.ll
@@ -45,18 +45,18 @@ define i128 @test2(i128 %x) nounwind {
; X86-NEXT: andl $-16, %esp
; X86-NEXT: subl $144, %esp
; X86-NEXT: movl 32(%ebp), %esi
-; X86-NEXT: movl 36(%ebp), %edi
+; X86-NEXT: movl 36(%ebp), %ebx
; X86-NEXT: movl 28(%ebp), %ecx
-; X86-NEXT: testl %edi, %edi
+; X86-NEXT: testl %ebx, %ebx
; X86-NEXT: jne .LBB1_1
; X86-NEXT: # %bb.2: # %_udiv-special-cases
-; X86-NEXT: bsrl %esi, %ebx
-; X86-NEXT: xorl $31, %ebx
-; X86-NEXT: orl $32, %ebx
+; X86-NEXT: bsrl %esi, %edi
+; X86-NEXT: xorl $31, %edi
+; X86-NEXT: orl $32, %edi
; X86-NEXT: jmp .LBB1_3
; X86-NEXT: .LBB1_1:
-; X86-NEXT: bsrl %edi, %ebx
-; X86-NEXT: xorl $31, %ebx
+; X86-NEXT: bsrl %ebx, %edi
+; X86-NEXT: xorl $31, %edi
; X86-NEXT: .LBB1_3: # %_udiv-special-cases
; X86-NEXT: movl 24(%ebp), %edx
; X86-NEXT: testl %ecx, %ecx
@@ -70,82 +70,82 @@ define i128 @test2(i128 %x) nounwind {
; X86-NEXT: bsrl %ecx, %eax
; X86-NEXT: xorl $31, %eax
; X86-NEXT: .LBB1_6: # %_udiv-special-cases
-; X86-NEXT: orl %edi, %ecx
+; X86-NEXT: orl %ebx, %ecx
; X86-NEXT: orl %esi, %edx
-; X86-NEXT: orl %edi, %esi
+; X86-NEXT: orl %ebx, %esi
; X86-NEXT: jne .LBB1_8
; X86-NEXT: # %bb.7: # %_udiv-special-cases
; X86-NEXT: orl $64, %eax
-; X86-NEXT: movl %eax, %ebx
+; X86-NEXT: movl %eax, %edi
; X86-NEXT: .LBB1_8: # %_udiv-special-cases
-; X86-NEXT: negl %ebx
-; X86-NEXT: movl %ebx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT: negl %edi
; X86-NEXT: movl $0, %esi
; X86-NEXT: sbbl %esi, %esi
; X86-NEXT: movl $0, %ebx
; X86-NEXT: sbbl %ebx, %ebx
-; X86-NEXT: movl $0, %edi
-; X86-NEXT: sbbl %edi, %edi
+; X86-NEXT: movl $0, %eax
+; X86-NEXT: sbbl %eax, %eax
; X86-NEXT: orl %ecx, %edx
-; X86-NEXT: movl %esi, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT: movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
; X86-NEXT: movl %edi, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT: movl %esi, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
; X86-NEXT: je .LBB1_9
; X86-NEXT: # %bb.10: # %select.false.sink
-; X86-NEXT: xorl %eax, %eax
+; X86-NEXT: xorl %edx, %edx
; X86-NEXT: movl $127, %ecx
-; X86-NEXT: cmpl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Folded Reload
+; X86-NEXT: cmpl %edi, %ecx
; X86-NEXT: movl $0, %ecx
; X86-NEXT: sbbl %esi, %ecx
; X86-NEXT: movl $0, %ecx
; X86-NEXT: sbbl %ebx, %ecx
-; X86-NEXT: sbbl %edi, %eax
+; X86-NEXT: sbbl %eax, %edx
; X86-NEXT: setb %cl
; X86-NEXT: .LBB1_11: # %select.end
; X86-NEXT: movl %ebx, %esi
; X86-NEXT: movl 8(%ebp), %eax
-; X86-NEXT: movl $0, (%esp) # 4-byte Folded Spill
-; X86-NEXT: testb %cl, %cl
; X86-NEXT: movl $0, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Folded Spill
+; X86-NEXT: testb %cl, %cl
; X86-NEXT: movl $0, %edi
+; X86-NEXT: movl $0, %ebx
; X86-NEXT: movl $0, %ecx
; X86-NEXT: jne .LBB1_13
; X86-NEXT: # %bb.12: # %select.end
; X86-NEXT: movl 28(%ebp), %ebx
-; X86-NEXT: movl %ebx, (%esp) # 4-byte Spill
-; X86-NEXT: movl 24(%ebp), %ecx
-; X86-NEXT: movl %ecx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NEXT: movl 32(%ebp), %edi
+; X86-NEXT: movl %ebx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT: movl 24(%ebp), %edi
+; X86-NEXT: movl 32(%ebp), %ebx
; X86-NEXT: movl 36(%ebp), %ecx
; X86-NEXT: .LBB1_13: # %select.end
-; X86-NEXT: movl %ecx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT: movl %ecx, (%esp) # 4-byte Spill
; X86-NEXT: jne .LBB1_14
; X86-NEXT: # %bb.20: # %select.end
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Reload
-; X86-NEXT: movl %ecx, %ebx
; X86-NEXT: xorl $127, %ecx
; X86-NEXT: orl %esi, %ecx
; X86-NEXT: movl %esi, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %edx # 4-byte Reload
; X86-NEXT: orl {{[-0-9]+}}(%e{{[sb]}}p), %edx # 4-byte Folded Reload
; X86-NEXT: orl %ecx, %edx
-; X86-NEXT: movl (%esp), %ecx # 4-byte Reload
-; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %edx # 4-byte Reload
-; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %esi # 4-byte Reload
+; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Reload
+; X86-NEXT: movl (%esp), %edx # 4-byte Reload
+; X86-NEXT: movl %edi, %esi
; X86-NEXT: je .LBB1_21
; X86-NEXT: # %bb.18: # %udiv-bb1
; X86-NEXT: movl 24(%ebp), %eax
; X86-NEXT: movl %eax, {{[0-9]+}}(%esp)
; X86-NEXT: movl 28(%ebp), %eax
; X86-NEXT: movl %eax, {{[0-9]+}}(%esp)
-; X86-NEXT: movl 32(%ebp), %eax
-; X86-NEXT: movl %eax, {{[0-9]+}}(%esp)
+; X86-NEXT: movl 32(%ebp), %esi
+; X86-NEXT: movl %esi, {{[0-9]+}}(%esp)
; X86-NEXT: movl 36(%ebp), %eax
; X86-NEXT: movl %eax, {{[0-9]+}}(%esp)
; X86-NEXT: movl $0, {{[0-9]+}}(%esp)
; X86-NEXT: movl $0, {{[0-9]+}}(%esp)
; X86-NEXT: movl $0, {{[0-9]+}}(%esp)
; X86-NEXT: movl $0, {{[0-9]+}}(%esp)
-; X86-NEXT: movl %ebx, %ecx
+; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
+; X86-NEXT: movl %eax, %ecx
+; X86-NEXT: movl %eax, %edi
; X86-NEXT: xorb $127, %cl
; X86-NEXT: movl %ecx, %eax
; X86-NEXT: shrb $3, %al
@@ -153,19 +153,20 @@ define i128 @test2(i128 %x) nounwind {
; X86-NEXT: negb %al
; X86-NEXT: movsbl %al, %eax
; X86-NEXT: movl 120(%esp,%eax), %edx
-; X86-NEXT: movl 124(%esp,%eax), %edi
-; X86-NEXT: shldl %cl, %edx, %edi
-; X86-NEXT: movl %edi, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NEXT: movl 112(%esp,%eax), %edi
+; X86-NEXT: movl 124(%esp,%eax), %ebx
+; X86-NEXT: shldl %cl, %edx, %ebx
+; X86-NEXT: movl %ebx, (%esp) # 4-byte Spill
+; X86-NEXT: movl %esi, %ebx
+; X86-NEXT: movl 112(%esp,%eax), %esi
; X86-NEXT: movl 116(%esp,%eax), %eax
; X86-NEXT: shldl %cl, %eax, %edx
; X86-NEXT: movl %edx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NEXT: shldl %cl, %edi, %eax
-; X86-NEXT: movl %eax, (%esp) # 4-byte Spill
-; X86-NEXT: shll %cl, %edi
+; X86-NEXT: shldl %cl, %esi, %eax
+; X86-NEXT: movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT: shll %cl, %esi
+; X86-NEXT: movl %esi, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT: addl $1, %edi
; X86-NEXT: movl %edi, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NEXT: addl $1, %ebx
-; X86-NEXT: movl %ebx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
; X86-NEXT: adcl $0, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Folded Spill
; X86-NEXT: adcl $0, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Folded Spill
; X86-NEXT: adcl $0, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Folded Spill
@@ -175,8 +176,7 @@ define i128 @test2(i128 %x) nounwind {
; X86-NEXT: movl %eax, {{[0-9]+}}(%esp)
; X86-NEXT: movl 28(%ebp), %eax
; X86-NEXT: movl %eax, {{[0-9]+}}(%esp)
-; X86-NEXT: movl 32(%ebp), %eax
-; X86-NEXT: movl %eax, {{[0-9]+}}(%esp)
+; X86-NEXT: movl %ebx, {{[0-9]+}}(%esp)
; X86-NEXT: movl 36(%ebp), %eax
; X86-NEXT: movl %eax, {{[0-9]+}}(%esp)
; X86-NEXT: movl $0, {{[0-9]+}}(%esp)
@@ -210,38 +210,38 @@ define i128 @test2(i128 %x) nounwind {
; X86-NEXT: adcl $-1, %eax
; X86-NEXT: movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
; X86-NEXT: movl $0, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Folded Spill
-; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %edi # 4-byte Reload
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %edx # 4-byte Reload
+; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %ebx # 4-byte Reload
+; X86-NEXT: movl (%esp), %edi # 4-byte Reload
; X86-NEXT: movl $0, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Folded Spill
; X86-NEXT: .p2align 4
; X86-NEXT: .LBB1_16: # %udiv-do-while
; X86-NEXT: # =>This Inner Loop Header: Depth=1
+; X86-NEXT: movl %ebx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
; X86-NEXT: movl %edx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
; X86-NEXT: movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NEXT: movl %edi, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %ebx # 4-byte Reload
; X86-NEXT: shldl $1, %ebx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Folded Spill
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %esi # 4-byte Reload
; X86-NEXT: shldl $1, %esi, %ebx
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Reload
; X86-NEXT: shldl $1, %ecx, %esi
-; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %edi # 4-byte Reload
; X86-NEXT: shldl $1, %edi, %ecx
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
; X86-NEXT: shldl $1, %eax, %edi
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %edx # 4-byte Reload
; X86-NEXT: orl %edx, %edi
-; X86-NEXT: movl %edi, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NEXT: movl (%esp), %edi # 4-byte Reload
+; X86-NEXT: movl %edi, (%esp) # 4-byte Spill
+; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %edi # 4-byte Reload
; X86-NEXT: shldl $1, %edi, %eax
; X86-NEXT: orl %edx, %eax
; X86-NEXT: movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
; X86-NEXT: shldl $1, %eax, %edi
; X86-NEXT: orl %edx, %edi
-; X86-NEXT: movl %edi, (%esp) # 4-byte Spill
-; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %edi # 4-byte Reload
+; X86-NEXT: movl %edi, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT: movl (%esp), %edi # 4-byte Reload
; X86-NEXT: addl %eax, %eax
; X86-NEXT: orl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Folded Reload
; X86-NEXT: movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
@@ -253,52 +253,52 @@ define i128 @test2(i128 %x) nounwind {
; X86-NEXT: sbbl %esi, %ecx
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Reload
; X86-NEXT: sbbl %ebx, %ecx
-; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Reload
+; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %esi # 4-byte Reload
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %edx # 4-byte Reload
-; X86-NEXT: sbbl %edx, %ecx
-; X86-NEXT: sarl $31, %ecx
-; X86-NEXT: movl %ecx, %eax
+; X86-NEXT: sbbl %edx, %esi
+; X86-NEXT: sarl $31, %esi
+; X86-NEXT: movl %esi, %eax
; X86-NEXT: andl $1, %eax
; X86-NEXT: movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NEXT: movl $0, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Folded Spill
-; X86-NEXT: movl %ecx, %esi
+; X86-NEXT: movl %esi, %ecx
; X86-NEXT: movl $-1, %eax
-; X86-NEXT: andl %eax, %esi
-; X86-NEXT: movl $-4, %eax
; X86-NEXT: andl %eax, %ecx
+; X86-NEXT: movl $-4, %eax
+; X86-NEXT: andl %eax, %esi
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
-; X86-NEXT: subl %ecx, %ebx
+; X86-NEXT: movl $0, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Folded Spill
+; X86-NEXT: subl %esi, %ebx
; X86-NEXT: movl %ebx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NEXT: movl %edx, %ecx
-; X86-NEXT: sbbl %esi, %ecx
-; X86-NEXT: movl %ecx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %ebx # 4-byte Reload
+; X86-NEXT: sbbl %ecx, %edx
+; X86-NEXT: movl %edx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %edx # 4-byte Reload
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %esi # 4-byte Reload
; X86-NEXT: addl $-1, %esi
-; X86-NEXT: adcl $-1, %eax
; X86-NEXT: adcl $-1, %edx
-; X86-NEXT: adcl $-1, %edi
-; X86-NEXT: movl %eax, %ecx
-; X86-NEXT: orl %edi, %ecx
+; X86-NEXT: adcl $-1, %ebx
+; X86-NEXT: adcl $-1, %eax
+; X86-NEXT: movl %edx, %ecx
+; X86-NEXT: orl %eax, %ecx
; X86-NEXT: movl %esi, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NEXT: orl %edx, %esi
+; X86-NEXT: orl %ebx, %esi
; X86-NEXT: orl %ecx, %esi
; X86-NEXT: jne .LBB1_16
; X86-NEXT: .LBB1_17: # %udiv-loop-exit
-; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %edi # 4-byte Reload
-; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %edx # 4-byte Reload
-; X86-NEXT: shldl $1, %edi, %edx
-; X86-NEXT: movl (%esp), %ecx # 4-byte Reload
-; X86-NEXT: shldl $1, %ecx, %edi
+; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %ebx # 4-byte Reload
+; X86-NEXT: shldl $1, %ebx, %edi
+; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Reload
+; X86-NEXT: shldl $1, %ecx, %ebx
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
; X86-NEXT: shldl $1, %eax, %ecx
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %esi # 4-byte Reload
; X86-NEXT: leal (%esi,%eax,2), %esi
; X86-NEXT: movl 8(%ebp), %eax
+; X86-NEXT: movl %edi, %edx
; X86-NEXT: .LBB1_21: # %udiv-end
; X86-NEXT: movl %esi, (%eax)
; X86-NEXT: movl %ecx, 4(%eax)
-; X86-NEXT: movl %edi, 8(%eax)
+; X86-NEXT: movl %ebx, 8(%eax)
; X86-NEXT: movl %edx, 12(%eax)
; X86-NEXT: leal -12(%ebp), %esp
; X86-NEXT: popl %esi
@@ -310,12 +310,13 @@ define i128 @test2(i128 %x) nounwind {
; X86-NEXT: movb $1, %cl
; X86-NEXT: jmp .LBB1_11
; X86-NEXT: .LBB1_19:
+; X86-NEXT: movl (%esp), %edi # 4-byte Reload
; X86-NEXT: movl $0, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Folded Spill
; X86-NEXT: jmp .LBB1_17
; X86-NEXT: .LBB1_14:
-; X86-NEXT: movl (%esp), %ecx # 4-byte Reload
-; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %edx # 4-byte Reload
-; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %esi # 4-byte Reload
+; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Reload
+; X86-NEXT: movl (%esp), %edx # 4-byte Reload
+; X86-NEXT: movl %edi, %esi
; X86-NEXT: jmp .LBB1_21
;
; X64-LABEL: test2:
@@ -576,7 +577,6 @@ define i128 @test3(i128 %x) nounwind {
; X86-NEXT: movl %ecx, %edx
; X86-NEXT: andl $1, %edx
; X86-NEXT: movl %edx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NEXT: movl $0, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Folded Spill
; X86-NEXT: movl %ecx, %edi
; X86-NEXT: movl $-1, %esi
; X86-NEXT: andl %esi, %edi
@@ -586,6 +586,7 @@ define i128 @test3(i128 %x) nounwind {
; X86-NEXT: movl $-3, %edx
; X86-NEXT: andl %edx, %ecx
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %edx # 4-byte Reload
+; X86-NEXT: movl $0, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Folded Spill
; X86-NEXT: subl %ecx, %ebx
; X86-NEXT: movl %ebx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
; X86-NEXT: sbbl %edi, %eax
@@ -907,10 +908,10 @@ define i128 @div_by_7(i128 %x) nounwind {
; X86-NEXT: movl %ecx, %eax
; X86-NEXT: andl $1, %eax
; X86-NEXT: movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NEXT: movl $0, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Folded Spill
; X86-NEXT: movl $7, %eax
; X86-NEXT: andl %eax, %ecx
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
+; X86-NEXT: movl $0, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Folded Spill
; X86-NEXT: subl %ecx, %edx
; X86-NEXT: movl %edx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
; X86-NEXT: sbbl $0, %edi
@@ -1233,10 +1234,10 @@ define i128 @div_by_11(i128 %x) nounwind {
; X86-NEXT: movl %ecx, %eax
; X86-NEXT: andl $1, %eax
; X86-NEXT: movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NEXT: movl $0, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Folded Spill
; X86-NEXT: movl $11, %eax
; X86-NEXT: andl %eax, %ecx
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
+; X86-NEXT: movl $0, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Folded Spill
; X86-NEXT: subl %ecx, %edx
; X86-NEXT: movl %edx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
; X86-NEXT: sbbl $0, %edi
@@ -1557,10 +1558,10 @@ define i128 @div_by_22(i128 %x) nounwind {
; X86-NEXT: movl %ecx, %eax
; X86-NEXT: andl $1, %eax
; X86-NEXT: movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NEXT: movl $0, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Folded Spill
; X86-NEXT: movl $22, %eax
; X86-NEXT: andl %eax, %ecx
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
+; X86-NEXT: movl $0, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Folded Spill
; X86-NEXT: subl %ecx, %edx
; X86-NEXT: movl %edx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
; X86-NEXT: sbbl $0, %edi
@@ -1883,10 +1884,10 @@ define i128 @div_by_56(i128 %x) nounwind {
; X86-NEXT: movl %ecx, %eax
; X86-NEXT: andl $1, %eax
; X86-NEXT: movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NEXT: movl $0, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Folded Spill
; X86-NEXT: movl $56, %eax
; X86-NEXT: andl %eax, %ecx
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
+; X86-NEXT: movl $0, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Folded Spill
; X86-NEXT: subl %ecx, %edx
; X86-NEXT: movl %edx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
; X86-NEXT: sbbl $0, %edi
@@ -2880,10 +2881,10 @@ define i128 @div_by_67(i128 %x) nounwind {
; X86-NEXT: movl %ecx, %eax
; X86-NEXT: andl $1, %eax
; X86-NEXT: movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NEXT: movl $0, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Folded Spill
; X86-NEXT: movl $67, %eax
; X86-NEXT: andl %eax, %ecx
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
+; X86-NEXT: movl $0, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Folded Spill
; X86-NEXT: subl %ecx, %edx
; X86-NEXT: movl %edx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
; X86-NEXT: sbbl $0, %edi
diff --git a/llvm/test/CodeGen/X86/mmx-arith.ll b/llvm/test/CodeGen/X86/mmx-arith.ll
index 8f97d2652bc53..a573e20fb7270 100644
--- a/llvm/test/CodeGen/X86/mmx-arith.ll
+++ b/llvm/test/CodeGen/X86/mmx-arith.ll
@@ -412,12 +412,13 @@ define <1 x i64> @test3(ptr %a, ptr %b, i32 %count) nounwind {
; X86-NEXT: .p2align 4
; X86-NEXT: .LBB3_3: # %bb26
; X86-NEXT: # =>This Inner Loop Header: Depth=1
-; X86-NEXT: movl {{[0-9]+}}(%esp), %edi
-; X86-NEXT: movl (%edi,%ebx,8), %ebp
-; X86-NEXT: movl %ecx, %esi
-; X86-NEXT: movl 4(%edi,%ebx,8), %ecx
+; X86-NEXT: movl {{[0-9]+}}(%esp), %esi
+; X86-NEXT: movl (%esi,%ebx,8), %ebp
; X86-NEXT: movl {{[0-9]+}}(%esp), %edi
; X86-NEXT: addl (%edi,%ebx,8), %ebp
+; X86-NEXT: movl %ecx, %esi
+; X86-NEXT: movl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT: movl 4(%ecx,%ebx,8), %ecx
; X86-NEXT: adcl 4(%edi,%ebx,8), %ecx
; X86-NEXT: addl %ebp, %eax
; X86-NEXT: adcl %ecx, %edx
diff --git a/llvm/test/CodeGen/X86/pr38539.ll b/llvm/test/CodeGen/X86/pr38539.ll
index eecd15dd2afe9..7edfcd419a676 100644
--- a/llvm/test/CodeGen/X86/pr38539.ll
+++ b/llvm/test/CodeGen/X86/pr38539.ll
@@ -193,14 +193,15 @@ define void @f() nounwind {
; X86-NEXT: leal (%esi,%ebx,2), %ebx
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %esi # 4-byte Reload
; X86-NEXT: shldl $1, %esi, %edi
-; X86-NEXT: orl {{[-0-9]+}}(%e{{[sb]}}p), %edi # 4-byte Folded Reload
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
; X86-NEXT: shldl $1, %eax, %esi
-; X86-NEXT: orl {{[-0-9]+}}(%e{{[sb]}}p), %esi # 4-byte Folded Reload
-; X86-NEXT: movl %esi, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
; X86-NEXT: addl %eax, %eax
; X86-NEXT: orl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Folded Reload
; X86-NEXT: movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
+; X86-NEXT: orl %eax, %edi
+; X86-NEXT: orl %eax, %esi
+; X86-NEXT: movl %esi, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
; X86-NEXT: andl $3, %edi
; X86-NEXT: movl %edi, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
; X86-NEXT: cmpl %ebx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Folded Reload
diff --git a/llvm/test/CodeGen/X86/sdiv_fix.ll b/llvm/test/CodeGen/X86/sdiv_fix.ll
index 392bc83d9d5d8..00bb7f3dbe4ce 100644
--- a/llvm/test/CodeGen/X86/sdiv_fix.ll
+++ b/llvm/test/CodeGen/X86/sdiv_fix.ll
@@ -309,24 +309,25 @@ define i64 @func5(i64 %x, i64 %y) nounwind {
; X86-NEXT: andl $-16, %esp
; X86-NEXT: subl $112, %esp
; X86-NEXT: movl 8(%ebp), %ecx
-; X86-NEXT: movl 12(%ebp), %edi
-; X86-NEXT: movl 16(%ebp), %eax
-; X86-NEXT: movl 20(%ebp), %edx
-; X86-NEXT: movl %edx, %ebx
+; X86-NEXT: movl 12(%ebp), %edx
+; X86-NEXT: movl 16(%ebp), %esi
+; X86-NEXT: movl 20(%ebp), %edi
+; X86-NEXT: leal {{[0-9]+}}(%esp), %eax
+; X86-NEXT: movl %eax, (%esp)
+; X86-NEXT: movl %edi, %ebx
; X86-NEXT: sarl $31, %ebx
; X86-NEXT: movl %ebx, {{[0-9]+}}(%esp)
; X86-NEXT: movl %ebx, {{[0-9]+}}(%esp)
-; X86-NEXT: movl %edx, {{[0-9]+}}(%esp)
-; X86-NEXT: movl %eax, {{[0-9]+}}(%esp)
-; X86-NEXT: movl %edi, %esi
-; X86-NEXT: sarl $31, %esi
+; X86-NEXT: movl %edi, {{[0-9]+}}(%esp)
; X86-NEXT: movl %esi, {{[0-9]+}}(%esp)
-; X86-NEXT: movl %esi, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NEXT: leal {{[0-9]+}}(%esp), %eax
-; X86-NEXT: movl %eax, (%esp)
-; X86-NEXT: shldl $31, %edi, %esi
-; X86-NEXT: shldl $31, %ecx, %edi
+; X86-NEXT: movl %edx, %edi
+; X86-NEXT: sarl $31, %edi
; X86-NEXT: movl %edi, {{[0-9]+}}(%esp)
+; X86-NEXT: movl %edi, %esi
+; X86-NEXT: shldl $31, %edx, %esi
+; X86-NEXT: shldl $31, %ecx, %edx
+; X86-NEXT: movl %edx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT: movl %edx, {{[0-9]+}}(%esp)
; X86-NEXT: shll $31, %ecx
; X86-NEXT: movl %ecx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
; X86-NEXT: movl %ecx, {{[0-9]+}}(%esp)
@@ -339,23 +340,24 @@ define i64 @func5(i64 %x, i64 %y) nounwind {
; X86-NEXT: movl %eax, {{[0-9]+}}(%esp)
; X86-NEXT: movl %ebx, {{[0-9]+}}(%esp)
; X86-NEXT: movl %ebx, {{[0-9]+}}(%esp)
-; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Reload
-; X86-NEXT: movl %ecx, {{[0-9]+}}(%esp)
; X86-NEXT: movl %edi, {{[0-9]+}}(%esp)
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
; X86-NEXT: movl %eax, {{[0-9]+}}(%esp)
+; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
+; X86-NEXT: movl %eax, {{[0-9]+}}(%esp)
; X86-NEXT: leal {{[0-9]+}}(%esp), %eax
; X86-NEXT: movl %eax, (%esp)
; X86-NEXT: movl %esi, {{[0-9]+}}(%esp)
+; X86-NEXT: movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT: movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
; X86-NEXT: movl {{[0-9]+}}(%esp), %esi
; X86-NEXT: movl %esi, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NEXT: movl {{[0-9]+}}(%esp), %edi
-; X86-NEXT: movl %edi, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NEXT: subl $1, %esi
-; X86-NEXT: sbbl $0, %edi
+; X86-NEXT: subl $1, %eax
+; X86-NEXT: movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT: sbbl $0, %esi
; X86-NEXT: testl %ebx, %ebx
; X86-NEXT: sets %al
-; X86-NEXT: testl %ecx, %ecx
+; X86-NEXT: testl %edi, %edi
; X86-NEXT: sets %bl
; X86-NEXT: xorb %al, %bl
; X86-NEXT: calll __modti3
@@ -367,10 +369,10 @@ define i64 @func5(i64 %x, i64 %y) nounwind {
; X86-NEXT: orl %eax, %ecx
; X86-NEXT: setne %al
; X86-NEXT: testb %bl, %al
+; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
+; X86-NEXT: cmovel {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Folded Reload
; X86-NEXT: cmovel {{[-0-9]+}}(%e{{[sb]}}p), %esi # 4-byte Folded Reload
-; X86-NEXT: cmovel {{[-0-9]+}}(%e{{[sb]}}p), %edi # 4-byte Folded Reload
-; X86-NEXT: movl %esi, %eax
-; X86-NEXT: movl %edi, %edx
+; X86-NEXT: movl %esi, %edx
; X86-NEXT: leal -12(%ebp), %esp
; X86-NEXT: popl %esi
; X86-NEXT: popl %edi
diff --git a/llvm/test/CodeGen/X86/sdiv_fix_sat.ll b/llvm/test/CodeGen/X86/sdiv_fix_sat.ll
index e7d41d5bebc8a..4e2a3d9ba8d9e 100644
--- a/llvm/test/CodeGen/X86/sdiv_fix_sat.ll
+++ b/llvm/test/CodeGen/X86/sdiv_fix_sat.ll
@@ -371,62 +371,62 @@ define i64 @func5(i64 %x, i64 %y) nounwind {
; X86-NEXT: pushl %esi
; X86-NEXT: andl $-16, %esp
; X86-NEXT: subl $128, %esp
-; X86-NEXT: movl 8(%ebp), %esi
-; X86-NEXT: movl 12(%ebp), %edi
-; X86-NEXT: movl 16(%ebp), %ecx
-; X86-NEXT: movl 20(%ebp), %edx
-; X86-NEXT: movl %edx, %eax
-; X86-NEXT: sarl $31, %eax
-; X86-NEXT: movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NEXT: movl %eax, {{[0-9]+}}(%esp)
-; X86-NEXT: movl %eax, {{[0-9]+}}(%esp)
-; X86-NEXT: movl %edx, {{[0-9]+}}(%esp)
-; X86-NEXT: movl %ecx, {{[0-9]+}}(%esp)
-; X86-NEXT: movl %edi, %ebx
-; X86-NEXT: sarl $31, %ebx
-; X86-NEXT: movl %ebx, {{[0-9]+}}(%esp)
-; X86-NEXT: movl %ebx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT: movl 8(%ebp), %ecx
+; X86-NEXT: movl 12(%ebp), %ebx
+; X86-NEXT: movl 16(%ebp), %edx
+; X86-NEXT: movl 20(%ebp), %esi
; X86-NEXT: leal {{[0-9]+}}(%esp), %eax
; X86-NEXT: movl %eax, (%esp)
-; X86-NEXT: shldl $31, %edi, %ebx
-; X86-NEXT: shldl $31, %esi, %edi
+; X86-NEXT: movl %esi, %edi
+; X86-NEXT: sarl $31, %edi
+; X86-NEXT: movl %edi, {{[0-9]+}}(%esp)
; X86-NEXT: movl %edi, {{[0-9]+}}(%esp)
-; X86-NEXT: shll $31, %esi
; X86-NEXT: movl %esi, {{[0-9]+}}(%esp)
+; X86-NEXT: movl %edx, {{[0-9]+}}(%esp)
+; X86-NEXT: movl %ebx, %esi
+; X86-NEXT: sarl $31, %esi
+; X86-NEXT: movl %esi, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT: movl %esi, {{[0-9]+}}(%esp)
+; X86-NEXT: shldl $31, %ebx, %esi
+; X86-NEXT: shldl $31, %ecx, %ebx
; X86-NEXT: movl %ebx, {{[0-9]+}}(%esp)
+; X86-NEXT: shll $31, %ecx
+; X86-NEXT: movl %ecx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT: movl %ecx, {{[0-9]+}}(%esp)
+; X86-NEXT: movl %esi, {{[0-9]+}}(%esp)
; X86-NEXT: calll __divti3
; X86-NEXT: subl $4, %esp
; X86-NEXT: movl 20(%ebp), %eax
; X86-NEXT: movl %eax, {{[0-9]+}}(%esp)
; X86-NEXT: movl 16(%ebp), %eax
; X86-NEXT: movl %eax, {{[0-9]+}}(%esp)
+; X86-NEXT: movl %edi, {{[0-9]+}}(%esp)
+; X86-NEXT: movl %edi, {{[0-9]+}}(%esp)
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Reload
; X86-NEXT: movl %ecx, {{[0-9]+}}(%esp)
-; X86-NEXT: movl %ecx, {{[0-9]+}}(%esp)
-; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %edx # 4-byte Reload
-; X86-NEXT: movl %edx, {{[0-9]+}}(%esp)
-; X86-NEXT: movl %edi, {{[0-9]+}}(%esp)
-; X86-NEXT: movl %esi, {{[0-9]+}}(%esp)
+; X86-NEXT: movl %ebx, {{[0-9]+}}(%esp)
+; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
+; X86-NEXT: movl %eax, {{[0-9]+}}(%esp)
; X86-NEXT: leal {{[0-9]+}}(%esp), %eax
; X86-NEXT: movl %eax, (%esp)
-; X86-NEXT: movl %ebx, {{[0-9]+}}(%esp)
-; X86-NEXT: movl {{[0-9]+}}(%esp), %edi
-; X86-NEXT: movl %edi, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NEXT: movl %esi, {{[0-9]+}}(%esp)
+; X86-NEXT: movl {{[0-9]+}}(%esp), %esi
+; X86-NEXT: movl %esi, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
; X86-NEXT: movl {{[0-9]+}}(%esp), %eax
; X86-NEXT: movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NEXT: subl $1, %edi
+; X86-NEXT: subl $1, %esi
; X86-NEXT: sbbl $0, %eax
; X86-NEXT: movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
; X86-NEXT: movl {{[0-9]+}}(%esp), %eax
; X86-NEXT: movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
; X86-NEXT: sbbl $0, %eax
; X86-NEXT: movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NEXT: movl {{[0-9]+}}(%esp), %esi
-; X86-NEXT: movl %esi, %ebx
+; X86-NEXT: movl {{[0-9]+}}(%esp), %ebx
+; X86-NEXT: movl %ebx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
; X86-NEXT: sbbl $0, %ebx
-; X86-NEXT: testl %ecx, %ecx
+; X86-NEXT: testl %edi, %edi
; X86-NEXT: sets %al
-; X86-NEXT: testl %edx, %edx
+; X86-NEXT: testl %ecx, %ecx
; X86-NEXT: sets %cl
; X86-NEXT: xorb %al, %cl
; X86-NEXT: movb %cl, {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Spill
@@ -439,38 +439,40 @@ define i64 @func5(i64 %x, i64 %y) nounwind {
; X86-NEXT: orl %eax, %ecx
; X86-NEXT: setne %al
; X86-NEXT: testb %al, {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Folded Reload
-; X86-NEXT: cmovel %esi, %ebx
+; X86-NEXT: cmovel {{[-0-9]+}}(%e{{[sb]}}p), %ebx # 4-byte Folded Reload
+; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %edx # 4-byte Reload
+; X86-NEXT: cmovel {{[-0-9]+}}(%e{{[sb]}}p), %edx # 4-byte Folded Reload
; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
; X86-NEXT: cmovel {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Folded Reload
-; X86-NEXT: movl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Reload
-; X86-NEXT: cmovel {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Folded Reload
-; X86-NEXT: movl %ecx, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
-; X86-NEXT: cmovel {{[-0-9]+}}(%e{{[sb]}}p), %edi # 4-byte Folded Reload
-; X86-NEXT: cmpl $-1, %edi
-; X86-NEXT: sbbl $2147483647, %ecx # imm = 0x7FFFFFFF
+; X86-NEXT: cmovel {{[-0-9]+}}(%e{{[sb]}}p), %esi # 4-byte Folded Reload
+; X86-NEXT: cmpl $-1, %esi
; X86-NEXT: movl %eax, %ecx
+; X86-NEXT: movl %eax, %edi
+; X86-NEXT: sbbl $2147483647, %ecx # imm = 0x7FFFFFFF
+; X86-NEXT: movl %edx, %ecx
+; X86-NEXT: movl %edx, %eax
; X86-NEXT: sbbl $0, %ecx
; X86-NEXT: movl %ebx, %ecx
; X86-NEXT: sbbl $0, %ecx
; X86-NEXT: movl $2147483647, %edx # imm = 0x7FFFFFFF
-; X86-NEXT: cmovll {{[-0-9]+}}(%e{{[sb]}}p), %edx # 4-byte Folded Reload
+; X86-NEXT: cmovll %edi, %edx
; X86-NEXT: movl $0, %ecx
; X86-NEXT: cmovgel %ecx, %ebx
; X86-NEXT: cmovgel %ecx, %eax
; X86-NEXT: movl $-1, %ecx
-; X86-NEXT: cmovgel %ecx, %edi
-; X86-NEXT: movl %edi, %esi
-; X86-NEXT: negl %esi
-; X86-NEXT: movl $-2147483648, %esi # imm = 0x80000000
-; X86-NEXT: sbbl %edx, %esi
-; X86-NEXT: movl $-1, %esi
-; X86-NEXT: sbbl %eax, %esi
+; X86-NEXT: cmovgel %ecx, %esi
+; X86-NEXT: movl %esi, %edi
+; X86-NEXT: negl %edi
+; X86-NEXT: movl $-2147483648, %edi # imm = 0x80000000
+; X86-NEXT: sbbl %edx, %edi
+; X86-NEXT: movl $-1, %edi
+; X86-NEXT: sbbl %eax, %edi
; X86-NEXT: sbbl %ebx, %ecx
; X86-NEXT: movl $0, %eax
-; X86-NEXT: cmovgel %eax, %edi
+; X86-NEXT: cmovgel %eax, %esi
; X86-NEXT: movl $-2147483648, %eax # imm = 0x80000000
; X86-NEXT: cmovgel %eax, %edx
-; X86-NEXT: movl %edi, %eax
+; X86-NEXT: movl %esi, %eax
; X86-NEXT: leal -12(%ebp), %esp
; X86-NEXT: popl %esi
; X86-NEXT: popl %edi
diff --git a/llvm/test/CodeGen/X86/vector-shift-by-select-loop.ll b/llvm/test/CodeGen/X86/vector-shift-by-select-loop.ll
index b453f925b94e8..0a5ce805e1fde 100644
--- a/llvm/test/CodeGen/X86/vector-shift-by-select-loop.ll
+++ b/llvm/test/CodeGen/X86/vector-shift-by-select-loop.ll
@@ -159,10 +159,8 @@ define void @vector_variable_shift_left_loop(ptr nocapture %arr, ptr nocapture r
; AVX1-NEXT: vmovdqa %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
; AVX1-NEXT: vpmovzxdq {{.*#+}} xmm0 = xmm8[0],zero,xmm8[1],zero
; AVX1-NEXT: vmovdqa %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
-; AVX1-NEXT: vpmovzxdq {{.*#+}} xmm0 = xmm7[0],zero,xmm7[1],zero
-; AVX1-NEXT: vmovdqa %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
-; AVX1-NEXT: vpmovzxdq {{.*#+}} xmm0 = xmm8[0],zero,xmm8[1],zero
-; AVX1-NEXT: vmovdqa %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; AVX1-NEXT: vpmovzxdq {{.*#+}} xmm3 = xmm7[0],zero,xmm7[1],zero
+; AVX1-NEXT: vpmovzxdq {{.*#+}} xmm4 = xmm8[0],zero,xmm8[1],zero
; AVX1-NEXT: vpmovzxdq {{.*#+}} xmm5 = xmm7[0],zero,xmm7[1],zero
; AVX1-NEXT: vpmovzxdq {{.*#+}} xmm6 = xmm8[0],zero,xmm8[1],zero
; AVX1-NEXT: vpmovzxdq {{.*#+}} xmm7 = xmm7[0],zero,xmm7[1],zero
@@ -173,68 +171,66 @@ define void @vector_variable_shift_left_loop(ptr nocapture %arr, ptr nocapture r
; AVX1-NEXT: vmovq {{.*#+}} xmm9 = mem[0],zero
; AVX1-NEXT: vmovq {{.*#+}} xmm10 = mem[0],zero
; AVX1-NEXT: vmovq {{.*#+}} xmm11 = mem[0],zero
-; AVX1-NEXT: vmovq {{.*#+}} xmm12 = mem[0],zero
-; AVX1-NEXT: vpxor %xmm2, %xmm2, %xmm2
-; AVX1-NEXT: vpcmpeqb %xmm2, %xmm9, %xmm9
-; AVX1-NEXT: vpmovsxbd %xmm9, %xmm14
-; AVX1-NEXT: vpshufd {{.*#+}} xmm9 = xmm9[1,1,1,1]
-; AVX1-NEXT: vpmovsxbd %xmm9, %xmm15
-; AVX1-NEXT: vpcmpeqb %xmm2, %xmm10, %xmm9
-; AVX1-NEXT: vpmovsxbd %xmm9, %xmm0
+; AVX1-NEXT: vpxor %xmm1, %xmm1, %xmm1
+; AVX1-NEXT: vpcmpeqb %xmm1, %xmm9, %xmm9
+; AVX1-NEXT: vpmovsxbd %xmm9, %xmm12
; AVX1-NEXT: vpshufd {{.*#+}} xmm9 = xmm9[1,1,1,1]
-; AVX1-NEXT: vpmovsxbd %xmm9, %xmm1
-; AVX1-NEXT: vpcmpeqb %xmm2, %xmm11, %xmm9
; AVX1-NEXT: vpmovsxbd %xmm9, %xmm13
+; AVX1-NEXT: vpcmpeqb %xmm1, %xmm10, %xmm9
+; AVX1-NEXT: vpmovsxbd %xmm9, %xmm14
; AVX1-NEXT: vpshufd {{.*#+}} xmm9 = xmm9[1,1,1,1]
-; AVX1-NEXT: vpmovsxbd %xmm9, %xmm11
-; AVX1-NEXT: vpcmpeqb %xmm2, %xmm12, %xmm9
-; AVX1-NEXT: vpmovsxbd %xmm9, %xmm10
-; AVX1-NEXT: vpshufd {{.*#+}} xmm9 = xmm9[1,1,1,1]
-; AVX1-NEXT: vpmovsxbd %xmm9, %xmm9
-; AVX1-NEXT: vmovdqu (%rdi,%rcx,4), %xmm12
-; AVX1-NEXT: vmovdqa {{[-0-9]+}}(%r{{[sb]}}p), %xmm3 # 16-byte Reload
-; AVX1-NEXT: vpslld %xmm3, %xmm12, %xmm2
-; AVX1-NEXT: vmovdqa {{[-0-9]+}}(%r{{[sb]}}p), %xmm4 # 16-byte Reload
+; AVX1-NEXT: vpmovsxbd %xmm9, %xmm15
+; AVX1-NEXT: vpcmpeqb %xmm1, %xmm11, %xmm11
+; AVX1-NEXT: vmovdqu (%rdi,%rcx,4), %xmm9
+; AVX1-NEXT: vmovdqa {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; AVX1-NEXT: vpslld %xmm0, %xmm9, %xmm10
+; AVX1-NEXT: vmovdqa {{[-0-9]+}}(%r{{[sb]}}p), %xmm2 # 16-byte Reload
+; AVX1-NEXT: vpslld %xmm2, %xmm9, %xmm9
+; AVX1-NEXT: vblendvps %xmm12, %xmm10, %xmm9, %xmm9
+; AVX1-NEXT: vmovdqu 16(%rdi,%rcx,4), %xmm10
+; AVX1-NEXT: vpslld %xmm0, %xmm10, %xmm12
+; AVX1-NEXT: vpslld %xmm2, %xmm10, %xmm10
+; AVX1-NEXT: vblendvps %xmm13, %xmm12, %xmm10, %xmm10
+; AVX1-NEXT: vmovdqu 32(%rdi,%rcx,4), %xmm12
+; AVX1-NEXT: vpslld %xmm3, %xmm12, %xmm13
; AVX1-NEXT: vpslld %xmm4, %xmm12, %xmm12
-; AVX1-NEXT: vblendvps %xmm14, %xmm2, %xmm12, %xmm12
-; AVX1-NEXT: vmovdqu 16(%rdi,%rcx,4), %xmm2
-; AVX1-NEXT: vpslld %xmm3, %xmm2, %xmm14
-; AVX1-NEXT: vpslld %xmm4, %xmm2, %xmm2
-; AVX1-NEXT: vblendvps %xmm15, %xmm14, %xmm2, %xmm2
-; AVX1-NEXT: vmovdqu 32(%rdi,%rcx,4), %xmm14
-; AVX1-NEXT: vmovdqa {{[-0-9]+}}(%r{{[sb]}}p), %xmm3 # 16-byte Reload
-; AVX1-NEXT: vpslld %xmm3, %xmm14, %xmm15
-; AVX1-NEXT: vmovdqa {{[-0-9]+}}(%r{{[sb]}}p), %xmm4 # 16-byte Reload
-; AVX1-NEXT: vpslld %xmm4, %xmm14, %xmm14
-; AVX1-NEXT: vblendvps %xmm0, %xmm15, %xmm14, %xmm0
-; AVX1-NEXT: vmovdqu 48(%rdi,%rcx,4), %xmm14
-; AVX1-NEXT: vpslld %xmm3, %xmm14, %xmm15
-; AVX1-NEXT: vpslld %xmm4, %xmm14, %xmm14
-; AVX1-NEXT: vblendvps %xmm1, %xmm15, %xmm14, %xmm1
+; AVX1-NEXT: vblendvps %xmm14, %xmm13, %xmm12, %xmm12
+; AVX1-NEXT: vmovdqu 48(%rdi,%rcx,4), %xmm13
+; AVX1-NEXT: vpslld %xmm3, %xmm13, %xmm14
+; AVX1-NEXT: vpslld %xmm4, %xmm13, %xmm13
+; AVX1-NEXT: vblendvps %xmm15, %xmm14, %xmm13, %xmm13
; AVX1-NEXT: vmovdqu 64(%rdi,%rcx,4), %xmm14
; AVX1-NEXT: vpslld %xmm5, %xmm14, %xmm15
; AVX1-NEXT: vpslld %xmm6, %xmm14, %xmm14
-; AVX1-NEXT: vblendvps %xmm13, %xmm15, %xmm14, %xmm13
+; AVX1-NEXT: vpmovsxbd %xmm11, %xmm0
+; AVX1-NEXT: vblendvps %xmm0, %xmm15, %xmm14, %xmm0
+; AVX1-NEXT: vpshufd {{.*#+}} xmm11 = xmm11[1,1,1,1]
+; AVX1-NEXT: vpmovsxbd %xmm11, %xmm11
; AVX1-NEXT: vmovdqu 80(%rdi,%rcx,4), %xmm14
; AVX1-NEXT: vpslld %xmm5, %xmm14, %xmm15
; AVX1-NEXT: vpslld %xmm6, %xmm14, %xmm14
; AVX1-NEXT: vblendvps %xmm11, %xmm15, %xmm14, %xmm11
-; AVX1-NEXT: vmovdqu 96(%rdi,%rcx,4), %xmm14
-; AVX1-NEXT: vpslld %xmm7, %xmm14, %xmm15
-; AVX1-NEXT: vpslld %xmm8, %xmm14, %xmm14
-; AVX1-NEXT: vblendvps %xmm10, %xmm15, %xmm14, %xmm10
+; AVX1-NEXT: vmovq {{.*#+}} xmm14 = mem[0],zero
+; AVX1-NEXT: vpcmpeqb %xmm1, %xmm14, %xmm14
+; AVX1-NEXT: vmovdqu 96(%rdi,%rcx,4), %xmm15
+; AVX1-NEXT: vpslld %xmm7, %xmm15, %xmm1
+; AVX1-NEXT: vpslld %xmm8, %xmm15, %xmm15
+; AVX1-NEXT: vpmovsxbd %xmm14, %xmm2
+; AVX1-NEXT: vblendvps %xmm2, %xmm1, %xmm15, %xmm1
+; AVX1-NEXT: vpshufd {{.*#+}} xmm2 = xmm14[1,1,1,1]
+; AVX1-NEXT: vpmovsxbd %xmm2, %xmm2
; AVX1-NEXT: vmovdqu 112(%rdi,%rcx,4), %xmm14
; AVX1-NEXT: vpslld %xmm7, %xmm14, %xmm15
; AVX1-NEXT: vpslld %xmm8, %xmm14, %xmm14
-; AVX1-NEXT: vblendvps %xmm9, %xmm15, %xmm14, %xmm9
-; AVX1-NEXT: vmovups %xmm12, (%rdi,%rcx,4)
-; AVX1-NEXT: vmovups %xmm2, 16(%rdi,%rcx,4)
-; AVX1-NEXT: vmovups %xmm0, 32(%rdi,%rcx,4)
-; AVX1-NEXT: vmovups %xmm1, 48(%rdi,%rcx,4)
-; AVX1-NEXT: vmovups %xmm13, 64(%rdi,%rcx,4)
+; AVX1-NEXT: vblendvps %xmm2, %xmm15, %xmm14, %xmm2
+; AVX1-NEXT: vmovups %xmm9, (%rdi,%rcx,4)
+; AVX1-NEXT: vmovups %xmm10, 16(%rdi,%rcx,4)
+; AVX1-NEXT: vmovups %xmm12, 32(%rdi,%rcx,4)
+; AVX1-NEXT: vmovups %xmm13, 48(%rdi,%rcx,4)
+; AVX1-NEXT: vmovups %xmm0, 64(%rdi,%rcx,4)
; AVX1-NEXT: vmovups %xmm11, 80(%rdi,%rcx,4)
-; AVX1-NEXT: vmovups %xmm10, 96(%rdi,%rcx,4)
-; AVX1-NEXT: vmovups %xmm9, 112(%rdi,%rcx,4)
+; AVX1-NEXT: vmovups %xmm1, 96(%rdi,%rcx,4)
+; AVX1-NEXT: vmovups %xmm2, 112(%rdi,%rcx,4)
; AVX1-NEXT: addq $32, %rcx
; AVX1-NEXT: cmpq %rcx, %rdx
; AVX1-NEXT: jne .LBB0_4
More information about the llvm-commits
mailing list