[llvm] RegisterPressure: Detect dead physreg defs from LiveIntervals (PR #225079)
Matt Arsenault via llvm-commits
llvm-commits at lists.llvm.org
Mon Sep 21 06:02:25 PDT 2026
https://github.com/arsenm created https://github.com/llvm/llvm-project/pull/225079
This reverts the remainder of #222627, which was partially reverted by
on dead flags. This is a prerequisite to deleting LiveVariables.
When constructing the PressureDiff for an instruction during scheduling
DAG construction, dead defs were only recognized from the dead flag on the
operand. This implicitly relied on preprocessing done by LiveVariables to
fixup inconsistent dead flags with overlapping registers in other operands.
Dead flags have no verifier-enforced rules and are thus unreliable.
Before LiveVariables, consider this example:
dead $eax = MOV32r0 implicit-def dead $eflags, implicit-def $rax
; $rax is never used
$rax is never used, but only the $eax def is dead-flagged and the overlapping
implicit-def $rax is not. The shared $eax register units are covered by the
non-dead $rax def and so are counted as live defs. That shared unit is then
decremented on recede without a matching increment, tripping the "PSet
overflow/underflow" assertion in getUpwardPressureDelta.
Operand dead flags only describe physreg liveness reliably when
LiveVariables produced them. For a wide def that is only partly used,
HandlePhysRegKill marks the superregister def dead and adds a separate,
explicitly non-dead def of the used sub-register. In the example, LiveVariables
fixes up the flags to liven $eax, and make $rax dead:
$eax = MOV32r0 implicit-def dead $eflags, implicit-def dead $rax
The rule is a regunit unit is dead iff every def covering it is dead. When
LiveVariables does not run, the superregister dead flag is simply missing and
the flags no longer describe liveness.
Detect dead defs from LiveIntervals instead, matching what the pressure
tracker already does on the recede path. We need to not take a conservative no
answer if getCachedRegUnit doesn't already have the computed interval, so add
a parameter to not use the cache. This is the source of most of the churn, which
needs to de-constify the LiveIntervals passed around.
I'm somewhat dissatisfied with relying on LiveIntervals and ignoring the flags. I'm
separately working on adding some verifier rules for dead flags, though I'm not
sure that will be sufficient to avoid this.
Co-authored-by: Claude claude-opus-4.8 <noreply at anthropic.com>
>From 5d2c4e85b1a513886f008c407e264618e9c9e3f8 Mon Sep 17 00:00:00 2001
From: Matt Arsenault <Matthew.Arsenault at amd.com>
Date: Fri, 18 Sep 2026 12:27:36 +0200
Subject: [PATCH] RegisterPressure: Detect dead physreg defs from LiveIntervals
This reverts the remainder of #222627, which was partially reverted by
on dead flags. This is a prerequisite to deleting LiveVariables.
When constructing the PressureDiff for an instruction during scheduling
DAG construction, dead defs were only recognized from the dead flag on the
operand. This implicitly relied on preprocessing done by LiveVariables to
fixup inconsistent dead flags with overlapping registers in other operands.
Dead flags have no verifier-enforced rules and are thus unreliable.
Before LiveVariables, consider this example:
dead $eax = MOV32r0 implicit-def dead $eflags, implicit-def $rax
; $rax is never used
$rax is never used, but only the $eax def is dead-flagged and the overlapping
implicit-def $rax is not. The shared $eax register units are covered by the
non-dead $rax def and so are counted as live defs. That shared unit is then
decremented on recede without a matching increment, tripping the "PSet
overflow/underflow" assertion in getUpwardPressureDelta.
Operand dead flags only describe physreg liveness reliably when
LiveVariables produced them. For a wide def that is only partly used,
HandlePhysRegKill marks the superregister def dead and adds a separate,
explicitly non-dead def of the used sub-register. In the example, LiveVariables
fixes up the flags to liven $eax, and make $rax dead:
$eax = MOV32r0 implicit-def dead $eflags, implicit-def dead $rax
The rule is a regunit unit is dead iff every def covering it is dead. When
LiveVariables does not run, the superregister dead flag is simply missing and
the flags no longer describe liveness.
Detect dead defs from LiveIntervals instead, matching what the pressure
tracker already does on the recede path. We need to not take a conservative no
answer if getCachedRegUnit doesn't already have the computed interval, so add
a parameter to not use the cache. This is the source of most of the churn, which
needs to de-constify the LiveIntervals passed around.
I'm somewhat dissatisfied with relying on LiveIntervals and ignoring the flags. I'm
separately working on adding some verifier rules for dead flags, though I'm not
sure that will be sufficient to avoid this.
Co-authored-by: Claude claude-opus-4.8 <noreply at anthropic.com>
---
llvm/include/llvm/CodeGen/RegisterPressure.h | 12 +-
llvm/lib/CodeGen/RegisterPressure.cpp | 48 +-
llvm/lib/CodeGen/ScheduleDAGInstrs.cpp | 4 +
.../Target/AMDGPU/AMDGPUNextUseAnalysis.cpp | 9 +-
llvm/lib/Target/AMDGPU/GCNRegPressure.cpp | 2 +-
llvm/lib/Target/AMDGPU/GCNRegPressure.h | 8 +-
llvm/test/CodeGen/X86/masked-udiv.ll | 433 +++++++++---------
.../CodeGen/X86/min-legal-vector-width.ll | 64 +--
...misched-pressure-dead-physreg-superreg.mir | 60 +++
llvm/test/CodeGen/X86/ssub_sat_plus.ll | 4 +-
llvm/test/CodeGen/X86/statepoint-ra.ll | 5 +-
.../statepoint-vreg-unlimited-tied-opnds.ll | 41 +-
llvm/test/CodeGen/X86/udiv_fix.ll | 34 +-
llvm/test/CodeGen/X86/udiv_fix_sat.ll | 82 ++--
llvm/test/CodeGen/X86/usub_sat_plus.ll | 4 +-
llvm/test/CodeGen/X86/vector-idiv-strictfp.ll | 191 ++++----
llvm/test/CodeGen/X86/vector-idiv-udiv-512.ll | 132 +++---
llvm/test/CodeGen/X86/xmulo.ll | 12 +-
18 files changed, 584 insertions(+), 561 deletions(-)
create mode 100644 llvm/test/CodeGen/X86/misched-pressure-dead-physreg-superreg.mir
diff --git a/llvm/include/llvm/CodeGen/RegisterPressure.h b/llvm/include/llvm/CodeGen/RegisterPressure.h
index 47c2a31d4317a..bd6b89df9c049 100644
--- a/llvm/include/llvm/CodeGen/RegisterPressure.h
+++ b/llvm/include/llvm/CodeGen/RegisterPressure.h
@@ -187,19 +187,19 @@ class RegisterOperands {
/// Use liveness information to find dead defs at \p MI's dead slot not marked
/// with a dead flag and move them to the DeadDefs vector. This only considers
/// the merged live interval for defs, not the per-lane sub-ranges.
- LLVM_ABI void detectDeadDefs(const MachineInstr &MI, const LiveIntervals &LIS,
+ LLVM_ABI void detectDeadDefs(const MachineInstr &MI, LiveIntervals &LIS,
const MachineRegisterInfo &MRI);
/// Use liveness information to find out which uses/defs are partially
/// undefined/dead at \p Pos and adjust the VRegMaskOrUnits accordingly.
- LLVM_ABI void adjustLaneLiveness(const LiveIntervals &LIS,
+ LLVM_ABI void adjustLaneLiveness(LiveIntervals &LIS,
const MachineRegisterInfo &MRI,
SlotIndex Pos);
/// Use liveness information to find out which uses/defs are partially
/// undefined/dead at the \p MI's position and adjust the VRegMaskOrUnits
/// accordingly. Missing read-undef and dead flags are added to \p MI.
- LLVM_ABI void adjustLaneLiveness(const LiveIntervals &LIS,
+ LLVM_ABI void adjustLaneLiveness(LiveIntervals &LIS,
const MachineRegisterInfo &MRI,
MachineInstr &MI);
@@ -211,7 +211,7 @@ class RegisterOperands {
VRegMaskOrUnit *adjustDef(VRegMaskOrUnit &Def, LaneBitmask LiveAfterDef);
/// Use liveness information at \p Pos to adjust the lanemask of all uses.
- void adjustUses(const LiveIntervals &LIS, const MachineRegisterInfo &MRI,
+ void adjustUses(LiveIntervals &LIS, const MachineRegisterInfo &MRI,
SlotIndex Pos);
};
@@ -382,7 +382,7 @@ class RegPressureTracker {
const TargetRegisterInfo *TRI = nullptr;
const RegisterClassInfo *RCI = nullptr;
const MachineRegisterInfo *MRI = nullptr;
- const LiveIntervals *LIS = nullptr;
+ LiveIntervals *LIS = nullptr;
/// We currently only allow pressure tracking within a block.
const MachineBasicBlock *MBB = nullptr;
@@ -423,7 +423,7 @@ class RegPressureTracker {
LLVM_ABI void reset();
LLVM_ABI void init(const MachineFunction *mf, const RegisterClassInfo *rci,
- const LiveIntervals *lis, const MachineBasicBlock *mbb,
+ LiveIntervals *lis, const MachineBasicBlock *mbb,
MachineBasicBlock::const_iterator pos, bool TrackLaneMasks,
bool TrackUntiedDefs);
diff --git a/llvm/lib/CodeGen/RegisterPressure.cpp b/llvm/lib/CodeGen/RegisterPressure.cpp
index 10a0d9b02d9f4..40e3c003e10fb 100644
--- a/llvm/lib/CodeGen/RegisterPressure.cpp
+++ b/llvm/lib/CodeGen/RegisterPressure.cpp
@@ -254,7 +254,7 @@ void RegPressureTracker::reset() {
/// TODO: Add support for pressure without LiveIntervals.
void RegPressureTracker::init(const MachineFunction *mf,
const RegisterClassInfo *rci,
- const LiveIntervals *lis,
+ LiveIntervals *lis,
const MachineBasicBlock *mbb,
MachineBasicBlock::const_iterator pos,
bool TrackLaneMasks, bool TrackUntiedDefs) {
@@ -411,10 +411,11 @@ static void removeRegLanes(SmallVectorImpl<VRegMaskOrUnit> &RegUnits,
}
static LaneBitmask
-getLanesWithProperty(const LiveIntervals &LIS, const MachineRegisterInfo &MRI,
+getLanesWithProperty(LiveIntervals &LIS, const MachineRegisterInfo &MRI,
bool TrackLaneMasks, VirtRegOrUnit VRegOrUnit,
SlotIndex Pos, LaneBitmask SafeDefault,
- bool (*Property)(const LiveRange &LR, SlotIndex Pos)) {
+ bool (*Property)(const LiveRange &LR, SlotIndex Pos),
+ bool ComputePhysRegs = false) {
if (VRegOrUnit.isVirtualReg()) {
const LiveInterval &LI = LIS.getInterval(VRegOrUnit.asVirtualReg());
LaneBitmask Result;
@@ -431,22 +432,27 @@ getLanesWithProperty(const LiveIntervals &LIS, const MachineRegisterInfo &MRI,
return Result;
} else {
- const LiveRange *LR = LIS.getCachedRegUnit(VRegOrUnit.asMCRegUnit());
- // Be prepared for missing liveranges: We usually do not compute liveranges
- // for physical registers on targets with many registers (GPUs).
+ MCRegUnit Unit = VRegOrUnit.asMCRegUnit();
+ // We usually do not compute liveranges for physical registers on targets
+ // with many registers (GPUs), so the cached range may be absent. Callers
+ // that require an authoritative answer pass ComputePhysRegs to force the
+ // range to be computed on demand.
+ const LiveRange *LR =
+ ComputePhysRegs ? &LIS.getRegUnit(Unit) : LIS.getCachedRegUnit(Unit);
if (LR == nullptr)
return SafeDefault;
return Property(*LR, Pos) ? LaneBitmask::getAll() : LaneBitmask::getNone();
}
}
-static LaneBitmask getLiveLanesAt(const LiveIntervals &LIS,
+static LaneBitmask getLiveLanesAt(LiveIntervals &LIS,
const MachineRegisterInfo &MRI,
bool TrackLaneMasks, VirtRegOrUnit VRegOrUnit,
- SlotIndex Pos) {
+ SlotIndex Pos, bool ComputePhysRegs = false) {
return getLanesWithProperty(
LIS, MRI, TrackLaneMasks, VRegOrUnit, Pos, LaneBitmask::getAll(),
- [](const LiveRange &LR, SlotIndex Pos) { return LR.liveAt(Pos); });
+ [](const LiveRange &LR, SlotIndex Pos) { return LR.liveAt(Pos); },
+ ComputePhysRegs);
}
namespace {
@@ -472,12 +478,9 @@ class RegisterOperandsCollector {
for (ConstMIBundleOperands OperI(MI); OperI.isValid(); ++OperI)
collectOperand(*OperI);
- // An instruction can have overlapping defs where only some carry the dead
- // flag, for example a dead super-register def alongside a live sub-register
- // def. A register unit is dead if any def covering it is dead, so subtract
- // the dead defs from the live defs.
- for (const VRegMaskOrUnit &P : RegOpers.DeadDefs)
- removeRegLanes(RegOpers.Defs, P);
+ // Remove redundant physreg dead defs.
+ for (const VRegMaskOrUnit &P : RegOpers.Defs)
+ removeRegLanes(RegOpers.DeadDefs, P);
}
void collectInstrLanes(const MachineInstr &MI) const {
@@ -573,17 +576,20 @@ void RegisterOperands::collect(const MachineInstr &MI,
}
void RegisterOperands::detectDeadDefs(const MachineInstr &MI,
- const LiveIntervals &LIS,
+ LiveIntervals &LIS,
const MachineRegisterInfo &MRI) {
SlotIndex DeadSlotIdx = LIS.getInstructionIndex(MI).getDeadSlot();
for (auto *I = Defs.begin(); I != Defs.end(); /*empty*/) {
- LaneBitmask LiveAfter = getLiveLanesAt(LIS, MRI, /*TrackLaneMasks=*/false,
- I->VRegOrUnit, DeadSlotIdx);
+ // Force physreg unit ranges to be computed, we need to accurately know if a
+ // physreg is dead.
+ LaneBitmask LiveAfter =
+ getLiveLanesAt(LIS, MRI, /*TrackLaneMasks=*/false, I->VRegOrUnit,
+ DeadSlotIdx, /*ComputePhysRegs=*/true);
I = adjustDef(*I, LiveAfter);
}
}
-void RegisterOperands::adjustLaneLiveness(const LiveIntervals &LIS,
+void RegisterOperands::adjustLaneLiveness(LiveIntervals &LIS,
const MachineRegisterInfo &MRI,
SlotIndex Pos) {
for (auto *I = Defs.begin(); I != Defs.end(); /*empty*/) {
@@ -594,7 +600,7 @@ void RegisterOperands::adjustLaneLiveness(const LiveIntervals &LIS,
adjustUses(LIS, MRI, Pos.getBaseIndex());
}
-void RegisterOperands::adjustLaneLiveness(const LiveIntervals &LIS,
+void RegisterOperands::adjustLaneLiveness(LiveIntervals &LIS,
const MachineRegisterInfo &MRI,
MachineInstr &MI) {
SlotIndex Pos = LIS.getInstructionIndex(MI);
@@ -644,7 +650,7 @@ VRegMaskOrUnit *RegisterOperands::adjustDef(VRegMaskOrUnit &Def,
return &Def + 1;
}
-void RegisterOperands::adjustUses(const LiveIntervals &LIS,
+void RegisterOperands::adjustUses(LiveIntervals &LIS,
const MachineRegisterInfo &MRI,
SlotIndex Pos) {
for (auto &[VRegOrUnit, LaneMask] : Uses) {
diff --git a/llvm/lib/CodeGen/ScheduleDAGInstrs.cpp b/llvm/lib/CodeGen/ScheduleDAGInstrs.cpp
index c929276b219f7..b59898e4cd4e7 100644
--- a/llvm/lib/CodeGen/ScheduleDAGInstrs.cpp
+++ b/llvm/lib/CodeGen/ScheduleDAGInstrs.cpp
@@ -796,6 +796,10 @@ void ScheduleDAGInstrs::buildSchedGraph(AAResults *AA,
if (TrackLaneMasks) {
SlotIndex SlotIdx = LIS->getInstructionIndex(MI);
RegOpers.adjustLaneLiveness(*LIS, MRI, SlotIdx);
+ } else if (LIS) {
+ // Detect dead defs from LiveIntervals instead of trusting operand dead
+ // flags.
+ RegOpers.detectDeadDefs(MI, *LIS, MRI);
}
if (PDiffs != nullptr)
PDiffs->addInstruction(SU->NodeNum, RegOpers, MRI);
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUNextUseAnalysis.cpp b/llvm/lib/Target/AMDGPU/AMDGPUNextUseAnalysis.cpp
index a667b96c0d089..2a19add12b8e4 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUNextUseAnalysis.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUNextUseAnalysis.cpp
@@ -2414,7 +2414,7 @@ void printDistanceFromDefToUse(json::OStream &J, const MachineFunction &MF,
void printNextUseDistancesAsJson(json::OStream &J, const MachineFunction &MF,
const AMDGPUNextUseAnalysis &NUA,
const AMDGPUNextUseAnalysisImpl &NUAImpl,
- const LiveIntervals &LIS) {
+ LiveIntervals &LIS) {
using UseDistancePair = AMDGPUNextUseAnalysis::UseDistancePair;
const Function &F = MF.getFunction();
const Module *M = F.getParent();
@@ -2501,8 +2501,7 @@ void printNextUseDistancesAsJson(json::OStream &J, const MachineFunction &MF,
void printAsJson(raw_ostream &FallbackOS, TimerGroup &JsonTimerGroup,
Timer &JsonTimer, const MachineFunction &MF,
const AMDGPUNextUseAnalysis &NUA,
- const AMDGPUNextUseAnalysisImpl &NUAImpl,
- const LiveIntervals &LIS) {
+ const AMDGPUNextUseAnalysisImpl &NUAImpl, LiveIntervals &LIS) {
std::string FN = DumpNextUseDistanceAsJson;
auto dump = [&](raw_ostream &OS) {
@@ -2551,7 +2550,7 @@ bool AMDGPUNextUseAnalysisPrinterLegacyPass::runOnMachineFunction(
Timer JsonTimer("json", "Total time spent generating json", JsonTimerGroup);
JsonTimer.startTimer();
- const LiveIntervals &LIS = getAnalysis<LiveIntervalsWrapperPass>().getLIS();
+ LiveIntervals &LIS = getAnalysis<LiveIntervalsWrapperPass>().getLIS();
const AMDGPUNextUseAnalysis &NUA =
getAnalysis<AMDGPUNextUseAnalysisLegacyPass>().getNextUseAnalysis();
@@ -2600,7 +2599,7 @@ AMDGPUNextUseAnalysisPrinterPass::run(MachineFunction &MF,
Timer JsonTimer("json", "Total time spent generating json", JsonTimerGroup);
JsonTimer.startTimer();
- const LiveIntervals &LIS = MFAM.getResult<LiveIntervalsAnalysis>(MF);
+ LiveIntervals &LIS = MFAM.getResult<LiveIntervalsAnalysis>(MF);
const AMDGPUNextUseAnalysis &NUA =
MFAM.getResult<AMDGPUNextUseAnalysisPass>(MF);
diff --git a/llvm/lib/Target/AMDGPU/GCNRegPressure.cpp b/llvm/lib/Target/AMDGPU/GCNRegPressure.cpp
index 53617e89af757..312fa4cf852ae 100644
--- a/llvm/lib/Target/AMDGPU/GCNRegPressure.cpp
+++ b/llvm/lib/Target/AMDGPU/GCNRegPressure.cpp
@@ -971,7 +971,7 @@ getRegLiveThroughMask(const MachineRegisterInfo &MRI, const LiveIntervals &LIS,
bool GCNRegPressurePrinter::runOnMachineFunction(MachineFunction &MF) {
const MachineRegisterInfo &MRI = MF.getRegInfo();
const TargetRegisterInfo *TRI = MRI.getTargetRegisterInfo();
- const LiveIntervals &LIS = getAnalysis<LiveIntervalsWrapperPass>().getLIS();
+ LiveIntervals &LIS = getAnalysis<LiveIntervalsWrapperPass>().getLIS();
auto &OS = dbgs();
diff --git a/llvm/lib/Target/AMDGPU/GCNRegPressure.h b/llvm/lib/Target/AMDGPU/GCNRegPressure.h
index 021d879732832..5c8bd5a2cd76a 100644
--- a/llvm/lib/Target/AMDGPU/GCNRegPressure.h
+++ b/llvm/lib/Target/AMDGPU/GCNRegPressure.h
@@ -323,13 +323,13 @@ class GCNRPTracker {
using LiveRegSet = DenseMap<unsigned, LaneBitmask>;
protected:
- const LiveIntervals &LIS;
+ LiveIntervals &LIS;
LiveRegSet LiveRegs;
GCNRegPressure CurPressure, MaxPressure;
const MachineInstr *LastTrackedMI = nullptr;
mutable const MachineRegisterInfo *MRI = nullptr;
- GCNRPTracker(const LiveIntervals &LIS_) : LIS(LIS_) {}
+ GCNRPTracker(LiveIntervals &LIS_) : LIS(LIS_) {}
/// Resets tracker before or \p After the provided \p MI, which can be a debug
/// instruction.
@@ -370,7 +370,7 @@ getLiveRegs(SlotIndex SI, const LiveIntervals &LIS,
class GCNUpwardRPTracker : public GCNRPTracker {
public:
- GCNUpwardRPTracker(const LiveIntervals &LIS_) : GCNRPTracker(LIS_) {}
+ GCNUpwardRPTracker(LiveIntervals &LIS_) : GCNRPTracker(LIS_) {}
using GCNRPTracker::reset;
@@ -408,7 +408,7 @@ class GCNDownwardRPTracker : public GCNRPTracker {
MachineBasicBlock::const_iterator MBBEnd;
public:
- GCNDownwardRPTracker(const LiveIntervals &LIS_) : GCNRPTracker(LIS_) {}
+ GCNDownwardRPTracker(LiveIntervals &LIS_) : GCNRPTracker(LIS_) {}
using GCNRPTracker::reset;
diff --git a/llvm/test/CodeGen/X86/masked-udiv.ll b/llvm/test/CodeGen/X86/masked-udiv.ll
index 51114074232cb..7fbddc22cc38a 100644
--- a/llvm/test/CodeGen/X86/masked-udiv.ll
+++ b/llvm/test/CodeGen/X86/masked-udiv.ll
@@ -204,40 +204,39 @@ define <2 x i64> @udiv_v2i64(<2 x i64> %x, <2 x i64> %y, <2 x i1> %m) {
define <4 x i64> @udiv_v4i64(<4 x i64> %x, <4 x i64> %y, <4 x i1> %m) {
; SSE2-LABEL: udiv_v4i64:
; SSE2: # %bb.0:
-; SSE2-NEXT: pshufd {{.*#+}} xmm6 = xmm4[0,0,1,1]
+; SSE2-NEXT: movdqa %xmm0, %xmm5
+; SSE2-NEXT: pshufd {{.*#+}} xmm6 = xmm4[2,2,3,3]
; SSE2-NEXT: pslld $31, %xmm6
; SSE2-NEXT: psrad $31, %xmm6
-; SSE2-NEXT: movdqa {{.*#+}} xmm5 = [1,1]
-; SSE2-NEXT: pand %xmm6, %xmm2
-; SSE2-NEXT: pandn %xmm5, %xmm6
-; SSE2-NEXT: por %xmm2, %xmm6
-; SSE2-NEXT: movq %xmm6, %rcx
+; SSE2-NEXT: pshufd {{.*#+}} xmm4 = xmm4[0,0,1,1]
+; SSE2-NEXT: pslld $31, %xmm4
+; SSE2-NEXT: psrad $31, %xmm4
+; SSE2-NEXT: movdqa {{.*#+}} xmm7 = [1,1]
+; SSE2-NEXT: pand %xmm4, %xmm2
+; SSE2-NEXT: pandn %xmm7, %xmm4
+; SSE2-NEXT: por %xmm2, %xmm4
+; SSE2-NEXT: movq %xmm4, %rcx
; SSE2-NEXT: movq %xmm0, %rax
; SSE2-NEXT: xorl %edx, %edx
; SSE2-NEXT: divq %rcx
-; SSE2-NEXT: movq %rax, %rcx
-; SSE2-NEXT: pshufd {{.*#+}} xmm2 = xmm6[2,3,2,3]
-; SSE2-NEXT: movq %xmm2, %rsi
-; SSE2-NEXT: pshufd {{.*#+}} xmm0 = xmm0[2,3,2,3]
-; SSE2-NEXT: movq %xmm0, %rax
+; SSE2-NEXT: movq %rax, %xmm0
+; SSE2-NEXT: pshufd {{.*#+}} xmm2 = xmm4[2,3,2,3]
+; SSE2-NEXT: movq %xmm2, %rcx
+; SSE2-NEXT: pshufd {{.*#+}} xmm2 = xmm5[2,3,2,3]
+; SSE2-NEXT: movq %xmm2, %rax
; SSE2-NEXT: xorl %edx, %edx
-; SSE2-NEXT: divq %rsi
-; SSE2-NEXT: movq %rax, %rsi
-; SSE2-NEXT: pshufd {{.*#+}} xmm4 = xmm4[2,2,3,3]
-; SSE2-NEXT: pslld $31, %xmm4
-; SSE2-NEXT: psrad $31, %xmm4
-; SSE2-NEXT: pand %xmm4, %xmm3
-; SSE2-NEXT: pandn %xmm5, %xmm4
-; SSE2-NEXT: por %xmm3, %xmm4
-; SSE2-NEXT: movq %xmm4, %rdi
+; SSE2-NEXT: divq %rcx
+; SSE2-NEXT: movq %rax, %xmm2
+; SSE2-NEXT: punpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm2[0]
+; SSE2-NEXT: pand %xmm6, %xmm3
+; SSE2-NEXT: pandn %xmm7, %xmm6
+; SSE2-NEXT: por %xmm3, %xmm6
+; SSE2-NEXT: movq %xmm6, %rcx
; SSE2-NEXT: movq %xmm1, %rax
; SSE2-NEXT: xorl %edx, %edx
-; SSE2-NEXT: divq %rdi
-; SSE2-NEXT: movq %rcx, %xmm0
-; SSE2-NEXT: movq %rsi, %xmm2
-; SSE2-NEXT: punpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm2[0]
+; SSE2-NEXT: divq %rcx
; SSE2-NEXT: movq %rax, %xmm2
-; SSE2-NEXT: pshufd {{.*#+}} xmm3 = xmm4[2,3,2,3]
+; SSE2-NEXT: pshufd {{.*#+}} xmm3 = xmm6[2,3,2,3]
; SSE2-NEXT: movq %xmm3, %rcx
; SSE2-NEXT: pshufd {{.*#+}} xmm1 = xmm1[2,3,2,3]
; SSE2-NEXT: movq %xmm1, %rax
@@ -251,39 +250,38 @@ define <4 x i64> @udiv_v4i64(<4 x i64> %x, <4 x i64> %y, <4 x i1> %m) {
; SSE42-LABEL: udiv_v4i64:
; SSE42: # %bb.0:
; SSE42-NEXT: movdqa %xmm0, %xmm5
+; SSE42-NEXT: pshufd {{.*#+}} xmm6 = xmm4[2,2,3,3]
; SSE42-NEXT: pmovzxdq {{.*#+}} xmm0 = xmm4[0],zero,xmm4[1],zero
; SSE42-NEXT: psllq $63, %xmm0
-; SSE42-NEXT: movapd {{.*#+}} xmm6 = [1,1]
-; SSE42-NEXT: movapd %xmm6, %xmm7
+; SSE42-NEXT: movapd {{.*#+}} xmm4 = [1,1]
+; SSE42-NEXT: movapd %xmm4, %xmm7
; SSE42-NEXT: blendvpd %xmm0, %xmm2, %xmm7
; SSE42-NEXT: pextrq $1, %xmm7, %rcx
; SSE42-NEXT: pextrq $1, %xmm5, %rax
+; SSE42-NEXT: psllq $63, %xmm6
; SSE42-NEXT: xorl %edx, %edx
; SSE42-NEXT: divq %rcx
-; SSE42-NEXT: movq %rax, %rcx
-; SSE42-NEXT: pshufd {{.*#+}} xmm0 = xmm4[2,2,3,3]
-; SSE42-NEXT: psllq $63, %xmm0
-; SSE42-NEXT: movq %xmm7, %rdi
-; SSE42-NEXT: blendvpd %xmm0, %xmm3, %xmm6
-; SSE42-NEXT: pextrq $1, %xmm6, %r8
-; SSE42-NEXT: pextrq $1, %xmm1, %rsi
+; SSE42-NEXT: movq %rax, %xmm8
+; SSE42-NEXT: movq %xmm7, %rcx
; SSE42-NEXT: movq %xmm5, %rax
; SSE42-NEXT: xorl %edx, %edx
-; SSE42-NEXT: divq %rdi
-; SSE42-NEXT: movq %rax, %rdi
-; SSE42-NEXT: movq %rsi, %rax
-; SSE42-NEXT: xorl %edx, %edx
-; SSE42-NEXT: divq %r8
-; SSE42-NEXT: movq %rcx, %xmm2
-; SSE42-NEXT: movq %rdi, %xmm0
-; SSE42-NEXT: punpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm2[0]
+; SSE42-NEXT: divq %rcx
; SSE42-NEXT: movq %rax, %xmm2
-; SSE42-NEXT: movq %xmm6, %rcx
+; SSE42-NEXT: movdqa %xmm6, %xmm0
+; SSE42-NEXT: blendvpd %xmm0, %xmm3, %xmm4
+; SSE42-NEXT: pextrq $1, %xmm4, %rcx
+; SSE42-NEXT: pextrq $1, %xmm1, %rax
+; SSE42-NEXT: punpcklqdq {{.*#+}} xmm2 = xmm2[0],xmm8[0]
+; SSE42-NEXT: xorl %edx, %edx
+; SSE42-NEXT: divq %rcx
+; SSE42-NEXT: movq %rax, %xmm0
+; SSE42-NEXT: movq %xmm4, %rcx
; SSE42-NEXT: movq %xmm1, %rax
; SSE42-NEXT: xorl %edx, %edx
; SSE42-NEXT: divq %rcx
; SSE42-NEXT: movq %rax, %xmm1
-; SSE42-NEXT: punpcklqdq {{.*#+}} xmm1 = xmm1[0],xmm2[0]
+; SSE42-NEXT: punpcklqdq {{.*#+}} xmm1 = xmm1[0],xmm0[0]
+; SSE42-NEXT: movdqa %xmm2, %xmm0
; SSE42-NEXT: retq
;
; AVX2-LABEL: udiv_v4i64:
@@ -303,20 +301,19 @@ define <4 x i64> @udiv_v4i64(<4 x i64> %x, <4 x i64> %y, <4 x i1> %m) {
; AVX2-NEXT: vmovq %xmm3, %rax
; AVX2-NEXT: xorl %edx, %edx
; AVX2-NEXT: divq %rsi
-; AVX2-NEXT: movq %rax, %rsi
-; AVX2-NEXT: vpextrq $1, %xmm1, %rdi
+; AVX2-NEXT: vmovq %rcx, %xmm2
+; AVX2-NEXT: vpextrq $1, %xmm1, %rcx
+; AVX2-NEXT: vmovq %rax, %xmm3
; AVX2-NEXT: vpextrq $1, %xmm0, %rax
; AVX2-NEXT: xorl %edx, %edx
-; AVX2-NEXT: divq %rdi
-; AVX2-NEXT: movq %rax, %rdi
-; AVX2-NEXT: vmovq %rcx, %xmm2
-; AVX2-NEXT: vmovq %rsi, %xmm3
+; AVX2-NEXT: divq %rcx
+; AVX2-NEXT: movq %rax, %rcx
; AVX2-NEXT: vpunpcklqdq {{.*#+}} xmm2 = xmm3[0],xmm2[0]
-; AVX2-NEXT: vmovq %xmm1, %rcx
+; AVX2-NEXT: vmovq %xmm1, %rsi
; AVX2-NEXT: vmovq %xmm0, %rax
; AVX2-NEXT: xorl %edx, %edx
-; AVX2-NEXT: divq %rcx
-; AVX2-NEXT: vmovq %rdi, %xmm0
+; AVX2-NEXT: divq %rsi
+; AVX2-NEXT: vmovq %rcx, %xmm0
; AVX2-NEXT: vmovq %rax, %xmm1
; AVX2-NEXT: vpunpcklqdq {{.*#+}} xmm0 = xmm1[0],xmm0[0]
; AVX2-NEXT: vinserti128 $1, %xmm2, %ymm0, %ymm0
@@ -819,181 +816,169 @@ define <3 x i10> @udiv_v3i10(<3 x i10> %x, <3 x i10> %y, <3 x i1> %m) {
define <8 x i64> @udiv_v8i64(<8 x i64> %x, <8 x i64> %y, <8 x i1> %m) {
; SSE2-LABEL: udiv_v8i64:
; SSE2: # %bb.0:
-; SSE2-NEXT: movdqa {{[0-9]+}}(%rsp), %xmm8
-; SSE2-NEXT: pshufd {{.*#+}} xmm9 = xmm8[0,0,0,0]
+; SSE2-NEXT: movdqa %xmm0, %xmm8
+; SSE2-NEXT: movdqa {{[0-9]+}}(%rsp), %xmm0
+; SSE2-NEXT: pshufd {{.*#+}} xmm9 = xmm0[3,3,3,3]
; SSE2-NEXT: pshufhw {{.*#+}} xmm9 = xmm9[0,1,2,3,5,5,5,5]
-; SSE2-NEXT: pshufd {{.*#+}} xmm10 = xmm9[0,0,2,2]
+; SSE2-NEXT: pshufd {{.*#+}} xmm9 = xmm9[0,0,2,2]
+; SSE2-NEXT: pslld $31, %xmm9
+; SSE2-NEXT: psrad $31, %xmm9
+; SSE2-NEXT: pshufd {{.*#+}} xmm10 = xmm0[2,2,2,2]
+; SSE2-NEXT: pshufhw {{.*#+}} xmm10 = xmm10[0,1,2,3,5,5,5,5]
+; SSE2-NEXT: pshufd {{.*#+}} xmm10 = xmm10[0,0,2,2]
; SSE2-NEXT: pslld $31, %xmm10
; SSE2-NEXT: psrad $31, %xmm10
-; SSE2-NEXT: movdqa {{.*#+}} xmm9 = [1,1]
-; SSE2-NEXT: pand %xmm10, %xmm4
-; SSE2-NEXT: pandn %xmm9, %xmm10
-; SSE2-NEXT: por %xmm4, %xmm10
-; SSE2-NEXT: movq %xmm10, %rcx
-; SSE2-NEXT: movq %xmm0, %rax
+; SSE2-NEXT: pshufd {{.*#+}} xmm11 = xmm0[1,1,1,1]
+; SSE2-NEXT: pshufhw {{.*#+}} xmm11 = xmm11[0,1,2,3,5,5,5,5]
+; SSE2-NEXT: pshufd {{.*#+}} xmm12 = xmm11[0,0,2,2]
+; SSE2-NEXT: pslld $31, %xmm12
+; SSE2-NEXT: psrad $31, %xmm12
+; SSE2-NEXT: pshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; SSE2-NEXT: pshufhw {{.*#+}} xmm0 = xmm0[0,1,2,3,5,5,5,5]
+; SSE2-NEXT: pshufd {{.*#+}} xmm13 = xmm0[0,0,2,2]
+; SSE2-NEXT: pslld $31, %xmm13
+; SSE2-NEXT: psrad $31, %xmm13
+; SSE2-NEXT: movdqa {{.*#+}} xmm11 = [1,1]
+; SSE2-NEXT: pand %xmm13, %xmm4
+; SSE2-NEXT: pandn %xmm11, %xmm13
+; SSE2-NEXT: por %xmm4, %xmm13
+; SSE2-NEXT: movq %xmm13, %rcx
+; SSE2-NEXT: movq %xmm8, %rax
; SSE2-NEXT: xorl %edx, %edx
; SSE2-NEXT: divq %rcx
-; SSE2-NEXT: movq %rax, %rcx
-; SSE2-NEXT: pshufd {{.*#+}} xmm4 = xmm10[2,3,2,3]
-; SSE2-NEXT: movq %xmm4, %rsi
-; SSE2-NEXT: pshufd {{.*#+}} xmm0 = xmm0[2,3,2,3]
-; SSE2-NEXT: movq %xmm0, %rax
+; SSE2-NEXT: movq %rax, %xmm0
+; SSE2-NEXT: pshufd {{.*#+}} xmm4 = xmm13[2,3,2,3]
+; SSE2-NEXT: movq %xmm4, %rcx
+; SSE2-NEXT: pshufd {{.*#+}} xmm4 = xmm8[2,3,2,3]
+; SSE2-NEXT: movq %xmm4, %rax
; SSE2-NEXT: xorl %edx, %edx
-; SSE2-NEXT: divq %rsi
-; SSE2-NEXT: movq %rax, %rsi
-; SSE2-NEXT: pshufd {{.*#+}} xmm0 = xmm8[1,1,1,1]
-; SSE2-NEXT: pshufhw {{.*#+}} xmm0 = xmm0[0,1,2,3,5,5,5,5]
-; SSE2-NEXT: pshufd {{.*#+}} xmm0 = xmm0[0,0,2,2]
-; SSE2-NEXT: pslld $31, %xmm0
-; SSE2-NEXT: psrad $31, %xmm0
-; SSE2-NEXT: pand %xmm0, %xmm5
-; SSE2-NEXT: pandn %xmm9, %xmm0
-; SSE2-NEXT: por %xmm5, %xmm0
-; SSE2-NEXT: movq %xmm0, %rdi
+; SSE2-NEXT: divq %rcx
+; SSE2-NEXT: movq %rax, %xmm4
+; SSE2-NEXT: punpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm4[0]
+; SSE2-NEXT: pand %xmm12, %xmm5
+; SSE2-NEXT: pandn %xmm11, %xmm12
+; SSE2-NEXT: por %xmm5, %xmm12
+; SSE2-NEXT: movq %xmm12, %rcx
; SSE2-NEXT: movq %xmm1, %rax
; SSE2-NEXT: xorl %edx, %edx
-; SSE2-NEXT: divq %rdi
-; SSE2-NEXT: movq %rax, %rdi
-; SSE2-NEXT: pshufd {{.*#+}} xmm0 = xmm0[2,3,2,3]
-; SSE2-NEXT: movq %xmm0, %r8
-; SSE2-NEXT: pshufd {{.*#+}} xmm0 = xmm1[2,3,2,3]
-; SSE2-NEXT: movq %xmm0, %rax
+; SSE2-NEXT: divq %rcx
+; SSE2-NEXT: movq %rax, %xmm4
+; SSE2-NEXT: pshufd {{.*#+}} xmm5 = xmm12[2,3,2,3]
+; SSE2-NEXT: movq %xmm5, %rcx
+; SSE2-NEXT: pshufd {{.*#+}} xmm1 = xmm1[2,3,2,3]
+; SSE2-NEXT: movq %xmm1, %rax
; SSE2-NEXT: xorl %edx, %edx
-; SSE2-NEXT: divq %r8
-; SSE2-NEXT: movq %rax, %r8
-; SSE2-NEXT: pshufd {{.*#+}} xmm0 = xmm8[2,2,2,2]
-; SSE2-NEXT: pshufhw {{.*#+}} xmm0 = xmm0[0,1,2,3,5,5,5,5]
-; SSE2-NEXT: pshufd {{.*#+}} xmm0 = xmm0[0,0,2,2]
-; SSE2-NEXT: pslld $31, %xmm0
-; SSE2-NEXT: psrad $31, %xmm0
-; SSE2-NEXT: pand %xmm0, %xmm6
-; SSE2-NEXT: pandn %xmm9, %xmm0
-; SSE2-NEXT: por %xmm6, %xmm0
-; SSE2-NEXT: movq %xmm0, %r9
+; SSE2-NEXT: divq %rcx
+; SSE2-NEXT: movq %rax, %xmm1
+; SSE2-NEXT: punpcklqdq {{.*#+}} xmm4 = xmm4[0],xmm1[0]
+; SSE2-NEXT: pand %xmm10, %xmm6
+; SSE2-NEXT: pandn %xmm11, %xmm10
+; SSE2-NEXT: por %xmm6, %xmm10
+; SSE2-NEXT: movq %xmm10, %rcx
; SSE2-NEXT: movq %xmm2, %rax
; SSE2-NEXT: xorl %edx, %edx
-; SSE2-NEXT: divq %r9
-; SSE2-NEXT: movq %rax, %r9
-; SSE2-NEXT: pshufd {{.*#+}} xmm0 = xmm0[2,3,2,3]
-; SSE2-NEXT: movq %xmm0, %r10
-; SSE2-NEXT: pshufd {{.*#+}} xmm0 = xmm2[2,3,2,3]
-; SSE2-NEXT: movq %xmm0, %rax
+; SSE2-NEXT: divq %rcx
+; SSE2-NEXT: movq %rax, %xmm5
+; SSE2-NEXT: pshufd {{.*#+}} xmm1 = xmm10[2,3,2,3]
+; SSE2-NEXT: movq %xmm1, %rcx
+; SSE2-NEXT: pshufd {{.*#+}} xmm1 = xmm2[2,3,2,3]
+; SSE2-NEXT: movq %xmm1, %rax
; SSE2-NEXT: xorl %edx, %edx
-; SSE2-NEXT: divq %r10
-; SSE2-NEXT: movq %rax, %r10
-; SSE2-NEXT: pshufd {{.*#+}} xmm0 = xmm8[3,3,3,3]
-; SSE2-NEXT: pshufhw {{.*#+}} xmm0 = xmm0[0,1,2,3,5,5,5,5]
-; SSE2-NEXT: pshufd {{.*#+}} xmm5 = xmm0[0,0,2,2]
-; SSE2-NEXT: pslld $31, %xmm5
-; SSE2-NEXT: psrad $31, %xmm5
-; SSE2-NEXT: pand %xmm5, %xmm7
-; SSE2-NEXT: pandn %xmm9, %xmm5
-; SSE2-NEXT: por %xmm7, %xmm5
-; SSE2-NEXT: movq %xmm5, %r11
+; SSE2-NEXT: divq %rcx
+; SSE2-NEXT: movq %rax, %xmm1
+; SSE2-NEXT: punpcklqdq {{.*#+}} xmm5 = xmm5[0],xmm1[0]
+; SSE2-NEXT: pand %xmm9, %xmm7
+; SSE2-NEXT: pandn %xmm11, %xmm9
+; SSE2-NEXT: por %xmm7, %xmm9
+; SSE2-NEXT: movq %xmm9, %rcx
; SSE2-NEXT: movq %xmm3, %rax
; SSE2-NEXT: xorl %edx, %edx
-; SSE2-NEXT: divq %r11
-; SSE2-NEXT: movq %rcx, %xmm0
-; SSE2-NEXT: movq %rsi, %xmm4
-; SSE2-NEXT: movq %rdi, %xmm1
-; SSE2-NEXT: movq %r8, %xmm6
-; SSE2-NEXT: movq %r9, %xmm2
-; SSE2-NEXT: punpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm4[0]
-; SSE2-NEXT: punpcklqdq {{.*#+}} xmm1 = xmm1[0],xmm6[0]
-; SSE2-NEXT: movq %r10, %xmm4
-; SSE2-NEXT: punpcklqdq {{.*#+}} xmm2 = xmm2[0],xmm4[0]
-; SSE2-NEXT: movq %rax, %xmm4
-; SSE2-NEXT: pshufd {{.*#+}} xmm5 = xmm5[2,3,2,3]
-; SSE2-NEXT: movq %xmm5, %rcx
-; SSE2-NEXT: pshufd {{.*#+}} xmm3 = xmm3[2,3,2,3]
-; SSE2-NEXT: movq %xmm3, %rax
+; SSE2-NEXT: divq %rcx
+; SSE2-NEXT: movq %rax, %xmm6
+; SSE2-NEXT: pshufd {{.*#+}} xmm1 = xmm9[2,3,2,3]
+; SSE2-NEXT: movq %xmm1, %rcx
+; SSE2-NEXT: pshufd {{.*#+}} xmm1 = xmm3[2,3,2,3]
+; SSE2-NEXT: movq %xmm1, %rax
; SSE2-NEXT: xorl %edx, %edx
; SSE2-NEXT: divq %rcx
-; SSE2-NEXT: movq %rax, %xmm3
-; SSE2-NEXT: punpcklqdq {{.*#+}} xmm4 = xmm4[0],xmm3[0]
-; SSE2-NEXT: movdqa %xmm4, %xmm3
+; SSE2-NEXT: movq %rax, %xmm1
+; SSE2-NEXT: punpcklqdq {{.*#+}} xmm6 = xmm6[0],xmm1[0]
+; SSE2-NEXT: movdqa %xmm4, %xmm1
+; SSE2-NEXT: movdqa %xmm5, %xmm2
+; SSE2-NEXT: movdqa %xmm6, %xmm3
; SSE2-NEXT: retq
;
; SSE42-LABEL: udiv_v8i64:
; SSE42: # %bb.0:
-; SSE42-NEXT: pushq %rbx
-; SSE42-NEXT: .cfi_def_cfa_offset 16
-; SSE42-NEXT: .cfi_offset %rbx, -16
-; SSE42-NEXT: movdqa %xmm0, %xmm8
-; SSE42-NEXT: movdqa {{[0-9]+}}(%rsp), %xmm10
-; SSE42-NEXT: pmovzxwq {{.*#+}} xmm0 = xmm10[0],zero,zero,zero,xmm10[1],zero,zero,zero
+; SSE42-NEXT: movdqa %xmm0, %xmm11
+; SSE42-NEXT: movdqa {{[0-9]+}}(%rsp), %xmm0
+; SSE42-NEXT: pshufd {{.*#+}} xmm8 = xmm0[3,3,3,3]
+; SSE42-NEXT: pmovzxwq {{.*#+}} xmm8 = xmm8[0],zero,zero,zero,xmm8[1],zero,zero,zero
+; SSE42-NEXT: psllq $63, %xmm8
+; SSE42-NEXT: pshufd {{.*#+}} xmm9 = xmm0[2,3,2,3]
+; SSE42-NEXT: pmovzxwq {{.*#+}} xmm9 = xmm9[0],zero,zero,zero,xmm9[1],zero,zero,zero
+; SSE42-NEXT: psllq $63, %xmm9
+; SSE42-NEXT: pshufd {{.*#+}} xmm10 = xmm0[1,1,1,1]
+; SSE42-NEXT: pmovzxwq {{.*#+}} xmm10 = xmm10[0],zero,zero,zero,xmm10[1],zero,zero,zero
+; SSE42-NEXT: pmovzxwq {{.*#+}} xmm0 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero
; SSE42-NEXT: psllq $63, %xmm0
-; SSE42-NEXT: movapd {{.*#+}} xmm9 = [1,1]
-; SSE42-NEXT: movapd %xmm9, %xmm11
-; SSE42-NEXT: blendvpd %xmm0, %xmm4, %xmm11
-; SSE42-NEXT: pextrq $1, %xmm11, %rcx
-; SSE42-NEXT: pextrq $1, %xmm8, %rax
+; SSE42-NEXT: movapd {{.*#+}} xmm12 = [1,1]
+; SSE42-NEXT: movapd %xmm12, %xmm13
+; SSE42-NEXT: blendvpd %xmm0, %xmm4, %xmm13
+; SSE42-NEXT: pextrq $1, %xmm13, %rcx
+; SSE42-NEXT: pextrq $1, %xmm11, %rax
+; SSE42-NEXT: psllq $63, %xmm10
; SSE42-NEXT: xorl %edx, %edx
; SSE42-NEXT: divq %rcx
-; SSE42-NEXT: movq %rax, %rcx
-; SSE42-NEXT: movq %xmm11, %rsi
-; SSE42-NEXT: movq %xmm8, %rax
+; SSE42-NEXT: movq %rax, %xmm0
+; SSE42-NEXT: movq %xmm13, %rcx
+; SSE42-NEXT: movq %xmm11, %rax
; SSE42-NEXT: xorl %edx, %edx
-; SSE42-NEXT: divq %rsi
-; SSE42-NEXT: movq %rax, %rsi
-; SSE42-NEXT: pshufd {{.*#+}} xmm0 = xmm10[1,1,1,1]
-; SSE42-NEXT: pmovzxwq {{.*#+}} xmm0 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero
-; SSE42-NEXT: psllq $63, %xmm0
-; SSE42-NEXT: movapd %xmm9, %xmm4
-; SSE42-NEXT: blendvpd %xmm0, %xmm5, %xmm4
-; SSE42-NEXT: pextrq $1, %xmm4, %rdi
+; SSE42-NEXT: divq %rcx
+; SSE42-NEXT: movq %rax, %xmm4
+; SSE42-NEXT: punpcklqdq {{.*#+}} xmm4 = xmm4[0],xmm0[0]
+; SSE42-NEXT: movapd %xmm12, %xmm11
+; SSE42-NEXT: movdqa %xmm10, %xmm0
+; SSE42-NEXT: blendvpd %xmm0, %xmm5, %xmm11
+; SSE42-NEXT: pextrq $1, %xmm11, %rcx
; SSE42-NEXT: pextrq $1, %xmm1, %rax
; SSE42-NEXT: xorl %edx, %edx
-; SSE42-NEXT: divq %rdi
-; SSE42-NEXT: movq %rax, %rdi
-; SSE42-NEXT: pshufd {{.*#+}} xmm0 = xmm10[2,3,2,3]
-; SSE42-NEXT: pmovzxwq {{.*#+}} xmm0 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero
-; SSE42-NEXT: psllq $63, %xmm0
-; SSE42-NEXT: movq %xmm4, %r8
-; SSE42-NEXT: movapd %xmm9, %xmm4
-; SSE42-NEXT: blendvpd %xmm0, %xmm6, %xmm4
-; SSE42-NEXT: pextrq $1, %xmm4, %r10
-; SSE42-NEXT: pextrq $1, %xmm2, %r9
+; SSE42-NEXT: divq %rcx
+; SSE42-NEXT: movq %rax, %xmm5
+; SSE42-NEXT: movq %xmm11, %rcx
; SSE42-NEXT: movq %xmm1, %rax
; SSE42-NEXT: xorl %edx, %edx
-; SSE42-NEXT: divq %r8
-; SSE42-NEXT: movq %rax, %r8
-; SSE42-NEXT: movq %r9, %rax
+; SSE42-NEXT: divq %rcx
+; SSE42-NEXT: movq %rax, %xmm1
+; SSE42-NEXT: movapd %xmm12, %xmm10
+; SSE42-NEXT: movdqa %xmm9, %xmm0
+; SSE42-NEXT: blendvpd %xmm0, %xmm6, %xmm10
+; SSE42-NEXT: pextrq $1, %xmm10, %rcx
+; SSE42-NEXT: punpcklqdq {{.*#+}} xmm1 = xmm1[0],xmm5[0]
+; SSE42-NEXT: pextrq $1, %xmm2, %rax
; SSE42-NEXT: xorl %edx, %edx
-; SSE42-NEXT: divq %r10
-; SSE42-NEXT: movq %rax, %r9
-; SSE42-NEXT: pshufd {{.*#+}} xmm0 = xmm10[3,3,3,3]
-; SSE42-NEXT: pmovzxwq {{.*#+}} xmm0 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero
-; SSE42-NEXT: psllq $63, %xmm0
-; SSE42-NEXT: movq %xmm4, %r11
-; SSE42-NEXT: blendvpd %xmm0, %xmm7, %xmm9
-; SSE42-NEXT: pextrq $1, %xmm9, %rbx
-; SSE42-NEXT: pextrq $1, %xmm3, %r10
+; SSE42-NEXT: divq %rcx
+; SSE42-NEXT: movq %rax, %xmm5
+; SSE42-NEXT: movq %xmm10, %rcx
; SSE42-NEXT: movq %xmm2, %rax
; SSE42-NEXT: xorl %edx, %edx
-; SSE42-NEXT: divq %r11
-; SSE42-NEXT: movq %rax, %r11
-; SSE42-NEXT: movq %r10, %rax
-; SSE42-NEXT: xorl %edx, %edx
-; SSE42-NEXT: divq %rbx
-; SSE42-NEXT: movq %rcx, %xmm2
-; SSE42-NEXT: movq %rsi, %xmm0
-; SSE42-NEXT: movq %rdi, %xmm4
-; SSE42-NEXT: movq %r8, %xmm1
-; SSE42-NEXT: movq %r9, %xmm5
-; SSE42-NEXT: punpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm2[0]
-; SSE42-NEXT: punpcklqdq {{.*#+}} xmm1 = xmm1[0],xmm4[0]
-; SSE42-NEXT: movq %r11, %xmm2
+; SSE42-NEXT: divq %rcx
+; SSE42-NEXT: movq %rax, %xmm2
+; SSE42-NEXT: movdqa %xmm8, %xmm0
+; SSE42-NEXT: blendvpd %xmm0, %xmm7, %xmm12
+; SSE42-NEXT: pextrq $1, %xmm12, %rcx
; SSE42-NEXT: punpcklqdq {{.*#+}} xmm2 = xmm2[0],xmm5[0]
-; SSE42-NEXT: movq %rax, %xmm4
-; SSE42-NEXT: movq %xmm9, %rcx
+; SSE42-NEXT: pextrq $1, %xmm3, %rax
+; SSE42-NEXT: xorl %edx, %edx
+; SSE42-NEXT: divq %rcx
+; SSE42-NEXT: movq %rax, %xmm0
+; SSE42-NEXT: movq %xmm12, %rcx
; SSE42-NEXT: movq %xmm3, %rax
; SSE42-NEXT: xorl %edx, %edx
; SSE42-NEXT: divq %rcx
; SSE42-NEXT: movq %rax, %xmm3
-; SSE42-NEXT: punpcklqdq {{.*#+}} xmm3 = xmm3[0],xmm4[0]
-; SSE42-NEXT: popq %rbx
-; SSE42-NEXT: .cfi_def_cfa_offset 8
+; SSE42-NEXT: punpcklqdq {{.*#+}} xmm3 = xmm3[0],xmm0[0]
+; SSE42-NEXT: movdqa %xmm4, %xmm0
; SSE42-NEXT: retq
;
; AVX2-LABEL: udiv_v8i64:
@@ -1010,57 +995,55 @@ define <8 x i64> @udiv_v8i64(<8 x i64> %x, <8 x i64> %y, <8 x i1> %m) {
; AVX2-NEXT: xorl %edx, %edx
; AVX2-NEXT: divq %rcx
; AVX2-NEXT: movq %rax, %rcx
+; AVX2-NEXT: vpunpckhwd {{.*#+}} xmm4 = xmm4[4,4,5,5,6,6,7,7]
; AVX2-NEXT: vmovq %xmm6, %rsi
; AVX2-NEXT: vmovq %xmm7, %rax
; AVX2-NEXT: xorl %edx, %edx
; AVX2-NEXT: divq %rsi
; AVX2-NEXT: movq %rax, %rsi
+; AVX2-NEXT: vpslld $31, %xmm4, %xmm4
+; AVX2-NEXT: vpmovsxdq %xmm4, %ymm4
; AVX2-NEXT: vpextrq $1, %xmm2, %rdi
+; AVX2-NEXT: vmovq %rcx, %xmm6
; AVX2-NEXT: vpextrq $1, %xmm0, %rax
; AVX2-NEXT: xorl %edx, %edx
; AVX2-NEXT: divq %rdi
-; AVX2-NEXT: movq %rax, %rdi
-; AVX2-NEXT: vmovq %xmm2, %r8
+; AVX2-NEXT: movq %rax, %rcx
+; AVX2-NEXT: vmovq %rsi, %xmm7
+; AVX2-NEXT: vmovq %xmm2, %rsi
; AVX2-NEXT: vmovq %xmm0, %rax
; AVX2-NEXT: xorl %edx, %edx
-; AVX2-NEXT: divq %r8
-; AVX2-NEXT: movq %rax, %r8
-; AVX2-NEXT: vpunpckhwd {{.*#+}} xmm0 = xmm4[4,4,5,5,6,6,7,7]
-; AVX2-NEXT: vpslld $31, %xmm0, %xmm0
-; AVX2-NEXT: vpmovsxdq %xmm0, %ymm0
-; AVX2-NEXT: vblendvpd %ymm0, %ymm3, %ymm5, %ymm2
-; AVX2-NEXT: vextractf128 $1, %ymm2, %xmm0
-; AVX2-NEXT: vpextrq $1, %xmm0, %r9
-; AVX2-NEXT: vextracti128 $1, %ymm1, %xmm3
-; AVX2-NEXT: vpextrq $1, %xmm3, %rax
+; AVX2-NEXT: divq %rsi
+; AVX2-NEXT: vpunpcklqdq {{.*#+}} xmm0 = xmm7[0],xmm6[0]
+; AVX2-NEXT: vmovq %rcx, %xmm6
+; AVX2-NEXT: vblendvpd %ymm4, %ymm3, %ymm5, %ymm2
+; AVX2-NEXT: vmovq %rax, %xmm3
+; AVX2-NEXT: vextractf128 $1, %ymm2, %xmm4
+; AVX2-NEXT: vpextrq $1, %xmm4, %rcx
+; AVX2-NEXT: vextracti128 $1, %ymm1, %xmm5
+; AVX2-NEXT: vpextrq $1, %xmm5, %rax
; AVX2-NEXT: xorl %edx, %edx
-; AVX2-NEXT: divq %r9
-; AVX2-NEXT: movq %rax, %r9
-; AVX2-NEXT: vmovq %xmm0, %r10
-; AVX2-NEXT: vmovq %xmm3, %rax
+; AVX2-NEXT: divq %rcx
+; AVX2-NEXT: movq %rax, %rcx
+; AVX2-NEXT: vpunpcklqdq {{.*#+}} xmm3 = xmm3[0],xmm6[0]
+; AVX2-NEXT: vmovq %xmm4, %rsi
+; AVX2-NEXT: vmovq %xmm5, %rax
; AVX2-NEXT: xorl %edx, %edx
-; AVX2-NEXT: divq %r10
-; AVX2-NEXT: movq %rax, %r10
-; AVX2-NEXT: vpextrq $1, %xmm2, %r11
+; AVX2-NEXT: divq %rsi
+; AVX2-NEXT: vinserti128 $1, %xmm0, %ymm3, %ymm0
+; AVX2-NEXT: vmovq %rcx, %xmm3
+; AVX2-NEXT: vpextrq $1, %xmm2, %rcx
+; AVX2-NEXT: vmovq %rax, %xmm4
; AVX2-NEXT: vpextrq $1, %xmm1, %rax
; AVX2-NEXT: xorl %edx, %edx
-; AVX2-NEXT: divq %r11
-; AVX2-NEXT: movq %rax, %r11
-; AVX2-NEXT: vmovq %rcx, %xmm0
-; AVX2-NEXT: vmovq %rsi, %xmm3
-; AVX2-NEXT: vmovq %rdi, %xmm4
-; AVX2-NEXT: vmovq %r8, %xmm5
-; AVX2-NEXT: vpunpcklqdq {{.*#+}} xmm0 = xmm3[0],xmm0[0]
-; AVX2-NEXT: vpunpcklqdq {{.*#+}} xmm3 = xmm5[0],xmm4[0]
-; AVX2-NEXT: vinserti128 $1, %xmm0, %ymm3, %ymm0
-; AVX2-NEXT: vmovq %r9, %xmm3
-; AVX2-NEXT: vmovq %r10, %xmm4
+; AVX2-NEXT: divq %rcx
+; AVX2-NEXT: movq %rax, %rcx
; AVX2-NEXT: vpunpcklqdq {{.*#+}} xmm3 = xmm4[0],xmm3[0]
-; AVX2-NEXT: vmovq %xmm2, %rcx
+; AVX2-NEXT: vmovq %xmm2, %rsi
; AVX2-NEXT: vmovq %xmm1, %rax
; AVX2-NEXT: xorl %edx, %edx
-; AVX2-NEXT: divq %rcx
-; AVX2-NEXT: vmovq %r11, %xmm1
+; AVX2-NEXT: divq %rsi
+; AVX2-NEXT: vmovq %rcx, %xmm1
; AVX2-NEXT: vmovq %rax, %xmm2
; AVX2-NEXT: vpunpcklqdq {{.*#+}} xmm1 = xmm2[0],xmm1[0]
; AVX2-NEXT: vinserti128 $1, %xmm3, %ymm1, %ymm1
diff --git a/llvm/test/CodeGen/X86/min-legal-vector-width.ll b/llvm/test/CodeGen/X86/min-legal-vector-width.ll
index b4f5a50ba1b46..5d80435618dd5 100644
--- a/llvm/test/CodeGen/X86/min-legal-vector-width.ll
+++ b/llvm/test/CodeGen/X86/min-legal-vector-width.ll
@@ -2072,54 +2072,54 @@ define <8 x i64> @udiv_v8i64_prefer256(<8 x i64> %x, <8 x i64> %y) nounwind "min
; CHECK-NEXT: vpextrq $1, %xmm5, %rcx
; CHECK-NEXT: xorl %edx, %edx
; CHECK-NEXT: divq %rcx
-; CHECK-NEXT: movq %rax, %rcx
+; CHECK-NEXT: movq %rax, %rdi
; CHECK-NEXT: vmovq %xmm4, %rax
-; CHECK-NEXT: vmovq %xmm5, %rsi
+; CHECK-NEXT: vmovq %xmm5, %rcx
; CHECK-NEXT: xorl %edx, %edx
-; CHECK-NEXT: divq %rsi
-; CHECK-NEXT: movq %rax, %rsi
+; CHECK-NEXT: divq %rcx
+; CHECK-NEXT: movq %rax, %rcx
; CHECK-NEXT: vpextrq $1, %xmm0, %rax
-; CHECK-NEXT: vpextrq $1, %xmm2, %rdi
+; CHECK-NEXT: vpextrq $1, %xmm2, %rsi
; CHECK-NEXT: xorl %edx, %edx
-; CHECK-NEXT: divq %rdi
-; CHECK-NEXT: movq %rax, %rdi
+; CHECK-NEXT: divq %rsi
+; CHECK-NEXT: movq %rax, %rsi
; CHECK-NEXT: vmovq %xmm0, %rax
; CHECK-NEXT: vmovq %xmm2, %r8
; CHECK-NEXT: xorl %edx, %edx
; CHECK-NEXT: divq %r8
; CHECK-NEXT: movq %rax, %r8
-; CHECK-NEXT: vextracti128 $1, %ymm1, %xmm0
-; CHECK-NEXT: vpextrq $1, %xmm0, %rax
-; CHECK-NEXT: vextracti128 $1, %ymm3, %xmm2
-; CHECK-NEXT: vpextrq $1, %xmm2, %r9
-; CHECK-NEXT: xorl %edx, %edx
-; CHECK-NEXT: divq %r9
-; CHECK-NEXT: movq %rax, %r9
-; CHECK-NEXT: vmovq %xmm0, %rax
-; CHECK-NEXT: vmovq %xmm2, %r10
+; CHECK-NEXT: vmovq %rdi, %xmm0
+; CHECK-NEXT: vextracti128 $1, %ymm1, %xmm2
+; CHECK-NEXT: vpextrq $1, %xmm2, %rax
+; CHECK-NEXT: vextracti128 $1, %ymm3, %xmm4
+; CHECK-NEXT: vpextrq $1, %xmm4, %rdi
; CHECK-NEXT: xorl %edx, %edx
-; CHECK-NEXT: divq %r10
-; CHECK-NEXT: movq %rax, %r10
-; CHECK-NEXT: vpextrq $1, %xmm1, %rax
-; CHECK-NEXT: vpextrq $1, %xmm3, %r11
+; CHECK-NEXT: divq %rdi
+; CHECK-NEXT: movq %rax, %rdi
+; CHECK-NEXT: vmovq %rcx, %xmm5
+; CHECK-NEXT: vmovq %xmm2, %rax
+; CHECK-NEXT: vmovq %xmm4, %rcx
; CHECK-NEXT: xorl %edx, %edx
-; CHECK-NEXT: divq %r11
-; CHECK-NEXT: movq %rax, %r11
-; CHECK-NEXT: vmovq %rcx, %xmm0
+; CHECK-NEXT: divq %rcx
+; CHECK-NEXT: movq %rax, %rcx
+; CHECK-NEXT: vpunpcklqdq {{.*#+}} xmm0 = xmm5[0],xmm0[0]
; CHECK-NEXT: vmovq %rsi, %xmm2
-; CHECK-NEXT: vmovq %rdi, %xmm4
-; CHECK-NEXT: vpunpcklqdq {{.*#+}} xmm0 = xmm2[0],xmm0[0]
-; CHECK-NEXT: vmovq %r8, %xmm2
-; CHECK-NEXT: vpunpcklqdq {{.*#+}} xmm2 = xmm2[0],xmm4[0]
+; CHECK-NEXT: vmovq %r8, %xmm4
+; CHECK-NEXT: vpunpcklqdq {{.*#+}} xmm2 = xmm4[0],xmm2[0]
+; CHECK-NEXT: vpextrq $1, %xmm1, %rax
; CHECK-NEXT: vinserti128 $1, %xmm0, %ymm2, %ymm0
-; CHECK-NEXT: vmovq %r9, %xmm2
+; CHECK-NEXT: vpextrq $1, %xmm3, %rsi
+; CHECK-NEXT: xorl %edx, %edx
+; CHECK-NEXT: divq %rsi
+; CHECK-NEXT: movq %rax, %rsi
+; CHECK-NEXT: vmovq %rdi, %xmm2
; CHECK-NEXT: vmovq %xmm1, %rax
-; CHECK-NEXT: vmovq %xmm3, %rcx
+; CHECK-NEXT: vmovq %xmm3, %rdi
; CHECK-NEXT: xorl %edx, %edx
-; CHECK-NEXT: divq %rcx
-; CHECK-NEXT: vmovq %r10, %xmm1
+; CHECK-NEXT: divq %rdi
+; CHECK-NEXT: vmovq %rcx, %xmm1
; CHECK-NEXT: vpunpcklqdq {{.*#+}} xmm1 = xmm1[0],xmm2[0]
-; CHECK-NEXT: vmovq %r11, %xmm2
+; CHECK-NEXT: vmovq %rsi, %xmm2
; CHECK-NEXT: vmovq %rax, %xmm3
; CHECK-NEXT: vpunpcklqdq {{.*#+}} xmm2 = xmm3[0],xmm2[0]
; CHECK-NEXT: vinserti128 $1, %xmm1, %ymm2, %ymm1
diff --git a/llvm/test/CodeGen/X86/misched-pressure-dead-physreg-superreg.mir b/llvm/test/CodeGen/X86/misched-pressure-dead-physreg-superreg.mir
new file mode 100644
index 0000000000000..f445c938d196e
--- /dev/null
+++ b/llvm/test/CodeGen/X86/misched-pressure-dead-physreg-superreg.mir
@@ -0,0 +1,60 @@
+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -mtriple=x86_64-- -run-pass=machine-scheduler -o - %s | FileCheck %s
+
+# MUL8r defines $al and its superregister $ax, but only $al is used so
+# the $ax def is dead without a dead flag on the operand. When the
+# machine scheduler builds the PressureDiff from LiveIntervals (not
+# from operand dead flags), the overlapping $al/$ax register units
+# must be recognized as a dead def, otherwise the pressure delta
+# underflows and trips the "PSet overflow/underflow" assertion in
+# getUpwardPressureDelta.
+
+---
+name: func4
+tracksRegLiveness: true
+body: |
+ bb.0:
+ liveins: $edi, $esi, $edx
+
+ ; CHECK-LABEL: name: func4
+ ; CHECK: liveins: $edi, $esi, $edx
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: [[COPY:%[0-9]+]]:gr32 = COPY $edx
+ ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gr32 = COPY $esi
+ ; CHECK-NEXT: [[COPY2:%[0-9]+]]:gr32 = COPY $edi
+ ; CHECK-NEXT: $al = COPY [[COPY1]].sub_8bit
+ ; CHECK-NEXT: MUL8r [[COPY]].sub_8bit, implicit-def $al, implicit-def dead $eflags, implicit-def $ax, implicit $al
+ ; CHECK-NEXT: [[COPY3:%[0-9]+]]:gr8 = COPY $al
+ ; CHECK-NEXT: [[COPY3:%[0-9]+]]:gr8 = SHL8ri [[COPY3]], 4, implicit-def dead $eflags
+ ; CHECK-NEXT: [[COPY3:%[0-9]+]]:gr8 = SAR8ri [[COPY3]], 4, implicit-def dead $eflags
+ ; CHECK-NEXT: [[COPY2:%[0-9]+]].sub_8bit:gr32 = SUB8rr [[COPY2]].sub_8bit, [[COPY3]], implicit-def dead $eflags
+ ; CHECK-NEXT: [[MOVZX32rr8_:%[0-9]+]]:gr32 = MOVZX32rr8 [[COPY2]].sub_8bit
+ ; CHECK-NEXT: CMP8ri [[MOVZX32rr8_]].sub_8bit, 7, implicit-def $eflags
+ ; CHECK-NEXT: [[MOV32ri:%[0-9]+]]:gr32 = MOV32ri 7
+ ; CHECK-NEXT: [[MOV32ri:%[0-9]+]]:gr32 = CMOV32rr [[MOV32ri]], [[MOVZX32rr8_]], 12, implicit $eflags
+ ; CHECK-NEXT: CMP8ri [[MOV32ri]].sub_8bit, -7, implicit-def $eflags
+ ; CHECK-NEXT: [[MOV32ri1:%[0-9]+]]:gr32 = MOV32ri 248
+ ; CHECK-NEXT: [[MOV32ri1:%[0-9]+]]:gr32 = CMOV32rr [[MOV32ri1]], [[MOV32ri]], 13, implicit $eflags
+ ; CHECK-NEXT: [[MOVSX32rr8_:%[0-9]+]]:gr32 = MOVSX32rr8 [[MOV32ri1]].sub_8bit
+ ; CHECK-NEXT: $eax = COPY [[MOVSX32rr8_]]
+ ; CHECK-NEXT: RET 0, $eax
+ %2:gr32 = COPY $edx
+ %1:gr32 = COPY $esi
+ %0:gr32 = COPY $edi
+ $al = COPY %1.sub_8bit
+ MUL8r %2.sub_8bit, implicit-def $al, implicit-def dead $eflags, implicit-def $ax, implicit $al
+ %8:gr8 = COPY $al
+ %8:gr8 = SHL8ri %8, 4, implicit-def dead $eflags
+ %8:gr8 = SAR8ri %8, 4, implicit-def dead $eflags
+ %0.sub_8bit:gr32 = SUB8rr %0.sub_8bit, %8, implicit-def dead $eflags
+ %10:gr32 = MOVZX32rr8 %0.sub_8bit
+ CMP8ri %10.sub_8bit, 7, implicit-def $eflags
+ %13:gr32 = MOV32ri 7
+ %13:gr32 = CMOV32rr %13, %10, 12, implicit $eflags
+ CMP8ri %13.sub_8bit, -7, implicit-def $eflags
+ %17:gr32 = MOV32ri 248
+ %17:gr32 = CMOV32rr %17, %13, 13, implicit $eflags
+ %19:gr32 = MOVSX32rr8 %17.sub_8bit
+ $eax = COPY %19
+ RET 0, $eax
+...
diff --git a/llvm/test/CodeGen/X86/ssub_sat_plus.ll b/llvm/test/CodeGen/X86/ssub_sat_plus.ll
index 8b96a8050e65e..5baf7a1dac74c 100644
--- a/llvm/test/CodeGen/X86/ssub_sat_plus.ll
+++ b/llvm/test/CodeGen/X86/ssub_sat_plus.ll
@@ -105,9 +105,9 @@ define signext i16 @func16(i16 signext %x, i16 signext %y, i16 signext %z) nounw
define signext i8 @func8(i8 signext %x, i8 signext %y, i8 signext %z) nounwind {
; X86-LABEL: func8:
; X86: # %bb.0:
+; X86-NEXT: movzbl {{[0-9]+}}(%esp), %edx
; X86-NEXT: movzbl {{[0-9]+}}(%esp), %eax
; X86-NEXT: mulb {{[0-9]+}}(%esp)
-; X86-NEXT: movzbl {{[0-9]+}}(%esp), %edx
; X86-NEXT: xorl %ecx, %ecx
; X86-NEXT: cmpb %al, %dl
; X86-NEXT: setns %cl
@@ -140,9 +140,9 @@ define signext i8 @func8(i8 signext %x, i8 signext %y, i8 signext %z) nounwind {
define signext i4 @func4(i4 signext %x, i4 signext %y, i4 signext %z) nounwind {
; X86-LABEL: func4:
; X86: # %bb.0:
+; X86-NEXT: movzbl {{[0-9]+}}(%esp), %ecx
; X86-NEXT: movzbl {{[0-9]+}}(%esp), %eax
; X86-NEXT: mulb {{[0-9]+}}(%esp)
-; X86-NEXT: movzbl {{[0-9]+}}(%esp), %ecx
; X86-NEXT: shlb $4, %al
; X86-NEXT: sarb $4, %al
; X86-NEXT: subb %al, %cl
diff --git a/llvm/test/CodeGen/X86/statepoint-ra.ll b/llvm/test/CodeGen/X86/statepoint-ra.ll
index 5a4e04dd70553..f8728b572e58d 100644
--- a/llvm/test/CodeGen/X86/statepoint-ra.ll
+++ b/llvm/test/CodeGen/X86/statepoint-ra.ll
@@ -124,6 +124,8 @@ declare token @llvm.experimental.gc.statepoint.p0(i64 , i32 , ptr, i32 , i32 , .
;CHECK: successors: %bb.5(0x00000000), %bb.6(0x80000000)
;CHECK: EH_LABEL <mcsymbol >
;CHECK: ADJCALLSTACKDOWN64 0, 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
+;CHECK: dead $edi = IMPLICIT_DEF
+;CHECK: dead $rsi = IMPLICIT_DEF
;CHECK: $edx = MOV32r0 implicit-def dead $eflags
;CHECK: STATEPOINT 1, 16, 3, undef %29:gr64, undef $edi, undef $rsi, $edx, 2, 0, 2, 0, 2, 105, 2, 0, 2, 2, 2, 0, 2, 97, 2, 0, 2, 26, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 0, 2, 2, 2, 3, 2, 0, 2, 20, 2, 0, 2, 0, 2, 4278124286, 2, 4, 1, 8, %stack.6, 0, 2, 7, 2, 0, 2, 4, 1, 8, %stack.5, 0, 2, 7, 2, 0, 2, 4, 1, 8, %stack.4, 0, 2, 7, 2, 0, 2, 4, 1, 8, %stack.2, 0, 2, 7, 2, 0, 2, 4, 1, 8, %stack.1, 0, 2, 7, 2, 0, 2, 4, 1, 8, %stack.3, 0, 2, 7, 2, 0, 2, 4, 1, 8, %fixed-stack.3, 0, 2, 7, 2, 0, 2, 4, 1, 8, %fixed-stack.2, 0, 2, 7, 2, 0, 2, 4, 1, 8, %stack.7, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 1, 2, 4278124286, 2, 0, 2, 1, 0, 0, csr_64, implicit-def $rsp, implicit-def $ssp, implicit-def dead $eax :: (load (s64) from %stack.1), (load (s64) from %stack.2), (load (s64) from %stack.3), (load (s64) from %stack.4), (load (s64) from %stack.5), (load (s64) from %stack.6), (load (s64) from %fixed-stack.2), (load (s64) from %fixed-stack.3, align 16), (load (s64) from %stack.7)
;CHECK: ADJCALLSTACKUP64 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
@@ -145,7 +147,7 @@ declare token @llvm.experimental.gc.statepoint.p0(i64 , i32 , ptr, i32 , i32 , .
;CHECK: $esi = MOV32ri 51
;CHECK: %75:fr64 = MOVSDrm_alt %fixed-stack.2, 1, $noreg, 0, $noreg :: (load (s64) from %fixed-stack.2)
;CHECK: $xmm7 = COPY %75
-;CHECK: STATEPOINT 2, 5, 10, undef %36:gr64, undef $rdi, $xmm0, $xmm1, $xmm2, $xmm3, $xmm4, $xmm5, $xmm6, $xmm7, killed $esi, 2, 0, 2, 0, 2, 105, 2, 0, 2, 2, 2, 0, 2, 97, 2, 0, 2, 26, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 2, 2, 2, 2, 46, 2, 0, 2, 20, 2, 0, 2, 0, 2, 4278124286, 2, 4, 1, 8, %stack.6, 0, 2, 7, 2, 0, 2, 4, 1, 8, %stack.5, 0, 2, 7, 2, 0, 2, 4, 1, 8, %stack.4, 0, 2, 7, 2, 0, 2, 4, 1, 8, %stack.2, 0, 2, 7, 2, 0, 2, 4, 1, 8, %stack.1, 0, 2, 7, 2, 0, 2, 4, 1, 8, %stack.3, 0, 2, 7, 2, 0, 2, 4, 1, 8, %fixed-stack.3, 0, 2, 7, 2, 0, 2, 4, 1, 8, %fixed-stack.2, 0, 2, 7, 2, 0, 2, 4, 1, 8, %stack.7, 0, 2, 7, 2, 0, 2, 3, 2, 51, 2, 1, 2, 4278124286, 2, 0, 2, 1, 0, 0, csr_64, implicit-def $rsp, implicit-def $ssp :: (load (s64) from %stack.1), (load (s64) from %stack.2), (load (s64) from %stack.3), (load (s64) from %stack.4), (load (s64) from %stack.5), (load (s64) from %stack.6), (load (s64) from %fixed-stack.2), (load (s64) from %fixed-stack.3, align 16), (load (s64) from %stack.7)
+;CHECK: STATEPOINT 2, 5, 10, undef %36:gr64, undef $rdi, $xmm0, $xmm1, $xmm2, $xmm3, $xmm4, $xmm5, $xmm6, $xmm7, $esi, 2, 0, 2, 0, 2, 105, 2, 0, 2, 2, 2, 0, 2, 97, 2, 0, 2, 26, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 7, 2, 0, 2, 2, 2, 2, 2, 46, 2, 0, 2, 20, 2, 0, 2, 0, 2, 4278124286, 2, 4, 1, 8, %stack.6, 0, 2, 7, 2, 0, 2, 4, 1, 8, %stack.5, 0, 2, 7, 2, 0, 2, 4, 1, 8, %stack.4, 0, 2, 7, 2, 0, 2, 4, 1, 8, %stack.2, 0, 2, 7, 2, 0, 2, 4, 1, 8, %stack.1, 0, 2, 7, 2, 0, 2, 4, 1, 8, %stack.3, 0, 2, 7, 2, 0, 2, 4, 1, 8, %fixed-stack.3, 0, 2, 7, 2, 0, 2, 4, 1, 8, %fixed-stack.2, 0, 2, 7, 2, 0, 2, 4, 1, 8, %stack.7, 0, 2, 7, 2, 0, 2, 3, 2, 51, 2, 1, 2, 4278124286, 2, 0, 2, 1, 0, 0, csr_64, implicit-def $rsp, implicit-def $ssp :: (load (s64) from %stack.1), (load (s64) from %stack.2), (load (s64) from %stack.3), (load (s64) from %stack.4), (load (s64) from %stack.5), (load (s64) from %stack.6), (load (s64) from %fixed-stack.2), (load (s64) from %fixed-stack.3, align 16), (load (s64) from %stack.7)
;CHECK: ADJCALLSTACKUP64 8, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
;CHECK: bb.6.bb23 (landing-pad):
;CHECK: liveins: $rax, $rdx
@@ -153,3 +155,4 @@ declare token @llvm.experimental.gc.statepoint.p0(i64 , i32 , ptr, i32 , i32 , .
;CHECK: RET 0
;CHECK: bb.7.bb25:
;CHECK: RET 0
+;CHECK: ...
diff --git a/llvm/test/CodeGen/X86/statepoint-vreg-unlimited-tied-opnds.ll b/llvm/test/CodeGen/X86/statepoint-vreg-unlimited-tied-opnds.ll
index 0594f2fbc0a35..a813c70b569c5 100644
--- a/llvm/test/CodeGen/X86/statepoint-vreg-unlimited-tied-opnds.ll
+++ b/llvm/test/CodeGen/X86/statepoint-vreg-unlimited-tied-opnds.ll
@@ -31,28 +31,29 @@ define i32 @test_spill(
; CHECK-VREG-NEXT: [[MOV64rm10:%[0-9]+]]:gr64 = MOV64rm %fixed-stack.1, 1, $noreg, 0, $noreg :: (load (s64) from %fixed-stack.1, align 16)
; CHECK-VREG-NEXT: [[MOV64rm11:%[0-9]+]]:gr64 = MOV64rm %fixed-stack.0, 1, $noreg, 0, $noreg :: (load (s64) from %fixed-stack.0)
; CHECK-VREG-NEXT: ADJCALLSTACKDOWN64 0, 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
- ; CHECK-VREG-NEXT: [[MOV64rm11]]:gr64, [[MOV64rm10]]:gr64, [[MOV64rm9]]:gr64, [[MOV64rm8]]:gr64, [[MOV64rm7]]:gr64, [[MOV64rm6]]:gr64, [[MOV64rm5]]:gr64, [[MOV64rm4]]:gr64, [[MOV64rm3]]:gr64, [[MOV64rm2]]:gr64, [[MOV64rm1]]:gr64, [[MOV64rm]]:gr64, [[COPY]]:gr64, [[COPY1]]:gr64, [[COPY2]]:gr64, [[COPY3]]:gr64, [[COPY4]]:gr64, [[COPY5]]:gr64 = STATEPOINT 0, 0, 0, @func, 2, 0, 2, 0, 2, 0, 2, 18, [[MOV64rm11]](tied-def 0), [[MOV64rm10]](tied-def 1), [[MOV64rm9]](tied-def 2), [[MOV64rm8]](tied-def 3), [[MOV64rm7]](tied-def 4), [[MOV64rm6]](tied-def 5), [[MOV64rm5]](tied-def 6), [[MOV64rm4]](tied-def 7), [[MOV64rm3]](tied-def 8), [[MOV64rm2]](tied-def 9), [[MOV64rm1]](tied-def 10), [[MOV64rm]](tied-def 11), [[COPY]](tied-def 12), [[COPY1]](tied-def 13), [[COPY2]](tied-def 14), [[COPY3]](tied-def 15), [[COPY4]](tied-def 16), [[COPY5]](tied-def 17), 2, 0, 2, 18, 0, 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8, 8, 9, 9, 10, 10, 11, 11, 12, 12, 13, 13, 14, 14, 15, 15, 16, 16, 17, 17, csr_64, implicit-def $rsp, implicit-def $ssp
+ ; CHECK-VREG-NEXT: [[MOV64rm11:%[0-9]+]]:gr64, [[MOV64rm10:%[0-9]+]]:gr64, [[MOV64rm9:%[0-9]+]]:gr64, [[MOV64rm8:%[0-9]+]]:gr64, [[MOV64rm7:%[0-9]+]]:gr64, [[MOV64rm6:%[0-9]+]]:gr64, [[MOV64rm5:%[0-9]+]]:gr64, [[MOV64rm4:%[0-9]+]]:gr64, [[MOV64rm3:%[0-9]+]]:gr64, [[MOV64rm2:%[0-9]+]]:gr64, [[MOV64rm1:%[0-9]+]]:gr64, [[MOV64rm:%[0-9]+]]:gr64, [[COPY:%[0-9]+]]:gr64, [[COPY1:%[0-9]+]]:gr64, [[COPY2:%[0-9]+]]:gr64, [[COPY3:%[0-9]+]]:gr64, [[COPY4:%[0-9]+]]:gr64, [[COPY5:%[0-9]+]]:gr64 = STATEPOINT 0, 0, 0, @func, 2, 0, 2, 0, 2, 0, 2, 18, [[MOV64rm11]](tied-def 0), [[MOV64rm10]](tied-def 1), [[MOV64rm9]](tied-def 2), [[MOV64rm8]](tied-def 3), [[MOV64rm7]](tied-def 4), [[MOV64rm6]](tied-def 5), [[MOV64rm5]](tied-def 6), [[MOV64rm4]](tied-def 7), [[MOV64rm3]](tied-def 8), [[MOV64rm2]](tied-def 9), [[MOV64rm1]](tied-def 10), [[MOV64rm]](tied-def 11), [[COPY]](tied-def 12), [[COPY1]](tied-def 13), [[COPY2]](tied-def 14), [[COPY3]](tied-def 15), [[COPY4]](tied-def 16), [[COPY5]](tied-def 17), 2, 0, 2, 18, 0, 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8, 8, 9, 9, 10, 10, 11, 11, 12, 12, 13, 13, 14, 14, 15, 15, 16, 16, 17, 17, csr_64, implicit-def $rsp, implicit-def $ssp
; CHECK-VREG-NEXT: ADJCALLSTACKUP64 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
; CHECK-VREG-NEXT: [[MOV32rm:%[0-9]+]]:gr32 = MOV32rm [[COPY5]], 1, $noreg, 4, $noreg :: (load (s32) from %ir.gep00, addrspace 1)
- ; CHECK-VREG-NEXT: [[ADD32rm:%[0-9]+]]:gr32 = ADD32rm [[ADD32rm]], [[COPY4]], 1, $noreg, 8, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep01, addrspace 1)
- ; CHECK-VREG-NEXT: [[ADD32rm1:%[0-9]+]]:gr32 = ADD32rm [[ADD32rm1]], [[COPY3]], 1, $noreg, 12, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep02, addrspace 1)
- ; CHECK-VREG-NEXT: [[ADD32rm1:%[0-9]+]]:gr32 = ADD32rm [[ADD32rm1]], [[COPY2]], 1, $noreg, 16, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep03, addrspace 1)
- ; CHECK-VREG-NEXT: [[ADD32rm1:%[0-9]+]]:gr32 = ADD32rm [[ADD32rm1]], [[COPY1]], 1, $noreg, 20, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep04, addrspace 1)
- ; CHECK-VREG-NEXT: [[ADD32rm1:%[0-9]+]]:gr32 = ADD32rm [[ADD32rm1]], [[COPY]], 1, $noreg, 24, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep05, addrspace 1)
- ; CHECK-VREG-NEXT: [[ADD32rm1:%[0-9]+]]:gr32 = ADD32rm [[ADD32rm1]], [[MOV64rm]], 1, $noreg, 28, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep06, addrspace 1)
- ; CHECK-VREG-NEXT: [[ADD32rm1:%[0-9]+]]:gr32 = ADD32rm [[ADD32rm1]], [[MOV64rm1]], 1, $noreg, 32, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep07, addrspace 1)
- ; CHECK-VREG-NEXT: [[ADD32rm1:%[0-9]+]]:gr32 = ADD32rm [[ADD32rm1]], [[MOV64rm2]], 1, $noreg, 36, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep08, addrspace 1)
- ; CHECK-VREG-NEXT: [[ADD32rm1:%[0-9]+]]:gr32 = ADD32rm [[ADD32rm1]], [[MOV64rm3]], 1, $noreg, 40, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep09, addrspace 1)
- ; CHECK-VREG-NEXT: [[ADD32rm1:%[0-9]+]]:gr32 = ADD32rm [[ADD32rm1]], [[MOV64rm4]], 1, $noreg, 44, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep10, addrspace 1)
- ; CHECK-VREG-NEXT: [[ADD32rm1:%[0-9]+]]:gr32 = ADD32rm [[ADD32rm1]], [[MOV64rm5]], 1, $noreg, 48, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep11, addrspace 1)
- ; CHECK-VREG-NEXT: [[ADD32rm1:%[0-9]+]]:gr32 = ADD32rm [[ADD32rm1]], [[MOV64rm6]], 1, $noreg, 52, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep12, addrspace 1)
- ; CHECK-VREG-NEXT: [[ADD32rm1:%[0-9]+]]:gr32 = ADD32rm [[ADD32rm1]], [[MOV64rm7]], 1, $noreg, 56, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep13, addrspace 1)
- ; CHECK-VREG-NEXT: [[ADD32rm1:%[0-9]+]]:gr32 = ADD32rm [[ADD32rm1]], [[MOV64rm8]], 1, $noreg, 60, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep14, addrspace 1)
- ; CHECK-VREG-NEXT: [[ADD32rm1:%[0-9]+]]:gr32 = ADD32rm [[ADD32rm1]], [[MOV64rm9]], 1, $noreg, 64, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep15, addrspace 1)
- ; CHECK-VREG-NEXT: [[ADD32rm1:%[0-9]+]]:gr32 = ADD32rm [[ADD32rm1]], [[MOV64rm10]], 1, $noreg, 68, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep16, addrspace 1)
- ; CHECK-VREG-NEXT: [[ADD32rm1:%[0-9]+]]:gr32 = ADD32rm [[ADD32rm1]], [[MOV64rm11]], 1, $noreg, 72, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep17, addrspace 1)
- ; CHECK-VREG-NEXT: $eax = COPY [[ADD32rm1]]
- ; CHECK-VREG-NEXT: RET 0, killed $eax
+ ; CHECK-VREG-NEXT: [[MOV32rm:%[0-9]+]]:gr32 = ADD32rm [[MOV32rm]], [[COPY4]], 1, $noreg, 8, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep01, addrspace 1)
+ ; CHECK-VREG-NEXT: [[MOV32rm:%[0-9]+]]:gr32 = ADD32rm [[MOV32rm]], [[COPY3]], 1, $noreg, 12, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep02, addrspace 1)
+ ; CHECK-VREG-NEXT: [[MOV32rm:%[0-9]+]]:gr32 = ADD32rm [[MOV32rm]], [[COPY2]], 1, $noreg, 16, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep03, addrspace 1)
+ ; CHECK-VREG-NEXT: [[MOV32rm:%[0-9]+]]:gr32 = ADD32rm [[MOV32rm]], [[COPY1]], 1, $noreg, 20, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep04, addrspace 1)
+ ; CHECK-VREG-NEXT: [[MOV32rm:%[0-9]+]]:gr32 = ADD32rm [[MOV32rm]], [[COPY]], 1, $noreg, 24, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep05, addrspace 1)
+ ; CHECK-VREG-NEXT: [[MOV32rm:%[0-9]+]]:gr32 = ADD32rm [[MOV32rm]], [[MOV64rm]], 1, $noreg, 28, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep06, addrspace 1)
+ ; CHECK-VREG-NEXT: [[MOV32rm:%[0-9]+]]:gr32 = ADD32rm [[MOV32rm]], [[MOV64rm1]], 1, $noreg, 32, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep07, addrspace 1)
+ ; CHECK-VREG-NEXT: [[MOV32rm:%[0-9]+]]:gr32 = ADD32rm [[MOV32rm]], [[MOV64rm2]], 1, $noreg, 36, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep08, addrspace 1)
+ ; CHECK-VREG-NEXT: [[MOV32rm:%[0-9]+]]:gr32 = ADD32rm [[MOV32rm]], [[MOV64rm3]], 1, $noreg, 40, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep09, addrspace 1)
+ ; CHECK-VREG-NEXT: [[MOV32rm:%[0-9]+]]:gr32 = ADD32rm [[MOV32rm]], [[MOV64rm4]], 1, $noreg, 44, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep10, addrspace 1)
+ ; CHECK-VREG-NEXT: [[MOV32rm:%[0-9]+]]:gr32 = ADD32rm [[MOV32rm]], [[MOV64rm5]], 1, $noreg, 48, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep11, addrspace 1)
+ ; CHECK-VREG-NEXT: [[MOV32rm:%[0-9]+]]:gr32 = ADD32rm [[MOV32rm]], [[MOV64rm6]], 1, $noreg, 52, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep12, addrspace 1)
+ ; CHECK-VREG-NEXT: [[MOV32rm:%[0-9]+]]:gr32 = ADD32rm [[MOV32rm]], [[MOV64rm7]], 1, $noreg, 56, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep13, addrspace 1)
+ ; CHECK-VREG-NEXT: [[MOV32rm:%[0-9]+]]:gr32 = ADD32rm [[MOV32rm]], [[MOV64rm8]], 1, $noreg, 60, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep14, addrspace 1)
+ ; CHECK-VREG-NEXT: [[MOV32rm:%[0-9]+]]:gr32 = ADD32rm [[MOV32rm]], [[MOV64rm9]], 1, $noreg, 64, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep15, addrspace 1)
+ ; CHECK-VREG-NEXT: [[MOV32rm:%[0-9]+]]:gr32 = ADD32rm [[MOV32rm]], [[MOV64rm10]], 1, $noreg, 68, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep16, addrspace 1)
+ ; CHECK-VREG-NEXT: [[MOV32rm:%[0-9]+]]:gr32 = ADD32rm [[MOV32rm]], [[MOV64rm11]], 1, $noreg, 72, $noreg, implicit-def dead $eflags :: (load (s32) from %ir.gep17, addrspace 1)
+ ; CHECK-VREG-NEXT: $eax = COPY [[MOV32rm]]
+ ; CHECK-VREG-NEXT: RET 0, $eax
+ ;
; CHECK-PREG-LABEL: name: test_spill
; CHECK-PREG: bb.0 (%ir-block.0):
; CHECK-PREG-NEXT: liveins: $rcx, $rdi, $rdx, $rsi, $r8, $r9
diff --git a/llvm/test/CodeGen/X86/udiv_fix.ll b/llvm/test/CodeGen/X86/udiv_fix.ll
index 8c3698c535b00..b2ad846b1be7e 100644
--- a/llvm/test/CodeGen/X86/udiv_fix.ll
+++ b/llvm/test/CodeGen/X86/udiv_fix.ll
@@ -230,33 +230,31 @@ define <4 x i32> @vec(<4 x i32> %x, <4 x i32> %y) nounwind {
; X64-LABEL: vec:
; X64: # %bb.0:
; X64-NEXT: pxor %xmm2, %xmm2
-; X64-NEXT: movdqa %xmm0, %xmm3
+; X64-NEXT: movdqa %xmm0, %xmm4
+; X64-NEXT: punpckhdq {{.*#+}} xmm4 = xmm4[2],xmm2[2],xmm4[3],xmm2[3]
+; X64-NEXT: psllq $31, %xmm4
+; X64-NEXT: movq %xmm4, %rax
+; X64-NEXT: movdqa %xmm1, %xmm3
; X64-NEXT: punpckhdq {{.*#+}} xmm3 = xmm3[2],xmm2[2],xmm3[3],xmm2[3]
-; X64-NEXT: psllq $31, %xmm3
-; X64-NEXT: movq %xmm3, %rax
+; X64-NEXT: movd %xmm3, %ecx
+; X64-NEXT: xorl %edx, %edx
+; X64-NEXT: divq %rcx
+; X64-NEXT: movq %rax, %xmm3
+; X64-NEXT: pshufd {{.*#+}} xmm4 = xmm4[2,3,2,3]
+; X64-NEXT: movq %xmm4, %rax
; X64-NEXT: movdqa %xmm1, %xmm4
-; X64-NEXT: punpckhdq {{.*#+}} xmm4 = xmm4[2],xmm2[2],xmm4[3],xmm2[3]
+; X64-NEXT: psrldq {{.*#+}} xmm4 = xmm4[12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero
; X64-NEXT: movd %xmm4, %ecx
; X64-NEXT: xorl %edx, %edx
; X64-NEXT: divq %rcx
-; X64-NEXT: movq %rax, %rcx
-; X64-NEXT: pshufd {{.*#+}} xmm3 = xmm3[2,3,2,3]
-; X64-NEXT: movq %xmm3, %rax
-; X64-NEXT: movdqa %xmm1, %xmm3
-; X64-NEXT: psrldq {{.*#+}} xmm3 = xmm3[12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero
-; X64-NEXT: movd %xmm3, %esi
-; X64-NEXT: xorl %edx, %edx
-; X64-NEXT: divq %rsi
-; X64-NEXT: movq %rax, %rsi
+; X64-NEXT: movq %rax, %xmm4
+; X64-NEXT: punpcklqdq {{.*#+}} xmm3 = xmm3[0],xmm4[0]
; X64-NEXT: punpckldq {{.*#+}} xmm0 = xmm0[0],xmm2[0],xmm0[1],xmm2[1]
; X64-NEXT: psllq $31, %xmm0
; X64-NEXT: movq %xmm0, %rax
-; X64-NEXT: movd %xmm1, %edi
+; X64-NEXT: movd %xmm1, %ecx
; X64-NEXT: xorl %edx, %edx
-; X64-NEXT: divq %rdi
-; X64-NEXT: movq %rcx, %xmm3
-; X64-NEXT: movq %rsi, %xmm2
-; X64-NEXT: punpcklqdq {{.*#+}} xmm3 = xmm3[0],xmm2[0]
+; X64-NEXT: divq %rcx
; X64-NEXT: movq %rax, %xmm2
; X64-NEXT: pshufd {{.*#+}} xmm0 = xmm0[2,3,2,3]
; X64-NEXT: movq %xmm0, %rax
diff --git a/llvm/test/CodeGen/X86/udiv_fix_sat.ll b/llvm/test/CodeGen/X86/udiv_fix_sat.ll
index 659ab7e69e9ae..565918e5b159b 100644
--- a/llvm/test/CodeGen/X86/udiv_fix_sat.ll
+++ b/llvm/test/CodeGen/X86/udiv_fix_sat.ll
@@ -302,61 +302,59 @@ define <4 x i32> @vec(<4 x i32> %x, <4 x i32> %y) nounwind {
; X64-NEXT: movd %xmm4, %ecx
; X64-NEXT: xorl %edx, %edx
; X64-NEXT: divq %rcx
-; X64-NEXT: movq %rax, %rcx
+; X64-NEXT: movq %rax, %xmm8
; X64-NEXT: pshufd {{.*#+}} xmm3 = xmm3[2,3,2,3]
; X64-NEXT: movq %xmm3, %rax
; X64-NEXT: movdqa %xmm1, %xmm3
; X64-NEXT: psrldq {{.*#+}} xmm3 = xmm3[12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero
-; X64-NEXT: movd %xmm3, %esi
+; X64-NEXT: movd %xmm3, %ecx
; X64-NEXT: xorl %edx, %edx
-; X64-NEXT: divq %rsi
-; X64-NEXT: movq %rax, %rsi
-; X64-NEXT: punpckldq {{.*#+}} xmm2 = xmm2[0],xmm0[0],xmm2[1],xmm0[1]
-; X64-NEXT: movq %xmm2, %rax
-; X64-NEXT: movd %xmm1, %edi
-; X64-NEXT: xorl %edx, %edx
-; X64-NEXT: divq %rdi
-; X64-NEXT: movq %rcx, %xmm7
-; X64-NEXT: movq %rsi, %xmm0
-; X64-NEXT: punpcklqdq {{.*#+}} xmm7 = xmm7[0],xmm0[0]
-; X64-NEXT: movdqa {{.*#+}} xmm0 = [9223372039002259456,9223372039002259456]
-; X64-NEXT: movdqa %xmm7, %xmm3
-; X64-NEXT: pxor %xmm0, %xmm3
-; X64-NEXT: pshufd {{.*#+}} xmm4 = xmm3[1,1,3,3]
-; X64-NEXT: movdqa {{.*#+}} xmm6 = [2147483649,2147483649,2147483649,2147483649]
-; X64-NEXT: pcmpeqd %xmm6, %xmm4
-; X64-NEXT: movdqa {{.*#+}} xmm5 = [9223372043297226751,9223372043297226751]
-; X64-NEXT: movdqa %xmm5, %xmm8
-; X64-NEXT: pcmpgtd %xmm3, %xmm8
-; X64-NEXT: pshufd {{.*#+}} xmm9 = xmm8[0,0,2,2]
-; X64-NEXT: pand %xmm4, %xmm9
-; X64-NEXT: pshufd {{.*#+}} xmm3 = xmm8[1,1,3,3]
-; X64-NEXT: por %xmm9, %xmm3
-; X64-NEXT: pcmpeqd %xmm4, %xmm4
-; X64-NEXT: pand %xmm3, %xmm7
+; X64-NEXT: divq %rcx
+; X64-NEXT: movq %rax, %xmm3
+; X64-NEXT: punpcklqdq {{.*#+}} xmm8 = xmm8[0],xmm3[0]
+; X64-NEXT: movdqa {{.*#+}} xmm4 = [9223372039002259456,9223372039002259456]
+; X64-NEXT: movdqa %xmm8, %xmm3
; X64-NEXT: pxor %xmm4, %xmm3
-; X64-NEXT: por %xmm7, %xmm3
+; X64-NEXT: pshufd {{.*#+}} xmm6 = xmm3[1,1,3,3]
+; X64-NEXT: movdqa {{.*#+}} xmm7 = [2147483649,2147483649,2147483649,2147483649]
+; X64-NEXT: pcmpeqd %xmm7, %xmm6
+; X64-NEXT: movdqa {{.*#+}} xmm5 = [9223372043297226751,9223372043297226751]
+; X64-NEXT: movdqa %xmm5, %xmm9
+; X64-NEXT: pcmpgtd %xmm3, %xmm9
+; X64-NEXT: pshufd {{.*#+}} xmm10 = xmm9[0,0,2,2]
+; X64-NEXT: pand %xmm6, %xmm10
+; X64-NEXT: pshufd {{.*#+}} xmm3 = xmm9[1,1,3,3]
+; X64-NEXT: por %xmm10, %xmm3
+; X64-NEXT: pcmpeqd %xmm6, %xmm6
+; X64-NEXT: pand %xmm3, %xmm8
+; X64-NEXT: pxor %xmm6, %xmm3
+; X64-NEXT: por %xmm8, %xmm3
; X64-NEXT: psrlq $1, %xmm3
-; X64-NEXT: movq %rax, %xmm7
-; X64-NEXT: pshufd {{.*#+}} xmm2 = xmm2[2,3,2,3]
+; X64-NEXT: punpckldq {{.*#+}} xmm2 = xmm2[0],xmm0[0],xmm2[1],xmm0[1]
; X64-NEXT: movq %xmm2, %rax
+; X64-NEXT: movd %xmm1, %ecx
+; X64-NEXT: xorl %edx, %edx
+; X64-NEXT: divq %rcx
+; X64-NEXT: movq %rax, %xmm8
+; X64-NEXT: pshufd {{.*#+}} xmm0 = xmm2[2,3,2,3]
+; X64-NEXT: movq %xmm0, %rax
; X64-NEXT: psrlq $32, %xmm1
; X64-NEXT: movd %xmm1, %ecx
; X64-NEXT: xorl %edx, %edx
; X64-NEXT: divq %rcx
-; X64-NEXT: movq %rax, %xmm1
-; X64-NEXT: punpcklqdq {{.*#+}} xmm7 = xmm7[0],xmm1[0]
-; X64-NEXT: pxor %xmm7, %xmm0
-; X64-NEXT: pshufd {{.*#+}} xmm1 = xmm0[1,1,3,3]
-; X64-NEXT: pcmpeqd %xmm6, %xmm1
-; X64-NEXT: pcmpgtd %xmm0, %xmm5
-; X64-NEXT: pshufd {{.*#+}} xmm2 = xmm5[0,0,2,2]
-; X64-NEXT: pand %xmm1, %xmm2
+; X64-NEXT: movq %rax, %xmm0
+; X64-NEXT: punpcklqdq {{.*#+}} xmm8 = xmm8[0],xmm0[0]
+; X64-NEXT: pxor %xmm8, %xmm4
+; X64-NEXT: pshufd {{.*#+}} xmm0 = xmm4[1,1,3,3]
+; X64-NEXT: pcmpeqd %xmm7, %xmm0
+; X64-NEXT: pcmpgtd %xmm4, %xmm5
+; X64-NEXT: pshufd {{.*#+}} xmm1 = xmm5[0,0,2,2]
+; X64-NEXT: pand %xmm0, %xmm1
; X64-NEXT: pshufd {{.*#+}} xmm0 = xmm5[1,1,3,3]
-; X64-NEXT: por %xmm2, %xmm0
-; X64-NEXT: pxor %xmm0, %xmm4
-; X64-NEXT: pand %xmm7, %xmm0
-; X64-NEXT: por %xmm4, %xmm0
+; X64-NEXT: por %xmm1, %xmm0
+; X64-NEXT: pxor %xmm0, %xmm6
+; X64-NEXT: pand %xmm8, %xmm0
+; X64-NEXT: por %xmm6, %xmm0
; X64-NEXT: psrlq $1, %xmm0
; X64-NEXT: shufps {{.*#+}} xmm0 = xmm0[0,2],xmm3[0,2]
; X64-NEXT: retq
diff --git a/llvm/test/CodeGen/X86/usub_sat_plus.ll b/llvm/test/CodeGen/X86/usub_sat_plus.ll
index 725de4401cb87..0fb14ad5cf7b0 100644
--- a/llvm/test/CodeGen/X86/usub_sat_plus.ll
+++ b/llvm/test/CodeGen/X86/usub_sat_plus.ll
@@ -82,9 +82,9 @@ define zeroext i16 @func16(i16 zeroext %x, i16 zeroext %y, i16 zeroext %z) nounw
define zeroext i8 @func8(i8 zeroext %x, i8 zeroext %y, i8 zeroext %z) nounwind {
; X86-LABEL: func8:
; X86: # %bb.0:
+; X86-NEXT: movzbl {{[0-9]+}}(%esp), %ecx
; X86-NEXT: movzbl {{[0-9]+}}(%esp), %eax
; X86-NEXT: mulb {{[0-9]+}}(%esp)
-; X86-NEXT: movzbl {{[0-9]+}}(%esp), %ecx
; X86-NEXT: xorl %edx, %edx
; X86-NEXT: subb %al, %cl
; X86-NEXT: movzbl %cl, %eax
@@ -111,9 +111,9 @@ define zeroext i8 @func8(i8 zeroext %x, i8 zeroext %y, i8 zeroext %z) nounwind {
define zeroext i4 @func4(i4 zeroext %x, i4 zeroext %y, i4 zeroext %z) nounwind {
; X86-LABEL: func4:
; X86: # %bb.0:
+; X86-NEXT: movzbl {{[0-9]+}}(%esp), %ecx
; X86-NEXT: movzbl {{[0-9]+}}(%esp), %eax
; X86-NEXT: mulb {{[0-9]+}}(%esp)
-; X86-NEXT: movzbl {{[0-9]+}}(%esp), %ecx
; X86-NEXT: andb $15, %al
; X86-NEXT: xorl %edx, %edx
; X86-NEXT: subb %al, %cl
diff --git a/llvm/test/CodeGen/X86/vector-idiv-strictfp.ll b/llvm/test/CodeGen/X86/vector-idiv-strictfp.ll
index 7e4fbaec1019e..972e759d1763c 100644
--- a/llvm/test/CodeGen/X86/vector-idiv-strictfp.ll
+++ b/llvm/test/CodeGen/X86/vector-idiv-strictfp.ll
@@ -293,50 +293,44 @@ define <8 x i64> @test_divv_8i64_strictfp(<8 x i64> %a, <8 x i64> %b) nounwind s
; NODQ-NEXT: vpextrq $1, %xmm3, %rcx
; NODQ-NEXT: xorl %edx, %edx
; NODQ-NEXT: divq %rcx
-; NODQ-NEXT: movq %rax, %rcx
+; NODQ-NEXT: vmovq %rax, %xmm4
; NODQ-NEXT: vmovq %xmm2, %rax
-; NODQ-NEXT: vmovq %xmm3, %rsi
+; NODQ-NEXT: vmovq %xmm3, %rcx
; NODQ-NEXT: xorl %edx, %edx
-; NODQ-NEXT: divq %rsi
-; NODQ-NEXT: movq %rax, %rsi
-; NODQ-NEXT: vextracti32x4 $2, %zmm0, %xmm2
-; NODQ-NEXT: vpextrq $1, %xmm2, %rax
-; NODQ-NEXT: vextracti32x4 $2, %zmm1, %xmm3
-; NODQ-NEXT: vpextrq $1, %xmm3, %rdi
+; NODQ-NEXT: divq %rcx
+; NODQ-NEXT: vmovq %rax, %xmm2
+; NODQ-NEXT: vpunpcklqdq {{.*#+}} xmm2 = xmm2[0],xmm4[0]
+; NODQ-NEXT: vextracti32x4 $2, %zmm0, %xmm3
+; NODQ-NEXT: vpextrq $1, %xmm3, %rax
+; NODQ-NEXT: vextracti32x4 $2, %zmm1, %xmm4
+; NODQ-NEXT: vpextrq $1, %xmm4, %rcx
; NODQ-NEXT: xorl %edx, %edx
-; NODQ-NEXT: divq %rdi
-; NODQ-NEXT: movq %rax, %rdi
-; NODQ-NEXT: vmovq %xmm2, %rax
-; NODQ-NEXT: vmovq %xmm3, %r8
+; NODQ-NEXT: divq %rcx
+; NODQ-NEXT: vmovq %rax, %xmm5
+; NODQ-NEXT: vmovq %xmm3, %rax
+; NODQ-NEXT: vmovq %xmm4, %rcx
; NODQ-NEXT: xorl %edx, %edx
-; NODQ-NEXT: divq %r8
-; NODQ-NEXT: movq %rax, %r8
-; NODQ-NEXT: vextracti128 $1, %ymm0, %xmm2
-; NODQ-NEXT: vpextrq $1, %xmm2, %rax
-; NODQ-NEXT: vextracti128 $1, %ymm1, %xmm3
-; NODQ-NEXT: vpextrq $1, %xmm3, %r9
+; NODQ-NEXT: divq %rcx
+; NODQ-NEXT: vmovq %rax, %xmm3
+; NODQ-NEXT: vpunpcklqdq {{.*#+}} xmm3 = xmm3[0],xmm5[0]
+; NODQ-NEXT: vinserti128 $1, %xmm2, %ymm3, %ymm2
+; NODQ-NEXT: vextracti128 $1, %ymm0, %xmm3
+; NODQ-NEXT: vpextrq $1, %xmm3, %rax
+; NODQ-NEXT: vextracti128 $1, %ymm1, %xmm4
+; NODQ-NEXT: vpextrq $1, %xmm4, %rcx
; NODQ-NEXT: xorl %edx, %edx
-; NODQ-NEXT: divq %r9
-; NODQ-NEXT: movq %rax, %r9
-; NODQ-NEXT: vmovq %xmm2, %rax
-; NODQ-NEXT: vmovq %xmm3, %r10
+; NODQ-NEXT: divq %rcx
+; NODQ-NEXT: vmovq %rax, %xmm5
+; NODQ-NEXT: vmovq %xmm3, %rax
+; NODQ-NEXT: vmovq %xmm4, %rcx
; NODQ-NEXT: xorl %edx, %edx
-; NODQ-NEXT: divq %r10
-; NODQ-NEXT: movq %rax, %r10
+; NODQ-NEXT: divq %rcx
+; NODQ-NEXT: vmovq %rax, %xmm3
+; NODQ-NEXT: vpunpcklqdq {{.*#+}} xmm3 = xmm3[0],xmm5[0]
; NODQ-NEXT: vpextrq $1, %xmm0, %rax
-; NODQ-NEXT: vpextrq $1, %xmm1, %r11
+; NODQ-NEXT: vpextrq $1, %xmm1, %rcx
; NODQ-NEXT: xorl %edx, %edx
-; NODQ-NEXT: divq %r11
-; NODQ-NEXT: vmovq %rcx, %xmm2
-; NODQ-NEXT: vmovq %rsi, %xmm3
-; NODQ-NEXT: vmovq %rdi, %xmm4
-; NODQ-NEXT: vmovq %r8, %xmm5
-; NODQ-NEXT: vpunpcklqdq {{.*#+}} xmm2 = xmm3[0],xmm2[0]
-; NODQ-NEXT: vmovq %r9, %xmm3
-; NODQ-NEXT: vpunpcklqdq {{.*#+}} xmm4 = xmm5[0],xmm4[0]
-; NODQ-NEXT: vinserti128 $1, %xmm2, %ymm4, %ymm2
-; NODQ-NEXT: vmovq %r10, %xmm4
-; NODQ-NEXT: vpunpcklqdq {{.*#+}} xmm3 = xmm4[0],xmm3[0]
+; NODQ-NEXT: divq %rcx
; NODQ-NEXT: vmovq %rax, %xmm4
; NODQ-NEXT: vmovq %xmm0, %rax
; NODQ-NEXT: vmovq %xmm1, %rcx
@@ -382,50 +376,44 @@ define <8 x i64> @test_divv_8i64_narrow_strictfp(<8 x i64> %a, <8 x i64> %b) nou
; NODQ-NEXT: vpextrq $1, %xmm3, %rcx
; NODQ-NEXT: xorl %edx, %edx
; NODQ-NEXT: divq %rcx
-; NODQ-NEXT: movq %rax, %rcx
+; NODQ-NEXT: vmovq %rax, %xmm4
; NODQ-NEXT: vmovq %xmm2, %rax
-; NODQ-NEXT: vmovq %xmm3, %rsi
+; NODQ-NEXT: vmovq %xmm3, %rcx
; NODQ-NEXT: xorl %edx, %edx
-; NODQ-NEXT: divq %rsi
-; NODQ-NEXT: movq %rax, %rsi
-; NODQ-NEXT: vextracti32x4 $2, %zmm0, %xmm2
-; NODQ-NEXT: vpextrq $1, %xmm2, %rax
-; NODQ-NEXT: vextracti32x4 $2, %zmm1, %xmm3
-; NODQ-NEXT: vpextrq $1, %xmm3, %rdi
+; NODQ-NEXT: divq %rcx
+; NODQ-NEXT: vmovq %rax, %xmm2
+; NODQ-NEXT: vpunpcklqdq {{.*#+}} xmm2 = xmm2[0],xmm4[0]
+; NODQ-NEXT: vextracti32x4 $2, %zmm0, %xmm3
+; NODQ-NEXT: vpextrq $1, %xmm3, %rax
+; NODQ-NEXT: vextracti32x4 $2, %zmm1, %xmm4
+; NODQ-NEXT: vpextrq $1, %xmm4, %rcx
; NODQ-NEXT: xorl %edx, %edx
-; NODQ-NEXT: divq %rdi
-; NODQ-NEXT: movq %rax, %rdi
-; NODQ-NEXT: vmovq %xmm2, %rax
-; NODQ-NEXT: vmovq %xmm3, %r8
+; NODQ-NEXT: divq %rcx
+; NODQ-NEXT: vmovq %rax, %xmm5
+; NODQ-NEXT: vmovq %xmm3, %rax
+; NODQ-NEXT: vmovq %xmm4, %rcx
; NODQ-NEXT: xorl %edx, %edx
-; NODQ-NEXT: divq %r8
-; NODQ-NEXT: movq %rax, %r8
-; NODQ-NEXT: vextracti128 $1, %ymm0, %xmm2
-; NODQ-NEXT: vpextrq $1, %xmm2, %rax
-; NODQ-NEXT: vextracti128 $1, %ymm1, %xmm3
-; NODQ-NEXT: vpextrq $1, %xmm3, %r9
+; NODQ-NEXT: divq %rcx
+; NODQ-NEXT: vmovq %rax, %xmm3
+; NODQ-NEXT: vpunpcklqdq {{.*#+}} xmm3 = xmm3[0],xmm5[0]
+; NODQ-NEXT: vinserti128 $1, %xmm2, %ymm3, %ymm2
+; NODQ-NEXT: vextracti128 $1, %ymm0, %xmm3
+; NODQ-NEXT: vpextrq $1, %xmm3, %rax
+; NODQ-NEXT: vextracti128 $1, %ymm1, %xmm4
+; NODQ-NEXT: vpextrq $1, %xmm4, %rcx
; NODQ-NEXT: xorl %edx, %edx
-; NODQ-NEXT: divq %r9
-; NODQ-NEXT: movq %rax, %r9
-; NODQ-NEXT: vmovq %xmm2, %rax
-; NODQ-NEXT: vmovq %xmm3, %r10
+; NODQ-NEXT: divq %rcx
+; NODQ-NEXT: vmovq %rax, %xmm5
+; NODQ-NEXT: vmovq %xmm3, %rax
+; NODQ-NEXT: vmovq %xmm4, %rcx
; NODQ-NEXT: xorl %edx, %edx
-; NODQ-NEXT: divq %r10
-; NODQ-NEXT: movq %rax, %r10
+; NODQ-NEXT: divq %rcx
+; NODQ-NEXT: vmovq %rax, %xmm3
+; NODQ-NEXT: vpunpcklqdq {{.*#+}} xmm3 = xmm3[0],xmm5[0]
; NODQ-NEXT: vpextrq $1, %xmm0, %rax
-; NODQ-NEXT: vpextrq $1, %xmm1, %r11
+; NODQ-NEXT: vpextrq $1, %xmm1, %rcx
; NODQ-NEXT: xorl %edx, %edx
-; NODQ-NEXT: divq %r11
-; NODQ-NEXT: vmovq %rcx, %xmm2
-; NODQ-NEXT: vmovq %rsi, %xmm3
-; NODQ-NEXT: vmovq %rdi, %xmm4
-; NODQ-NEXT: vmovq %r8, %xmm5
-; NODQ-NEXT: vpunpcklqdq {{.*#+}} xmm2 = xmm3[0],xmm2[0]
-; NODQ-NEXT: vmovq %r9, %xmm3
-; NODQ-NEXT: vpunpcklqdq {{.*#+}} xmm4 = xmm5[0],xmm4[0]
-; NODQ-NEXT: vinserti128 $1, %xmm2, %ymm4, %ymm2
-; NODQ-NEXT: vmovq %r10, %xmm4
-; NODQ-NEXT: vpunpcklqdq {{.*#+}} xmm3 = xmm4[0],xmm3[0]
+; NODQ-NEXT: divq %rcx
; NODQ-NEXT: vmovq %rax, %xmm4
; NODQ-NEXT: vmovq %xmm0, %rax
; NODQ-NEXT: vmovq %xmm1, %rcx
@@ -461,44 +449,39 @@ define void @test_divv_7i64_narrow_strictfp(<7 x i64> %a, <7 x i64> %b, ptr %p)
; NODQ-NEXT: vpextrq $1, %xmm0, %rcx
; NODQ-NEXT: xorl %edx, %edx
; NODQ-NEXT: divq %rcx
-; NODQ-NEXT: movq %rax, %rcx
+; NODQ-NEXT: vmovq %rax, %xmm1
; NODQ-NEXT: vmovq %xmm2, %rax
-; NODQ-NEXT: vmovq %xmm0, %rsi
+; NODQ-NEXT: vmovq %xmm0, %rcx
; NODQ-NEXT: xorl %edx, %edx
-; NODQ-NEXT: divq %rsi
-; NODQ-NEXT: movq %rax, %rsi
-; NODQ-NEXT: vextracti128 $1, %ymm2, %xmm1
-; NODQ-NEXT: vpextrq $1, %xmm1, %rax
-; NODQ-NEXT: vextracti128 $1, %ymm0, %xmm3
-; NODQ-NEXT: vpextrq $1, %xmm3, %r8
+; NODQ-NEXT: divq %rcx
+; NODQ-NEXT: vmovq %rax, %xmm3
+; NODQ-NEXT: vpunpcklqdq {{.*#+}} xmm1 = xmm3[0],xmm1[0]
+; NODQ-NEXT: vextracti128 $1, %ymm2, %xmm3
+; NODQ-NEXT: vpextrq $1, %xmm3, %rax
+; NODQ-NEXT: vextracti128 $1, %ymm0, %xmm4
+; NODQ-NEXT: vpextrq $1, %xmm4, %rcx
; NODQ-NEXT: xorl %edx, %edx
-; NODQ-NEXT: divq %r8
-; NODQ-NEXT: movq %rax, %r8
-; NODQ-NEXT: vmovq %xmm1, %rax
-; NODQ-NEXT: vmovq %xmm3, %r9
+; NODQ-NEXT: divq %rcx
+; NODQ-NEXT: vmovq %rax, %xmm5
+; NODQ-NEXT: vmovq %xmm3, %rax
+; NODQ-NEXT: vmovq %xmm4, %rcx
; NODQ-NEXT: xorl %edx, %edx
-; NODQ-NEXT: divq %r9
-; NODQ-NEXT: movq %rax, %r9
-; NODQ-NEXT: vextracti32x4 $2, %zmm2, %xmm1
-; NODQ-NEXT: vpextrq $1, %xmm1, %rax
-; NODQ-NEXT: vextracti32x4 $2, %zmm0, %xmm3
-; NODQ-NEXT: vpextrq $1, %xmm3, %r10
+; NODQ-NEXT: divq %rcx
+; NODQ-NEXT: vmovq %rax, %xmm3
+; NODQ-NEXT: vpunpcklqdq {{.*#+}} xmm3 = xmm3[0],xmm5[0]
+; NODQ-NEXT: vextracti32x4 $2, %zmm2, %xmm4
+; NODQ-NEXT: vpextrq $1, %xmm4, %rax
+; NODQ-NEXT: vextracti32x4 $2, %zmm0, %xmm5
+; NODQ-NEXT: vpextrq $1, %xmm5, %rcx
; NODQ-NEXT: xorl %edx, %edx
-; NODQ-NEXT: divq %r10
-; NODQ-NEXT: movq %rax, %r10
-; NODQ-NEXT: vmovq %xmm1, %rax
-; NODQ-NEXT: vmovq %xmm3, %r11
+; NODQ-NEXT: divq %rcx
+; NODQ-NEXT: vmovq %rax, %xmm6
+; NODQ-NEXT: vmovq %xmm4, %rax
+; NODQ-NEXT: vmovq %xmm5, %rcx
; NODQ-NEXT: xorl %edx, %edx
-; NODQ-NEXT: divq %r11
-; NODQ-NEXT: vmovq %rcx, %xmm1
-; NODQ-NEXT: vmovq %rsi, %xmm3
-; NODQ-NEXT: vmovq %r8, %xmm4
-; NODQ-NEXT: vmovq %r9, %xmm5
-; NODQ-NEXT: vpunpcklqdq {{.*#+}} xmm1 = xmm3[0],xmm1[0]
-; NODQ-NEXT: vpunpcklqdq {{.*#+}} xmm3 = xmm5[0],xmm4[0]
-; NODQ-NEXT: vmovq %r10, %xmm4
-; NODQ-NEXT: vmovq %rax, %xmm5
-; NODQ-NEXT: vpunpcklqdq {{.*#+}} xmm4 = xmm5[0],xmm4[0]
+; NODQ-NEXT: divq %rcx
+; NODQ-NEXT: vmovq %rax, %xmm4
+; NODQ-NEXT: vpunpcklqdq {{.*#+}} xmm4 = xmm4[0],xmm6[0]
; NODQ-NEXT: vextracti32x4 $3, %zmm2, %xmm2
; NODQ-NEXT: vmovq %xmm2, %rax
; NODQ-NEXT: vextracti32x4 $3, %zmm0, %xmm0
diff --git a/llvm/test/CodeGen/X86/vector-idiv-udiv-512.ll b/llvm/test/CodeGen/X86/vector-idiv-udiv-512.ll
index e3918f0bddf90..5fde70d7b083e 100644
--- a/llvm/test/CodeGen/X86/vector-idiv-udiv-512.ll
+++ b/llvm/test/CodeGen/X86/vector-idiv-udiv-512.ll
@@ -253,50 +253,44 @@ define <8 x i64> @test_divv_8i64(<8 x i64> %a, <8 x i64> %b) nounwind {
; NODQ-NEXT: vpextrq $1, %xmm3, %rcx
; NODQ-NEXT: xorl %edx, %edx
; NODQ-NEXT: divq %rcx
-; NODQ-NEXT: movq %rax, %rcx
+; NODQ-NEXT: vmovq %rax, %xmm4
; NODQ-NEXT: vmovq %xmm2, %rax
-; NODQ-NEXT: vmovq %xmm3, %rsi
+; NODQ-NEXT: vmovq %xmm3, %rcx
; NODQ-NEXT: xorl %edx, %edx
-; NODQ-NEXT: divq %rsi
-; NODQ-NEXT: movq %rax, %rsi
-; NODQ-NEXT: vextracti32x4 $2, %zmm0, %xmm2
-; NODQ-NEXT: vpextrq $1, %xmm2, %rax
-; NODQ-NEXT: vextracti32x4 $2, %zmm1, %xmm3
-; NODQ-NEXT: vpextrq $1, %xmm3, %rdi
+; NODQ-NEXT: divq %rcx
+; NODQ-NEXT: vmovq %rax, %xmm2
+; NODQ-NEXT: vpunpcklqdq {{.*#+}} xmm2 = xmm2[0],xmm4[0]
+; NODQ-NEXT: vextracti32x4 $2, %zmm0, %xmm3
+; NODQ-NEXT: vpextrq $1, %xmm3, %rax
+; NODQ-NEXT: vextracti32x4 $2, %zmm1, %xmm4
+; NODQ-NEXT: vpextrq $1, %xmm4, %rcx
; NODQ-NEXT: xorl %edx, %edx
-; NODQ-NEXT: divq %rdi
-; NODQ-NEXT: movq %rax, %rdi
-; NODQ-NEXT: vmovq %xmm2, %rax
-; NODQ-NEXT: vmovq %xmm3, %r8
+; NODQ-NEXT: divq %rcx
+; NODQ-NEXT: vmovq %rax, %xmm5
+; NODQ-NEXT: vmovq %xmm3, %rax
+; NODQ-NEXT: vmovq %xmm4, %rcx
; NODQ-NEXT: xorl %edx, %edx
-; NODQ-NEXT: divq %r8
-; NODQ-NEXT: movq %rax, %r8
-; NODQ-NEXT: vextracti128 $1, %ymm0, %xmm2
-; NODQ-NEXT: vpextrq $1, %xmm2, %rax
-; NODQ-NEXT: vextracti128 $1, %ymm1, %xmm3
-; NODQ-NEXT: vpextrq $1, %xmm3, %r9
+; NODQ-NEXT: divq %rcx
+; NODQ-NEXT: vmovq %rax, %xmm3
+; NODQ-NEXT: vpunpcklqdq {{.*#+}} xmm3 = xmm3[0],xmm5[0]
+; NODQ-NEXT: vinserti128 $1, %xmm2, %ymm3, %ymm2
+; NODQ-NEXT: vextracti128 $1, %ymm0, %xmm3
+; NODQ-NEXT: vpextrq $1, %xmm3, %rax
+; NODQ-NEXT: vextracti128 $1, %ymm1, %xmm4
+; NODQ-NEXT: vpextrq $1, %xmm4, %rcx
; NODQ-NEXT: xorl %edx, %edx
-; NODQ-NEXT: divq %r9
-; NODQ-NEXT: movq %rax, %r9
-; NODQ-NEXT: vmovq %xmm2, %rax
-; NODQ-NEXT: vmovq %xmm3, %r10
+; NODQ-NEXT: divq %rcx
+; NODQ-NEXT: vmovq %rax, %xmm5
+; NODQ-NEXT: vmovq %xmm3, %rax
+; NODQ-NEXT: vmovq %xmm4, %rcx
; NODQ-NEXT: xorl %edx, %edx
-; NODQ-NEXT: divq %r10
-; NODQ-NEXT: movq %rax, %r10
+; NODQ-NEXT: divq %rcx
+; NODQ-NEXT: vmovq %rax, %xmm3
+; NODQ-NEXT: vpunpcklqdq {{.*#+}} xmm3 = xmm3[0],xmm5[0]
; NODQ-NEXT: vpextrq $1, %xmm0, %rax
-; NODQ-NEXT: vpextrq $1, %xmm1, %r11
+; NODQ-NEXT: vpextrq $1, %xmm1, %rcx
; NODQ-NEXT: xorl %edx, %edx
-; NODQ-NEXT: divq %r11
-; NODQ-NEXT: vmovq %rcx, %xmm2
-; NODQ-NEXT: vmovq %rsi, %xmm3
-; NODQ-NEXT: vmovq %rdi, %xmm4
-; NODQ-NEXT: vmovq %r8, %xmm5
-; NODQ-NEXT: vpunpcklqdq {{.*#+}} xmm2 = xmm3[0],xmm2[0]
-; NODQ-NEXT: vmovq %r9, %xmm3
-; NODQ-NEXT: vpunpcklqdq {{.*#+}} xmm4 = xmm5[0],xmm4[0]
-; NODQ-NEXT: vinserti128 $1, %xmm2, %ymm4, %ymm2
-; NODQ-NEXT: vmovq %r10, %xmm4
-; NODQ-NEXT: vpunpcklqdq {{.*#+}} xmm3 = xmm4[0],xmm3[0]
+; NODQ-NEXT: divq %rcx
; NODQ-NEXT: vmovq %rax, %xmm4
; NODQ-NEXT: vmovq %xmm0, %rax
; NODQ-NEXT: vmovq %xmm1, %rcx
@@ -746,50 +740,44 @@ define <8 x i64> @test_divv_8i64_narrow(<8 x i64> %a, <8 x i64> %b) nounwind {
; NODQ-NEXT: vpextrq $1, %xmm3, %rcx
; NODQ-NEXT: xorl %edx, %edx
; NODQ-NEXT: divq %rcx
-; NODQ-NEXT: movq %rax, %rcx
+; NODQ-NEXT: vmovq %rax, %xmm4
; NODQ-NEXT: vmovq %xmm2, %rax
-; NODQ-NEXT: vmovq %xmm3, %rsi
+; NODQ-NEXT: vmovq %xmm3, %rcx
; NODQ-NEXT: xorl %edx, %edx
-; NODQ-NEXT: divq %rsi
-; NODQ-NEXT: movq %rax, %rsi
-; NODQ-NEXT: vextracti32x4 $2, %zmm0, %xmm2
-; NODQ-NEXT: vpextrq $1, %xmm2, %rax
-; NODQ-NEXT: vextracti32x4 $2, %zmm1, %xmm3
-; NODQ-NEXT: vpextrq $1, %xmm3, %rdi
+; NODQ-NEXT: divq %rcx
+; NODQ-NEXT: vmovq %rax, %xmm2
+; NODQ-NEXT: vpunpcklqdq {{.*#+}} xmm2 = xmm2[0],xmm4[0]
+; NODQ-NEXT: vextracti32x4 $2, %zmm0, %xmm3
+; NODQ-NEXT: vpextrq $1, %xmm3, %rax
+; NODQ-NEXT: vextracti32x4 $2, %zmm1, %xmm4
+; NODQ-NEXT: vpextrq $1, %xmm4, %rcx
; NODQ-NEXT: xorl %edx, %edx
-; NODQ-NEXT: divq %rdi
-; NODQ-NEXT: movq %rax, %rdi
-; NODQ-NEXT: vmovq %xmm2, %rax
-; NODQ-NEXT: vmovq %xmm3, %r8
+; NODQ-NEXT: divq %rcx
+; NODQ-NEXT: vmovq %rax, %xmm5
+; NODQ-NEXT: vmovq %xmm3, %rax
+; NODQ-NEXT: vmovq %xmm4, %rcx
; NODQ-NEXT: xorl %edx, %edx
-; NODQ-NEXT: divq %r8
-; NODQ-NEXT: movq %rax, %r8
-; NODQ-NEXT: vextracti128 $1, %ymm0, %xmm2
-; NODQ-NEXT: vpextrq $1, %xmm2, %rax
-; NODQ-NEXT: vextracti128 $1, %ymm1, %xmm3
-; NODQ-NEXT: vpextrq $1, %xmm3, %r9
+; NODQ-NEXT: divq %rcx
+; NODQ-NEXT: vmovq %rax, %xmm3
+; NODQ-NEXT: vpunpcklqdq {{.*#+}} xmm3 = xmm3[0],xmm5[0]
+; NODQ-NEXT: vinserti128 $1, %xmm2, %ymm3, %ymm2
+; NODQ-NEXT: vextracti128 $1, %ymm0, %xmm3
+; NODQ-NEXT: vpextrq $1, %xmm3, %rax
+; NODQ-NEXT: vextracti128 $1, %ymm1, %xmm4
+; NODQ-NEXT: vpextrq $1, %xmm4, %rcx
; NODQ-NEXT: xorl %edx, %edx
-; NODQ-NEXT: divq %r9
-; NODQ-NEXT: movq %rax, %r9
-; NODQ-NEXT: vmovq %xmm2, %rax
-; NODQ-NEXT: vmovq %xmm3, %r10
+; NODQ-NEXT: divq %rcx
+; NODQ-NEXT: vmovq %rax, %xmm5
+; NODQ-NEXT: vmovq %xmm3, %rax
+; NODQ-NEXT: vmovq %xmm4, %rcx
; NODQ-NEXT: xorl %edx, %edx
-; NODQ-NEXT: divq %r10
-; NODQ-NEXT: movq %rax, %r10
+; NODQ-NEXT: divq %rcx
+; NODQ-NEXT: vmovq %rax, %xmm3
+; NODQ-NEXT: vpunpcklqdq {{.*#+}} xmm3 = xmm3[0],xmm5[0]
; NODQ-NEXT: vpextrq $1, %xmm0, %rax
-; NODQ-NEXT: vpextrq $1, %xmm1, %r11
+; NODQ-NEXT: vpextrq $1, %xmm1, %rcx
; NODQ-NEXT: xorl %edx, %edx
-; NODQ-NEXT: divq %r11
-; NODQ-NEXT: vmovq %rcx, %xmm2
-; NODQ-NEXT: vmovq %rsi, %xmm3
-; NODQ-NEXT: vmovq %rdi, %xmm4
-; NODQ-NEXT: vmovq %r8, %xmm5
-; NODQ-NEXT: vpunpcklqdq {{.*#+}} xmm2 = xmm3[0],xmm2[0]
-; NODQ-NEXT: vmovq %r9, %xmm3
-; NODQ-NEXT: vpunpcklqdq {{.*#+}} xmm4 = xmm5[0],xmm4[0]
-; NODQ-NEXT: vinserti128 $1, %xmm2, %ymm4, %ymm2
-; NODQ-NEXT: vmovq %r10, %xmm4
-; NODQ-NEXT: vpunpcklqdq {{.*#+}} xmm3 = xmm4[0],xmm3[0]
+; NODQ-NEXT: divq %rcx
; NODQ-NEXT: vmovq %rax, %xmm4
; NODQ-NEXT: vmovq %xmm0, %rax
; NODQ-NEXT: vmovq %xmm1, %rcx
diff --git a/llvm/test/CodeGen/X86/xmulo.ll b/llvm/test/CodeGen/X86/xmulo.ll
index d680ae11cea8a..3dee19784c461 100644
--- a/llvm/test/CodeGen/X86/xmulo.ll
+++ b/llvm/test/CodeGen/X86/xmulo.ll
@@ -90,9 +90,9 @@ define zeroext i1 @smuloi8(i8 %v1, i8 %v2, ptr %res) {
;
; WIN32-LABEL: smuloi8:
; WIN32: # %bb.0:
+; WIN32-NEXT: movl {{[0-9]+}}(%esp), %edx
; WIN32-NEXT: movzbl {{[0-9]+}}(%esp), %eax
; WIN32-NEXT: imulb {{[0-9]+}}(%esp)
-; WIN32-NEXT: movl {{[0-9]+}}(%esp), %edx
; WIN32-NEXT: setb %cl
; WIN32-NEXT: movb %al, (%edx)
; WIN32-NEXT: movl %ecx, %eax
@@ -303,9 +303,9 @@ define zeroext i1 @umuloi8(i8 %v1, i8 %v2, ptr %res) {
;
; WIN32-LABEL: umuloi8:
; WIN32: # %bb.0:
+; WIN32-NEXT: movl {{[0-9]+}}(%esp), %edx
; WIN32-NEXT: movzbl {{[0-9]+}}(%esp), %eax
; WIN32-NEXT: mulb {{[0-9]+}}(%esp)
-; WIN32-NEXT: movl {{[0-9]+}}(%esp), %edx
; WIN32-NEXT: setb %cl
; WIN32-NEXT: movb %al, (%edx)
; WIN32-NEXT: movl %ecx, %eax
@@ -1386,10 +1386,10 @@ define zeroext i1 @smuloi8_load(ptr %ptr1, i8 %v2, ptr %res) {
;
; WIN32-LABEL: smuloi8_load:
; WIN32: # %bb.0:
+; WIN32-NEXT: movl {{[0-9]+}}(%esp), %edx
; WIN32-NEXT: movl {{[0-9]+}}(%esp), %eax
; WIN32-NEXT: movzbl (%eax), %eax
; WIN32-NEXT: imulb {{[0-9]+}}(%esp)
-; WIN32-NEXT: movl {{[0-9]+}}(%esp), %edx
; WIN32-NEXT: setb %cl
; WIN32-NEXT: movb %al, (%edx)
; WIN32-NEXT: movl %ecx, %eax
@@ -1435,10 +1435,10 @@ define zeroext i1 @smuloi8_load2(i8 %v1, ptr %ptr2, ptr %res) {
;
; WIN32-LABEL: smuloi8_load2:
; WIN32: # %bb.0:
+; WIN32-NEXT: movl {{[0-9]+}}(%esp), %edx
; WIN32-NEXT: movzbl {{[0-9]+}}(%esp), %eax
; WIN32-NEXT: movl {{[0-9]+}}(%esp), %ecx
; WIN32-NEXT: imulb (%ecx)
-; WIN32-NEXT: movl {{[0-9]+}}(%esp), %edx
; WIN32-NEXT: setb %cl
; WIN32-NEXT: movb %al, (%edx)
; WIN32-NEXT: movl %ecx, %eax
@@ -1829,10 +1829,10 @@ define zeroext i1 @umuloi8_load(ptr %ptr1, i8 %v2, ptr %res) {
;
; WIN32-LABEL: umuloi8_load:
; WIN32: # %bb.0:
+; WIN32-NEXT: movl {{[0-9]+}}(%esp), %edx
; WIN32-NEXT: movl {{[0-9]+}}(%esp), %eax
; WIN32-NEXT: movzbl (%eax), %eax
; WIN32-NEXT: mulb {{[0-9]+}}(%esp)
-; WIN32-NEXT: movl {{[0-9]+}}(%esp), %edx
; WIN32-NEXT: setb %cl
; WIN32-NEXT: movb %al, (%edx)
; WIN32-NEXT: movl %ecx, %eax
@@ -1878,10 +1878,10 @@ define zeroext i1 @umuloi8_load2(i8 %v1, ptr %ptr2, ptr %res) {
;
; WIN32-LABEL: umuloi8_load2:
; WIN32: # %bb.0:
+; WIN32-NEXT: movl {{[0-9]+}}(%esp), %edx
; WIN32-NEXT: movzbl {{[0-9]+}}(%esp), %eax
; WIN32-NEXT: movl {{[0-9]+}}(%esp), %ecx
; WIN32-NEXT: mulb (%ecx)
-; WIN32-NEXT: movl {{[0-9]+}}(%esp), %edx
; WIN32-NEXT: setb %cl
; WIN32-NEXT: movb %al, (%edx)
; WIN32-NEXT: movl %ecx, %eax
More information about the llvm-commits
mailing list