[llvm] r265317 - Revert r265309 and r265312 because they caused some errors I need to investigate.
Quentin Colombet via llvm-commits
llvm-commits at lists.llvm.org
Mon Apr 4 11:45:53 PDT 2016
> On Apr 4, 2016, at 11:43 AM, Wei Mi <wmi at google.com> wrote:
>
> My patch caused llvm bootstrap error on several targets. I am looking
> at it. Will fix the error as soon as possible.
Do you have a link?
Q.
>
> Wei.
>
> On Mon, Apr 4, 2016 at 11:39 AM, Quentin Colombet <qcolombet at apple.com> wrote:
>>
>> On Apr 4, 2016, at 11:35 AM, Chandler Carruth <chandlerc at google.com> wrote:
>>
>> On Mon, Apr 4, 2016 at 10:50 AM Wei Mi via llvm-commits
>> <llvm-commits at lists.llvm.org> wrote:
>>>
>>> Author: wmi
>>> Date: Mon Apr 4 12:45:03 2016
>>> New Revision: 265317
>>>
>>> URL: http://llvm.org/viewvc/llvm-project?rev=265317&view=rev
>>> Log:
>>> Revert r265309 and r265312 because they caused some errors I need to
>>> investigate.
>>
>>
>> Ouch.
>>
>> So, we were really counting on these to address a severe compile time
>> regression for the sanatizers with Quentin's patch in r263460.
>>
>> Quentin, I wonder if it would make sense to revert r263460 until Wei's fixes
>> can land? Our ASan users are really having trouble because of the widespread
>> compile time regressions this causes. While I know you're patch fixes
>> compile time issues as well, Richard had indicated that we really should
>> prioritize not *regressing* compile time... Thoughts?
>>
>>
>> Works for me as long as we keep pushing this one further to reland the whole
>> thing.
>>
>> Q.
>>
>>
>> -Chandler
>>
>>>
>>>
>>> Added:
>>> llvm/trunk/test/CodeGen/AArch64/aarch64-deferred-spilling.ll
>>> - copied unchanged from r265308,
>>> llvm/trunk/test/CodeGen/AArch64/aarch64-deferred-spilling.ll
>>> Removed:
>>> llvm/trunk/test/CodeGen/X86/hoist-spill.ll
>>> llvm/trunk/test/CodeGen/X86/new-remat.ll
>>> Modified:
>>> llvm/trunk/include/llvm/CodeGen/LiveRangeEdit.h
>>> llvm/trunk/lib/CodeGen/InlineSpiller.cpp
>>> llvm/trunk/lib/CodeGen/LiveRangeEdit.cpp
>>> llvm/trunk/lib/CodeGen/RegAllocBase.cpp
>>> llvm/trunk/lib/CodeGen/RegAllocBase.h
>>> llvm/trunk/lib/CodeGen/RegAllocBasic.cpp
>>> llvm/trunk/lib/CodeGen/RegAllocGreedy.cpp
>>> llvm/trunk/lib/CodeGen/RegAllocPBQP.cpp
>>> llvm/trunk/lib/CodeGen/Spiller.h
>>> llvm/trunk/lib/CodeGen/SplitKit.cpp
>>> llvm/trunk/lib/CodeGen/SplitKit.h
>>> llvm/trunk/test/CodeGen/X86/fp128-compare.ll
>>> llvm/trunk/test/CodeGen/X86/ragreedy-hoist-spill.ll
>>>
>>> Modified: llvm/trunk/include/llvm/CodeGen/LiveRangeEdit.h
>>> URL:
>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/LiveRangeEdit.h?rev=265317&r1=265316&r2=265317&view=diff
>>>
>>> ==============================================================================
>>> --- llvm/trunk/include/llvm/CodeGen/LiveRangeEdit.h (original)
>>> +++ llvm/trunk/include/llvm/CodeGen/LiveRangeEdit.h Mon Apr 4 12:45:03
>>> 2016
>>> @@ -72,10 +72,6 @@ private:
>>> /// ScannedRemattable - true when remattable values have been
>>> identified.
>>> bool ScannedRemattable;
>>>
>>> - /// DeadRemats - The saved instructions which have already been dead
>>> after
>>> - /// rematerialization but not deleted yet -- to be done in
>>> postOptimization.
>>> - SmallPtrSet<MachineInstr *, 32> *DeadRemats;
>>> -
>>> /// Remattable - Values defined by remattable instructions as
>>> identified by
>>> /// tii.isTriviallyReMaterializable().
>>> SmallPtrSet<const VNInfo*,4> Remattable;
>>> @@ -120,16 +116,13 @@ public:
>>> /// @param vrm Map of virtual registers to physical registers for this
>>> /// function. If NULL, no virtual register map updates will
>>> /// be done. This could be the case if called before
>>> Regalloc.
>>> - /// @param deadRemats The collection of all the instructions defining
>>> an
>>> - /// original reg and are dead after remat.
>>> LiveRangeEdit(LiveInterval *parent, SmallVectorImpl<unsigned> &newRegs,
>>> MachineFunction &MF, LiveIntervals &lis, VirtRegMap *vrm,
>>> - Delegate *delegate = nullptr,
>>> - SmallPtrSet<MachineInstr *, 32> *deadRemats = nullptr)
>>> + Delegate *delegate = nullptr)
>>> : Parent(parent), NewRegs(newRegs), MRI(MF.getRegInfo()), LIS(lis),
>>> - VRM(vrm), TII(*MF.getSubtarget().getInstrInfo()),
>>> TheDelegate(delegate),
>>> - FirstNew(newRegs.size()), ScannedRemattable(false),
>>> - DeadRemats(deadRemats) {
>>> + VRM(vrm), TII(*MF.getSubtarget().getInstrInfo()),
>>> + TheDelegate(delegate), FirstNew(newRegs.size()),
>>> + ScannedRemattable(false) {
>>> MRI.setDelegate(this);
>>> }
>>>
>>> @@ -149,16 +142,6 @@ public:
>>> bool empty() const { return size() == 0; }
>>> unsigned get(unsigned idx) const { return NewRegs[idx+FirstNew]; }
>>>
>>> - /// pop_back - It allows LiveRangeEdit users to drop new registers.
>>> - /// The context is when an original def instruction of a register is
>>> - /// dead after rematerialization, we still want to keep it for
>>> following
>>> - /// rematerializations. We save the def instruction in DeadRemats,
>>> - /// and replace the original dst register with a new dummy register so
>>> - /// the live range of original dst register can be shrinked normally.
>>> - /// We don't want to allocate phys register for the dummy register, so
>>> - /// we want to drop it from the NewRegs set.
>>> - void pop_back() { NewRegs.pop_back(); }
>>> -
>>> ArrayRef<unsigned> regs() const {
>>> return makeArrayRef(NewRegs).slice(FirstNew);
>>> }
>>> @@ -192,15 +175,15 @@ public:
>>> /// Remat - Information needed to rematerialize at a specific location.
>>> struct Remat {
>>> VNInfo *ParentVNI; // parent_'s value at the remat location.
>>> - MachineInstr *OrigMI; // Instruction defining OrigVNI. It contains
>>> the
>>> - // real expr for remat.
>>> + MachineInstr *OrigMI; // Instruction defining ParentVNI.
>>> explicit Remat(VNInfo *ParentVNI) : ParentVNI(ParentVNI),
>>> OrigMI(nullptr) {}
>>> };
>>>
>>> /// canRematerializeAt - Determine if ParentVNI can be rematerialized
>>> at
>>> /// UseIdx. It is assumed that parent_.getVNINfoAt(UseIdx) ==
>>> ParentVNI.
>>> /// When cheapAsAMove is set, only cheap remats are allowed.
>>> - bool canRematerializeAt(Remat &RM, VNInfo *OrigVNI, SlotIndex UseIdx,
>>> + bool canRematerializeAt(Remat &RM,
>>> + SlotIndex UseIdx,
>>> bool cheapAsAMove);
>>>
>>> /// rematerializeAt - Rematerialize RM.ParentVNI into DestReg by
>>> inserting an
>>> @@ -225,12 +208,6 @@ public:
>>> return Rematted.count(ParentVNI);
>>> }
>>>
>>> - void markDeadRemat(MachineInstr *inst) {
>>> - // DeadRemats is an optional field.
>>> - if (DeadRemats)
>>> - DeadRemats->insert(inst);
>>> - }
>>> -
>>> /// eraseVirtReg - Notify the delegate that Reg is no longer in use,
>>> and try
>>> /// to erase it from LIS.
>>> void eraseVirtReg(unsigned Reg);
>>> @@ -241,11 +218,8 @@ public:
>>> /// RegsBeingSpilled lists registers currently being spilled by the
>>> register
>>> /// allocator. These registers should not be split into new intervals
>>> /// as currently those new intervals are not guaranteed to spill.
>>> - /// NoSplit indicates this func is used after the iterations of
>>> selectOrSplit
>>> - /// where registers should not be split into new intervals.
>>> - void eliminateDeadDefs(SmallVectorImpl<MachineInstr *> &Dead,
>>> - ArrayRef<unsigned> RegsBeingSpilled = None,
>>> - bool NoSplit = false);
>>> + void eliminateDeadDefs(SmallVectorImpl<MachineInstr*> &Dead,
>>> + ArrayRef<unsigned> RegsBeingSpilled = None);
>>>
>>> /// calculateRegClassAndHint - Recompute register class and hint for
>>> each new
>>> /// register.
>>>
>>> Modified: llvm/trunk/lib/CodeGen/InlineSpiller.cpp
>>> URL:
>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/InlineSpiller.cpp?rev=265317&r1=265316&r2=265317&view=diff
>>>
>>> ==============================================================================
>>> --- llvm/trunk/lib/CodeGen/InlineSpiller.cpp (original)
>>> +++ llvm/trunk/lib/CodeGen/InlineSpiller.cpp Mon Apr 4 12:45:03 2016
>>> @@ -48,77 +48,13 @@ STATISTIC(NumReloadsRemoved, "Number of
>>> STATISTIC(NumFolded, "Number of folded stack accesses");
>>> STATISTIC(NumFoldedLoads, "Number of folded loads");
>>> STATISTIC(NumRemats, "Number of rematerialized defs for
>>> spilling");
>>> +STATISTIC(NumOmitReloadSpill, "Number of omitted spills of reloads");
>>> +STATISTIC(NumHoists, "Number of hoisted spills");
>>>
>>> static cl::opt<bool> DisableHoisting("disable-spill-hoist", cl::Hidden,
>>> cl::desc("Disable inline spill
>>> hoisting"));
>>>
>>> namespace {
>>> -class HoistSpillHelper {
>>> - LiveIntervals &LIS;
>>> - LiveStacks &LSS;
>>> - AliasAnalysis *AA;
>>> - MachineDominatorTree &MDT;
>>> - MachineLoopInfo &Loops;
>>> - VirtRegMap &VRM;
>>> - MachineFrameInfo &MFI;
>>> - MachineRegisterInfo &MRI;
>>> - const TargetInstrInfo &TII;
>>> - const TargetRegisterInfo &TRI;
>>> - const MachineBlockFrequencyInfo &MBFI;
>>> -
>>> - // Map from StackSlot to its original register.
>>> - DenseMap<int, unsigned> StackSlotToReg;
>>> - // Map from pair of (StackSlot and Original VNI) to a set of spills
>>> which
>>> - // have the same stackslot and have equal values defined by Original
>>> VNI.
>>> - // These spills are mergeable and are hoist candiates.
>>> - typedef DenseMap<std::pair<int, VNInfo *>, SmallPtrSet<MachineInstr *,
>>> 16>>
>>> - MergeableSpillsMap;
>>> - MergeableSpillsMap MergeableSpills;
>>> -
>>> - /// This is the map from original register to a set containing all its
>>> - /// siblings. To hoist a spill to another BB, we need to find out a
>>> live
>>> - /// sibling there and use it as the source of the new spill.
>>> - DenseMap<unsigned, SmallSetVector<unsigned, 16>> Virt2SiblingsMap;
>>> -
>>> - bool isSpillCandBB(unsigned OrigReg, VNInfo &OrigVNI, MachineBasicBlock
>>> &BB,
>>> - unsigned &LiveReg);
>>> -
>>> - void rmRedundantSpills(
>>> - SmallPtrSet<MachineInstr *, 16> &Spills,
>>> - SmallVectorImpl<MachineInstr *> &SpillsToRm,
>>> - DenseMap<MachineDomTreeNode *, MachineInstr *> &SpillBBToSpill);
>>> -
>>> - void getVisitOrders(
>>> - MachineBasicBlock *Root, SmallPtrSet<MachineInstr *, 16> &Spills,
>>> - SmallVectorImpl<MachineDomTreeNode *> &Orders,
>>> - SmallVectorImpl<MachineInstr *> &SpillsToRm,
>>> - DenseMap<MachineDomTreeNode *, unsigned> &SpillsToKeep,
>>> - DenseMap<MachineDomTreeNode *, MachineInstr *> &SpillBBToSpill);
>>> -
>>> - void runHoistSpills(unsigned OrigReg, VNInfo &OrigVNI,
>>> - SmallPtrSet<MachineInstr *, 16> &Spills,
>>> - SmallVectorImpl<MachineInstr *> &SpillsToRm,
>>> - DenseMap<MachineBasicBlock *, unsigned>
>>> &SpillsToIns);
>>> -
>>> -public:
>>> - HoistSpillHelper(MachineFunctionPass &pass, MachineFunction &mf,
>>> - VirtRegMap &vrm)
>>> - : LIS(pass.getAnalysis<LiveIntervals>()),
>>> - LSS(pass.getAnalysis<LiveStacks>()),
>>> - AA(&pass.getAnalysis<AAResultsWrapperPass>().getAAResults()),
>>> - MDT(pass.getAnalysis<MachineDominatorTree>()),
>>> - Loops(pass.getAnalysis<MachineLoopInfo>()), VRM(vrm),
>>> - MFI(*mf.getFrameInfo()), MRI(mf.getRegInfo()),
>>> - TII(*mf.getSubtarget().getInstrInfo()),
>>> - TRI(*mf.getSubtarget().getRegisterInfo()),
>>> - MBFI(pass.getAnalysis<MachineBlockFrequencyInfo>()) {}
>>> -
>>> - void addToMergeableSpills(MachineInstr *Spill, int StackSlot,
>>> - unsigned Original);
>>> - bool rmFromMergeableSpills(MachineInstr *Spill, int StackSlot);
>>> - void hoistAllSpills(LiveRangeEdit &Edit);
>>> -};
>>> -
>>> class InlineSpiller : public Spiller {
>>> MachineFunction &MF;
>>> LiveIntervals &LIS;
>>> @@ -149,12 +85,56 @@ class InlineSpiller : public Spiller {
>>> // Values that failed to remat at some point.
>>> SmallPtrSet<VNInfo*, 8> UsedValues;
>>>
>>> +public:
>>> + // Information about a value that was defined by a copy from a sibling
>>> + // register.
>>> + struct SibValueInfo {
>>> + // True when all reaching defs were reloads: No spill is necessary.
>>> + bool AllDefsAreReloads;
>>> +
>>> + // True when value is defined by an original PHI not from splitting.
>>> + bool DefByOrigPHI;
>>> +
>>> + // True when the COPY defining this value killed its source.
>>> + bool KillsSource;
>>> +
>>> + // The preferred register to spill.
>>> + unsigned SpillReg;
>>> +
>>> + // The value of SpillReg that should be spilled.
>>> + VNInfo *SpillVNI;
>>> +
>>> + // The block where SpillVNI should be spilled. Currently, this must
>>> be the
>>> + // block containing SpillVNI->def.
>>> + MachineBasicBlock *SpillMBB;
>>> +
>>> + // A defining instruction that is not a sibling copy or a reload, or
>>> NULL.
>>> + // This can be used as a template for rematerialization.
>>> + MachineInstr *DefMI;
>>> +
>>> + // List of values that depend on this one. These values are actually
>>> the
>>> + // same, but live range splitting has placed them in different
>>> registers,
>>> + // or SSA update needed to insert PHI-defs to preserve SSA form.
>>> This is
>>> + // copies of the current value and phi-kills. Usually only phi-kills
>>> cause
>>> + // more than one dependent value.
>>> + TinyPtrVector<VNInfo*> Deps;
>>> +
>>> + SibValueInfo(unsigned Reg, VNInfo *VNI)
>>> + : AllDefsAreReloads(true), DefByOrigPHI(false), KillsSource(false),
>>> + SpillReg(Reg), SpillVNI(VNI), SpillMBB(nullptr), DefMI(nullptr)
>>> {}
>>> +
>>> + // Returns true when a def has been found.
>>> + bool hasDef() const { return DefByOrigPHI || DefMI; }
>>> + };
>>> +
>>> +private:
>>> + // Values in RegsToSpill defined by sibling copies.
>>> + typedef DenseMap<VNInfo*, SibValueInfo> SibValueMap;
>>> + SibValueMap SibValues;
>>> +
>>> // Dead defs generated during spilling.
>>> SmallVector<MachineInstr*, 8> DeadDefs;
>>>
>>> - // Object records spills information and does the hoisting.
>>> - HoistSpillHelper HSpiller;
>>> -
>>> ~InlineSpiller() override {}
>>>
>>> public:
>>> @@ -167,11 +147,9 @@ public:
>>> MFI(*mf.getFrameInfo()), MRI(mf.getRegInfo()),
>>> TII(*mf.getSubtarget().getInstrInfo()),
>>> TRI(*mf.getSubtarget().getRegisterInfo()),
>>> - MBFI(pass.getAnalysis<MachineBlockFrequencyInfo>()),
>>> - HSpiller(pass, mf, vrm) {}
>>> + MBFI(pass.getAnalysis<MachineBlockFrequencyInfo>()) {}
>>>
>>> void spill(LiveRangeEdit &) override;
>>> - void postOptimization();
>>>
>>> private:
>>> bool isSnippet(const LiveInterval &SnipLI);
>>> @@ -183,7 +161,11 @@ private:
>>> }
>>>
>>> bool isSibling(unsigned Reg);
>>> - bool hoistSpillInsideBB(LiveInterval &SpillLI, MachineInstr &CopyMI);
>>> + MachineInstr *traceSiblingValue(unsigned, VNInfo*, VNInfo*);
>>> + void propagateSiblingValue(SibValueMap::iterator, VNInfo *VNI =
>>> nullptr);
>>> + void analyzeSiblingValues();
>>> +
>>> + bool hoistSpill(LiveInterval &SpillLI, MachineInstr &CopyMI);
>>> void eliminateRedundantSpills(LiveInterval &LI, VNInfo *VNI);
>>>
>>> void markValueUsed(LiveInterval*, VNInfo*);
>>> @@ -315,44 +297,417 @@ void InlineSpiller::collectRegsToSpill()
>>> }
>>> }
>>>
>>> +
>>>
>>> +//===----------------------------------------------------------------------===//
>>> +// Sibling Values
>>>
>>> +//===----------------------------------------------------------------------===//
>>> +
>>> +// After live range splitting, some values to be spilled may be defined
>>> by
>>> +// copies from sibling registers. We trace the sibling copies back to the
>>> +// original value if it still exists. We need it for rematerialization.
>>> +//
>>> +// Even when the value can't be rematerialized, we still want to
>>> determine if
>>> +// the value has already been spilled, or we may want to hoist the spill
>>> from a
>>> +// loop.
>>> +
>>> bool InlineSpiller::isSibling(unsigned Reg) {
>>> return TargetRegisterInfo::isVirtualRegister(Reg) &&
>>> VRM.getOriginal(Reg) == Original;
>>> }
>>>
>>> -/// It is beneficial to spill to earlier place in the same BB in case
>>> -/// as follows:
>>> -/// There is an alternative def earlier in the same MBB.
>>> -/// Hoist the spill as far as possible in SpillMBB. This can ease
>>> -/// register pressure:
>>> +#ifndef NDEBUG
>>> +static raw_ostream &operator<<(raw_ostream &OS,
>>> + const InlineSpiller::SibValueInfo &SVI) {
>>> + OS << "spill " << PrintReg(SVI.SpillReg) << ':'
>>> + << SVI.SpillVNI->id << '@' << SVI.SpillVNI->def;
>>> + if (SVI.SpillMBB)
>>> + OS << " in BB#" << SVI.SpillMBB->getNumber();
>>> + if (SVI.AllDefsAreReloads)
>>> + OS << " all-reloads";
>>> + if (SVI.DefByOrigPHI)
>>> + OS << " orig-phi";
>>> + if (SVI.KillsSource)
>>> + OS << " kill";
>>> + OS << " deps[";
>>> + for (VNInfo *Dep : SVI.Deps)
>>> + OS << ' ' << Dep->id << '@' << Dep->def;
>>> + OS << " ]";
>>> + if (SVI.DefMI)
>>> + OS << " def: " << *SVI.DefMI;
>>> + else
>>> + OS << '\n';
>>> + return OS;
>>> +}
>>> +#endif
>>> +
>>> +/// propagateSiblingValue - Propagate the value in SVI to dependents if
>>> it is
>>> +/// known. Otherwise remember the dependency for later.
>>> ///
>>> -/// x = def
>>> -/// y = use x
>>> -/// s = copy x
>>> +/// @param SVIIter SibValues entry to propagate.
>>> +/// @param VNI Dependent value, or NULL to propagate to all saved
>>> dependents.
>>> +void InlineSpiller::propagateSiblingValue(SibValueMap::iterator SVIIter,
>>> + VNInfo *VNI) {
>>> + SibValueMap::value_type *SVI = &*SVIIter;
>>> +
>>> + // When VNI is non-NULL, add it to SVI's deps, and only propagate to
>>> that.
>>> + TinyPtrVector<VNInfo*> FirstDeps;
>>> + if (VNI) {
>>> + FirstDeps.push_back(VNI);
>>> + SVI->second.Deps.push_back(VNI);
>>> + }
>>> +
>>> + // Has the value been completely determined yet? If not, defer
>>> propagation.
>>> + if (!SVI->second.hasDef())
>>> + return;
>>> +
>>> + // Work list of values to propagate.
>>> + SmallSetVector<SibValueMap::value_type *, 8> WorkList;
>>> + WorkList.insert(SVI);
>>> +
>>> + do {
>>> + SVI = WorkList.pop_back_val();
>>> + TinyPtrVector<VNInfo*> *Deps = VNI ? &FirstDeps : &SVI->second.Deps;
>>> + VNI = nullptr;
>>> +
>>> + SibValueInfo &SV = SVI->second;
>>> + if (!SV.SpillMBB)
>>> + SV.SpillMBB = LIS.getMBBFromIndex(SV.SpillVNI->def);
>>> +
>>> + DEBUG(dbgs() << " prop to " << Deps->size() << ": "
>>> + << SVI->first->id << '@' << SVI->first->def << ":\t" <<
>>> SV);
>>> +
>>> + assert(SV.hasDef() && "Propagating undefined value");
>>> +
>>> + // Should this value be propagated as a preferred spill candidate?
>>> We don't
>>> + // propagate values of registers that are about to spill.
>>> + bool PropSpill = !DisableHoisting && !isRegToSpill(SV.SpillReg);
>>> + unsigned SpillDepth = ~0u;
>>> +
>>> + for (VNInfo *Dep : *Deps) {
>>> + SibValueMap::iterator DepSVI = SibValues.find(Dep);
>>> + assert(DepSVI != SibValues.end() && "Dependent value not in
>>> SibValues");
>>> + SibValueInfo &DepSV = DepSVI->second;
>>> + if (!DepSV.SpillMBB)
>>> + DepSV.SpillMBB = LIS.getMBBFromIndex(DepSV.SpillVNI->def);
>>> +
>>> + bool Changed = false;
>>> +
>>> + // Propagate defining instruction.
>>> + if (!DepSV.hasDef()) {
>>> + Changed = true;
>>> + DepSV.DefMI = SV.DefMI;
>>> + DepSV.DefByOrigPHI = SV.DefByOrigPHI;
>>> + }
>>> +
>>> + // Propagate AllDefsAreReloads. For PHI values, this computes an
>>> AND of
>>> + // all predecessors.
>>> + if (!SV.AllDefsAreReloads && DepSV.AllDefsAreReloads) {
>>> + Changed = true;
>>> + DepSV.AllDefsAreReloads = false;
>>> + }
>>> +
>>> + // Propagate best spill value.
>>> + if (PropSpill && SV.SpillVNI != DepSV.SpillVNI) {
>>> + if (SV.SpillMBB == DepSV.SpillMBB) {
>>> + // DepSV is in the same block. Hoist when dominated.
>>> + if (DepSV.KillsSource && SV.SpillVNI->def <
>>> DepSV.SpillVNI->def) {
>>> + // This is an alternative def earlier in the same MBB.
>>> + // Hoist the spill as far as possible in SpillMBB. This can
>>> ease
>>> + // register pressure:
>>> + //
>>> + // x = def
>>> + // y = use x
>>> + // s = copy x
>>> + //
>>> + // Hoisting the spill of s to immediately after the def
>>> removes the
>>> + // interference between x and y:
>>> + //
>>> + // x = def
>>> + // spill x
>>> + // y = use x<kill>
>>> + //
>>> + // This hoist only helps when the DepSV copy kills its
>>> source.
>>> + Changed = true;
>>> + DepSV.SpillReg = SV.SpillReg;
>>> + DepSV.SpillVNI = SV.SpillVNI;
>>> + DepSV.SpillMBB = SV.SpillMBB;
>>> + }
>>> + } else {
>>> + // DepSV is in a different block.
>>> + if (SpillDepth == ~0u)
>>> + SpillDepth = Loops.getLoopDepth(SV.SpillMBB);
>>> +
>>> + // Also hoist spills to blocks with smaller loop depth, but
>>> make sure
>>> + // that the new value dominates. Non-phi dependents are always
>>> + // dominated, phis need checking.
>>> +
>>> + const BranchProbability MarginProb(4, 5); // 80%
>>> + // Hoist a spill to outer loop if there are multiple dependents
>>> (it
>>> + // can be beneficial if more than one dependents are hoisted)
>>> or
>>> + // if DepSV (the hoisting source) is hotter than SV (the
>>> hoisting
>>> + // destination) (we add a 80% margin to bias a little towards
>>> + // loop depth).
>>> + bool HoistCondition =
>>> + (MBFI.getBlockFreq(DepSV.SpillMBB) >=
>>> + (MBFI.getBlockFreq(SV.SpillMBB) * MarginProb)) ||
>>> + Deps->size() > 1;
>>> +
>>> + if ((Loops.getLoopDepth(DepSV.SpillMBB) > SpillDepth) &&
>>> + HoistCondition &&
>>> + (!DepSVI->first->isPHIDef() ||
>>> + MDT.dominates(SV.SpillMBB, DepSV.SpillMBB))) {
>>> + Changed = true;
>>> + DepSV.SpillReg = SV.SpillReg;
>>> + DepSV.SpillVNI = SV.SpillVNI;
>>> + DepSV.SpillMBB = SV.SpillMBB;
>>> + }
>>> + }
>>> + }
>>> +
>>> + if (!Changed)
>>> + continue;
>>> +
>>> + // Something changed in DepSVI. Propagate to dependents.
>>> + WorkList.insert(&*DepSVI);
>>> +
>>> + DEBUG(dbgs() << " update " << DepSVI->first->id << '@'
>>> + << DepSVI->first->def << " to:\t" << DepSV);
>>> + }
>>> + } while (!WorkList.empty());
>>> +}
>>> +
>>> +/// traceSiblingValue - Trace a value that is about to be spilled back to
>>> the
>>> +/// real defining instructions by looking through sibling copies. Always
>>> stay
>>> +/// within the range of OrigVNI so the registers are known to carry the
>>> same
>>> +/// value.
>>> ///
>>> -/// Hoisting the spill of s to immediately after the def removes the
>>> -/// interference between x and y:
>>> +/// Determine if the value is defined by all reloads, so spilling isn't
>>> +/// necessary - the value is already in the stack slot.
>>> ///
>>> -/// x = def
>>> -/// spill x
>>> -/// y = use x<kill>
>>> +/// Return a defining instruction that may be a candidate for
>>> rematerialization.
>>> ///
>>> -/// This hoist only helps when the copy kills its source.
>>> +MachineInstr *InlineSpiller::traceSiblingValue(unsigned UseReg, VNInfo
>>> *UseVNI,
>>> + VNInfo *OrigVNI) {
>>> + // Check if a cached value already exists.
>>> + SibValueMap::iterator SVI;
>>> + bool Inserted;
>>> + std::tie(SVI, Inserted) =
>>> + SibValues.insert(std::make_pair(UseVNI, SibValueInfo(UseReg,
>>> UseVNI)));
>>> + if (!Inserted) {
>>> + DEBUG(dbgs() << "Cached value " << PrintReg(UseReg) << ':'
>>> + << UseVNI->id << '@' << UseVNI->def << ' ' <<
>>> SVI->second);
>>> + return SVI->second.DefMI;
>>> + }
>>> +
>>> + DEBUG(dbgs() << "Tracing value " << PrintReg(UseReg) << ':'
>>> + << UseVNI->id << '@' << UseVNI->def << '\n');
>>> +
>>> + // List of (Reg, VNI) that have been inserted into SibValues, but need
>>> to be
>>> + // processed.
>>> + SmallVector<std::pair<unsigned, VNInfo*>, 8> WorkList;
>>> + WorkList.push_back(std::make_pair(UseReg, UseVNI));
>>> +
>>> + LiveInterval &OrigLI = LIS.getInterval(Original);
>>> + do {
>>> + unsigned Reg;
>>> + VNInfo *VNI;
>>> + std::tie(Reg, VNI) = WorkList.pop_back_val();
>>> + DEBUG(dbgs() << " " << PrintReg(Reg) << ':' << VNI->id << '@' <<
>>> VNI->def
>>> + << ":\t");
>>> +
>>> + // First check if this value has already been computed.
>>> + SVI = SibValues.find(VNI);
>>> + assert(SVI != SibValues.end() && "Missing SibValues entry");
>>> +
>>> + // Trace through PHI-defs created by live range splitting.
>>> + if (VNI->isPHIDef()) {
>>> + // Stop at original PHIs. We don't know the value at the
>>> + // predecessors. Look up the VNInfo for the current definition
>>> + // in OrigLI, to properly determine whether or not this phi was
>>> + // added by splitting.
>>> + if (VNI->def == OrigLI.getVNInfoAt(VNI->def)->def) {
>>> + DEBUG(dbgs() << "orig phi value\n");
>>> + SVI->second.DefByOrigPHI = true;
>>> + SVI->second.AllDefsAreReloads = false;
>>> + propagateSiblingValue(SVI);
>>> + continue;
>>> + }
>>> +
>>> + // This is a PHI inserted by live range splitting. We could trace
>>> the
>>> + // live-out value from predecessor blocks, but that search can be
>>> very
>>> + // expensive if there are many predecessors and many more PHIs as
>>> + // generated by tail-dup when it sees an indirectbr. Instead, look
>>> at
>>> + // all the non-PHI defs that have the same value as OrigVNI. They
>>> must
>>> + // jointly dominate VNI->def. This is not optimal since VNI may
>>> actually
>>> + // be jointly dominated by a smaller subset of defs, so there is a
>>> change
>>> + // we will miss a AllDefsAreReloads optimization.
>>> +
>>> + // Separate all values dominated by OrigVNI into PHIs and non-PHIs.
>>> + SmallVector<VNInfo*, 8> PHIs, NonPHIs;
>>> + LiveInterval &LI = LIS.getInterval(Reg);
>>> +
>>> + for (LiveInterval::vni_iterator VI = LI.vni_begin(), VE =
>>> LI.vni_end();
>>> + VI != VE; ++VI) {
>>> + VNInfo *VNI2 = *VI;
>>> + if (VNI2->isUnused())
>>> + continue;
>>> + if (!OrigLI.containsOneValue() &&
>>> + OrigLI.getVNInfoAt(VNI2->def) != OrigVNI)
>>> + continue;
>>> + if (VNI2->isPHIDef() && VNI2->def != OrigVNI->def)
>>> + PHIs.push_back(VNI2);
>>> + else
>>> + NonPHIs.push_back(VNI2);
>>> + }
>>> + DEBUG(dbgs() << "split phi value, checking " << PHIs.size()
>>> + << " phi-defs, and " << NonPHIs.size()
>>> + << " non-phi/orig defs\n");
>>> +
>>> + // Create entries for all the PHIs. Don't add them to the
>>> worklist, we
>>> + // are processing all of them in one go here.
>>> + for (VNInfo *PHI : PHIs)
>>> + SibValues.insert(std::make_pair(PHI, SibValueInfo(Reg, PHI)));
>>> +
>>> + // Add every PHI as a dependent of all the non-PHIs.
>>> + for (VNInfo *NonPHI : NonPHIs) {
>>> + // Known value? Try an insertion.
>>> + std::tie(SVI, Inserted) =
>>> + SibValues.insert(std::make_pair(NonPHI, SibValueInfo(Reg,
>>> NonPHI)));
>>> + // Add all the PHIs as dependents of NonPHI.
>>> + SVI->second.Deps.insert(SVI->second.Deps.end(), PHIs.begin(),
>>> + PHIs.end());
>>> + // This is the first time we see NonPHI, add it to the worklist.
>>> + if (Inserted)
>>> + WorkList.push_back(std::make_pair(Reg, NonPHI));
>>> + else
>>> + // Propagate to all inserted PHIs, not just VNI.
>>> + propagateSiblingValue(SVI);
>>> + }
>>> +
>>> + // Next work list item.
>>> + continue;
>>> + }
>>> +
>>> + MachineInstr *MI = LIS.getInstructionFromIndex(VNI->def);
>>> + assert(MI && "Missing def");
>>> +
>>> + // Trace through sibling copies.
>>> + if (unsigned SrcReg = isFullCopyOf(MI, Reg)) {
>>> + if (isSibling(SrcReg)) {
>>> + LiveInterval &SrcLI = LIS.getInterval(SrcReg);
>>> + LiveQueryResult SrcQ = SrcLI.Query(VNI->def);
>>> + assert(SrcQ.valueIn() && "Copy from non-existing value");
>>> + // Check if this COPY kills its source.
>>> + SVI->second.KillsSource = SrcQ.isKill();
>>> + VNInfo *SrcVNI = SrcQ.valueIn();
>>> + DEBUG(dbgs() << "copy of " << PrintReg(SrcReg) << ':'
>>> + << SrcVNI->id << '@' << SrcVNI->def
>>> + << " kill=" << unsigned(SVI->second.KillsSource) <<
>>> '\n');
>>> + // Known sibling source value? Try an insertion.
>>> + std::tie(SVI, Inserted) = SibValues.insert(
>>> + std::make_pair(SrcVNI, SibValueInfo(SrcReg, SrcVNI)));
>>> + // This is the first time we see Src, add it to the worklist.
>>> + if (Inserted)
>>> + WorkList.push_back(std::make_pair(SrcReg, SrcVNI));
>>> + propagateSiblingValue(SVI, VNI);
>>> + // Next work list item.
>>> + continue;
>>> + }
>>> + }
>>> +
>>> + // Track reachable reloads.
>>> + SVI->second.DefMI = MI;
>>> + SVI->second.SpillMBB = MI->getParent();
>>> + int FI;
>>> + if (Reg == TII.isLoadFromStackSlot(MI, FI) && FI == StackSlot) {
>>> + DEBUG(dbgs() << "reload\n");
>>> + propagateSiblingValue(SVI);
>>> + // Next work list item.
>>> + continue;
>>> + }
>>> +
>>> + // Potential remat candidate.
>>> + DEBUG(dbgs() << "def " << *MI);
>>> + SVI->second.AllDefsAreReloads = false;
>>> + propagateSiblingValue(SVI);
>>> + } while (!WorkList.empty());
>>> +
>>> + // Look up the value we were looking for. We already did this lookup
>>> at the
>>> + // top of the function, but SibValues may have been invalidated.
>>> + SVI = SibValues.find(UseVNI);
>>> + assert(SVI != SibValues.end() && "Didn't compute requested info");
>>> + DEBUG(dbgs() << " traced to:\t" << SVI->second);
>>> + return SVI->second.DefMI;
>>> +}
>>> +
>>> +/// analyzeSiblingValues - Trace values defined by sibling copies back to
>>> +/// something that isn't a sibling copy.
>>> ///
>>> -bool InlineSpiller::hoistSpillInsideBB(LiveInterval &SpillLI,
>>> - MachineInstr &CopyMI) {
>>> +/// Keep track of values that may be rematerializable.
>>> +void InlineSpiller::analyzeSiblingValues() {
>>> + SibValues.clear();
>>> +
>>> + // No siblings at all?
>>> + if (Edit->getReg() == Original)
>>> + return;
>>> +
>>> + LiveInterval &OrigLI = LIS.getInterval(Original);
>>> + for (unsigned Reg : RegsToSpill) {
>>> + LiveInterval &LI = LIS.getInterval(Reg);
>>> + for (LiveInterval::const_vni_iterator VI = LI.vni_begin(),
>>> + VE = LI.vni_end(); VI != VE; ++VI) {
>>> + VNInfo *VNI = *VI;
>>> + if (VNI->isUnused())
>>> + continue;
>>> + MachineInstr *DefMI = nullptr;
>>> + if (!VNI->isPHIDef()) {
>>> + DefMI = LIS.getInstructionFromIndex(VNI->def);
>>> + assert(DefMI && "No defining instruction");
>>> + }
>>> + // Check possible sibling copies.
>>> + if (VNI->isPHIDef() || DefMI->isCopy()) {
>>> + VNInfo *OrigVNI = OrigLI.getVNInfoAt(VNI->def);
>>> + assert(OrigVNI && "Def outside original live range");
>>> + if (OrigVNI->def != VNI->def)
>>> + DefMI = traceSiblingValue(Reg, VNI, OrigVNI);
>>> + }
>>> + if (DefMI && Edit->checkRematerializable(VNI, DefMI, AA)) {
>>> + DEBUG(dbgs() << "Value " << PrintReg(Reg) << ':' << VNI->id <<
>>> '@'
>>> + << VNI->def << " may remat from " << *DefMI);
>>> + }
>>> + }
>>> + }
>>> +}
>>> +
>>> +/// hoistSpill - Given a sibling copy that defines a value to be spilled,
>>> insert
>>> +/// a spill at a better location.
>>> +bool InlineSpiller::hoistSpill(LiveInterval &SpillLI, MachineInstr
>>> &CopyMI) {
>>> SlotIndex Idx = LIS.getInstructionIndex(CopyMI);
>>> -#ifndef NDEBUG
>>> VNInfo *VNI = SpillLI.getVNInfoAt(Idx.getRegSlot());
>>> assert(VNI && VNI->def == Idx.getRegSlot() && "Not defined by copy");
>>> -#endif
>>> - unsigned SrcReg = CopyMI.getOperand(1).getReg();
>>> - LiveInterval &SrcLI = LIS.getInterval(SrcReg);
>>> - VNInfo *SrcVNI = SrcLI.getVNInfoAt(Idx);
>>> - LiveQueryResult SrcQ = SrcLI.Query(Idx);
>>> - MachineBasicBlock *DefMBB = LIS.getMBBFromIndex(SrcVNI->def);
>>> - if (DefMBB != CopyMI.getParent() || !SrcQ.isKill())
>>> + SibValueMap::iterator I = SibValues.find(VNI);
>>> + if (I == SibValues.end())
>>> + return false;
>>> +
>>> + const SibValueInfo &SVI = I->second;
>>> +
>>> + // Let the normal folding code deal with the boring case.
>>> + if (!SVI.AllDefsAreReloads && SVI.SpillVNI == VNI)
>>> + return false;
>>> +
>>> + // SpillReg may have been deleted by remat and DCE.
>>> + if (!LIS.hasInterval(SVI.SpillReg)) {
>>> + DEBUG(dbgs() << "Stale interval: " << PrintReg(SVI.SpillReg) <<
>>> '\n');
>>> + SibValues.erase(I);
>>> + return false;
>>> + }
>>> +
>>> + LiveInterval &SibLI = LIS.getInterval(SVI.SpillReg);
>>> + if (!SibLI.containsValue(SVI.SpillVNI)) {
>>> + DEBUG(dbgs() << "Stale value: " << PrintReg(SVI.SpillReg) << '\n');
>>> + SibValues.erase(I);
>>> return false;
>>> + }
>>>
>>> // Conservatively extend the stack slot range to the range of the
>>> original
>>> // value. We may be able to do better with stack slot coloring by being
>>> more
>>> @@ -364,29 +719,35 @@ bool InlineSpiller::hoistSpillInsideBB(L
>>> DEBUG(dbgs() << "\tmerged orig valno " << OrigVNI->id << ": "
>>> << *StackInt << '\n');
>>>
>>> - // We are going to spill SrcVNI immediately after its def, so clear out
>>> + // Already spilled everywhere.
>>> + if (SVI.AllDefsAreReloads) {
>>> + DEBUG(dbgs() << "\tno spill needed: " << SVI);
>>> + ++NumOmitReloadSpill;
>>> + return true;
>>> + }
>>> + // We are going to spill SVI.SpillVNI immediately after its def, so
>>> clear out
>>> // any later spills of the same value.
>>> - eliminateRedundantSpills(SrcLI, SrcVNI);
>>> + eliminateRedundantSpills(SibLI, SVI.SpillVNI);
>>>
>>> - MachineBasicBlock *MBB = LIS.getMBBFromIndex(SrcVNI->def);
>>> + MachineBasicBlock *MBB = LIS.getMBBFromIndex(SVI.SpillVNI->def);
>>> MachineBasicBlock::iterator MII;
>>> - if (SrcVNI->isPHIDef())
>>> + if (SVI.SpillVNI->isPHIDef())
>>> MII = MBB->SkipPHIsAndLabels(MBB->begin());
>>> else {
>>> - MachineInstr *DefMI = LIS.getInstructionFromIndex(SrcVNI->def);
>>> + MachineInstr *DefMI = LIS.getInstructionFromIndex(SVI.SpillVNI->def);
>>> assert(DefMI && "Defining instruction disappeared");
>>> MII = DefMI;
>>> ++MII;
>>> }
>>> // Insert spill without kill flag immediately after def.
>>> - TII.storeRegToStackSlot(*MBB, MII, SrcReg, false, StackSlot,
>>> - MRI.getRegClass(SrcReg), &TRI);
>>> + TII.storeRegToStackSlot(*MBB, MII, SVI.SpillReg, false, StackSlot,
>>> + MRI.getRegClass(SVI.SpillReg), &TRI);
>>> --MII; // Point to store instruction.
>>> LIS.InsertMachineInstrInMaps(*MII);
>>> - DEBUG(dbgs() << "\thoisted: " << SrcVNI->def << '\t' << *MII);
>>> + DEBUG(dbgs() << "\thoisted: " << SVI.SpillVNI->def << '\t' << *MII);
>>>
>>> - HSpiller.addToMergeableSpills(&(*MII), StackSlot, Original);
>>> ++NumSpills;
>>> + ++NumHoists;
>>> return true;
>>> }
>>>
>>> @@ -444,8 +805,7 @@ void InlineSpiller::eliminateRedundantSp
>>> MI->setDesc(TII.get(TargetOpcode::KILL));
>>> DeadDefs.push_back(MI);
>>> ++NumSpillsRemoved;
>>> - if (HSpiller.rmFromMergeableSpills(MI, StackSlot))
>>> - --NumSpills;
>>> + --NumSpills;
>>> }
>>> }
>>> } while (!WorkList.empty());
>>> @@ -516,12 +876,12 @@ bool InlineSpiller::reMaterializeFor(Liv
>>> if (SnippetCopies.count(&MI))
>>> return false;
>>>
>>> - LiveInterval &OrigLI = LIS.getInterval(Original);
>>> - VNInfo *OrigVNI = OrigLI.getVNInfoAt(UseIdx);
>>> + // Use an OrigVNI from traceSiblingValue when ParentVNI is a sibling
>>> copy.
>>> LiveRangeEdit::Remat RM(ParentVNI);
>>> - RM.OrigMI = LIS.getInstructionFromIndex(OrigVNI->def);
>>> -
>>> - if (!Edit->canRematerializeAt(RM, OrigVNI, UseIdx, false)) {
>>> + SibValueMap::const_iterator SibI = SibValues.find(ParentVNI);
>>> + if (SibI != SibValues.end())
>>> + RM.OrigMI = SibI->second.DefMI;
>>> + if (!Edit->canRematerializeAt(RM, UseIdx, false)) {
>>> markValueUsed(&VirtReg, ParentVNI);
>>> DEBUG(dbgs() << "\tcannot remat for " << UseIdx << '\t' << MI);
>>> return false;
>>> @@ -571,6 +931,7 @@ bool InlineSpiller::reMaterializeFor(Liv
>>> /// reMaterializeAll - Try to rematerialize as many uses as possible,
>>> /// and trim the live ranges after.
>>> void InlineSpiller::reMaterializeAll() {
>>> + // analyzeSiblingValues has already tested all relevant defining
>>> instructions.
>>> if (!Edit->anyRematerializable(AA))
>>> return;
>>>
>>> @@ -656,9 +1017,6 @@ bool InlineSpiller::coalesceStackAccess(
>>> if (InstrReg != Reg || FI != StackSlot)
>>> return false;
>>>
>>> - if (!IsLoad)
>>> - HSpiller.rmFromMergeableSpills(MI, StackSlot);
>>> -
>>> DEBUG(dbgs() << "Coalescing stack access: " << *MI);
>>> LIS.RemoveMachineInstrFromMaps(*MI);
>>> MI->eraseFromParent();
>>> @@ -783,9 +1141,6 @@ foldMemoryOperand(ArrayRef<std::pair<Mac
>>> LIS.removePhysRegDefAt(Reg, Idx);
>>> }
>>>
>>> - int FI;
>>> - if (TII.isStoreToStackSlot(MI, FI) &&
>>> HSpiller.rmFromMergeableSpills(MI, FI))
>>> - --NumSpills;
>>> LIS.ReplaceMachineInstrInMaps(*MI, *FoldMI);
>>> MI->eraseFromParent();
>>>
>>> @@ -811,10 +1166,9 @@ foldMemoryOperand(ArrayRef<std::pair<Mac
>>>
>>> if (!WasCopy)
>>> ++NumFolded;
>>> - else if (Ops.front().second == 0) {
>>> + else if (Ops.front().second == 0)
>>> ++NumSpills;
>>> - HSpiller.addToMergeableSpills(FoldMI, StackSlot, Original);
>>> - } else
>>> + else
>>> ++NumReloads;
>>> return true;
>>> }
>>> @@ -849,7 +1203,6 @@ void InlineSpiller::insertSpill(unsigned
>>> DEBUG(dumpMachineInstrRangeWithSlotIndex(std::next(MI), MIS.end(), LIS,
>>> "spill"));
>>> ++NumSpills;
>>> - HSpiller.addToMergeableSpills(std::next(MI), StackSlot, Original);
>>> }
>>>
>>> /// spillAroundUses - insert spill code around each use of Reg.
>>> @@ -913,7 +1266,8 @@ void InlineSpiller::spillAroundUses(unsi
>>> continue;
>>> }
>>> if (RI.Writes) {
>>> - if (hoistSpillInsideBB(OldLI, *MI)) {
>>> + // Hoist the spill of a sib-reg copy.
>>> + if (hoistSpill(OldLI, *MI)) {
>>> // This COPY is now dead, the value is already in the stack
>>> slot.
>>> MI->getOperand(0).setIsDead();
>>> DeadDefs.push_back(MI);
>>> @@ -1026,6 +1380,7 @@ void InlineSpiller::spill(LiveRangeEdit
>>> assert(DeadDefs.empty() && "Previous spill didn't remove dead defs");
>>>
>>> collectRegsToSpill();
>>> + analyzeSiblingValues();
>>> reMaterializeAll();
>>>
>>> // Remat may handle everything.
>>> @@ -1034,394 +1389,3 @@ void InlineSpiller::spill(LiveRangeEdit
>>>
>>> Edit->calculateRegClassAndHint(MF, Loops, MBFI);
>>> }
>>> -
>>> -/// Optimizations after all the reg selections and spills are done.
>>> -///
>>> -void InlineSpiller::postOptimization() {
>>> - SmallVector<unsigned, 4> NewVRegs;
>>> - LiveRangeEdit LRE(nullptr, NewVRegs, MF, LIS, &VRM, nullptr);
>>> - HSpiller.hoistAllSpills(LRE);
>>> - assert(NewVRegs.size() == 0 &&
>>> - "No new vregs should be generated in hoistAllSpills");
>>> -}
>>> -
>>> -/// When a spill is inserted, add the spill to MergeableSpills map.
>>> -///
>>> -void HoistSpillHelper::addToMergeableSpills(MachineInstr *Spill, int
>>> StackSlot,
>>> - unsigned Original) {
>>> - StackSlotToReg[StackSlot] = Original;
>>> - SlotIndex Idx = LIS.getInstructionIndex(*Spill);
>>> - VNInfo *OrigVNI =
>>> LIS.getInterval(Original).getVNInfoAt(Idx.getRegSlot());
>>> - std::pair<int, VNInfo *> MIdx = std::make_pair(StackSlot, OrigVNI);
>>> - MergeableSpills[MIdx].insert(Spill);
>>> -}
>>> -
>>> -/// When a spill is removed, remove the spill from MergeableSpills map.
>>> -/// Return true if the spill is removed successfully.
>>> -///
>>> -bool HoistSpillHelper::rmFromMergeableSpills(MachineInstr *Spill,
>>> - int StackSlot) {
>>> - int Original = StackSlotToReg[StackSlot];
>>> - if (!Original)
>>> - return false;
>>> - SlotIndex Idx = LIS.getInstructionIndex(*Spill);
>>> - VNInfo *OrigVNI =
>>> LIS.getInterval(Original).getVNInfoAt(Idx.getRegSlot());
>>> - std::pair<int, VNInfo *> MIdx = std::make_pair(StackSlot, OrigVNI);
>>> - return MergeableSpills[MIdx].erase(Spill);
>>> -}
>>> -
>>> -/// Check BB to see if it is a possible target BB to place a hoisted
>>> spill,
>>> -/// i.e., there should be a living sibling of OrigReg at the insert
>>> point.
>>> -///
>>> -bool HoistSpillHelper::isSpillCandBB(unsigned OrigReg, VNInfo &OrigVNI,
>>> - MachineBasicBlock &BB, unsigned
>>> &LiveReg) {
>>> - SlotIndex Idx;
>>> - MachineBasicBlock::iterator MI = BB.getFirstTerminator();
>>> - if (MI != BB.end())
>>> - Idx = LIS.getInstructionIndex(*MI);
>>> - else
>>> - Idx = LIS.getMBBEndIdx(&BB).getPrevSlot();
>>> - SmallSetVector<unsigned, 16> &Siblings = Virt2SiblingsMap[OrigReg];
>>> - assert((LIS.getInterval(OrigReg)).getVNInfoAt(Idx) == &OrigVNI &&
>>> - "Unexpected VNI");
>>> -
>>> - for (auto const SibReg : Siblings) {
>>> - LiveInterval &LI = LIS.getInterval(SibReg);
>>> - VNInfo *VNI = LI.getVNInfoAt(Idx);
>>> - if (VNI) {
>>> - LiveReg = SibReg;
>>> - return true;
>>> - }
>>> - }
>>> - return false;
>>> -}
>>> -
>>> -/// Remove redundent spills in the same BB. Save those redundent spills
>>> in
>>> -/// SpillsToRm, and save the spill to keep and its BB in SpillBBToSpill
>>> map.
>>> -///
>>> -void HoistSpillHelper::rmRedundantSpills(
>>> - SmallPtrSet<MachineInstr *, 16> &Spills,
>>> - SmallVectorImpl<MachineInstr *> &SpillsToRm,
>>> - DenseMap<MachineDomTreeNode *, MachineInstr *> &SpillBBToSpill) {
>>> - // For each spill saw, check SpillBBToSpill[] and see if its BB already
>>> has
>>> - // another spill inside. If a BB contains more than one spill, only
>>> keep the
>>> - // earlier spill with smaller SlotIndex.
>>> - for (const auto CurrentSpill : Spills) {
>>> - MachineBasicBlock *Block = CurrentSpill->getParent();
>>> - MachineDomTreeNode *Node = MDT.DT->getNode(Block);
>>> - MachineInstr *PrevSpill = SpillBBToSpill[Node];
>>> - if (PrevSpill) {
>>> - SlotIndex PIdx = LIS.getInstructionIndex(*PrevSpill);
>>> - SlotIndex CIdx = LIS.getInstructionIndex(*CurrentSpill);
>>> - MachineInstr *SpillToRm = (CIdx > PIdx) ? CurrentSpill : PrevSpill;
>>> - MachineInstr *SpillToKeep = (CIdx > PIdx) ? PrevSpill :
>>> CurrentSpill;
>>> - SpillsToRm.push_back(SpillToRm);
>>> - SpillBBToSpill[MDT.DT->getNode(Block)] = SpillToKeep;
>>> - } else {
>>> - SpillBBToSpill[MDT.DT->getNode(Block)] = CurrentSpill;
>>> - }
>>> - }
>>> - for (const auto SpillToRm : SpillsToRm)
>>> - Spills.erase(SpillToRm);
>>> -}
>>> -
>>> -/// Starting from \p Root find a top-down traversal order of the
>>> dominator
>>> -/// tree to visit all basic blocks containing the elements of \p Spills.
>>> -/// Redundant spills will be found and put into \p SpillsToRm at the same
>>> -/// time. \p SpillBBToSpill will be populated as part of the process and
>>> -/// maps a basic block to the first store occurring in the basic block.
>>> -/// \post SpillsToRm.union(Spills at post) == Spills at pre
>>> -///
>>> -void HoistSpillHelper::getVisitOrders(
>>> - MachineBasicBlock *Root, SmallPtrSet<MachineInstr *, 16> &Spills,
>>> - SmallVectorImpl<MachineDomTreeNode *> &Orders,
>>> - SmallVectorImpl<MachineInstr *> &SpillsToRm,
>>> - DenseMap<MachineDomTreeNode *, unsigned> &SpillsToKeep,
>>> - DenseMap<MachineDomTreeNode *, MachineInstr *> &SpillBBToSpill) {
>>> - // The set contains all the possible BB nodes to which we may hoist
>>> - // original spills.
>>> - SmallPtrSet<MachineDomTreeNode *, 8> WorkSet;
>>> - // Save the BB nodes on the path from the first BB node containing
>>> - // non-redundent spill to the Root node.
>>> - SmallPtrSet<MachineDomTreeNode *, 8> NodesOnPath;
>>> - // All the spills to be hoisted must originate from a single def
>>> instruction
>>> - // to the OrigReg. It means the def instruction should dominate all the
>>> spills
>>> - // to be hoisted. We choose the BB where the def instruction is located
>>> as
>>> - // the Root.
>>> - MachineDomTreeNode *RootIDomNode = MDT[Root]->getIDom();
>>> - // For every node on the dominator tree with spill, walk up on the
>>> dominator
>>> - // tree towards the Root node until it is reached. If there is other
>>> node
>>> - // containing spill in the middle of the path, the previous spill saw
>>> will
>>> - // be redundent and the node containing it will be removed. All the
>>> nodes on
>>> - // the path starting from the first node with non-redundent spill to
>>> the Root
>>> - // node will be added to the WorkSet, which will contain all the
>>> possible
>>> - // locations where spills may be hoisted to after the loop below is
>>> done.
>>> - for (const auto Spill : Spills) {
>>> - MachineBasicBlock *Block = Spill->getParent();
>>> - MachineDomTreeNode *Node = MDT[Block];
>>> - MachineInstr *SpillToRm = nullptr;
>>> - while (Node != RootIDomNode) {
>>> - // If Node dominates Block, and it already contains a spill, the
>>> spill in
>>> - // Block will be redundent.
>>> - if (Node != MDT[Block] && SpillBBToSpill[Node]) {
>>> - SpillToRm = SpillBBToSpill[MDT[Block]];
>>> - break;
>>> - /// If we see the Node already in WorkSet, the path from the Node
>>> to
>>> - /// the Root node must already be traversed by another spill.
>>> - /// Then no need to repeat.
>>> - } else if (WorkSet.count(Node)) {
>>> - break;
>>> - } else {
>>> - NodesOnPath.insert(Node);
>>> - }
>>> - Node = Node->getIDom();
>>> - }
>>> - if (SpillToRm) {
>>> - SpillsToRm.push_back(SpillToRm);
>>> - } else {
>>> - // Add a BB containing the original spills to SpillsToKeep -- i.e.,
>>> - // set the initial status before hoisting start. The value of BBs
>>> - // containing original spills is set to 0, in order to descriminate
>>> - // with BBs containing hoisted spills which will be inserted to
>>> - // SpillsToKeep later during hoisting.
>>> - SpillsToKeep[MDT[Block]] = 0;
>>> - WorkSet.insert(NodesOnPath.begin(), NodesOnPath.end());
>>> - }
>>> - NodesOnPath.clear();
>>> - }
>>> -
>>> - // Sort the nodes in WorkSet in top-down order and save the nodes
>>> - // in Orders. Orders will be used for hoisting in runHoistSpills.
>>> - unsigned idx = 0;
>>> - Orders.push_back(MDT.DT->getNode(Root));
>>> - do {
>>> - MachineDomTreeNode *Node = Orders[idx++];
>>> - const std::vector<MachineDomTreeNode *> &Children =
>>> Node->getChildren();
>>> - unsigned NumChildren = Children.size();
>>> - for (unsigned i = 0; i != NumChildren; ++i) {
>>> - MachineDomTreeNode *Child = Children[i];
>>> - if (WorkSet.count(Child))
>>> - Orders.push_back(Child);
>>> - }
>>> - } while (idx != Orders.size());
>>> - assert(Orders.size() == WorkSet.size() &&
>>> - "Orders have different size with WorkSet");
>>> -
>>> -#ifndef NDEBUG
>>> - DEBUG(dbgs() << "Orders size is " << Orders.size() << "\n");
>>> - SmallVector<MachineDomTreeNode *, 32>::reverse_iterator RIt =
>>> Orders.rbegin();
>>> - for (; RIt != Orders.rend(); RIt++)
>>> - DEBUG(dbgs() << "BB" << (*RIt)->getBlock()->getNumber() << ",");
>>> - DEBUG(dbgs() << "\n");
>>> -#endif
>>> -}
>>> -
>>> -/// Try to hoist spills according to BB hotness. The spills to removed
>>> will
>>> -/// be saved in \p SpillsToRm. The spills to be inserted will be saved in
>>> -/// \p SpillsToIns.
>>> -///
>>> -void HoistSpillHelper::runHoistSpills(
>>> - unsigned OrigReg, VNInfo &OrigVNI, SmallPtrSet<MachineInstr *, 16>
>>> &Spills,
>>> - SmallVectorImpl<MachineInstr *> &SpillsToRm,
>>> - DenseMap<MachineBasicBlock *, unsigned> &SpillsToIns) {
>>> - // Visit order of dominator tree nodes.
>>> - SmallVector<MachineDomTreeNode *, 32> Orders;
>>> - // SpillsToKeep contains all the nodes where spills are to be inserted
>>> - // during hoisting. If the spill to be inserted is an original spill
>>> - // (not a hoisted one), the value of the map entry is 0. If the spill
>>> - // is a hoisted spill, the value of the map entry is the VReg to be
>>> used
>>> - // as the source of the spill.
>>> - DenseMap<MachineDomTreeNode *, unsigned> SpillsToKeep;
>>> - // Map from BB to the first spill inside of it.
>>> - DenseMap<MachineDomTreeNode *, MachineInstr *> SpillBBToSpill;
>>> -
>>> - rmRedundantSpills(Spills, SpillsToRm, SpillBBToSpill);
>>> -
>>> - MachineBasicBlock *Root = LIS.getMBBFromIndex(OrigVNI.def);
>>> - getVisitOrders(Root, Spills, Orders, SpillsToRm, SpillsToKeep,
>>> - SpillBBToSpill);
>>> -
>>> - // SpillsInSubTree keeps the map from a dom tree node to a pair of
>>> - // nodes set and the cost of all the spills inside those nodes.
>>> - // The nodes set are the locations where spills are to be inserted
>>> - // in the subtree of current node.
>>> - typedef std::pair<SmallPtrSet<MachineDomTreeNode *, 16>,
>>> BlockFrequency>
>>> - NodesCostPair;
>>> - DenseMap<MachineDomTreeNode *, NodesCostPair> SpillsInSubTreeMap;
>>> - // Iterate Orders set in reverse order, which will be a bottom-up order
>>> - // in the dominator tree. Once we visit a dom tree node, we know its
>>> - // children have already been visited and the spill locations in the
>>> - // subtrees of all the children have been determined.
>>> - SmallVector<MachineDomTreeNode *, 32>::reverse_iterator RIt =
>>> Orders.rbegin();
>>> - for (; RIt != Orders.rend(); RIt++) {
>>> - MachineBasicBlock *Block = (*RIt)->getBlock();
>>> - SmallPtrSet<MachineDomTreeNode *, 16> &SpillsInSubTree =
>>> - SpillsInSubTreeMap[*RIt].first;
>>> - // Total spill costs inside the sub tree.
>>> - BlockFrequency &SubTreeCost = SpillsInSubTreeMap[*RIt].second;
>>> -
>>> - // If Block contains an original spill, simply continue.
>>> - if (SpillsToKeep.find(*RIt) != SpillsToKeep.end() &&
>>> !SpillsToKeep[*RIt]) {
>>> - SpillsInSubTree.insert(*RIt);
>>> - SubTreeCost = MBFI.getBlockFreq(Block);
>>> - continue;
>>> - }
>>> -
>>> - // Collect spills in subtree of current node (*RIt) to
>>> - // SpillsInSubTree.
>>> - const std::vector<MachineDomTreeNode *> &Children =
>>> (*RIt)->getChildren();
>>> - unsigned NumChildren = Children.size();
>>> - for (unsigned i = 0; i != NumChildren; ++i) {
>>> - MachineDomTreeNode *Child = Children[i];
>>> - SpillsInSubTree.insert(SpillsInSubTreeMap[Child].first.begin(),
>>> - SpillsInSubTreeMap[Child].first.end());
>>> - SubTreeCost += SpillsInSubTreeMap[Child].second;
>>> - SpillsInSubTreeMap.erase(Child);
>>> - }
>>> -
>>> - // No spills in subtree, simply continue.
>>> - if (SpillsInSubTree.empty())
>>> - continue;
>>> -
>>> - // Check whether Block is a possible candidate to insert spill.
>>> - unsigned LiveReg = 0;
>>> - if (!isSpillCandBB(OrigReg, OrigVNI, *Block, LiveReg))
>>> - continue;
>>> -
>>> - // If there are multiple spills that could be merged, bias a little
>>> - // to hoist the spill.
>>> - BranchProbability MarginProb = (SpillsInSubTree.size() > 1)
>>> - ? BranchProbability(9, 10)
>>> - : BranchProbability(1, 1);
>>> - if (SubTreeCost > MBFI.getBlockFreq(Block) * MarginProb) {
>>> - // Hoist: Move spills to current Block.
>>> - for (const auto SpillBB : SpillsInSubTree) {
>>> - // When SpillBB is a BB contains original spill, insert the spill
>>> - // to SpillsToRm.
>>> - if (SpillsToKeep.find(SpillBB) != SpillsToKeep.end() &&
>>> - !SpillsToKeep[SpillBB]) {
>>> - MachineInstr *SpillToRm = SpillBBToSpill[SpillBB];
>>> - SpillsToRm.push_back(SpillToRm);
>>> - }
>>> - // SpillBB will not contain spill anymore, remove it from
>>> SpillsToKeep.
>>> - SpillsToKeep.erase(SpillBB);
>>> - }
>>> - // Current Block is the BB containing the new hoisted spill. Add it
>>> to
>>> - // SpillsToKeep. LiveReg is the source of the new spill.
>>> - SpillsToKeep[*RIt] = LiveReg;
>>> - DEBUG({
>>> - dbgs() << "spills in BB: ";
>>> - for (const auto Rspill : SpillsInSubTree)
>>> - dbgs() << Rspill->getBlock()->getNumber() << " ";
>>> - dbgs() << "were promoted to BB" <<
>>> (*RIt)->getBlock()->getNumber()
>>> - << "\n";
>>> - });
>>> - SpillsInSubTree.clear();
>>> - SpillsInSubTree.insert(*RIt);
>>> - SubTreeCost = MBFI.getBlockFreq(Block);
>>> - }
>>> - }
>>> - // For spills in SpillsToKeep with LiveReg set (i.e., not original
>>> spill),
>>> - // save them to SpillsToIns.
>>> - for (const auto Ent : SpillsToKeep) {
>>> - if (Ent.second)
>>> - SpillsToIns[Ent.first->getBlock()] = Ent.second;
>>> - }
>>> -}
>>> -
>>> -/// For spills with equal values, remove redundent spills and hoist the
>>> left
>>> -/// to less hot spots.
>>> -///
>>> -/// Spills with equal values will be collected into the same set in
>>> -/// MergeableSpills when spill is inserted. These equal spills are
>>> originated
>>> -/// from the same define instruction and are dominated by the
>>> instruction.
>>> -/// Before hoisting all the equal spills, redundent spills inside in the
>>> same
>>> -/// BB is first marked to be deleted. Then starting from spills left,
>>> walk up
>>> -/// on the dominator tree towards the Root node where the define
>>> instruction
>>> -/// is located, mark the dominated spills to be deleted along the way and
>>> -/// collect the BB nodes on the path from non-dominated spills to the
>>> define
>>> -/// instruction into a WorkSet. The nodes in WorkSet are the candidate
>>> places
>>> -/// where we consider to hoist the spills. We iterate the WorkSet in
>>> bottom-up
>>> -/// order, and for each node, we will decide whether to hoist spills
>>> inside
>>> -/// its subtree to that node. In this way, we can get benefit locally
>>> even if
>>> -/// hoisting all the equal spills to one cold place is impossible.
>>> -///
>>> -void HoistSpillHelper::hoistAllSpills(LiveRangeEdit &Edit) {
>>> - // Save the mapping between stackslot and its original reg.
>>> - DenseMap<int, unsigned> SlotToOrigReg;
>>> - for (unsigned i = 0, e = MRI.getNumVirtRegs(); i != e; ++i) {
>>> - unsigned Reg = TargetRegisterInfo::index2VirtReg(i);
>>> - int Slot = VRM.getStackSlot(Reg);
>>> - if (Slot != VirtRegMap::NO_STACK_SLOT)
>>> - SlotToOrigReg[Slot] = VRM.getOriginal(Reg);
>>> - unsigned Original = VRM.getPreSplitReg(Reg);
>>> - if (!MRI.def_empty(Reg))
>>> - Virt2SiblingsMap[Original].insert(Reg);
>>> - }
>>> -
>>> - // Each entry in MergeableSpills contains a spill set with equal
>>> values.
>>> - for (auto &Ent : MergeableSpills) {
>>> - int Slot = Ent.first.first;
>>> - unsigned OrigReg = SlotToOrigReg[Slot];
>>> - VNInfo *OrigVNI = Ent.first.second;
>>> - SmallPtrSet<MachineInstr *, 16> &EqValSpills = Ent.second;
>>> - if (Ent.second.empty())
>>> - continue;
>>> -
>>> - DEBUG({
>>> - dbgs() << "\nFor Slot" << Slot << " and VN" << OrigVNI->id << ":\n"
>>> - << "Equal spills in BB: ";
>>> - for (const auto spill : EqValSpills)
>>> - dbgs() << spill->getParent()->getNumber() << " ";
>>> - dbgs() << "\n";
>>> - });
>>> -
>>> - // SpillsToRm is the spill set to be removed from EqValSpills.
>>> - SmallVector<MachineInstr *, 16> SpillsToRm;
>>> - // SpillsToIns is the spill set to be newly inserted after hoisting.
>>> - DenseMap<MachineBasicBlock *, unsigned> SpillsToIns;
>>> -
>>> - runHoistSpills(OrigReg, *OrigVNI, EqValSpills, SpillsToRm,
>>> SpillsToIns);
>>> -
>>> - DEBUG({
>>> - dbgs() << "Finally inserted spills in BB: ";
>>> - for (const auto Ispill : SpillsToIns)
>>> - dbgs() << Ispill.first->getNumber() << " ";
>>> - dbgs() << "\nFinally removed spills in BB: ";
>>> - for (const auto Rspill : SpillsToRm)
>>> - dbgs() << Rspill->getParent()->getNumber() << " ";
>>> - dbgs() << "\n";
>>> - });
>>> -
>>> - // Stack live range update.
>>> - LiveInterval &StackIntvl = LSS.getInterval(Slot);
>>> - if (!SpillsToIns.empty() || !SpillsToRm.empty()) {
>>> - LiveInterval &OrigLI = LIS.getInterval(OrigReg);
>>> - StackIntvl.MergeValueInAsValue(OrigLI, OrigVNI,
>>> - StackIntvl.getValNumInfo(0));
>>> - }
>>> -
>>> - // Insert hoisted spills.
>>> - for (auto const Insert : SpillsToIns) {
>>> - MachineBasicBlock *BB = Insert.first;
>>> - unsigned LiveReg = Insert.second;
>>> - MachineBasicBlock::iterator MI = BB->getFirstTerminator();
>>> - TII.storeRegToStackSlot(*BB, MI, LiveReg, false, Slot,
>>> - MRI.getRegClass(LiveReg), &TRI);
>>> - LIS.InsertMachineInstrRangeInMaps(std::prev(MI), MI);
>>> - ++NumSpills;
>>> - }
>>> -
>>> - // Remove redundent spills or change them to dead instructions.
>>> - NumSpills -= SpillsToRm.size();
>>> - for (auto const RMEnt : SpillsToRm) {
>>> - RMEnt->setDesc(TII.get(TargetOpcode::KILL));
>>> - for (unsigned i = RMEnt->getNumOperands(); i; --i) {
>>> - MachineOperand &MO = RMEnt->getOperand(i - 1);
>>> - if (MO.isReg() && MO.isImplicit() && MO.isDef() && !MO.isDead())
>>> - RMEnt->RemoveOperand(i - 1);
>>> - }
>>> - }
>>> - Edit.eliminateDeadDefs(SpillsToRm, None, true);
>>> - }
>>> -}
>>>
>>> Modified: llvm/trunk/lib/CodeGen/LiveRangeEdit.cpp
>>> URL:
>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/LiveRangeEdit.cpp?rev=265317&r1=265316&r2=265317&view=diff
>>>
>>> ==============================================================================
>>> --- llvm/trunk/lib/CodeGen/LiveRangeEdit.cpp (original)
>>> +++ llvm/trunk/lib/CodeGen/LiveRangeEdit.cpp Mon Apr 4 12:45:03 2016
>>> @@ -63,13 +63,10 @@ void LiveRangeEdit::scanRemattable(Alias
>>> for (VNInfo *VNI : getParent().valnos) {
>>> if (VNI->isUnused())
>>> continue;
>>> - unsigned Original = VRM->getOriginal(getReg());
>>> - LiveInterval &OrigLI = LIS.getInterval(Original);
>>> - VNInfo *OrigVNI = OrigLI.getVNInfoAt(VNI->def);
>>> - MachineInstr *DefMI = LIS.getInstructionFromIndex(OrigVNI->def);
>>> + MachineInstr *DefMI = LIS.getInstructionFromIndex(VNI->def);
>>> if (!DefMI)
>>> continue;
>>> - checkRematerializable(OrigVNI, DefMI, aa);
>>> + checkRematerializable(VNI, DefMI, aa);
>>> }
>>> ScannedRemattable = true;
>>> }
>>> @@ -116,18 +113,24 @@ bool LiveRangeEdit::allUsesAvailableAt(c
>>> return true;
>>> }
>>>
>>> -bool LiveRangeEdit::canRematerializeAt(Remat &RM, VNInfo *OrigVNI,
>>> - SlotIndex UseIdx, bool
>>> cheapAsAMove) {
>>> +bool LiveRangeEdit::canRematerializeAt(Remat &RM,
>>> + SlotIndex UseIdx,
>>> + bool cheapAsAMove) {
>>> assert(ScannedRemattable && "Call anyRematerializable first");
>>>
>>> // Use scanRemattable info.
>>> - if (!Remattable.count(OrigVNI))
>>> + if (!Remattable.count(RM.ParentVNI))
>>> return false;
>>>
>>> // No defining instruction provided.
>>> SlotIndex DefIdx;
>>> - assert(RM.OrigMI && "No defining instruction for remattable value");
>>> - DefIdx = LIS.getInstructionIndex(*RM.OrigMI);
>>> + if (RM.OrigMI)
>>> + DefIdx = LIS.getInstructionIndex(*RM.OrigMI);
>>> + else {
>>> + DefIdx = RM.ParentVNI->def;
>>> + RM.OrigMI = LIS.getInstructionFromIndex(DefIdx);
>>> + assert(RM.OrigMI && "No defining instruction for remattable value");
>>> + }
>>>
>>> // If only cheap remats were requested, bail out early.
>>> if (cheapAsAMove && !TII.isAsCheapAsAMove(RM.OrigMI))
>>> @@ -258,15 +261,6 @@ void LiveRangeEdit::eliminateDeadDef(Mac
>>> // Collect virtual registers to be erased after MI is gone.
>>> SmallVector<unsigned, 8> RegsToErase;
>>> bool ReadsPhysRegs = false;
>>> - bool isOrigDef = false;
>>> - unsigned Dest;
>>> - if (VRM && MI->getOperand(0).isReg()) {
>>> - Dest = MI->getOperand(0).getReg();
>>> - unsigned Original = VRM->getOriginal(Dest);
>>> - LiveInterval &OrigLI = LIS.getInterval(Original);
>>> - VNInfo *OrigVNI = OrigLI.getVNInfoAt(Idx);
>>> - isOrigDef = SlotIndex::isSameInstr(OrigVNI->def, Idx);
>>> - }
>>>
>>> // Check for live intervals that may shrink
>>> for (MachineInstr::mop_iterator MOI = MI->operands_begin(),
>>> @@ -320,24 +314,11 @@ void LiveRangeEdit::eliminateDeadDef(Mac
>>> }
>>> DEBUG(dbgs() << "Converted physregs to:\t" << *MI);
>>> } else {
>>> - // If the dest of MI is an original reg, don't delete the inst.
>>> Replace
>>> - // the dest with a new reg, keep the inst for remat of other
>>> siblings.
>>> - // The inst is saved in LiveRangeEdit::DeadRemats and will be deleted
>>> - // after all the allocations of the func are done.
>>> - if (isOrigDef) {
>>> - unsigned NewDest = createFrom(Dest);
>>> - pop_back();
>>> - markDeadRemat(MI);
>>> - const TargetRegisterInfo &TRI = *MRI.getTargetRegisterInfo();
>>> - MI->substituteRegister(Dest, NewDest, 0, TRI);
>>> - MI->getOperand(0).setIsDead(false);
>>> - } else {
>>> - if (TheDelegate)
>>> - TheDelegate->LRE_WillEraseInstruction(MI);
>>> - LIS.RemoveMachineInstrFromMaps(*MI);
>>> - MI->eraseFromParent();
>>> - ++NumDCEDeleted;
>>> - }
>>> + if (TheDelegate)
>>> + TheDelegate->LRE_WillEraseInstruction(MI);
>>> + LIS.RemoveMachineInstrFromMaps(*MI);
>>> + MI->eraseFromParent();
>>> + ++NumDCEDeleted;
>>> }
>>>
>>> // Erase any virtregs that are now empty and unused. There may be
>>> <undef>
>>> @@ -351,9 +332,8 @@ void LiveRangeEdit::eliminateDeadDef(Mac
>>> }
>>> }
>>>
>>> -void LiveRangeEdit::eliminateDeadDefs(SmallVectorImpl<MachineInstr *>
>>> &Dead,
>>> - ArrayRef<unsigned>
>>> RegsBeingSpilled,
>>> - bool NoSplit) {
>>> +void LiveRangeEdit::eliminateDeadDefs(SmallVectorImpl<MachineInstr*>
>>> &Dead,
>>> + ArrayRef<unsigned>
>>> RegsBeingSpilled) {
>>> ToShrinkSet ToShrink;
>>>
>>> for (;;) {
>>> @@ -375,9 +355,6 @@ void LiveRangeEdit::eliminateDeadDefs(Sm
>>> if (!LIS.shrinkToUses(LI, &Dead))
>>> continue;
>>>
>>> - if (NoSplit)
>>> - continue;
>>> -
>>> // Don't create new intervals for a register being spilled.
>>> // The new intervals would have to be spilled anyway so its not worth
>>> it.
>>> // Also they currently aren't spilled so creating them and not
>>> spilling
>>>
>>> Modified: llvm/trunk/lib/CodeGen/RegAllocBase.cpp
>>> URL:
>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/RegAllocBase.cpp?rev=265317&r1=265316&r2=265317&view=diff
>>>
>>> ==============================================================================
>>> --- llvm/trunk/lib/CodeGen/RegAllocBase.cpp (original)
>>> +++ llvm/trunk/lib/CodeGen/RegAllocBase.cpp Mon Apr 4 12:45:03 2016
>>> @@ -153,12 +153,3 @@ void RegAllocBase::allocatePhysRegs() {
>>> }
>>> }
>>> }
>>> -
>>> -void RegAllocBase::postOptimization() {
>>> - spiller().postOptimization();
>>> - for (auto DeadInst : DeadRemats) {
>>> - LIS->RemoveMachineInstrFromMaps(*DeadInst);
>>> - DeadInst->eraseFromParent();
>>> - }
>>> - DeadRemats.clear();
>>> -}
>>>
>>> Modified: llvm/trunk/lib/CodeGen/RegAllocBase.h
>>> URL:
>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/RegAllocBase.h?rev=265317&r1=265316&r2=265317&view=diff
>>>
>>> ==============================================================================
>>> --- llvm/trunk/lib/CodeGen/RegAllocBase.h (original)
>>> +++ llvm/trunk/lib/CodeGen/RegAllocBase.h Mon Apr 4 12:45:03 2016
>>> @@ -65,12 +65,6 @@ protected:
>>> LiveRegMatrix *Matrix;
>>> RegisterClassInfo RegClassInfo;
>>>
>>> - /// Inst which is a def of an original reg and whose defs are already
>>> all
>>> - /// dead after remat is saved in DeadRemats. The deletion of such inst
>>> is
>>> - /// postponed till all the allocations are done, so its remat expr is
>>> - /// always available for the remat of all the siblings of the original
>>> reg.
>>> - SmallPtrSet<MachineInstr *, 32> DeadRemats;
>>> -
>>> RegAllocBase()
>>> : TRI(nullptr), MRI(nullptr), VRM(nullptr), LIS(nullptr),
>>> Matrix(nullptr) {}
>>>
>>> @@ -83,10 +77,6 @@ protected:
>>> // physical register assignments.
>>> void allocatePhysRegs();
>>>
>>> - // Include spiller post optimization and removing dead defs left
>>> because of
>>> - // rematerialization.
>>> - virtual void postOptimization();
>>> -
>>> // Get a temporary reference to a Spiller instance.
>>> virtual Spiller &spiller() = 0;
>>>
>>>
>>> Modified: llvm/trunk/lib/CodeGen/RegAllocBasic.cpp
>>> URL:
>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/RegAllocBasic.cpp?rev=265317&r1=265316&r2=265317&view=diff
>>>
>>> ==============================================================================
>>> --- llvm/trunk/lib/CodeGen/RegAllocBasic.cpp (original)
>>> +++ llvm/trunk/lib/CodeGen/RegAllocBasic.cpp Mon Apr 4 12:45:03 2016
>>> @@ -199,7 +199,7 @@ bool RABasic::spillInterferences(LiveInt
>>> Matrix->unassign(Spill);
>>>
>>> // Spill the extracted interval.
>>> - LiveRangeEdit LRE(&Spill, SplitVRegs, *MF, *LIS, VRM, nullptr,
>>> &DeadRemats);
>>> + LiveRangeEdit LRE(&Spill, SplitVRegs, *MF, *LIS, VRM);
>>> spiller().spill(LRE);
>>> }
>>> return true;
>>> @@ -258,7 +258,7 @@ unsigned RABasic::selectOrSplit(LiveInte
>>> DEBUG(dbgs() << "spilling: " << VirtReg << '\n');
>>> if (!VirtReg.isSpillable())
>>> return ~0u;
>>> - LiveRangeEdit LRE(&VirtReg, SplitVRegs, *MF, *LIS, VRM, nullptr,
>>> &DeadRemats);
>>> + LiveRangeEdit LRE(&VirtReg, SplitVRegs, *MF, *LIS, VRM);
>>> spiller().spill(LRE);
>>>
>>> // The live virtual register requesting allocation was spilled, so tell
>>> @@ -283,7 +283,6 @@ bool RABasic::runOnMachineFunction(Machi
>>> SpillerInstance.reset(createInlineSpiller(*this, *MF, *VRM));
>>>
>>> allocatePhysRegs();
>>> - postOptimization();
>>>
>>> // Diagnostic output before rewriting
>>> DEBUG(dbgs() << "Post alloc VirtRegMap:\n" << *VRM << "\n");
>>>
>>> Modified: llvm/trunk/lib/CodeGen/RegAllocGreedy.cpp
>>> URL:
>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/RegAllocGreedy.cpp?rev=265317&r1=265316&r2=265317&view=diff
>>>
>>> ==============================================================================
>>> --- llvm/trunk/lib/CodeGen/RegAllocGreedy.cpp (original)
>>> +++ llvm/trunk/lib/CodeGen/RegAllocGreedy.cpp Mon Apr 4 12:45:03 2016
>>> @@ -12,6 +12,7 @@
>>> //
>>>
>>> //===----------------------------------------------------------------------===//
>>>
>>> +#include "llvm/CodeGen/Passes.h"
>>> #include "AllocationOrder.h"
>>> #include "InterferenceCache.h"
>>> #include "LiveDebugVariables.h"
>>> @@ -32,7 +33,6 @@
>>> #include "llvm/CodeGen/MachineFunctionPass.h"
>>> #include "llvm/CodeGen/MachineLoopInfo.h"
>>> #include "llvm/CodeGen/MachineRegisterInfo.h"
>>> -#include "llvm/CodeGen/Passes.h"
>>> #include "llvm/CodeGen/RegAllocRegistry.h"
>>> #include "llvm/CodeGen/RegisterClassInfo.h"
>>> #include "llvm/CodeGen/VirtRegMap.h"
>>> @@ -44,7 +44,6 @@
>>> #include "llvm/Support/ErrorHandling.h"
>>> #include "llvm/Support/Timer.h"
>>> #include "llvm/Support/raw_ostream.h"
>>> -#include "llvm/Target/TargetInstrInfo.h"
>>> #include "llvm/Target/TargetSubtargetInfo.h"
>>> #include <queue>
>>>
>>> @@ -56,14 +55,14 @@ STATISTIC(NumGlobalSplits, "Number of sp
>>> STATISTIC(NumLocalSplits, "Number of split local live ranges");
>>> STATISTIC(NumEvicted, "Number of interferences evicted");
>>>
>>> -static cl::opt<SplitEditor::ComplementSpillMode> SplitSpillMode(
>>> - "split-spill-mode", cl::Hidden,
>>> - cl::desc("Spill mode for splitting live ranges"),
>>> - cl::values(clEnumValN(SplitEditor::SM_Partition, "default",
>>> "Default"),
>>> - clEnumValN(SplitEditor::SM_Size, "size", "Optimize for
>>> size"),
>>> - clEnumValN(SplitEditor::SM_Speed, "speed", "Optimize for
>>> speed"),
>>> - clEnumValEnd),
>>> - cl::init(SplitEditor::SM_Speed));
>>> +static cl::opt<SplitEditor::ComplementSpillMode>
>>> +SplitSpillMode("split-spill-mode", cl::Hidden,
>>> + cl::desc("Spill mode for splitting live ranges"),
>>> + cl::values(clEnumValN(SplitEditor::SM_Partition, "default", "Default"),
>>> + clEnumValN(SplitEditor::SM_Size, "size", "Optimize for
>>> size"),
>>> + clEnumValN(SplitEditor::SM_Speed, "speed", "Optimize for
>>> speed"),
>>> + clEnumValEnd),
>>> + cl::init(SplitEditor::SM_Partition));
>>>
>>> static cl::opt<unsigned>
>>> LastChanceRecoloringMaxDepth("lcr-max-depth", cl::Hidden,
>>> @@ -1466,7 +1465,7 @@ unsigned RAGreedy::doRegionSplit(LiveInt
>>> SmallVectorImpl<unsigned> &NewVRegs) {
>>> SmallVector<unsigned, 8> UsedCands;
>>> // Prepare split editor.
>>> - LiveRangeEdit LREdit(&VirtReg, NewVRegs, *MF, *LIS, VRM, this,
>>> &DeadRemats);
>>> + LiveRangeEdit LREdit(&VirtReg, NewVRegs, *MF, *LIS, VRM, this);
>>> SE->reset(LREdit, SplitSpillMode);
>>>
>>> // Assign all edge bundles to the preferred candidate, or NoCand.
>>> @@ -1514,7 +1513,7 @@ unsigned RAGreedy::tryBlockSplit(LiveInt
>>> assert(&SA->getParent() == &VirtReg && "Live range wasn't analyzed");
>>> unsigned Reg = VirtReg.reg;
>>> bool SingleInstrs =
>>> RegClassInfo.isProperSubClass(MRI->getRegClass(Reg));
>>> - LiveRangeEdit LREdit(&VirtReg, NewVRegs, *MF, *LIS, VRM, this,
>>> &DeadRemats);
>>> + LiveRangeEdit LREdit(&VirtReg, NewVRegs, *MF, *LIS, VRM, this);
>>> SE->reset(LREdit, SplitSpillMode);
>>> ArrayRef<SplitAnalysis::BlockInfo> UseBlocks = SA->getUseBlocks();
>>> for (unsigned i = 0; i != UseBlocks.size(); ++i) {
>>> @@ -1586,7 +1585,7 @@ RAGreedy::tryInstructionSplit(LiveInterv
>>>
>>> // Always enable split spill mode, since we're effectively spilling to
>>> a
>>> // register.
>>> - LiveRangeEdit LREdit(&VirtReg, NewVRegs, *MF, *LIS, VRM, this,
>>> &DeadRemats);
>>> + LiveRangeEdit LREdit(&VirtReg, NewVRegs, *MF, *LIS, VRM, this);
>>> SE->reset(LREdit, SplitEditor::SM_Size);
>>>
>>> ArrayRef<SlotIndex> Uses = SA->getUseSlots();
>>> @@ -1909,7 +1908,7 @@ unsigned RAGreedy::tryLocalSplit(LiveInt
>>> << '-' << Uses[BestAfter] << ", " << BestDiff
>>> << ", " << (BestAfter - BestBefore + 1) << " instrs\n");
>>>
>>> - LiveRangeEdit LREdit(&VirtReg, NewVRegs, *MF, *LIS, VRM, this,
>>> &DeadRemats);
>>> + LiveRangeEdit LREdit(&VirtReg, NewVRegs, *MF, *LIS, VRM, this);
>>> SE->reset(LREdit);
>>>
>>> SE->openIntv();
>>> @@ -2552,7 +2551,7 @@ unsigned RAGreedy::selectOrSplitImpl(Liv
>>> NewVRegs.push_back(VirtReg.reg);
>>> } else {
>>> NamedRegionTimer T("Spiller", TimerGroupName, TimePassesIsEnabled);
>>> - LiveRangeEdit LRE(&VirtReg, NewVRegs, *MF, *LIS, VRM, this,
>>> &DeadRemats);
>>> + LiveRangeEdit LRE(&VirtReg, NewVRegs, *MF, *LIS, VRM, this);
>>> spiller().spill(LRE);
>>> setStage(NewVRegs.begin(), NewVRegs.end(), RS_Done);
>>>
>>> @@ -2610,8 +2609,6 @@ bool RAGreedy::runOnMachineFunction(Mach
>>>
>>> allocatePhysRegs();
>>> tryHintsRecoloring();
>>> - postOptimization();
>>> -
>>> releaseMemory();
>>> return true;
>>> }
>>>
>>> Modified: llvm/trunk/lib/CodeGen/RegAllocPBQP.cpp
>>> URL:
>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/RegAllocPBQP.cpp?rev=265317&r1=265316&r2=265317&view=diff
>>>
>>> ==============================================================================
>>> --- llvm/trunk/lib/CodeGen/RegAllocPBQP.cpp (original)
>>> +++ llvm/trunk/lib/CodeGen/RegAllocPBQP.cpp Mon Apr 4 12:45:03 2016
>>> @@ -123,12 +123,6 @@ private:
>>>
>>> RegSet VRegsToAlloc, EmptyIntervalVRegs;
>>>
>>> - /// Inst which is a def of an original reg and whose defs are already
>>> all
>>> - /// dead after remat is saved in DeadRemats. The deletion of such inst
>>> is
>>> - /// postponed till all the allocations are done, so its remat expr is
>>> - /// always available for the remat of all the siblings of the original
>>> reg.
>>> - SmallPtrSet<MachineInstr *, 32> DeadRemats;
>>> -
>>> /// \brief Finds the initial set of vreg intervals to allocate.
>>> void findVRegIntervalsToAlloc(const MachineFunction &MF, LiveIntervals
>>> &LIS);
>>>
>>> @@ -152,7 +146,6 @@ private:
>>> void finalizeAlloc(MachineFunction &MF, L
>>
>>
More information about the llvm-commits
mailing list