<div dir="ltr">No worries, sounds like we have a good path forward to keep the tree healthy until you have stuff sorted out. Don't hesitate to reach out for help if you need it!</div><br><div class="gmail_quote"><div dir="ltr">On Mon, Apr 4, 2016 at 11:43 AM Wei Mi <<a href="mailto:wmi@google.com">wmi@google.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">My patch caused llvm bootstrap error on several targets. I am looking<br>
at it. Will fix the error as soon as possible.<br>
<br>
Wei.<br>
<br>
On Mon, Apr 4, 2016 at 11:39 AM, Quentin Colombet <<a href="mailto:qcolombet@apple.com" target="_blank">qcolombet@apple.com</a>> wrote:<br>
><br>
> On Apr 4, 2016, at 11:35 AM, Chandler Carruth <<a href="mailto:chandlerc@google.com" target="_blank">chandlerc@google.com</a>> wrote:<br>
><br>
> On Mon, Apr 4, 2016 at 10:50 AM Wei Mi via llvm-commits<br>
> <<a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a>> wrote:<br>
>><br>
>> Author: wmi<br>
>> Date: Mon Apr 4 12:45:03 2016<br>
>> New Revision: 265317<br>
>><br>
>> URL: <a href="http://llvm.org/viewvc/llvm-project?rev=265317&view=rev" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project?rev=265317&view=rev</a><br>
>> Log:<br>
>> Revert r265309 and r265312 because they caused some errors I need to<br>
>> investigate.<br>
><br>
><br>
> Ouch.<br>
><br>
> So, we were really counting on these to address a severe compile time<br>
> regression for the sanatizers with Quentin's patch in r263460.<br>
><br>
> Quentin, I wonder if it would make sense to revert r263460 until Wei's fixes<br>
> can land? Our ASan users are really having trouble because of the widespread<br>
> compile time regressions this causes. While I know you're patch fixes<br>
> compile time issues as well, Richard had indicated that we really should<br>
> prioritize not *regressing* compile time... Thoughts?<br>
><br>
><br>
> Works for me as long as we keep pushing this one further to reland the whole<br>
> thing.<br>
><br>
> Q.<br>
><br>
><br>
> -Chandler<br>
><br>
>><br>
>><br>
>> Added:<br>
>> llvm/trunk/test/CodeGen/AArch64/aarch64-deferred-spilling.ll<br>
>> - copied unchanged from r265308,<br>
>> llvm/trunk/test/CodeGen/AArch64/aarch64-deferred-spilling.ll<br>
>> Removed:<br>
>> llvm/trunk/test/CodeGen/X86/hoist-spill.ll<br>
>> llvm/trunk/test/CodeGen/X86/new-remat.ll<br>
>> Modified:<br>
>> llvm/trunk/include/llvm/CodeGen/LiveRangeEdit.h<br>
>> llvm/trunk/lib/CodeGen/InlineSpiller.cpp<br>
>> llvm/trunk/lib/CodeGen/LiveRangeEdit.cpp<br>
>> llvm/trunk/lib/CodeGen/RegAllocBase.cpp<br>
>> llvm/trunk/lib/CodeGen/RegAllocBase.h<br>
>> llvm/trunk/lib/CodeGen/RegAllocBasic.cpp<br>
>> llvm/trunk/lib/CodeGen/RegAllocGreedy.cpp<br>
>> llvm/trunk/lib/CodeGen/RegAllocPBQP.cpp<br>
>> llvm/trunk/lib/CodeGen/Spiller.h<br>
>> llvm/trunk/lib/CodeGen/SplitKit.cpp<br>
>> llvm/trunk/lib/CodeGen/SplitKit.h<br>
>> llvm/trunk/test/CodeGen/X86/fp128-compare.ll<br>
>> llvm/trunk/test/CodeGen/X86/ragreedy-hoist-spill.ll<br>
>><br>
>> Modified: llvm/trunk/include/llvm/CodeGen/LiveRangeEdit.h<br>
>> URL:<br>
>> <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/LiveRangeEdit.h?rev=265317&r1=265316&r2=265317&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/LiveRangeEdit.h?rev=265317&r1=265316&r2=265317&view=diff</a><br>
>><br>
>> ==============================================================================<br>
>> --- llvm/trunk/include/llvm/CodeGen/LiveRangeEdit.h (original)<br>
>> +++ llvm/trunk/include/llvm/CodeGen/LiveRangeEdit.h Mon Apr 4 12:45:03<br>
>> 2016<br>
>> @@ -72,10 +72,6 @@ private:<br>
>> /// ScannedRemattable - true when remattable values have been<br>
>> identified.<br>
>> bool ScannedRemattable;<br>
>><br>
>> - /// DeadRemats - The saved instructions which have already been dead<br>
>> after<br>
>> - /// rematerialization but not deleted yet -- to be done in<br>
>> postOptimization.<br>
>> - SmallPtrSet<MachineInstr *, 32> *DeadRemats;<br>
>> -<br>
>> /// Remattable - Values defined by remattable instructions as<br>
>> identified by<br>
>> /// tii.isTriviallyReMaterializable().<br>
>> SmallPtrSet<const VNInfo*,4> Remattable;<br>
>> @@ -120,16 +116,13 @@ public:<br>
>> /// @param vrm Map of virtual registers to physical registers for this<br>
>> /// function. If NULL, no virtual register map updates will<br>
>> /// be done. This could be the case if called before<br>
>> Regalloc.<br>
>> - /// @param deadRemats The collection of all the instructions defining<br>
>> an<br>
>> - /// original reg and are dead after remat.<br>
>> LiveRangeEdit(LiveInterval *parent, SmallVectorImpl<unsigned> &newRegs,<br>
>> MachineFunction &MF, LiveIntervals &lis, VirtRegMap *vrm,<br>
>> - Delegate *delegate = nullptr,<br>
>> - SmallPtrSet<MachineInstr *, 32> *deadRemats = nullptr)<br>
>> + Delegate *delegate = nullptr)<br>
>> : Parent(parent), NewRegs(newRegs), MRI(MF.getRegInfo()), LIS(lis),<br>
>> - VRM(vrm), TII(*MF.getSubtarget().getInstrInfo()),<br>
>> TheDelegate(delegate),<br>
>> - FirstNew(newRegs.size()), ScannedRemattable(false),<br>
>> - DeadRemats(deadRemats) {<br>
>> + VRM(vrm), TII(*MF.getSubtarget().getInstrInfo()),<br>
>> + TheDelegate(delegate), FirstNew(newRegs.size()),<br>
>> + ScannedRemattable(false) {<br>
>> MRI.setDelegate(this);<br>
>> }<br>
>><br>
>> @@ -149,16 +142,6 @@ public:<br>
>> bool empty() const { return size() == 0; }<br>
>> unsigned get(unsigned idx) const { return NewRegs[idx+FirstNew]; }<br>
>><br>
>> - /// pop_back - It allows LiveRangeEdit users to drop new registers.<br>
>> - /// The context is when an original def instruction of a register is<br>
>> - /// dead after rematerialization, we still want to keep it for<br>
>> following<br>
>> - /// rematerializations. We save the def instruction in DeadRemats,<br>
>> - /// and replace the original dst register with a new dummy register so<br>
>> - /// the live range of original dst register can be shrinked normally.<br>
>> - /// We don't want to allocate phys register for the dummy register, so<br>
>> - /// we want to drop it from the NewRegs set.<br>
>> - void pop_back() { NewRegs.pop_back(); }<br>
>> -<br>
>> ArrayRef<unsigned> regs() const {<br>
>> return makeArrayRef(NewRegs).slice(FirstNew);<br>
>> }<br>
>> @@ -192,15 +175,15 @@ public:<br>
>> /// Remat - Information needed to rematerialize at a specific location.<br>
>> struct Remat {<br>
>> VNInfo *ParentVNI; // parent_'s value at the remat location.<br>
>> - MachineInstr *OrigMI; // Instruction defining OrigVNI. It contains<br>
>> the<br>
>> - // real expr for remat.<br>
>> + MachineInstr *OrigMI; // Instruction defining ParentVNI.<br>
>> explicit Remat(VNInfo *ParentVNI) : ParentVNI(ParentVNI),<br>
>> OrigMI(nullptr) {}<br>
>> };<br>
>><br>
>> /// canRematerializeAt - Determine if ParentVNI can be rematerialized<br>
>> at<br>
>> /// UseIdx. It is assumed that parent_.getVNINfoAt(UseIdx) ==<br>
>> ParentVNI.<br>
>> /// When cheapAsAMove is set, only cheap remats are allowed.<br>
>> - bool canRematerializeAt(Remat &RM, VNInfo *OrigVNI, SlotIndex UseIdx,<br>
>> + bool canRematerializeAt(Remat &RM,<br>
>> + SlotIndex UseIdx,<br>
>> bool cheapAsAMove);<br>
>><br>
>> /// rematerializeAt - Rematerialize RM.ParentVNI into DestReg by<br>
>> inserting an<br>
>> @@ -225,12 +208,6 @@ public:<br>
>> return Rematted.count(ParentVNI);<br>
>> }<br>
>><br>
>> - void markDeadRemat(MachineInstr *inst) {<br>
>> - // DeadRemats is an optional field.<br>
>> - if (DeadRemats)<br>
>> - DeadRemats->insert(inst);<br>
>> - }<br>
>> -<br>
>> /// eraseVirtReg - Notify the delegate that Reg is no longer in use,<br>
>> and try<br>
>> /// to erase it from LIS.<br>
>> void eraseVirtReg(unsigned Reg);<br>
>> @@ -241,11 +218,8 @@ public:<br>
>> /// RegsBeingSpilled lists registers currently being spilled by the<br>
>> register<br>
>> /// allocator. These registers should not be split into new intervals<br>
>> /// as currently those new intervals are not guaranteed to spill.<br>
>> - /// NoSplit indicates this func is used after the iterations of<br>
>> selectOrSplit<br>
>> - /// where registers should not be split into new intervals.<br>
>> - void eliminateDeadDefs(SmallVectorImpl<MachineInstr *> &Dead,<br>
>> - ArrayRef<unsigned> RegsBeingSpilled = None,<br>
>> - bool NoSplit = false);<br>
>> + void eliminateDeadDefs(SmallVectorImpl<MachineInstr*> &Dead,<br>
>> + ArrayRef<unsigned> RegsBeingSpilled = None);<br>
>><br>
>> /// calculateRegClassAndHint - Recompute register class and hint for<br>
>> each new<br>
>> /// register.<br>
>><br>
>> Modified: llvm/trunk/lib/CodeGen/InlineSpiller.cpp<br>
>> URL:<br>
>> <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/InlineSpiller.cpp?rev=265317&r1=265316&r2=265317&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/InlineSpiller.cpp?rev=265317&r1=265316&r2=265317&view=diff</a><br>
>><br>
>> ==============================================================================<br>
>> --- llvm/trunk/lib/CodeGen/InlineSpiller.cpp (original)<br>
>> +++ llvm/trunk/lib/CodeGen/InlineSpiller.cpp Mon Apr 4 12:45:03 2016<br>
>> @@ -48,77 +48,13 @@ STATISTIC(NumReloadsRemoved, "Number of<br>
>> STATISTIC(NumFolded, "Number of folded stack accesses");<br>
>> STATISTIC(NumFoldedLoads, "Number of folded loads");<br>
>> STATISTIC(NumRemats, "Number of rematerialized defs for<br>
>> spilling");<br>
>> +STATISTIC(NumOmitReloadSpill, "Number of omitted spills of reloads");<br>
>> +STATISTIC(NumHoists, "Number of hoisted spills");<br>
>><br>
>> static cl::opt<bool> DisableHoisting("disable-spill-hoist", cl::Hidden,<br>
>> cl::desc("Disable inline spill<br>
>> hoisting"));<br>
>><br>
>> namespace {<br>
>> -class HoistSpillHelper {<br>
>> - LiveIntervals &LIS;<br>
>> - LiveStacks &LSS;<br>
>> - AliasAnalysis *AA;<br>
>> - MachineDominatorTree &MDT;<br>
>> - MachineLoopInfo &Loops;<br>
>> - VirtRegMap &VRM;<br>
>> - MachineFrameInfo &MFI;<br>
>> - MachineRegisterInfo &MRI;<br>
>> - const TargetInstrInfo &TII;<br>
>> - const TargetRegisterInfo &TRI;<br>
>> - const MachineBlockFrequencyInfo &MBFI;<br>
>> -<br>
>> - // Map from StackSlot to its original register.<br>
>> - DenseMap<int, unsigned> StackSlotToReg;<br>
>> - // Map from pair of (StackSlot and Original VNI) to a set of spills<br>
>> which<br>
>> - // have the same stackslot and have equal values defined by Original<br>
>> VNI.<br>
>> - // These spills are mergeable and are hoist candiates.<br>
>> - typedef DenseMap<std::pair<int, VNInfo *>, SmallPtrSet<MachineInstr *,<br>
>> 16>><br>
>> - MergeableSpillsMap;<br>
>> - MergeableSpillsMap MergeableSpills;<br>
>> -<br>
>> - /// This is the map from original register to a set containing all its<br>
>> - /// siblings. To hoist a spill to another BB, we need to find out a<br>
>> live<br>
>> - /// sibling there and use it as the source of the new spill.<br>
>> - DenseMap<unsigned, SmallSetVector<unsigned, 16>> Virt2SiblingsMap;<br>
>> -<br>
>> - bool isSpillCandBB(unsigned OrigReg, VNInfo &OrigVNI, MachineBasicBlock<br>
>> &BB,<br>
>> - unsigned &LiveReg);<br>
>> -<br>
>> - void rmRedundantSpills(<br>
>> - SmallPtrSet<MachineInstr *, 16> &Spills,<br>
>> - SmallVectorImpl<MachineInstr *> &SpillsToRm,<br>
>> - DenseMap<MachineDomTreeNode *, MachineInstr *> &SpillBBToSpill);<br>
>> -<br>
>> - void getVisitOrders(<br>
>> - MachineBasicBlock *Root, SmallPtrSet<MachineInstr *, 16> &Spills,<br>
>> - SmallVectorImpl<MachineDomTreeNode *> &Orders,<br>
>> - SmallVectorImpl<MachineInstr *> &SpillsToRm,<br>
>> - DenseMap<MachineDomTreeNode *, unsigned> &SpillsToKeep,<br>
>> - DenseMap<MachineDomTreeNode *, MachineInstr *> &SpillBBToSpill);<br>
>> -<br>
>> - void runHoistSpills(unsigned OrigReg, VNInfo &OrigVNI,<br>
>> - SmallPtrSet<MachineInstr *, 16> &Spills,<br>
>> - SmallVectorImpl<MachineInstr *> &SpillsToRm,<br>
>> - DenseMap<MachineBasicBlock *, unsigned><br>
>> &SpillsToIns);<br>
>> -<br>
>> -public:<br>
>> - HoistSpillHelper(MachineFunctionPass &pass, MachineFunction &mf,<br>
>> - VirtRegMap &vrm)<br>
>> - : LIS(pass.getAnalysis<LiveIntervals>()),<br>
>> - LSS(pass.getAnalysis<LiveStacks>()),<br>
>> - AA(&pass.getAnalysis<AAResultsWrapperPass>().getAAResults()),<br>
>> - MDT(pass.getAnalysis<MachineDominatorTree>()),<br>
>> - Loops(pass.getAnalysis<MachineLoopInfo>()), VRM(vrm),<br>
>> - MFI(*mf.getFrameInfo()), MRI(mf.getRegInfo()),<br>
>> - TII(*mf.getSubtarget().getInstrInfo()),<br>
>> - TRI(*mf.getSubtarget().getRegisterInfo()),<br>
>> - MBFI(pass.getAnalysis<MachineBlockFrequencyInfo>()) {}<br>
>> -<br>
>> - void addToMergeableSpills(MachineInstr *Spill, int StackSlot,<br>
>> - unsigned Original);<br>
>> - bool rmFromMergeableSpills(MachineInstr *Spill, int StackSlot);<br>
>> - void hoistAllSpills(LiveRangeEdit &Edit);<br>
>> -};<br>
>> -<br>
>> class InlineSpiller : public Spiller {<br>
>> MachineFunction &MF;<br>
>> LiveIntervals &LIS;<br>
>> @@ -149,12 +85,56 @@ class InlineSpiller : public Spiller {<br>
>> // Values that failed to remat at some point.<br>
>> SmallPtrSet<VNInfo*, 8> UsedValues;<br>
>><br>
>> +public:<br>
>> + // Information about a value that was defined by a copy from a sibling<br>
>> + // register.<br>
>> + struct SibValueInfo {<br>
>> + // True when all reaching defs were reloads: No spill is necessary.<br>
>> + bool AllDefsAreReloads;<br>
>> +<br>
>> + // True when value is defined by an original PHI not from splitting.<br>
>> + bool DefByOrigPHI;<br>
>> +<br>
>> + // True when the COPY defining this value killed its source.<br>
>> + bool KillsSource;<br>
>> +<br>
>> + // The preferred register to spill.<br>
>> + unsigned SpillReg;<br>
>> +<br>
>> + // The value of SpillReg that should be spilled.<br>
>> + VNInfo *SpillVNI;<br>
>> +<br>
>> + // The block where SpillVNI should be spilled. Currently, this must<br>
>> be the<br>
>> + // block containing SpillVNI->def.<br>
>> + MachineBasicBlock *SpillMBB;<br>
>> +<br>
>> + // A defining instruction that is not a sibling copy or a reload, or<br>
>> NULL.<br>
>> + // This can be used as a template for rematerialization.<br>
>> + MachineInstr *DefMI;<br>
>> +<br>
>> + // List of values that depend on this one. These values are actually<br>
>> the<br>
>> + // same, but live range splitting has placed them in different<br>
>> registers,<br>
>> + // or SSA update needed to insert PHI-defs to preserve SSA form.<br>
>> This is<br>
>> + // copies of the current value and phi-kills. Usually only phi-kills<br>
>> cause<br>
>> + // more than one dependent value.<br>
>> + TinyPtrVector<VNInfo*> Deps;<br>
>> +<br>
>> + SibValueInfo(unsigned Reg, VNInfo *VNI)<br>
>> + : AllDefsAreReloads(true), DefByOrigPHI(false), KillsSource(false),<br>
>> + SpillReg(Reg), SpillVNI(VNI), SpillMBB(nullptr), DefMI(nullptr)<br>
>> {}<br>
>> +<br>
>> + // Returns true when a def has been found.<br>
>> + bool hasDef() const { return DefByOrigPHI || DefMI; }<br>
>> + };<br>
>> +<br>
>> +private:<br>
>> + // Values in RegsToSpill defined by sibling copies.<br>
>> + typedef DenseMap<VNInfo*, SibValueInfo> SibValueMap;<br>
>> + SibValueMap SibValues;<br>
>> +<br>
>> // Dead defs generated during spilling.<br>
>> SmallVector<MachineInstr*, 8> DeadDefs;<br>
>><br>
>> - // Object records spills information and does the hoisting.<br>
>> - HoistSpillHelper HSpiller;<br>
>> -<br>
>> ~InlineSpiller() override {}<br>
>><br>
>> public:<br>
>> @@ -167,11 +147,9 @@ public:<br>
>> MFI(*mf.getFrameInfo()), MRI(mf.getRegInfo()),<br>
>> TII(*mf.getSubtarget().getInstrInfo()),<br>
>> TRI(*mf.getSubtarget().getRegisterInfo()),<br>
>> - MBFI(pass.getAnalysis<MachineBlockFrequencyInfo>()),<br>
>> - HSpiller(pass, mf, vrm) {}<br>
>> + MBFI(pass.getAnalysis<MachineBlockFrequencyInfo>()) {}<br>
>><br>
>> void spill(LiveRangeEdit &) override;<br>
>> - void postOptimization();<br>
>><br>
>> private:<br>
>> bool isSnippet(const LiveInterval &SnipLI);<br>
>> @@ -183,7 +161,11 @@ private:<br>
>> }<br>
>><br>
>> bool isSibling(unsigned Reg);<br>
>> - bool hoistSpillInsideBB(LiveInterval &SpillLI, MachineInstr &CopyMI);<br>
>> + MachineInstr *traceSiblingValue(unsigned, VNInfo*, VNInfo*);<br>
>> + void propagateSiblingValue(SibValueMap::iterator, VNInfo *VNI =<br>
>> nullptr);<br>
>> + void analyzeSiblingValues();<br>
>> +<br>
>> + bool hoistSpill(LiveInterval &SpillLI, MachineInstr &CopyMI);<br>
>> void eliminateRedundantSpills(LiveInterval &LI, VNInfo *VNI);<br>
>><br>
>> void markValueUsed(LiveInterval*, VNInfo*);<br>
>> @@ -315,44 +297,417 @@ void InlineSpiller::collectRegsToSpill()<br>
>> }<br>
>> }<br>
>><br>
>> +<br>
>><br>
>> +//===----------------------------------------------------------------------===//<br>
>> +// Sibling Values<br>
>><br>
>> +//===----------------------------------------------------------------------===//<br>
>> +<br>
>> +// After live range splitting, some values to be spilled may be defined<br>
>> by<br>
>> +// copies from sibling registers. We trace the sibling copies back to the<br>
>> +// original value if it still exists. We need it for rematerialization.<br>
>> +//<br>
>> +// Even when the value can't be rematerialized, we still want to<br>
>> determine if<br>
>> +// the value has already been spilled, or we may want to hoist the spill<br>
>> from a<br>
>> +// loop.<br>
>> +<br>
>> bool InlineSpiller::isSibling(unsigned Reg) {<br>
>> return TargetRegisterInfo::isVirtualRegister(Reg) &&<br>
>> VRM.getOriginal(Reg) == Original;<br>
>> }<br>
>><br>
>> -/// It is beneficial to spill to earlier place in the same BB in case<br>
>> -/// as follows:<br>
>> -/// There is an alternative def earlier in the same MBB.<br>
>> -/// Hoist the spill as far as possible in SpillMBB. This can ease<br>
>> -/// register pressure:<br>
>> +#ifndef NDEBUG<br>
>> +static raw_ostream &operator<<(raw_ostream &OS,<br>
>> + const InlineSpiller::SibValueInfo &SVI) {<br>
>> + OS << "spill " << PrintReg(SVI.SpillReg) << ':'<br>
>> + << SVI.SpillVNI->id << '@' << SVI.SpillVNI->def;<br>
>> + if (SVI.SpillMBB)<br>
>> + OS << " in BB#" << SVI.SpillMBB->getNumber();<br>
>> + if (SVI.AllDefsAreReloads)<br>
>> + OS << " all-reloads";<br>
>> + if (SVI.DefByOrigPHI)<br>
>> + OS << " orig-phi";<br>
>> + if (SVI.KillsSource)<br>
>> + OS << " kill";<br>
>> + OS << " deps[";<br>
>> + for (VNInfo *Dep : SVI.Deps)<br>
>> + OS << ' ' << Dep->id << '@' << Dep->def;<br>
>> + OS << " ]";<br>
>> + if (SVI.DefMI)<br>
>> + OS << " def: " << *SVI.DefMI;<br>
>> + else<br>
>> + OS << '\n';<br>
>> + return OS;<br>
>> +}<br>
>> +#endif<br>
>> +<br>
>> +/// propagateSiblingValue - Propagate the value in SVI to dependents if<br>
>> it is<br>
>> +/// known. Otherwise remember the dependency for later.<br>
>> ///<br>
>> -/// x = def<br>
>> -/// y = use x<br>
>> -/// s = copy x<br>
>> +/// @param SVIIter SibValues entry to propagate.<br>
>> +/// @param VNI Dependent value, or NULL to propagate to all saved<br>
>> dependents.<br>
>> +void InlineSpiller::propagateSiblingValue(SibValueMap::iterator SVIIter,<br>
>> + VNInfo *VNI) {<br>
>> + SibValueMap::value_type *SVI = &*SVIIter;<br>
>> +<br>
>> + // When VNI is non-NULL, add it to SVI's deps, and only propagate to<br>
>> that.<br>
>> + TinyPtrVector<VNInfo*> FirstDeps;<br>
>> + if (VNI) {<br>
>> + FirstDeps.push_back(VNI);<br>
>> + SVI->second.Deps.push_back(VNI);<br>
>> + }<br>
>> +<br>
>> + // Has the value been completely determined yet? If not, defer<br>
>> propagation.<br>
>> + if (!SVI->second.hasDef())<br>
>> + return;<br>
>> +<br>
>> + // Work list of values to propagate.<br>
>> + SmallSetVector<SibValueMap::value_type *, 8> WorkList;<br>
>> + WorkList.insert(SVI);<br>
>> +<br>
>> + do {<br>
>> + SVI = WorkList.pop_back_val();<br>
>> + TinyPtrVector<VNInfo*> *Deps = VNI ? &FirstDeps : &SVI->second.Deps;<br>
>> + VNI = nullptr;<br>
>> +<br>
>> + SibValueInfo &SV = SVI->second;<br>
>> + if (!SV.SpillMBB)<br>
>> + SV.SpillMBB = LIS.getMBBFromIndex(SV.SpillVNI->def);<br>
>> +<br>
>> + DEBUG(dbgs() << " prop to " << Deps->size() << ": "<br>
>> + << SVI->first->id << '@' << SVI->first->def << ":\t" <<<br>
>> SV);<br>
>> +<br>
>> + assert(SV.hasDef() && "Propagating undefined value");<br>
>> +<br>
>> + // Should this value be propagated as a preferred spill candidate?<br>
>> We don't<br>
>> + // propagate values of registers that are about to spill.<br>
>> + bool PropSpill = !DisableHoisting && !isRegToSpill(SV.SpillReg);<br>
>> + unsigned SpillDepth = ~0u;<br>
>> +<br>
>> + for (VNInfo *Dep : *Deps) {<br>
>> + SibValueMap::iterator DepSVI = SibValues.find(Dep);<br>
>> + assert(DepSVI != SibValues.end() && "Dependent value not in<br>
>> SibValues");<br>
>> + SibValueInfo &DepSV = DepSVI->second;<br>
>> + if (!DepSV.SpillMBB)<br>
>> + DepSV.SpillMBB = LIS.getMBBFromIndex(DepSV.SpillVNI->def);<br>
>> +<br>
>> + bool Changed = false;<br>
>> +<br>
>> + // Propagate defining instruction.<br>
>> + if (!DepSV.hasDef()) {<br>
>> + Changed = true;<br>
>> + DepSV.DefMI = SV.DefMI;<br>
>> + DepSV.DefByOrigPHI = SV.DefByOrigPHI;<br>
>> + }<br>
>> +<br>
>> + // Propagate AllDefsAreReloads. For PHI values, this computes an<br>
>> AND of<br>
>> + // all predecessors.<br>
>> + if (!SV.AllDefsAreReloads && DepSV.AllDefsAreReloads) {<br>
>> + Changed = true;<br>
>> + DepSV.AllDefsAreReloads = false;<br>
>> + }<br>
>> +<br>
>> + // Propagate best spill value.<br>
>> + if (PropSpill && SV.SpillVNI != DepSV.SpillVNI) {<br>
>> + if (SV.SpillMBB == DepSV.SpillMBB) {<br>
>> + // DepSV is in the same block. Hoist when dominated.<br>
>> + if (DepSV.KillsSource && SV.SpillVNI->def <<br>
>> DepSV.SpillVNI->def) {<br>
>> + // This is an alternative def earlier in the same MBB.<br>
>> + // Hoist the spill as far as possible in SpillMBB. This can<br>
>> ease<br>
>> + // register pressure:<br>
>> + //<br>
>> + // x = def<br>
>> + // y = use x<br>
>> + // s = copy x<br>
>> + //<br>
>> + // Hoisting the spill of s to immediately after the def<br>
>> removes the<br>
>> + // interference between x and y:<br>
>> + //<br>
>> + // x = def<br>
>> + // spill x<br>
>> + // y = use x<kill><br>
>> + //<br>
>> + // This hoist only helps when the DepSV copy kills its<br>
>> source.<br>
>> + Changed = true;<br>
>> + DepSV.SpillReg = SV.SpillReg;<br>
>> + DepSV.SpillVNI = SV.SpillVNI;<br>
>> + DepSV.SpillMBB = SV.SpillMBB;<br>
>> + }<br>
>> + } else {<br>
>> + // DepSV is in a different block.<br>
>> + if (SpillDepth == ~0u)<br>
>> + SpillDepth = Loops.getLoopDepth(SV.SpillMBB);<br>
>> +<br>
>> + // Also hoist spills to blocks with smaller loop depth, but<br>
>> make sure<br>
>> + // that the new value dominates. Non-phi dependents are always<br>
>> + // dominated, phis need checking.<br>
>> +<br>
>> + const BranchProbability MarginProb(4, 5); // 80%<br>
>> + // Hoist a spill to outer loop if there are multiple dependents<br>
>> (it<br>
>> + // can be beneficial if more than one dependents are hoisted)<br>
>> or<br>
>> + // if DepSV (the hoisting source) is hotter than SV (the<br>
>> hoisting<br>
>> + // destination) (we add a 80% margin to bias a little towards<br>
>> + // loop depth).<br>
>> + bool HoistCondition =<br>
>> + (MBFI.getBlockFreq(DepSV.SpillMBB) >=<br>
>> + (MBFI.getBlockFreq(SV.SpillMBB) * MarginProb)) ||<br>
>> + Deps->size() > 1;<br>
>> +<br>
>> + if ((Loops.getLoopDepth(DepSV.SpillMBB) > SpillDepth) &&<br>
>> + HoistCondition &&<br>
>> + (!DepSVI->first->isPHIDef() ||<br>
>> + MDT.dominates(SV.SpillMBB, DepSV.SpillMBB))) {<br>
>> + Changed = true;<br>
>> + DepSV.SpillReg = SV.SpillReg;<br>
>> + DepSV.SpillVNI = SV.SpillVNI;<br>
>> + DepSV.SpillMBB = SV.SpillMBB;<br>
>> + }<br>
>> + }<br>
>> + }<br>
>> +<br>
>> + if (!Changed)<br>
>> + continue;<br>
>> +<br>
>> + // Something changed in DepSVI. Propagate to dependents.<br>
>> + WorkList.insert(&*DepSVI);<br>
>> +<br>
>> + DEBUG(dbgs() << " update " << DepSVI->first->id << '@'<br>
>> + << DepSVI->first->def << " to:\t" << DepSV);<br>
>> + }<br>
>> + } while (!WorkList.empty());<br>
>> +}<br>
>> +<br>
>> +/// traceSiblingValue - Trace a value that is about to be spilled back to<br>
>> the<br>
>> +/// real defining instructions by looking through sibling copies. Always<br>
>> stay<br>
>> +/// within the range of OrigVNI so the registers are known to carry the<br>
>> same<br>
>> +/// value.<br>
>> ///<br>
>> -/// Hoisting the spill of s to immediately after the def removes the<br>
>> -/// interference between x and y:<br>
>> +/// Determine if the value is defined by all reloads, so spilling isn't<br>
>> +/// necessary - the value is already in the stack slot.<br>
>> ///<br>
>> -/// x = def<br>
>> -/// spill x<br>
>> -/// y = use x<kill><br>
>> +/// Return a defining instruction that may be a candidate for<br>
>> rematerialization.<br>
>> ///<br>
>> -/// This hoist only helps when the copy kills its source.<br>
>> +MachineInstr *InlineSpiller::traceSiblingValue(unsigned UseReg, VNInfo<br>
>> *UseVNI,<br>
>> + VNInfo *OrigVNI) {<br>
>> + // Check if a cached value already exists.<br>
>> + SibValueMap::iterator SVI;<br>
>> + bool Inserted;<br>
>> + std::tie(SVI, Inserted) =<br>
>> + SibValues.insert(std::make_pair(UseVNI, SibValueInfo(UseReg,<br>
>> UseVNI)));<br>
>> + if (!Inserted) {<br>
>> + DEBUG(dbgs() << "Cached value " << PrintReg(UseReg) << ':'<br>
>> + << UseVNI->id << '@' << UseVNI->def << ' ' <<<br>
>> SVI->second);<br>
>> + return SVI->second.DefMI;<br>
>> + }<br>
>> +<br>
>> + DEBUG(dbgs() << "Tracing value " << PrintReg(UseReg) << ':'<br>
>> + << UseVNI->id << '@' << UseVNI->def << '\n');<br>
>> +<br>
>> + // List of (Reg, VNI) that have been inserted into SibValues, but need<br>
>> to be<br>
>> + // processed.<br>
>> + SmallVector<std::pair<unsigned, VNInfo*>, 8> WorkList;<br>
>> + WorkList.push_back(std::make_pair(UseReg, UseVNI));<br>
>> +<br>
>> + LiveInterval &OrigLI = LIS.getInterval(Original);<br>
>> + do {<br>
>> + unsigned Reg;<br>
>> + VNInfo *VNI;<br>
>> + std::tie(Reg, VNI) = WorkList.pop_back_val();<br>
>> + DEBUG(dbgs() << " " << PrintReg(Reg) << ':' << VNI->id << '@' <<<br>
>> VNI->def<br>
>> + << ":\t");<br>
>> +<br>
>> + // First check if this value has already been computed.<br>
>> + SVI = SibValues.find(VNI);<br>
>> + assert(SVI != SibValues.end() && "Missing SibValues entry");<br>
>> +<br>
>> + // Trace through PHI-defs created by live range splitting.<br>
>> + if (VNI->isPHIDef()) {<br>
>> + // Stop at original PHIs. We don't know the value at the<br>
>> + // predecessors. Look up the VNInfo for the current definition<br>
>> + // in OrigLI, to properly determine whether or not this phi was<br>
>> + // added by splitting.<br>
>> + if (VNI->def == OrigLI.getVNInfoAt(VNI->def)->def) {<br>
>> + DEBUG(dbgs() << "orig phi value\n");<br>
>> + SVI->second.DefByOrigPHI = true;<br>
>> + SVI->second.AllDefsAreReloads = false;<br>
>> + propagateSiblingValue(SVI);<br>
>> + continue;<br>
>> + }<br>
>> +<br>
>> + // This is a PHI inserted by live range splitting. We could trace<br>
>> the<br>
>> + // live-out value from predecessor blocks, but that search can be<br>
>> very<br>
>> + // expensive if there are many predecessors and many more PHIs as<br>
>> + // generated by tail-dup when it sees an indirectbr. Instead, look<br>
>> at<br>
>> + // all the non-PHI defs that have the same value as OrigVNI. They<br>
>> must<br>
>> + // jointly dominate VNI->def. This is not optimal since VNI may<br>
>> actually<br>
>> + // be jointly dominated by a smaller subset of defs, so there is a<br>
>> change<br>
>> + // we will miss a AllDefsAreReloads optimization.<br>
>> +<br>
>> + // Separate all values dominated by OrigVNI into PHIs and non-PHIs.<br>
>> + SmallVector<VNInfo*, 8> PHIs, NonPHIs;<br>
>> + LiveInterval &LI = LIS.getInterval(Reg);<br>
>> +<br>
>> + for (LiveInterval::vni_iterator VI = LI.vni_begin(), VE =<br>
>> LI.vni_end();<br>
>> + VI != VE; ++VI) {<br>
>> + VNInfo *VNI2 = *VI;<br>
>> + if (VNI2->isUnused())<br>
>> + continue;<br>
>> + if (!OrigLI.containsOneValue() &&<br>
>> + OrigLI.getVNInfoAt(VNI2->def) != OrigVNI)<br>
>> + continue;<br>
>> + if (VNI2->isPHIDef() && VNI2->def != OrigVNI->def)<br>
>> + PHIs.push_back(VNI2);<br>
>> + else<br>
>> + NonPHIs.push_back(VNI2);<br>
>> + }<br>
>> + DEBUG(dbgs() << "split phi value, checking " << PHIs.size()<br>
>> + << " phi-defs, and " << NonPHIs.size()<br>
>> + << " non-phi/orig defs\n");<br>
>> +<br>
>> + // Create entries for all the PHIs. Don't add them to the<br>
>> worklist, we<br>
>> + // are processing all of them in one go here.<br>
>> + for (VNInfo *PHI : PHIs)<br>
>> + SibValues.insert(std::make_pair(PHI, SibValueInfo(Reg, PHI)));<br>
>> +<br>
>> + // Add every PHI as a dependent of all the non-PHIs.<br>
>> + for (VNInfo *NonPHI : NonPHIs) {<br>
>> + // Known value? Try an insertion.<br>
>> + std::tie(SVI, Inserted) =<br>
>> + SibValues.insert(std::make_pair(NonPHI, SibValueInfo(Reg,<br>
>> NonPHI)));<br>
>> + // Add all the PHIs as dependents of NonPHI.<br>
>> + SVI->second.Deps.insert(SVI->second.Deps.end(), PHIs.begin(),<br>
>> + PHIs.end());<br>
>> + // This is the first time we see NonPHI, add it to the worklist.<br>
>> + if (Inserted)<br>
>> + WorkList.push_back(std::make_pair(Reg, NonPHI));<br>
>> + else<br>
>> + // Propagate to all inserted PHIs, not just VNI.<br>
>> + propagateSiblingValue(SVI);<br>
>> + }<br>
>> +<br>
>> + // Next work list item.<br>
>> + continue;<br>
>> + }<br>
>> +<br>
>> + MachineInstr *MI = LIS.getInstructionFromIndex(VNI->def);<br>
>> + assert(MI && "Missing def");<br>
>> +<br>
>> + // Trace through sibling copies.<br>
>> + if (unsigned SrcReg = isFullCopyOf(MI, Reg)) {<br>
>> + if (isSibling(SrcReg)) {<br>
>> + LiveInterval &SrcLI = LIS.getInterval(SrcReg);<br>
>> + LiveQueryResult SrcQ = SrcLI.Query(VNI->def);<br>
>> + assert(SrcQ.valueIn() && "Copy from non-existing value");<br>
>> + // Check if this COPY kills its source.<br>
>> + SVI->second.KillsSource = SrcQ.isKill();<br>
>> + VNInfo *SrcVNI = SrcQ.valueIn();<br>
>> + DEBUG(dbgs() << "copy of " << PrintReg(SrcReg) << ':'<br>
>> + << SrcVNI->id << '@' << SrcVNI->def<br>
>> + << " kill=" << unsigned(SVI->second.KillsSource) <<<br>
>> '\n');<br>
>> + // Known sibling source value? Try an insertion.<br>
>> + std::tie(SVI, Inserted) = SibValues.insert(<br>
>> + std::make_pair(SrcVNI, SibValueInfo(SrcReg, SrcVNI)));<br>
>> + // This is the first time we see Src, add it to the worklist.<br>
>> + if (Inserted)<br>
>> + WorkList.push_back(std::make_pair(SrcReg, SrcVNI));<br>
>> + propagateSiblingValue(SVI, VNI);<br>
>> + // Next work list item.<br>
>> + continue;<br>
>> + }<br>
>> + }<br>
>> +<br>
>> + // Track reachable reloads.<br>
>> + SVI->second.DefMI = MI;<br>
>> + SVI->second.SpillMBB = MI->getParent();<br>
>> + int FI;<br>
>> + if (Reg == TII.isLoadFromStackSlot(MI, FI) && FI == StackSlot) {<br>
>> + DEBUG(dbgs() << "reload\n");<br>
>> + propagateSiblingValue(SVI);<br>
>> + // Next work list item.<br>
>> + continue;<br>
>> + }<br>
>> +<br>
>> + // Potential remat candidate.<br>
>> + DEBUG(dbgs() << "def " << *MI);<br>
>> + SVI->second.AllDefsAreReloads = false;<br>
>> + propagateSiblingValue(SVI);<br>
>> + } while (!WorkList.empty());<br>
>> +<br>
>> + // Look up the value we were looking for. We already did this lookup<br>
>> at the<br>
>> + // top of the function, but SibValues may have been invalidated.<br>
>> + SVI = SibValues.find(UseVNI);<br>
>> + assert(SVI != SibValues.end() && "Didn't compute requested info");<br>
>> + DEBUG(dbgs() << " traced to:\t" << SVI->second);<br>
>> + return SVI->second.DefMI;<br>
>> +}<br>
>> +<br>
>> +/// analyzeSiblingValues - Trace values defined by sibling copies back to<br>
>> +/// something that isn't a sibling copy.<br>
>> ///<br>
>> -bool InlineSpiller::hoistSpillInsideBB(LiveInterval &SpillLI,<br>
>> - MachineInstr &CopyMI) {<br>
>> +/// Keep track of values that may be rematerializable.<br>
>> +void InlineSpiller::analyzeSiblingValues() {<br>
>> + SibValues.clear();<br>
>> +<br>
>> + // No siblings at all?<br>
>> + if (Edit->getReg() == Original)<br>
>> + return;<br>
>> +<br>
>> + LiveInterval &OrigLI = LIS.getInterval(Original);<br>
>> + for (unsigned Reg : RegsToSpill) {<br>
>> + LiveInterval &LI = LIS.getInterval(Reg);<br>
>> + for (LiveInterval::const_vni_iterator VI = LI.vni_begin(),<br>
>> + VE = LI.vni_end(); VI != VE; ++VI) {<br>
>> + VNInfo *VNI = *VI;<br>
>> + if (VNI->isUnused())<br>
>> + continue;<br>
>> + MachineInstr *DefMI = nullptr;<br>
>> + if (!VNI->isPHIDef()) {<br>
>> + DefMI = LIS.getInstructionFromIndex(VNI->def);<br>
>> + assert(DefMI && "No defining instruction");<br>
>> + }<br>
>> + // Check possible sibling copies.<br>
>> + if (VNI->isPHIDef() || DefMI->isCopy()) {<br>
>> + VNInfo *OrigVNI = OrigLI.getVNInfoAt(VNI->def);<br>
>> + assert(OrigVNI && "Def outside original live range");<br>
>> + if (OrigVNI->def != VNI->def)<br>
>> + DefMI = traceSiblingValue(Reg, VNI, OrigVNI);<br>
>> + }<br>
>> + if (DefMI && Edit->checkRematerializable(VNI, DefMI, AA)) {<br>
>> + DEBUG(dbgs() << "Value " << PrintReg(Reg) << ':' << VNI->id <<<br>
>> '@'<br>
>> + << VNI->def << " may remat from " << *DefMI);<br>
>> + }<br>
>> + }<br>
>> + }<br>
>> +}<br>
>> +<br>
>> +/// hoistSpill - Given a sibling copy that defines a value to be spilled,<br>
>> insert<br>
>> +/// a spill at a better location.<br>
>> +bool InlineSpiller::hoistSpill(LiveInterval &SpillLI, MachineInstr<br>
>> &CopyMI) {<br>
>> SlotIndex Idx = LIS.getInstructionIndex(CopyMI);<br>
>> -#ifndef NDEBUG<br>
>> VNInfo *VNI = SpillLI.getVNInfoAt(Idx.getRegSlot());<br>
>> assert(VNI && VNI->def == Idx.getRegSlot() && "Not defined by copy");<br>
>> -#endif<br>
>> - unsigned SrcReg = CopyMI.getOperand(1).getReg();<br>
>> - LiveInterval &SrcLI = LIS.getInterval(SrcReg);<br>
>> - VNInfo *SrcVNI = SrcLI.getVNInfoAt(Idx);<br>
>> - LiveQueryResult SrcQ = SrcLI.Query(Idx);<br>
>> - MachineBasicBlock *DefMBB = LIS.getMBBFromIndex(SrcVNI->def);<br>
>> - if (DefMBB != CopyMI.getParent() || !SrcQ.isKill())<br>
>> + SibValueMap::iterator I = SibValues.find(VNI);<br>
>> + if (I == SibValues.end())<br>
>> + return false;<br>
>> +<br>
>> + const SibValueInfo &SVI = I->second;<br>
>> +<br>
>> + // Let the normal folding code deal with the boring case.<br>
>> + if (!SVI.AllDefsAreReloads && SVI.SpillVNI == VNI)<br>
>> + return false;<br>
>> +<br>
>> + // SpillReg may have been deleted by remat and DCE.<br>
>> + if (!LIS.hasInterval(SVI.SpillReg)) {<br>
>> + DEBUG(dbgs() << "Stale interval: " << PrintReg(SVI.SpillReg) <<<br>
>> '\n');<br>
>> + SibValues.erase(I);<br>
>> + return false;<br>
>> + }<br>
>> +<br>
>> + LiveInterval &SibLI = LIS.getInterval(SVI.SpillReg);<br>
>> + if (!SibLI.containsValue(SVI.SpillVNI)) {<br>
>> + DEBUG(dbgs() << "Stale value: " << PrintReg(SVI.SpillReg) << '\n');<br>
>> + SibValues.erase(I);<br>
>> return false;<br>
>> + }<br>
>><br>
>> // Conservatively extend the stack slot range to the range of the<br>
>> original<br>
>> // value. We may be able to do better with stack slot coloring by being<br>
>> more<br>
>> @@ -364,29 +719,35 @@ bool InlineSpiller::hoistSpillInsideBB(L<br>
>> DEBUG(dbgs() << "\tmerged orig valno " << OrigVNI->id << ": "<br>
>> << *StackInt << '\n');<br>
>><br>
>> - // We are going to spill SrcVNI immediately after its def, so clear out<br>
>> + // Already spilled everywhere.<br>
>> + if (SVI.AllDefsAreReloads) {<br>
>> + DEBUG(dbgs() << "\tno spill needed: " << SVI);<br>
>> + ++NumOmitReloadSpill;<br>
>> + return true;<br>
>> + }<br>
>> + // We are going to spill SVI.SpillVNI immediately after its def, so<br>
>> clear out<br>
>> // any later spills of the same value.<br>
>> - eliminateRedundantSpills(SrcLI, SrcVNI);<br>
>> + eliminateRedundantSpills(SibLI, SVI.SpillVNI);<br>
>><br>
>> - MachineBasicBlock *MBB = LIS.getMBBFromIndex(SrcVNI->def);<br>
>> + MachineBasicBlock *MBB = LIS.getMBBFromIndex(SVI.SpillVNI->def);<br>
>> MachineBasicBlock::iterator MII;<br>
>> - if (SrcVNI->isPHIDef())<br>
>> + if (SVI.SpillVNI->isPHIDef())<br>
>> MII = MBB->SkipPHIsAndLabels(MBB->begin());<br>
>> else {<br>
>> - MachineInstr *DefMI = LIS.getInstructionFromIndex(SrcVNI->def);<br>
>> + MachineInstr *DefMI = LIS.getInstructionFromIndex(SVI.SpillVNI->def);<br>
>> assert(DefMI && "Defining instruction disappeared");<br>
>> MII = DefMI;<br>
>> ++MII;<br>
>> }<br>
>> // Insert spill without kill flag immediately after def.<br>
>> - TII.storeRegToStackSlot(*MBB, MII, SrcReg, false, StackSlot,<br>
>> - MRI.getRegClass(SrcReg), &TRI);<br>
>> + TII.storeRegToStackSlot(*MBB, MII, SVI.SpillReg, false, StackSlot,<br>
>> + MRI.getRegClass(SVI.SpillReg), &TRI);<br>
>> --MII; // Point to store instruction.<br>
>> LIS.InsertMachineInstrInMaps(*MII);<br>
>> - DEBUG(dbgs() << "\thoisted: " << SrcVNI->def << '\t' << *MII);<br>
>> + DEBUG(dbgs() << "\thoisted: " << SVI.SpillVNI->def << '\t' << *MII);<br>
>><br>
>> - HSpiller.addToMergeableSpills(&(*MII), StackSlot, Original);<br>
>> ++NumSpills;<br>
>> + ++NumHoists;<br>
>> return true;<br>
>> }<br>
>><br>
>> @@ -444,8 +805,7 @@ void InlineSpiller::eliminateRedundantSp<br>
>> MI->setDesc(TII.get(TargetOpcode::KILL));<br>
>> DeadDefs.push_back(MI);<br>
>> ++NumSpillsRemoved;<br>
>> - if (HSpiller.rmFromMergeableSpills(MI, StackSlot))<br>
>> - --NumSpills;<br>
>> + --NumSpills;<br>
>> }<br>
>> }<br>
>> } while (!WorkList.empty());<br>
>> @@ -516,12 +876,12 @@ bool InlineSpiller::reMaterializeFor(Liv<br>
>> if (SnippetCopies.count(&MI))<br>
>> return false;<br>
>><br>
>> - LiveInterval &OrigLI = LIS.getInterval(Original);<br>
>> - VNInfo *OrigVNI = OrigLI.getVNInfoAt(UseIdx);<br>
>> + // Use an OrigVNI from traceSiblingValue when ParentVNI is a sibling<br>
>> copy.<br>
>> LiveRangeEdit::Remat RM(ParentVNI);<br>
>> - RM.OrigMI = LIS.getInstructionFromIndex(OrigVNI->def);<br>
>> -<br>
>> - if (!Edit->canRematerializeAt(RM, OrigVNI, UseIdx, false)) {<br>
>> + SibValueMap::const_iterator SibI = SibValues.find(ParentVNI);<br>
>> + if (SibI != SibValues.end())<br>
>> + RM.OrigMI = SibI->second.DefMI;<br>
>> + if (!Edit->canRematerializeAt(RM, UseIdx, false)) {<br>
>> markValueUsed(&VirtReg, ParentVNI);<br>
>> DEBUG(dbgs() << "\tcannot remat for " << UseIdx << '\t' << MI);<br>
>> return false;<br>
>> @@ -571,6 +931,7 @@ bool InlineSpiller::reMaterializeFor(Liv<br>
>> /// reMaterializeAll - Try to rematerialize as many uses as possible,<br>
>> /// and trim the live ranges after.<br>
>> void InlineSpiller::reMaterializeAll() {<br>
>> + // analyzeSiblingValues has already tested all relevant defining<br>
>> instructions.<br>
>> if (!Edit->anyRematerializable(AA))<br>
>> return;<br>
>><br>
>> @@ -656,9 +1017,6 @@ bool InlineSpiller::coalesceStackAccess(<br>
>> if (InstrReg != Reg || FI != StackSlot)<br>
>> return false;<br>
>><br>
>> - if (!IsLoad)<br>
>> - HSpiller.rmFromMergeableSpills(MI, StackSlot);<br>
>> -<br>
>> DEBUG(dbgs() << "Coalescing stack access: " << *MI);<br>
>> LIS.RemoveMachineInstrFromMaps(*MI);<br>
>> MI->eraseFromParent();<br>
>> @@ -783,9 +1141,6 @@ foldMemoryOperand(ArrayRef<std::pair<Mac<br>
>> LIS.removePhysRegDefAt(Reg, Idx);<br>
>> }<br>
>><br>
>> - int FI;<br>
>> - if (TII.isStoreToStackSlot(MI, FI) &&<br>
>> HSpiller.rmFromMergeableSpills(MI, FI))<br>
>> - --NumSpills;<br>
>> LIS.ReplaceMachineInstrInMaps(*MI, *FoldMI);<br>
>> MI->eraseFromParent();<br>
>><br>
>> @@ -811,10 +1166,9 @@ foldMemoryOperand(ArrayRef<std::pair<Mac<br>
>><br>
>> if (!WasCopy)<br>
>> ++NumFolded;<br>
>> - else if (Ops.front().second == 0) {<br>
>> + else if (Ops.front().second == 0)<br>
>> ++NumSpills;<br>
>> - HSpiller.addToMergeableSpills(FoldMI, StackSlot, Original);<br>
>> - } else<br>
>> + else<br>
>> ++NumReloads;<br>
>> return true;<br>
>> }<br>
>> @@ -849,7 +1203,6 @@ void InlineSpiller::insertSpill(unsigned<br>
>> DEBUG(dumpMachineInstrRangeWithSlotIndex(std::next(MI), MIS.end(), LIS,<br>
>> "spill"));<br>
>> ++NumSpills;<br>
>> - HSpiller.addToMergeableSpills(std::next(MI), StackSlot, Original);<br>
>> }<br>
>><br>
>> /// spillAroundUses - insert spill code around each use of Reg.<br>
>> @@ -913,7 +1266,8 @@ void InlineSpiller::spillAroundUses(unsi<br>
>> continue;<br>
>> }<br>
>> if (RI.Writes) {<br>
>> - if (hoistSpillInsideBB(OldLI, *MI)) {<br>
>> + // Hoist the spill of a sib-reg copy.<br>
>> + if (hoistSpill(OldLI, *MI)) {<br>
>> // This COPY is now dead, the value is already in the stack<br>
>> slot.<br>
>> MI->getOperand(0).setIsDead();<br>
>> DeadDefs.push_back(MI);<br>
>> @@ -1026,6 +1380,7 @@ void InlineSpiller::spill(LiveRangeEdit<br>
>> assert(DeadDefs.empty() && "Previous spill didn't remove dead defs");<br>
>><br>
>> collectRegsToSpill();<br>
>> + analyzeSiblingValues();<br>
>> reMaterializeAll();<br>
>><br>
>> // Remat may handle everything.<br>
>> @@ -1034,394 +1389,3 @@ void InlineSpiller::spill(LiveRangeEdit<br>
>><br>
>> Edit->calculateRegClassAndHint(MF, Loops, MBFI);<br>
>> }<br>
>> -<br>
>> -/// Optimizations after all the reg selections and spills are done.<br>
>> -///<br>
>> -void InlineSpiller::postOptimization() {<br>
>> - SmallVector<unsigned, 4> NewVRegs;<br>
>> - LiveRangeEdit LRE(nullptr, NewVRegs, MF, LIS, &VRM, nullptr);<br>
>> - HSpiller.hoistAllSpills(LRE);<br>
>> - assert(NewVRegs.size() == 0 &&<br>
>> - "No new vregs should be generated in hoistAllSpills");<br>
>> -}<br>
>> -<br>
>> -/// When a spill is inserted, add the spill to MergeableSpills map.<br>
>> -///<br>
>> -void HoistSpillHelper::addToMergeableSpills(MachineInstr *Spill, int<br>
>> StackSlot,<br>
>> - unsigned Original) {<br>
>> - StackSlotToReg[StackSlot] = Original;<br>
>> - SlotIndex Idx = LIS.getInstructionIndex(*Spill);<br>
>> - VNInfo *OrigVNI =<br>
>> LIS.getInterval(Original).getVNInfoAt(Idx.getRegSlot());<br>
>> - std::pair<int, VNInfo *> MIdx = std::make_pair(StackSlot, OrigVNI);<br>
>> - MergeableSpills[MIdx].insert(Spill);<br>
>> -}<br>
>> -<br>
>> -/// When a spill is removed, remove the spill from MergeableSpills map.<br>
>> -/// Return true if the spill is removed successfully.<br>
>> -///<br>
>> -bool HoistSpillHelper::rmFromMergeableSpills(MachineInstr *Spill,<br>
>> - int StackSlot) {<br>
>> - int Original = StackSlotToReg[StackSlot];<br>
>> - if (!Original)<br>
>> - return false;<br>
>> - SlotIndex Idx = LIS.getInstructionIndex(*Spill);<br>
>> - VNInfo *OrigVNI =<br>
>> LIS.getInterval(Original).getVNInfoAt(Idx.getRegSlot());<br>
>> - std::pair<int, VNInfo *> MIdx = std::make_pair(StackSlot, OrigVNI);<br>
>> - return MergeableSpills[MIdx].erase(Spill);<br>
>> -}<br>
>> -<br>
>> -/// Check BB to see if it is a possible target BB to place a hoisted<br>
>> spill,<br>
>> -/// i.e., there should be a living sibling of OrigReg at the insert<br>
>> point.<br>
>> -///<br>
>> -bool HoistSpillHelper::isSpillCandBB(unsigned OrigReg, VNInfo &OrigVNI,<br>
>> - MachineBasicBlock &BB, unsigned<br>
>> &LiveReg) {<br>
>> - SlotIndex Idx;<br>
>> - MachineBasicBlock::iterator MI = BB.getFirstTerminator();<br>
>> - if (MI != BB.end())<br>
>> - Idx = LIS.getInstructionIndex(*MI);<br>
>> - else<br>
>> - Idx = LIS.getMBBEndIdx(&BB).getPrevSlot();<br>
>> - SmallSetVector<unsigned, 16> &Siblings = Virt2SiblingsMap[OrigReg];<br>
>> - assert((LIS.getInterval(OrigReg)).getVNInfoAt(Idx) == &OrigVNI &&<br>
>> - "Unexpected VNI");<br>
>> -<br>
>> - for (auto const SibReg : Siblings) {<br>
>> - LiveInterval &LI = LIS.getInterval(SibReg);<br>
>> - VNInfo *VNI = LI.getVNInfoAt(Idx);<br>
>> - if (VNI) {<br>
>> - LiveReg = SibReg;<br>
>> - return true;<br>
>> - }<br>
>> - }<br>
>> - return false;<br>
>> -}<br>
>> -<br>
>> -/// Remove redundent spills in the same BB. Save those redundent spills<br>
>> in<br>
>> -/// SpillsToRm, and save the spill to keep and its BB in SpillBBToSpill<br>
>> map.<br>
>> -///<br>
>> -void HoistSpillHelper::rmRedundantSpills(<br>
>> - SmallPtrSet<MachineInstr *, 16> &Spills,<br>
>> - SmallVectorImpl<MachineInstr *> &SpillsToRm,<br>
>> - DenseMap<MachineDomTreeNode *, MachineInstr *> &SpillBBToSpill) {<br>
>> - // For each spill saw, check SpillBBToSpill[] and see if its BB already<br>
>> has<br>
>> - // another spill inside. If a BB contains more than one spill, only<br>
>> keep the<br>
>> - // earlier spill with smaller SlotIndex.<br>
>> - for (const auto CurrentSpill : Spills) {<br>
>> - MachineBasicBlock *Block = CurrentSpill->getParent();<br>
>> - MachineDomTreeNode *Node = MDT.DT->getNode(Block);<br>
>> - MachineInstr *PrevSpill = SpillBBToSpill[Node];<br>
>> - if (PrevSpill) {<br>
>> - SlotIndex PIdx = LIS.getInstructionIndex(*PrevSpill);<br>
>> - SlotIndex CIdx = LIS.getInstructionIndex(*CurrentSpill);<br>
>> - MachineInstr *SpillToRm = (CIdx > PIdx) ? CurrentSpill : PrevSpill;<br>
>> - MachineInstr *SpillToKeep = (CIdx > PIdx) ? PrevSpill :<br>
>> CurrentSpill;<br>
>> - SpillsToRm.push_back(SpillToRm);<br>
>> - SpillBBToSpill[MDT.DT->getNode(Block)] = SpillToKeep;<br>
>> - } else {<br>
>> - SpillBBToSpill[MDT.DT->getNode(Block)] = CurrentSpill;<br>
>> - }<br>
>> - }<br>
>> - for (const auto SpillToRm : SpillsToRm)<br>
>> - Spills.erase(SpillToRm);<br>
>> -}<br>
>> -<br>
>> -/// Starting from \p Root find a top-down traversal order of the<br>
>> dominator<br>
>> -/// tree to visit all basic blocks containing the elements of \p Spills.<br>
>> -/// Redundant spills will be found and put into \p SpillsToRm at the same<br>
>> -/// time. \p SpillBBToSpill will be populated as part of the process and<br>
>> -/// maps a basic block to the first store occurring in the basic block.<br>
>> -/// \post SpillsToRm.union(Spills@post) == Spills@pre<br>
>> -///<br>
>> -void HoistSpillHelper::getVisitOrders(<br>
>> - MachineBasicBlock *Root, SmallPtrSet<MachineInstr *, 16> &Spills,<br>
>> - SmallVectorImpl<MachineDomTreeNode *> &Orders,<br>
>> - SmallVectorImpl<MachineInstr *> &SpillsToRm,<br>
>> - DenseMap<MachineDomTreeNode *, unsigned> &SpillsToKeep,<br>
>> - DenseMap<MachineDomTreeNode *, MachineInstr *> &SpillBBToSpill) {<br>
>> - // The set contains all the possible BB nodes to which we may hoist<br>
>> - // original spills.<br>
>> - SmallPtrSet<MachineDomTreeNode *, 8> WorkSet;<br>
>> - // Save the BB nodes on the path from the first BB node containing<br>
>> - // non-redundent spill to the Root node.<br>
>> - SmallPtrSet<MachineDomTreeNode *, 8> NodesOnPath;<br>
>> - // All the spills to be hoisted must originate from a single def<br>
>> instruction<br>
>> - // to the OrigReg. It means the def instruction should dominate all the<br>
>> spills<br>
>> - // to be hoisted. We choose the BB where the def instruction is located<br>
>> as<br>
>> - // the Root.<br>
>> - MachineDomTreeNode *RootIDomNode = MDT[Root]->getIDom();<br>
>> - // For every node on the dominator tree with spill, walk up on the<br>
>> dominator<br>
>> - // tree towards the Root node until it is reached. If there is other<br>
>> node<br>
>> - // containing spill in the middle of the path, the previous spill saw<br>
>> will<br>
>> - // be redundent and the node containing it will be removed. All the<br>
>> nodes on<br>
>> - // the path starting from the first node with non-redundent spill to<br>
>> the Root<br>
>> - // node will be added to the WorkSet, which will contain all the<br>
>> possible<br>
>> - // locations where spills may be hoisted to after the loop below is<br>
>> done.<br>
>> - for (const auto Spill : Spills) {<br>
>> - MachineBasicBlock *Block = Spill->getParent();<br>
>> - MachineDomTreeNode *Node = MDT[Block];<br>
>> - MachineInstr *SpillToRm = nullptr;<br>
>> - while (Node != RootIDomNode) {<br>
>> - // If Node dominates Block, and it already contains a spill, the<br>
>> spill in<br>
>> - // Block will be redundent.<br>
>> - if (Node != MDT[Block] && SpillBBToSpill[Node]) {<br>
>> - SpillToRm = SpillBBToSpill[MDT[Block]];<br>
>> - break;<br>
>> - /// If we see the Node already in WorkSet, the path from the Node<br>
>> to<br>
>> - /// the Root node must already be traversed by another spill.<br>
>> - /// Then no need to repeat.<br>
>> - } else if (WorkSet.count(Node)) {<br>
>> - break;<br>
>> - } else {<br>
>> - NodesOnPath.insert(Node);<br>
>> - }<br>
>> - Node = Node->getIDom();<br>
>> - }<br>
>> - if (SpillToRm) {<br>
>> - SpillsToRm.push_back(SpillToRm);<br>
>> - } else {<br>
>> - // Add a BB containing the original spills to SpillsToKeep -- i.e.,<br>
>> - // set the initial status before hoisting start. The value of BBs<br>
>> - // containing original spills is set to 0, in order to descriminate<br>
>> - // with BBs containing hoisted spills which will be inserted to<br>
>> - // SpillsToKeep later during hoisting.<br>
>> - SpillsToKeep[MDT[Block]] = 0;<br>
>> - WorkSet.insert(NodesOnPath.begin(), NodesOnPath.end());<br>
>> - }<br>
>> - NodesOnPath.clear();<br>
>> - }<br>
>> -<br>
>> - // Sort the nodes in WorkSet in top-down order and save the nodes<br>
>> - // in Orders. Orders will be used for hoisting in runHoistSpills.<br>
>> - unsigned idx = 0;<br>
>> - Orders.push_back(MDT.DT->getNode(Root));<br>
>> - do {<br>
>> - MachineDomTreeNode *Node = Orders[idx++];<br>
>> - const std::vector<MachineDomTreeNode *> &Children =<br>
>> Node->getChildren();<br>
>> - unsigned NumChildren = Children.size();<br>
>> - for (unsigned i = 0; i != NumChildren; ++i) {<br>
>> - MachineDomTreeNode *Child = Children[i];<br>
>> - if (WorkSet.count(Child))<br>
>> - Orders.push_back(Child);<br>
>> - }<br>
>> - } while (idx != Orders.size());<br>
>> - assert(Orders.size() == WorkSet.size() &&<br>
>> - "Orders have different size with WorkSet");<br>
>> -<br>
>> -#ifndef NDEBUG<br>
>> - DEBUG(dbgs() << "Orders size is " << Orders.size() << "\n");<br>
>> - SmallVector<MachineDomTreeNode *, 32>::reverse_iterator RIt =<br>
>> Orders.rbegin();<br>
>> - for (; RIt != Orders.rend(); RIt++)<br>
>> - DEBUG(dbgs() << "BB" << (*RIt)->getBlock()->getNumber() << ",");<br>
>> - DEBUG(dbgs() << "\n");<br>
>> -#endif<br>
>> -}<br>
>> -<br>
>> -/// Try to hoist spills according to BB hotness. The spills to removed<br>
>> will<br>
>> -/// be saved in \p SpillsToRm. The spills to be inserted will be saved in<br>
>> -/// \p SpillsToIns.<br>
>> -///<br>
>> -void HoistSpillHelper::runHoistSpills(<br>
>> - unsigned OrigReg, VNInfo &OrigVNI, SmallPtrSet<MachineInstr *, 16><br>
>> &Spills,<br>
>> - SmallVectorImpl<MachineInstr *> &SpillsToRm,<br>
>> - DenseMap<MachineBasicBlock *, unsigned> &SpillsToIns) {<br>
>> - // Visit order of dominator tree nodes.<br>
>> - SmallVector<MachineDomTreeNode *, 32> Orders;<br>
>> - // SpillsToKeep contains all the nodes where spills are to be inserted<br>
>> - // during hoisting. If the spill to be inserted is an original spill<br>
>> - // (not a hoisted one), the value of the map entry is 0. If the spill<br>
>> - // is a hoisted spill, the value of the map entry is the VReg to be<br>
>> used<br>
>> - // as the source of the spill.<br>
>> - DenseMap<MachineDomTreeNode *, unsigned> SpillsToKeep;<br>
>> - // Map from BB to the first spill inside of it.<br>
>> - DenseMap<MachineDomTreeNode *, MachineInstr *> SpillBBToSpill;<br>
>> -<br>
>> - rmRedundantSpills(Spills, SpillsToRm, SpillBBToSpill);<br>
>> -<br>
>> - MachineBasicBlock *Root = LIS.getMBBFromIndex(OrigVNI.def);<br>
>> - getVisitOrders(Root, Spills, Orders, SpillsToRm, SpillsToKeep,<br>
>> - SpillBBToSpill);<br>
>> -<br>
>> - // SpillsInSubTree keeps the map from a dom tree node to a pair of<br>
>> - // nodes set and the cost of all the spills inside those nodes.<br>
>> - // The nodes set are the locations where spills are to be inserted<br>
>> - // in the subtree of current node.<br>
>> - typedef std::pair<SmallPtrSet<MachineDomTreeNode *, 16>,<br>
>> BlockFrequency><br>
>> - NodesCostPair;<br>
>> - DenseMap<MachineDomTreeNode *, NodesCostPair> SpillsInSubTreeMap;<br>
>> - // Iterate Orders set in reverse order, which will be a bottom-up order<br>
>> - // in the dominator tree. Once we visit a dom tree node, we know its<br>
>> - // children have already been visited and the spill locations in the<br>
>> - // subtrees of all the children have been determined.<br>
>> - SmallVector<MachineDomTreeNode *, 32>::reverse_iterator RIt =<br>
>> Orders.rbegin();<br>
>> - for (; RIt != Orders.rend(); RIt++) {<br>
>> - MachineBasicBlock *Block = (*RIt)->getBlock();<br>
>> - SmallPtrSet<MachineDomTreeNode *, 16> &SpillsInSubTree =<br>
>> - SpillsInSubTreeMap[*RIt].first;<br>
>> - // Total spill costs inside the sub tree.<br>
>> - BlockFrequency &SubTreeCost = SpillsInSubTreeMap[*RIt].second;<br>
>> -<br>
>> - // If Block contains an original spill, simply continue.<br>
>> - if (SpillsToKeep.find(*RIt) != SpillsToKeep.end() &&<br>
>> !SpillsToKeep[*RIt]) {<br>
>> - SpillsInSubTree.insert(*RIt);<br>
>> - SubTreeCost = MBFI.getBlockFreq(Block);<br>
>> - continue;<br>
>> - }<br>
>> -<br>
>> - // Collect spills in subtree of current node (*RIt) to<br>
>> - // SpillsInSubTree.<br>
>> - const std::vector<MachineDomTreeNode *> &Children =<br>
>> (*RIt)->getChildren();<br>
>> - unsigned NumChildren = Children.size();<br>
>> - for (unsigned i = 0; i != NumChildren; ++i) {<br>
>> - MachineDomTreeNode *Child = Children[i];<br>
>> - SpillsInSubTree.insert(SpillsInSubTreeMap[Child].first.begin(),<br>
>> - SpillsInSubTreeMap[Child].first.end());<br>
>> - SubTreeCost += SpillsInSubTreeMap[Child].second;<br>
>> - SpillsInSubTreeMap.erase(Child);<br>
>> - }<br>
>> -<br>
>> - // No spills in subtree, simply continue.<br>
>> - if (SpillsInSubTree.empty())<br>
>> - continue;<br>
>> -<br>
>> - // Check whether Block is a possible candidate to insert spill.<br>
>> - unsigned LiveReg = 0;<br>
>> - if (!isSpillCandBB(OrigReg, OrigVNI, *Block, LiveReg))<br>
>> - continue;<br>
>> -<br>
>> - // If there are multiple spills that could be merged, bias a little<br>
>> - // to hoist the spill.<br>
>> - BranchProbability MarginProb = (SpillsInSubTree.size() > 1)<br>
>> - ? BranchProbability(9, 10)<br>
>> - : BranchProbability(1, 1);<br>
>> - if (SubTreeCost > MBFI.getBlockFreq(Block) * MarginProb) {<br>
>> - // Hoist: Move spills to current Block.<br>
>> - for (const auto SpillBB : SpillsInSubTree) {<br>
>> - // When SpillBB is a BB contains original spill, insert the spill<br>
>> - // to SpillsToRm.<br>
>> - if (SpillsToKeep.find(SpillBB) != SpillsToKeep.end() &&<br>
>> - !SpillsToKeep[SpillBB]) {<br>
>> - MachineInstr *SpillToRm = SpillBBToSpill[SpillBB];<br>
>> - SpillsToRm.push_back(SpillToRm);<br>
>> - }<br>
>> - // SpillBB will not contain spill anymore, remove it from<br>
>> SpillsToKeep.<br>
>> - SpillsToKeep.erase(SpillBB);<br>
>> - }<br>
>> - // Current Block is the BB containing the new hoisted spill. Add it<br>
>> to<br>
>> - // SpillsToKeep. LiveReg is the source of the new spill.<br>
>> - SpillsToKeep[*RIt] = LiveReg;<br>
>> - DEBUG({<br>
>> - dbgs() << "spills in BB: ";<br>
>> - for (const auto Rspill : SpillsInSubTree)<br>
>> - dbgs() << Rspill->getBlock()->getNumber() << " ";<br>
>> - dbgs() << "were promoted to BB" <<<br>
>> (*RIt)->getBlock()->getNumber()<br>
>> - << "\n";<br>
>> - });<br>
>> - SpillsInSubTree.clear();<br>
>> - SpillsInSubTree.insert(*RIt);<br>
>> - SubTreeCost = MBFI.getBlockFreq(Block);<br>
>> - }<br>
>> - }<br>
>> - // For spills in SpillsToKeep with LiveReg set (i.e., not original<br>
>> spill),<br>
>> - // save them to SpillsToIns.<br>
>> - for (const auto Ent : SpillsToKeep) {<br>
>> - if (Ent.second)<br>
>> - SpillsToIns[Ent.first->getBlock()] = Ent.second;<br>
>> - }<br>
>> -}<br>
>> -<br>
>> -/// For spills with equal values, remove redundent spills and hoist the<br>
>> left<br>
>> -/// to less hot spots.<br>
>> -///<br>
>> -/// Spills with equal values will be collected into the same set in<br>
>> -/// MergeableSpills when spill is inserted. These equal spills are<br>
>> originated<br>
>> -/// from the same define instruction and are dominated by the<br>
>> instruction.<br>
>> -/// Before hoisting all the equal spills, redundent spills inside in the<br>
>> same<br>
>> -/// BB is first marked to be deleted. Then starting from spills left,<br>
>> walk up<br>
>> -/// on the dominator tree towards the Root node where the define<br>
>> instruction<br>
>> -/// is located, mark the dominated spills to be deleted along the way and<br>
>> -/// collect the BB nodes on the path from non-dominated spills to the<br>
>> define<br>
>> -/// instruction into a WorkSet. The nodes in WorkSet are the candidate<br>
>> places<br>
>> -/// where we consider to hoist the spills. We iterate the WorkSet in<br>
>> bottom-up<br>
>> -/// order, and for each node, we will decide whether to hoist spills<br>
>> inside<br>
>> -/// its subtree to that node. In this way, we can get benefit locally<br>
>> even if<br>
>> -/// hoisting all the equal spills to one cold place is impossible.<br>
>> -///<br>
>> -void HoistSpillHelper::hoistAllSpills(LiveRangeEdit &Edit) {<br>
>> - // Save the mapping between stackslot and its original reg.<br>
>> - DenseMap<int, unsigned> SlotToOrigReg;<br>
>> - for (unsigned i = 0, e = MRI.getNumVirtRegs(); i != e; ++i) {<br>
>> - unsigned Reg = TargetRegisterInfo::index2VirtReg(i);<br>
>> - int Slot = VRM.getStackSlot(Reg);<br>
>> - if (Slot != VirtRegMap::NO_STACK_SLOT)<br>
>> - SlotToOrigReg[Slot] = VRM.getOriginal(Reg);<br>
>> - unsigned Original = VRM.getPreSplitReg(Reg);<br>
>> - if (!MRI.def_empty(Reg))<br>
>> - Virt2SiblingsMap[Original].insert(Reg);<br>
>> - }<br>
>> -<br>
>> - // Each entry in MergeableSpills contains a spill set with equal<br>
>> values.<br>
>> - for (auto &Ent : MergeableSpills) {<br>
>> - int Slot = Ent.first.first;<br>
>> - unsigned OrigReg = SlotToOrigReg[Slot];<br>
>> - VNInfo *OrigVNI = Ent.first.second;<br>
>> - SmallPtrSet<MachineInstr *, 16> &EqValSpills = Ent.second;<br>
>> - if (Ent.second.empty())<br>
>> - continue;<br>
>> -<br>
>> - DEBUG({<br>
>> - dbgs() << "\nFor Slot" << Slot << " and VN" << OrigVNI->id << ":\n"<br>
>> - << "Equal spills in BB: ";<br>
>> - for (const auto spill : EqValSpills)<br>
>> - dbgs() << spill->getParent()->getNumber() << " ";<br>
>> - dbgs() << "\n";<br>
>> - });<br>
>> -<br>
>> - // SpillsToRm is the spill set to be removed from EqValSpills.<br>
>> - SmallVector<MachineInstr *, 16> SpillsToRm;<br>
>> - // SpillsToIns is the spill set to be newly inserted after hoisting.<br>
>> - DenseMap<MachineBasicBlock *, unsigned> SpillsToIns;<br>
>> -<br>
>> - runHoistSpills(OrigReg, *OrigVNI, EqValSpills, SpillsToRm,<br>
>> SpillsToIns);<br>
>> -<br>
>> - DEBUG({<br>
>> - dbgs() << "Finally inserted spills in BB: ";<br>
>> - for (const auto Ispill : SpillsToIns)<br>
>> - dbgs() << Ispill.first->getNumber() << " ";<br>
>> - dbgs() << "\nFinally removed spills in BB: ";<br>
>> - for (const auto Rspill : SpillsToRm)<br>
>> - dbgs() << Rspill->getParent()->getNumber() << " ";<br>
>> - dbgs() << "\n";<br>
>> - });<br>
>> -<br>
>> - // Stack live range update.<br>
>> - LiveInterval &StackIntvl = LSS.getInterval(Slot);<br>
>> - if (!SpillsToIns.empty() || !SpillsToRm.empty()) {<br>
>> - LiveInterval &OrigLI = LIS.getInterval(OrigReg);<br>
>> - StackIntvl.MergeValueInAsValue(OrigLI, OrigVNI,<br>
>> - StackIntvl.getValNumInfo(0));<br>
>> - }<br>
>> -<br>
>> - // Insert hoisted spills.<br>
>> - for (auto const Insert : SpillsToIns) {<br>
>> - MachineBasicBlock *BB = Insert.first;<br>
>> - unsigned LiveReg = Insert.second;<br>
>> - MachineBasicBlock::iterator MI = BB->getFirstTerminator();<br>
>> - TII.storeRegToStackSlot(*BB, MI, LiveReg, false, Slot,<br>
>> - MRI.getRegClass(LiveReg), &TRI);<br>
>> - LIS.InsertMachineInstrRangeInMaps(std::prev(MI), MI);<br>
>> - ++NumSpills;<br>
>> - }<br>
>> -<br>
>> - // Remove redundent spills or change them to dead instructions.<br>
>> - NumSpills -= SpillsToRm.size();<br>
>> - for (auto const RMEnt : SpillsToRm) {<br>
>> - RMEnt->setDesc(TII.get(TargetOpcode::KILL));<br>
>> - for (unsigned i = RMEnt->getNumOperands(); i; --i) {<br>
>> - MachineOperand &MO = RMEnt->getOperand(i - 1);<br>
>> - if (MO.isReg() && MO.isImplicit() && MO.isDef() && !MO.isDead())<br>
>> - RMEnt->RemoveOperand(i - 1);<br>
>> - }<br>
>> - }<br>
>> - Edit.eliminateDeadDefs(SpillsToRm, None, true);<br>
>> - }<br>
>> -}<br>
>><br>
>> Modified: llvm/trunk/lib/CodeGen/LiveRangeEdit.cpp<br>
>> URL:<br>
>> <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/LiveRangeEdit.cpp?rev=265317&r1=265316&r2=265317&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/LiveRangeEdit.cpp?rev=265317&r1=265316&r2=265317&view=diff</a><br>
>><br>
>> ==============================================================================<br>
>> --- llvm/trunk/lib/CodeGen/LiveRangeEdit.cpp (original)<br>
>> +++ llvm/trunk/lib/CodeGen/LiveRangeEdit.cpp Mon Apr 4 12:45:03 2016<br>
>> @@ -63,13 +63,10 @@ void LiveRangeEdit::scanRemattable(Alias<br>
>> for (VNInfo *VNI : getParent().valnos) {<br>
>> if (VNI->isUnused())<br>
>> continue;<br>
>> - unsigned Original = VRM->getOriginal(getReg());<br>
>> - LiveInterval &OrigLI = LIS.getInterval(Original);<br>
>> - VNInfo *OrigVNI = OrigLI.getVNInfoAt(VNI->def);<br>
>> - MachineInstr *DefMI = LIS.getInstructionFromIndex(OrigVNI->def);<br>
>> + MachineInstr *DefMI = LIS.getInstructionFromIndex(VNI->def);<br>
>> if (!DefMI)<br>
>> continue;<br>
>> - checkRematerializable(OrigVNI, DefMI, aa);<br>
>> + checkRematerializable(VNI, DefMI, aa);<br>
>> }<br>
>> ScannedRemattable = true;<br>
>> }<br>
>> @@ -116,18 +113,24 @@ bool LiveRangeEdit::allUsesAvailableAt(c<br>
>> return true;<br>
>> }<br>
>><br>
>> -bool LiveRangeEdit::canRematerializeAt(Remat &RM, VNInfo *OrigVNI,<br>
>> - SlotIndex UseIdx, bool<br>
>> cheapAsAMove) {<br>
>> +bool LiveRangeEdit::canRematerializeAt(Remat &RM,<br>
>> + SlotIndex UseIdx,<br>
>> + bool cheapAsAMove) {<br>
>> assert(ScannedRemattable && "Call anyRematerializable first");<br>
>><br>
>> // Use scanRemattable info.<br>
>> - if (!Remattable.count(OrigVNI))<br>
>> + if (!Remattable.count(RM.ParentVNI))<br>
>> return false;<br>
>><br>
>> // No defining instruction provided.<br>
>> SlotIndex DefIdx;<br>
>> - assert(RM.OrigMI && "No defining instruction for remattable value");<br>
>> - DefIdx = LIS.getInstructionIndex(*RM.OrigMI);<br>
>> + if (RM.OrigMI)<br>
>> + DefIdx = LIS.getInstructionIndex(*RM.OrigMI);<br>
>> + else {<br>
>> + DefIdx = RM.ParentVNI->def;<br>
>> + RM.OrigMI = LIS.getInstructionFromIndex(DefIdx);<br>
>> + assert(RM.OrigMI && "No defining instruction for remattable value");<br>
>> + }<br>
>><br>
>> // If only cheap remats were requested, bail out early.<br>
>> if (cheapAsAMove && !TII.isAsCheapAsAMove(RM.OrigMI))<br>
>> @@ -258,15 +261,6 @@ void LiveRangeEdit::eliminateDeadDef(Mac<br>
>> // Collect virtual registers to be erased after MI is gone.<br>
>> SmallVector<unsigned, 8> RegsToErase;<br>
>> bool ReadsPhysRegs = false;<br>
>> - bool isOrigDef = false;<br>
>> - unsigned Dest;<br>
>> - if (VRM && MI->getOperand(0).isReg()) {<br>
>> - Dest = MI->getOperand(0).getReg();<br>
>> - unsigned Original = VRM->getOriginal(Dest);<br>
>> - LiveInterval &OrigLI = LIS.getInterval(Original);<br>
>> - VNInfo *OrigVNI = OrigLI.getVNInfoAt(Idx);<br>
>> - isOrigDef = SlotIndex::isSameInstr(OrigVNI->def, Idx);<br>
>> - }<br>
>><br>
>> // Check for live intervals that may shrink<br>
>> for (MachineInstr::mop_iterator MOI = MI->operands_begin(),<br>
>> @@ -320,24 +314,11 @@ void LiveRangeEdit::eliminateDeadDef(Mac<br>
>> }<br>
>> DEBUG(dbgs() << "Converted physregs to:\t" << *MI);<br>
>> } else {<br>
>> - // If the dest of MI is an original reg, don't delete the inst.<br>
>> Replace<br>
>> - // the dest with a new reg, keep the inst for remat of other<br>
>> siblings.<br>
>> - // The inst is saved in LiveRangeEdit::DeadRemats and will be deleted<br>
>> - // after all the allocations of the func are done.<br>
>> - if (isOrigDef) {<br>
>> - unsigned NewDest = createFrom(Dest);<br>
>> - pop_back();<br>
>> - markDeadRemat(MI);<br>
>> - const TargetRegisterInfo &TRI = *MRI.getTargetRegisterInfo();<br>
>> - MI->substituteRegister(Dest, NewDest, 0, TRI);<br>
>> - MI->getOperand(0).setIsDead(false);<br>
>> - } else {<br>
>> - if (TheDelegate)<br>
>> - TheDelegate->LRE_WillEraseInstruction(MI);<br>
>> - LIS.RemoveMachineInstrFromMaps(*MI);<br>
>> -</blockquote></div>