[llvm] r265547 - Recommit r265309 after fixed an invalid memory reference bug happened
Wei Mi via llvm-commits
llvm-commits at lists.llvm.org
Thu Apr 7 07:33:23 PDT 2016
Mikael, thanks for the detailed error analysis. I deliberately used a dummy
register to keep the dead instruction for use in rematerialization. I
missed MachineVerifier could complain about it. I am looking at how to fix
it.
Thanks,
Wei.
On Thu, Apr 7, 2016 at 1:59 AM, Mikael Holmén <mikael.holmen at ericsson.com>
wrote:
> Hi Wei,
>
> I'm having an issue with this in my out-of-tree target.
>
> The verifier, when run from RAGreedy::selectOrSplitImpl complains
>
> if (VerifyEnabled)
> MF->verify(this, "After spilling");
>
> *** Bad machine code: Virtual register has no Live interval ***
> - function: func1
> - basic block: BB#0 (0x24c0098) [0B;608B)
> - instruction: 88B %vreg53<def> = mv_dp16_nimm13_any16_noLo
> - operand 0: %vreg53<def>
>
> Just before the crash I see:
>
> Remat created 1 dead defs.
> Deleting dead def 88r %vreg7<def,dead> = mv_dp16_nimm13_any16_noLo
> <fi#-1>, pred:0, pred:%noreg, pred:0, %dp<imp-use>, %ac0<imp-use>,
> %ac1<imp-use>; mem:LD1[FixedStack-1] aNh_0_7:%vreg7
>
> And when this dead def is removed, it's actually not really removed at
> all, but instead %vreg7 is replaced with %vreg53, and then %vreg53 is
> inserted in the DeadRemats set by this code in LiveRangeEdit:
>
> // If the dest of MI is an original reg, don't delete the inst. Replace
> // the dest with a new reg, keep the inst for remat of other siblings.
> // The inst is saved in LiveRangeEdit::DeadRemats and will be deleted
> // after all the allocations of the func are done.
> if (isOrigDef) {
> unsigned NewDest = createFrom(Dest);
> pop_back();
> markDeadRemat(MI);
>
> However, the verifier above is run before the call to postOptimization, so
> the instruction defining %vreg53 is still there, and the verifier pukes on
> it.
>
> It seems all the verifiers that run "inside" the register allocator before
> postOptimization can now fail since the DeadRemats contains not yet removed
> instructions that the verifiers don't like.
>
> Regards,
> Mikael
>
> On 04/06/2016 05:41 PM, Wei Mi via llvm-commits wrote:
>
>> Author: wmi
>> Date: Wed Apr 6 10:41:07 2016
>> New Revision: 265547
>>
>> URL: http://llvm.org/viewvc/llvm-project?rev=265547&view=rev
>> Log:
>> Recommit r265309 after fixed an invalid memory reference bug happened
>> when DenseMap growed and moved memory. I verified it fixed the bootstrap
>> problem on x86_64-linux-gnu but I cannot verify whether it fixes
>> the bootstrap error on clang-ppc64be-linux. I will watch the build-bot
>> result closely.
>>
>> Replace analyzeSiblingValues with new algorithm to fix its compile
>> time issue. The patch is to solve PR17409 and its duplicates.
>>
>> analyzeSiblingValues is a N x N complexity algorithm where N is
>> the number of siblings generated by reg splitting. Although it
>> causes siginificant compile time issue when N is large, it is also
>> important for performance since it removes redundent spills and
>> enables rematerialization.
>>
>> To solve the compile time issue, the patch removes analyzeSiblingValues
>> and replaces it with lower cost alternatives containing two parts. The
>> first part creates a new spill hoisting method in postOptimization of
>> register allocation. It does spill hoisting at once after all the spills
>> are generated instead of inside every instance of selectOrSplit. The
>> second part queries the define expr of the original register for
>> rematerializaiton and keep it always available during register allocation
>> even if it is already dead. It deletes those dead instructions only in
>> postOptimization. With the two parts in the patch, it can remove
>> analyzeSiblingValues without sacrificing performance.
>>
>> Differential Revision: http://reviews.llvm.org/D15302
>>
>> Added:
>> llvm/trunk/test/CodeGen/X86/hoist-spill.ll
>> llvm/trunk/test/CodeGen/X86/new-remat.ll
>> Removed:
>> llvm/trunk/test/CodeGen/AArch64/aarch64-deferred-spilling.ll
>> Modified:
>> llvm/trunk/include/llvm/CodeGen/LiveRangeEdit.h
>> llvm/trunk/lib/CodeGen/InlineSpiller.cpp
>> llvm/trunk/lib/CodeGen/LiveRangeEdit.cpp
>> llvm/trunk/lib/CodeGen/RegAllocBase.cpp
>> llvm/trunk/lib/CodeGen/RegAllocBase.h
>> llvm/trunk/lib/CodeGen/RegAllocBasic.cpp
>> llvm/trunk/lib/CodeGen/RegAllocGreedy.cpp
>> llvm/trunk/lib/CodeGen/RegAllocPBQP.cpp
>> llvm/trunk/lib/CodeGen/Spiller.h
>> llvm/trunk/lib/CodeGen/SplitKit.cpp
>> llvm/trunk/lib/CodeGen/SplitKit.h
>> llvm/trunk/test/CodeGen/X86/fp128-compare.ll
>> llvm/trunk/test/CodeGen/X86/ragreedy-hoist-spill.ll
>>
>> Modified: llvm/trunk/include/llvm/CodeGen/LiveRangeEdit.h
>> URL:
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/LiveRangeEdit.h?rev=265547&r1=265546&r2=265547&view=diff
>>
>> ==============================================================================
>> --- llvm/trunk/include/llvm/CodeGen/LiveRangeEdit.h (original)
>> +++ llvm/trunk/include/llvm/CodeGen/LiveRangeEdit.h Wed Apr 6 10:41:07
>> 2016
>> @@ -72,6 +72,10 @@ private:
>> /// ScannedRemattable - true when remattable values have been
>> identified.
>> bool ScannedRemattable;
>>
>> + /// DeadRemats - The saved instructions which have already been dead
>> after
>> + /// rematerialization but not deleted yet -- to be done in
>> postOptimization.
>> + SmallPtrSet<MachineInstr *, 32> *DeadRemats;
>> +
>> /// Remattable - Values defined by remattable instructions as
>> identified by
>> /// tii.isTriviallyReMaterializable().
>> SmallPtrSet<const VNInfo*,4> Remattable;
>> @@ -116,13 +120,16 @@ public:
>> /// @param vrm Map of virtual registers to physical registers for this
>> /// function. If NULL, no virtual register map updates
>> will
>> /// be done. This could be the case if called before
>> Regalloc.
>> + /// @param deadRemats The collection of all the instructions defining
>> an
>> + /// original reg and are dead after remat.
>> LiveRangeEdit(LiveInterval *parent, SmallVectorImpl<unsigned>
>> &newRegs,
>> MachineFunction &MF, LiveIntervals &lis, VirtRegMap
>> *vrm,
>> - Delegate *delegate = nullptr)
>> + Delegate *delegate = nullptr,
>> + SmallPtrSet<MachineInstr *, 32> *deadRemats = nullptr)
>> : Parent(parent), NewRegs(newRegs), MRI(MF.getRegInfo()),
>> LIS(lis),
>> - VRM(vrm), TII(*MF.getSubtarget().getInstrInfo()),
>> - TheDelegate(delegate), FirstNew(newRegs.size()),
>> - ScannedRemattable(false) {
>> + VRM(vrm), TII(*MF.getSubtarget().getInstrInfo()),
>> TheDelegate(delegate),
>> + FirstNew(newRegs.size()), ScannedRemattable(false),
>> + DeadRemats(deadRemats) {
>> MRI.setDelegate(this);
>> }
>>
>> @@ -142,6 +149,16 @@ public:
>> bool empty() const { return size() == 0; }
>> unsigned get(unsigned idx) const { return NewRegs[idx+FirstNew]; }
>>
>> + /// pop_back - It allows LiveRangeEdit users to drop new registers.
>> + /// The context is when an original def instruction of a register is
>> + /// dead after rematerialization, we still want to keep it for
>> following
>> + /// rematerializations. We save the def instruction in DeadRemats,
>> + /// and replace the original dst register with a new dummy register so
>> + /// the live range of original dst register can be shrinked normally.
>> + /// We don't want to allocate phys register for the dummy register, so
>> + /// we want to drop it from the NewRegs set.
>> + void pop_back() { NewRegs.pop_back(); }
>> +
>> ArrayRef<unsigned> regs() const {
>> return makeArrayRef(NewRegs).slice(FirstNew);
>> }
>> @@ -175,15 +192,15 @@ public:
>> /// Remat - Information needed to rematerialize at a specific
>> location.
>> struct Remat {
>> VNInfo *ParentVNI; // parent_'s value at the remat location.
>> - MachineInstr *OrigMI; // Instruction defining ParentVNI.
>> + MachineInstr *OrigMI; // Instruction defining OrigVNI. It contains
>> the
>> + // real expr for remat.
>> explicit Remat(VNInfo *ParentVNI) : ParentVNI(ParentVNI),
>> OrigMI(nullptr) {}
>> };
>>
>> /// canRematerializeAt - Determine if ParentVNI can be rematerialized
>> at
>> /// UseIdx. It is assumed that parent_.getVNINfoAt(UseIdx) ==
>> ParentVNI.
>> /// When cheapAsAMove is set, only cheap remats are allowed.
>> - bool canRematerializeAt(Remat &RM,
>> - SlotIndex UseIdx,
>> + bool canRematerializeAt(Remat &RM, VNInfo *OrigVNI, SlotIndex UseIdx,
>> bool cheapAsAMove);
>>
>> /// rematerializeAt - Rematerialize RM.ParentVNI into DestReg by
>> inserting an
>> @@ -208,6 +225,12 @@ public:
>> return Rematted.count(ParentVNI);
>> }
>>
>> + void markDeadRemat(MachineInstr *inst) {
>> + // DeadRemats is an optional field.
>> + if (DeadRemats)
>> + DeadRemats->insert(inst);
>> + }
>> +
>> /// eraseVirtReg - Notify the delegate that Reg is no longer in use,
>> and try
>> /// to erase it from LIS.
>> void eraseVirtReg(unsigned Reg);
>> @@ -218,8 +241,11 @@ public:
>> /// RegsBeingSpilled lists registers currently being spilled by the
>> register
>> /// allocator. These registers should not be split into new intervals
>> /// as currently those new intervals are not guaranteed to spill.
>> - void eliminateDeadDefs(SmallVectorImpl<MachineInstr*> &Dead,
>> - ArrayRef<unsigned> RegsBeingSpilled = None);
>> + /// NoSplit indicates this func is used after the iterations of
>> selectOrSplit
>> + /// where registers should not be split into new intervals.
>> + void eliminateDeadDefs(SmallVectorImpl<MachineInstr *> &Dead,
>> + ArrayRef<unsigned> RegsBeingSpilled = None,
>> + bool NoSplit = false);
>>
>> /// calculateRegClassAndHint - Recompute register class and hint for
>> each new
>> /// register.
>>
>> Modified: llvm/trunk/lib/CodeGen/InlineSpiller.cpp
>> URL:
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/InlineSpiller.cpp?rev=265547&r1=265546&r2=265547&view=diff
>>
>> ==============================================================================
>> --- llvm/trunk/lib/CodeGen/InlineSpiller.cpp (original)
>> +++ llvm/trunk/lib/CodeGen/InlineSpiller.cpp Wed Apr 6 10:41:07 2016
>> @@ -48,13 +48,77 @@ STATISTIC(NumReloadsRemoved, "Number of
>> STATISTIC(NumFolded, "Number of folded stack accesses");
>> STATISTIC(NumFoldedLoads, "Number of folded loads");
>> STATISTIC(NumRemats, "Number of rematerialized defs for
>> spilling");
>> -STATISTIC(NumOmitReloadSpill, "Number of omitted spills of reloads");
>> -STATISTIC(NumHoists, "Number of hoisted spills");
>>
>> static cl::opt<bool> DisableHoisting("disable-spill-hoist", cl::Hidden,
>> cl::desc("Disable inline spill
>> hoisting"));
>>
>> namespace {
>> +class HoistSpillHelper {
>> + LiveIntervals &LIS;
>> + LiveStacks &LSS;
>> + AliasAnalysis *AA;
>> + MachineDominatorTree &MDT;
>> + MachineLoopInfo &Loops;
>> + VirtRegMap &VRM;
>> + MachineFrameInfo &MFI;
>> + MachineRegisterInfo &MRI;
>> + const TargetInstrInfo &TII;
>> + const TargetRegisterInfo &TRI;
>> + const MachineBlockFrequencyInfo &MBFI;
>> +
>> + // Map from StackSlot to its original register.
>> + DenseMap<int, unsigned> StackSlotToReg;
>> + // Map from pair of (StackSlot and Original VNI) to a set of spills
>> which
>> + // have the same stackslot and have equal values defined by Original
>> VNI.
>> + // These spills are mergeable and are hoist candiates.
>> + typedef DenseMap<std::pair<int, VNInfo *>, SmallPtrSet<MachineInstr *,
>> 16>>
>> + MergeableSpillsMap;
>> + MergeableSpillsMap MergeableSpills;
>> +
>> + /// This is the map from original register to a set containing all its
>> + /// siblings. To hoist a spill to another BB, we need to find out a
>> live
>> + /// sibling there and use it as the source of the new spill.
>> + DenseMap<unsigned, SmallSetVector<unsigned, 16>> Virt2SiblingsMap;
>> +
>> + bool isSpillCandBB(unsigned OrigReg, VNInfo &OrigVNI,
>> MachineBasicBlock &BB,
>> + unsigned &LiveReg);
>> +
>> + void rmRedundantSpills(
>> + SmallPtrSet<MachineInstr *, 16> &Spills,
>> + SmallVectorImpl<MachineInstr *> &SpillsToRm,
>> + DenseMap<MachineDomTreeNode *, MachineInstr *> &SpillBBToSpill);
>> +
>> + void getVisitOrders(
>> + MachineBasicBlock *Root, SmallPtrSet<MachineInstr *, 16> &Spills,
>> + SmallVectorImpl<MachineDomTreeNode *> &Orders,
>> + SmallVectorImpl<MachineInstr *> &SpillsToRm,
>> + DenseMap<MachineDomTreeNode *, unsigned> &SpillsToKeep,
>> + DenseMap<MachineDomTreeNode *, MachineInstr *> &SpillBBToSpill);
>> +
>> + void runHoistSpills(unsigned OrigReg, VNInfo &OrigVNI,
>> + SmallPtrSet<MachineInstr *, 16> &Spills,
>> + SmallVectorImpl<MachineInstr *> &SpillsToRm,
>> + DenseMap<MachineBasicBlock *, unsigned>
>> &SpillsToIns);
>> +
>> +public:
>> + HoistSpillHelper(MachineFunctionPass &pass, MachineFunction &mf,
>> + VirtRegMap &vrm)
>> + : LIS(pass.getAnalysis<LiveIntervals>()),
>> + LSS(pass.getAnalysis<LiveStacks>()),
>> + AA(&pass.getAnalysis<AAResultsWrapperPass>().getAAResults()),
>> + MDT(pass.getAnalysis<MachineDominatorTree>()),
>> + Loops(pass.getAnalysis<MachineLoopInfo>()), VRM(vrm),
>> + MFI(*mf.getFrameInfo()), MRI(mf.getRegInfo()),
>> + TII(*mf.getSubtarget().getInstrInfo()),
>> + TRI(*mf.getSubtarget().getRegisterInfo()),
>> + MBFI(pass.getAnalysis<MachineBlockFrequencyInfo>()) {}
>> +
>> + void addToMergeableSpills(MachineInstr *Spill, int StackSlot,
>> + unsigned Original);
>> + bool rmFromMergeableSpills(MachineInstr *Spill, int StackSlot);
>> + void hoistAllSpills(LiveRangeEdit &Edit);
>> +};
>> +
>> class InlineSpiller : public Spiller {
>> MachineFunction &MF;
>> LiveIntervals &LIS;
>> @@ -85,56 +149,12 @@ class InlineSpiller : public Spiller {
>> // Values that failed to remat at some point.
>> SmallPtrSet<VNInfo*, 8> UsedValues;
>>
>> -public:
>> - // Information about a value that was defined by a copy from a sibling
>> - // register.
>> - struct SibValueInfo {
>> - // True when all reaching defs were reloads: No spill is necessary.
>> - bool AllDefsAreReloads;
>> -
>> - // True when value is defined by an original PHI not from splitting.
>> - bool DefByOrigPHI;
>> -
>> - // True when the COPY defining this value killed its source.
>> - bool KillsSource;
>> -
>> - // The preferred register to spill.
>> - unsigned SpillReg;
>> -
>> - // The value of SpillReg that should be spilled.
>> - VNInfo *SpillVNI;
>> -
>> - // The block where SpillVNI should be spilled. Currently, this must
>> be the
>> - // block containing SpillVNI->def.
>> - MachineBasicBlock *SpillMBB;
>> -
>> - // A defining instruction that is not a sibling copy or a reload, or
>> NULL.
>> - // This can be used as a template for rematerialization.
>> - MachineInstr *DefMI;
>> -
>> - // List of values that depend on this one. These values are
>> actually the
>> - // same, but live range splitting has placed them in different
>> registers,
>> - // or SSA update needed to insert PHI-defs to preserve SSA form.
>> This is
>> - // copies of the current value and phi-kills. Usually only
>> phi-kills cause
>> - // more than one dependent value.
>> - TinyPtrVector<VNInfo*> Deps;
>> -
>> - SibValueInfo(unsigned Reg, VNInfo *VNI)
>> - : AllDefsAreReloads(true), DefByOrigPHI(false), KillsSource(false),
>> - SpillReg(Reg), SpillVNI(VNI), SpillMBB(nullptr), DefMI(nullptr)
>> {}
>> -
>> - // Returns true when a def has been found.
>> - bool hasDef() const { return DefByOrigPHI || DefMI; }
>> - };
>> -
>> -private:
>> - // Values in RegsToSpill defined by sibling copies.
>> - typedef DenseMap<VNInfo*, SibValueInfo> SibValueMap;
>> - SibValueMap SibValues;
>> -
>> // Dead defs generated during spilling.
>> SmallVector<MachineInstr*, 8> DeadDefs;
>>
>> + // Object records spills information and does the hoisting.
>> + HoistSpillHelper HSpiller;
>> +
>> ~InlineSpiller() override {}
>>
>> public:
>> @@ -147,9 +167,11 @@ public:
>> MFI(*mf.getFrameInfo()), MRI(mf.getRegInfo()),
>> TII(*mf.getSubtarget().getInstrInfo()),
>> TRI(*mf.getSubtarget().getRegisterInfo()),
>> - MBFI(pass.getAnalysis<MachineBlockFrequencyInfo>()) {}
>> + MBFI(pass.getAnalysis<MachineBlockFrequencyInfo>()),
>> + HSpiller(pass, mf, vrm) {}
>>
>> void spill(LiveRangeEdit &) override;
>> + void postOptimization() override;
>>
>> private:
>> bool isSnippet(const LiveInterval &SnipLI);
>> @@ -161,11 +183,7 @@ private:
>> }
>>
>> bool isSibling(unsigned Reg);
>> - MachineInstr *traceSiblingValue(unsigned, VNInfo*, VNInfo*);
>> - void propagateSiblingValue(SibValueMap::iterator, VNInfo *VNI =
>> nullptr);
>> - void analyzeSiblingValues();
>> -
>> - bool hoistSpill(LiveInterval &SpillLI, MachineInstr &CopyMI);
>> + bool hoistSpillInsideBB(LiveInterval &SpillLI, MachineInstr &CopyMI);
>> void eliminateRedundantSpills(LiveInterval &LI, VNInfo *VNI);
>>
>> void markValueUsed(LiveInterval*, VNInfo*);
>> @@ -297,417 +315,45 @@ void InlineSpiller::collectRegsToSpill()
>> }
>> }
>>
>> -
>>
>> -//===----------------------------------------------------------------------===//
>> -// Sibling Values
>>
>> -//===----------------------------------------------------------------------===//
>> -
>> -// After live range splitting, some values to be spilled may be defined
>> by
>> -// copies from sibling registers. We trace the sibling copies back to the
>> -// original value if it still exists. We need it for rematerialization.
>> -//
>> -// Even when the value can't be rematerialized, we still want to
>> determine if
>> -// the value has already been spilled, or we may want to hoist the spill
>> from a
>> -// loop.
>> -
>> bool InlineSpiller::isSibling(unsigned Reg) {
>> return TargetRegisterInfo::isVirtualRegister(Reg) &&
>> VRM.getOriginal(Reg) == Original;
>> }
>>
>> -#ifndef NDEBUG
>> -static raw_ostream &operator<<(raw_ostream &OS,
>> - const InlineSpiller::SibValueInfo &SVI) {
>> - OS << "spill " << PrintReg(SVI.SpillReg) << ':'
>> - << SVI.SpillVNI->id << '@' << SVI.SpillVNI->def;
>> - if (SVI.SpillMBB)
>> - OS << " in BB#" << SVI.SpillMBB->getNumber();
>> - if (SVI.AllDefsAreReloads)
>> - OS << " all-reloads";
>> - if (SVI.DefByOrigPHI)
>> - OS << " orig-phi";
>> - if (SVI.KillsSource)
>> - OS << " kill";
>> - OS << " deps[";
>> - for (VNInfo *Dep : SVI.Deps)
>> - OS << ' ' << Dep->id << '@' << Dep->def;
>> - OS << " ]";
>> - if (SVI.DefMI)
>> - OS << " def: " << *SVI.DefMI;
>> - else
>> - OS << '\n';
>> - return OS;
>> -}
>> -#endif
>> -
>> -/// propagateSiblingValue - Propagate the value in SVI to dependents if
>> it is
>> -/// known. Otherwise remember the dependency for later.
>> +/// It is beneficial to spill to earlier place in the same BB in case
>> +/// as follows:
>> +/// There is an alternative def earlier in the same MBB.
>> +/// Hoist the spill as far as possible in SpillMBB. This can ease
>> +/// register pressure:
>> ///
>> -/// @param SVIIter SibValues entry to propagate.
>> -/// @param VNI Dependent value, or NULL to propagate to all saved
>> dependents.
>> -void InlineSpiller::propagateSiblingValue(SibValueMap::iterator SVIIter,
>> - VNInfo *VNI) {
>> - SibValueMap::value_type *SVI = &*SVIIter;
>> -
>> - // When VNI is non-NULL, add it to SVI's deps, and only propagate to
>> that.
>> - TinyPtrVector<VNInfo*> FirstDeps;
>> - if (VNI) {
>> - FirstDeps.push_back(VNI);
>> - SVI->second.Deps.push_back(VNI);
>> - }
>> -
>> - // Has the value been completely determined yet? If not, defer
>> propagation.
>> - if (!SVI->second.hasDef())
>> - return;
>> -
>> - // Work list of values to propagate.
>> - SmallSetVector<SibValueMap::value_type *, 8> WorkList;
>> - WorkList.insert(SVI);
>> -
>> - do {
>> - SVI = WorkList.pop_back_val();
>> - TinyPtrVector<VNInfo*> *Deps = VNI ? &FirstDeps : &SVI->second.Deps;
>> - VNI = nullptr;
>> -
>> - SibValueInfo &SV = SVI->second;
>> - if (!SV.SpillMBB)
>> - SV.SpillMBB = LIS.getMBBFromIndex(SV.SpillVNI->def);
>> -
>> - DEBUG(dbgs() << " prop to " << Deps->size() << ": "
>> - << SVI->first->id << '@' << SVI->first->def << ":\t" <<
>> SV);
>> -
>> - assert(SV.hasDef() && "Propagating undefined value");
>> -
>> - // Should this value be propagated as a preferred spill candidate?
>> We don't
>> - // propagate values of registers that are about to spill.
>> - bool PropSpill = !DisableHoisting && !isRegToSpill(SV.SpillReg);
>> - unsigned SpillDepth = ~0u;
>> -
>> - for (VNInfo *Dep : *Deps) {
>> - SibValueMap::iterator DepSVI = SibValues.find(Dep);
>> - assert(DepSVI != SibValues.end() && "Dependent value not in
>> SibValues");
>> - SibValueInfo &DepSV = DepSVI->second;
>> - if (!DepSV.SpillMBB)
>> - DepSV.SpillMBB = LIS.getMBBFromIndex(DepSV.SpillVNI->def);
>> -
>> - bool Changed = false;
>> -
>> - // Propagate defining instruction.
>> - if (!DepSV.hasDef()) {
>> - Changed = true;
>> - DepSV.DefMI = SV.DefMI;
>> - DepSV.DefByOrigPHI = SV.DefByOrigPHI;
>> - }
>> -
>> - // Propagate AllDefsAreReloads. For PHI values, this computes an
>> AND of
>> - // all predecessors.
>> - if (!SV.AllDefsAreReloads && DepSV.AllDefsAreReloads) {
>> - Changed = true;
>> - DepSV.AllDefsAreReloads = false;
>> - }
>> -
>> - // Propagate best spill value.
>> - if (PropSpill && SV.SpillVNI != DepSV.SpillVNI) {
>> - if (SV.SpillMBB == DepSV.SpillMBB) {
>> - // DepSV is in the same block. Hoist when dominated.
>> - if (DepSV.KillsSource && SV.SpillVNI->def <
>> DepSV.SpillVNI->def) {
>> - // This is an alternative def earlier in the same MBB.
>> - // Hoist the spill as far as possible in SpillMBB. This can
>> ease
>> - // register pressure:
>> - //
>> - // x = def
>> - // y = use x
>> - // s = copy x
>> - //
>> - // Hoisting the spill of s to immediately after the def
>> removes the
>> - // interference between x and y:
>> - //
>> - // x = def
>> - // spill x
>> - // y = use x<kill>
>> - //
>> - // This hoist only helps when the DepSV copy kills its
>> source.
>> - Changed = true;
>> - DepSV.SpillReg = SV.SpillReg;
>> - DepSV.SpillVNI = SV.SpillVNI;
>> - DepSV.SpillMBB = SV.SpillMBB;
>> - }
>> - } else {
>> - // DepSV is in a different block.
>> - if (SpillDepth == ~0u)
>> - SpillDepth = Loops.getLoopDepth(SV.SpillMBB);
>> -
>> - // Also hoist spills to blocks with smaller loop depth, but
>> make sure
>> - // that the new value dominates. Non-phi dependents are always
>> - // dominated, phis need checking.
>> -
>> - const BranchProbability MarginProb(4, 5); // 80%
>> - // Hoist a spill to outer loop if there are multiple
>> dependents (it
>> - // can be beneficial if more than one dependents are hoisted)
>> or
>> - // if DepSV (the hoisting source) is hotter than SV (the
>> hoisting
>> - // destination) (we add a 80% margin to bias a little towards
>> - // loop depth).
>> - bool HoistCondition =
>> - (MBFI.getBlockFreq(DepSV.SpillMBB) >=
>> - (MBFI.getBlockFreq(SV.SpillMBB) * MarginProb)) ||
>> - Deps->size() > 1;
>> -
>> - if ((Loops.getLoopDepth(DepSV.SpillMBB) > SpillDepth) &&
>> - HoistCondition &&
>> - (!DepSVI->first->isPHIDef() ||
>> - MDT.dominates(SV.SpillMBB, DepSV.SpillMBB))) {
>> - Changed = true;
>> - DepSV.SpillReg = SV.SpillReg;
>> - DepSV.SpillVNI = SV.SpillVNI;
>> - DepSV.SpillMBB = SV.SpillMBB;
>> - }
>> - }
>> - }
>> -
>> - if (!Changed)
>> - continue;
>> -
>> - // Something changed in DepSVI. Propagate to dependents.
>> - WorkList.insert(&*DepSVI);
>> -
>> - DEBUG(dbgs() << " update " << DepSVI->first->id << '@'
>> - << DepSVI->first->def << " to:\t" << DepSV);
>> - }
>> - } while (!WorkList.empty());
>> -}
>> -
>> -/// traceSiblingValue - Trace a value that is about to be spilled back
>> to the
>> -/// real defining instructions by looking through sibling copies. Always
>> stay
>> -/// within the range of OrigVNI so the registers are known to carry the
>> same
>> -/// value.
>> +/// x = def
>> +/// y = use x
>> +/// s = copy x
>> ///
>> -/// Determine if the value is defined by all reloads, so spilling isn't
>> -/// necessary - the value is already in the stack slot.
>> +/// Hoisting the spill of s to immediately after the def removes the
>> +/// interference between x and y:
>> ///
>> -/// Return a defining instruction that may be a candidate for
>> rematerialization.
>> +/// x = def
>> +/// spill x
>> +/// y = use x<kill>
>> ///
>> -MachineInstr *InlineSpiller::traceSiblingValue(unsigned UseReg, VNInfo
>> *UseVNI,
>> - VNInfo *OrigVNI) {
>> - // Check if a cached value already exists.
>> - SibValueMap::iterator SVI;
>> - bool Inserted;
>> - std::tie(SVI, Inserted) =
>> - SibValues.insert(std::make_pair(UseVNI, SibValueInfo(UseReg,
>> UseVNI)));
>> - if (!Inserted) {
>> - DEBUG(dbgs() << "Cached value " << PrintReg(UseReg) << ':'
>> - << UseVNI->id << '@' << UseVNI->def << ' ' <<
>> SVI->second);
>> - return SVI->second.DefMI;
>> - }
>> -
>> - DEBUG(dbgs() << "Tracing value " << PrintReg(UseReg) << ':'
>> - << UseVNI->id << '@' << UseVNI->def << '\n');
>> -
>> - // List of (Reg, VNI) that have been inserted into SibValues, but need
>> to be
>> - // processed.
>> - SmallVector<std::pair<unsigned, VNInfo*>, 8> WorkList;
>> - WorkList.push_back(std::make_pair(UseReg, UseVNI));
>> -
>> - LiveInterval &OrigLI = LIS.getInterval(Original);
>> - do {
>> - unsigned Reg;
>> - VNInfo *VNI;
>> - std::tie(Reg, VNI) = WorkList.pop_back_val();
>> - DEBUG(dbgs() << " " << PrintReg(Reg) << ':' << VNI->id << '@' <<
>> VNI->def
>> - << ":\t");
>> -
>> - // First check if this value has already been computed.
>> - SVI = SibValues.find(VNI);
>> - assert(SVI != SibValues.end() && "Missing SibValues entry");
>> -
>> - // Trace through PHI-defs created by live range splitting.
>> - if (VNI->isPHIDef()) {
>> - // Stop at original PHIs. We don't know the value at the
>> - // predecessors. Look up the VNInfo for the current definition
>> - // in OrigLI, to properly determine whether or not this phi was
>> - // added by splitting.
>> - if (VNI->def == OrigLI.getVNInfoAt(VNI->def)->def) {
>> - DEBUG(dbgs() << "orig phi value\n");
>> - SVI->second.DefByOrigPHI = true;
>> - SVI->second.AllDefsAreReloads = false;
>> - propagateSiblingValue(SVI);
>> - continue;
>> - }
>> -
>> - // This is a PHI inserted by live range splitting. We could trace
>> the
>> - // live-out value from predecessor blocks, but that search can be
>> very
>> - // expensive if there are many predecessors and many more PHIs as
>> - // generated by tail-dup when it sees an indirectbr. Instead,
>> look at
>> - // all the non-PHI defs that have the same value as OrigVNI. They
>> must
>> - // jointly dominate VNI->def. This is not optimal since VNI may
>> actually
>> - // be jointly dominated by a smaller subset of defs, so there is a
>> change
>> - // we will miss a AllDefsAreReloads optimization.
>> -
>> - // Separate all values dominated by OrigVNI into PHIs and non-PHIs.
>> - SmallVector<VNInfo*, 8> PHIs, NonPHIs;
>> - LiveInterval &LI = LIS.getInterval(Reg);
>> -
>> - for (LiveInterval::vni_iterator VI = LI.vni_begin(), VE =
>> LI.vni_end();
>> - VI != VE; ++VI) {
>> - VNInfo *VNI2 = *VI;
>> - if (VNI2->isUnused())
>> - continue;
>> - if (!OrigLI.containsOneValue() &&
>> - OrigLI.getVNInfoAt(VNI2->def) != OrigVNI)
>> - continue;
>> - if (VNI2->isPHIDef() && VNI2->def != OrigVNI->def)
>> - PHIs.push_back(VNI2);
>> - else
>> - NonPHIs.push_back(VNI2);
>> - }
>> - DEBUG(dbgs() << "split phi value, checking " << PHIs.size()
>> - << " phi-defs, and " << NonPHIs.size()
>> - << " non-phi/orig defs\n");
>> -
>> - // Create entries for all the PHIs. Don't add them to the
>> worklist, we
>> - // are processing all of them in one go here.
>> - for (VNInfo *PHI : PHIs)
>> - SibValues.insert(std::make_pair(PHI, SibValueInfo(Reg, PHI)));
>> -
>> - // Add every PHI as a dependent of all the non-PHIs.
>> - for (VNInfo *NonPHI : NonPHIs) {
>> - // Known value? Try an insertion.
>> - std::tie(SVI, Inserted) =
>> - SibValues.insert(std::make_pair(NonPHI, SibValueInfo(Reg,
>> NonPHI)));
>> - // Add all the PHIs as dependents of NonPHI.
>> - SVI->second.Deps.insert(SVI->second.Deps.end(), PHIs.begin(),
>> - PHIs.end());
>> - // This is the first time we see NonPHI, add it to the worklist.
>> - if (Inserted)
>> - WorkList.push_back(std::make_pair(Reg, NonPHI));
>> - else
>> - // Propagate to all inserted PHIs, not just VNI.
>> - propagateSiblingValue(SVI);
>> - }
>> -
>> - // Next work list item.
>> - continue;
>> - }
>> -
>> - MachineInstr *MI = LIS.getInstructionFromIndex(VNI->def);
>> - assert(MI && "Missing def");
>> -
>> - // Trace through sibling copies.
>> - if (unsigned SrcReg = isFullCopyOf(MI, Reg)) {
>> - if (isSibling(SrcReg)) {
>> - LiveInterval &SrcLI = LIS.getInterval(SrcReg);
>> - LiveQueryResult SrcQ = SrcLI.Query(VNI->def);
>> - assert(SrcQ.valueIn() && "Copy from non-existing value");
>> - // Check if this COPY kills its source.
>> - SVI->second.KillsSource = SrcQ.isKill();
>> - VNInfo *SrcVNI = SrcQ.valueIn();
>> - DEBUG(dbgs() << "copy of " << PrintReg(SrcReg) << ':'
>> - << SrcVNI->id << '@' << SrcVNI->def
>> - << " kill=" << unsigned(SVI->second.KillsSource) <<
>> '\n');
>> - // Known sibling source value? Try an insertion.
>> - std::tie(SVI, Inserted) = SibValues.insert(
>> - std::make_pair(SrcVNI, SibValueInfo(SrcReg, SrcVNI)));
>> - // This is the first time we see Src, add it to the worklist.
>> - if (Inserted)
>> - WorkList.push_back(std::make_pair(SrcReg, SrcVNI));
>> - propagateSiblingValue(SVI, VNI);
>> - // Next work list item.
>> - continue;
>> - }
>> - }
>> -
>> - // Track reachable reloads.
>> - SVI->second.DefMI = MI;
>> - SVI->second.SpillMBB = MI->getParent();
>> - int FI;
>> - if (Reg == TII.isLoadFromStackSlot(MI, FI) && FI == StackSlot) {
>> - DEBUG(dbgs() << "reload\n");
>> - propagateSiblingValue(SVI);
>> - // Next work list item.
>> - continue;
>> - }
>> -
>> - // Potential remat candidate.
>> - DEBUG(dbgs() << "def " << *MI);
>> - SVI->second.AllDefsAreReloads = false;
>> - propagateSiblingValue(SVI);
>> - } while (!WorkList.empty());
>> -
>> - // Look up the value we were looking for. We already did this lookup
>> at the
>> - // top of the function, but SibValues may have been invalidated.
>> - SVI = SibValues.find(UseVNI);
>> - assert(SVI != SibValues.end() && "Didn't compute requested info");
>> - DEBUG(dbgs() << " traced to:\t" << SVI->second);
>> - return SVI->second.DefMI;
>> -}
>> -
>> -/// analyzeSiblingValues - Trace values defined by sibling copies back to
>> -/// something that isn't a sibling copy.
>> +/// This hoist only helps when the copy kills its source.
>> ///
>> -/// Keep track of values that may be rematerializable.
>> -void InlineSpiller::analyzeSiblingValues() {
>> - SibValues.clear();
>> -
>> - // No siblings at all?
>> - if (Edit->getReg() == Original)
>> - return;
>> -
>> - LiveInterval &OrigLI = LIS.getInterval(Original);
>> - for (unsigned Reg : RegsToSpill) {
>> - LiveInterval &LI = LIS.getInterval(Reg);
>> - for (LiveInterval::const_vni_iterator VI = LI.vni_begin(),
>> - VE = LI.vni_end(); VI != VE; ++VI) {
>> - VNInfo *VNI = *VI;
>> - if (VNI->isUnused())
>> - continue;
>> - MachineInstr *DefMI = nullptr;
>> - if (!VNI->isPHIDef()) {
>> - DefMI = LIS.getInstructionFromIndex(VNI->def);
>> - assert(DefMI && "No defining instruction");
>> - }
>> - // Check possible sibling copies.
>> - if (VNI->isPHIDef() || DefMI->isCopy()) {
>> - VNInfo *OrigVNI = OrigLI.getVNInfoAt(VNI->def);
>> - assert(OrigVNI && "Def outside original live range");
>> - if (OrigVNI->def != VNI->def)
>> - DefMI = traceSiblingValue(Reg, VNI, OrigVNI);
>> - }
>> - if (DefMI && Edit->checkRematerializable(VNI, DefMI, AA)) {
>> - DEBUG(dbgs() << "Value " << PrintReg(Reg) << ':' << VNI->id <<
>> '@'
>> - << VNI->def << " may remat from " << *DefMI);
>> - }
>> - }
>> - }
>> -}
>> -
>> -/// hoistSpill - Given a sibling copy that defines a value to be
>> spilled, insert
>> -/// a spill at a better location.
>> -bool InlineSpiller::hoistSpill(LiveInterval &SpillLI, MachineInstr
>> &CopyMI) {
>> +bool InlineSpiller::hoistSpillInsideBB(LiveInterval &SpillLI,
>> + MachineInstr &CopyMI) {
>> SlotIndex Idx = LIS.getInstructionIndex(CopyMI);
>> +#ifndef NDEBUG
>> VNInfo *VNI = SpillLI.getVNInfoAt(Idx.getRegSlot());
>> assert(VNI && VNI->def == Idx.getRegSlot() && "Not defined by copy");
>> - SibValueMap::iterator I = SibValues.find(VNI);
>> - if (I == SibValues.end())
>> - return false;
>> -
>> - const SibValueInfo &SVI = I->second;
>> -
>> - // Let the normal folding code deal with the boring case.
>> - if (!SVI.AllDefsAreReloads && SVI.SpillVNI == VNI)
>> - return false;
>> -
>> - // SpillReg may have been deleted by remat and DCE.
>> - if (!LIS.hasInterval(SVI.SpillReg)) {
>> - DEBUG(dbgs() << "Stale interval: " << PrintReg(SVI.SpillReg) <<
>> '\n');
>> - SibValues.erase(I);
>> - return false;
>> - }
>> +#endif
>>
>> - LiveInterval &SibLI = LIS.getInterval(SVI.SpillReg);
>> - if (!SibLI.containsValue(SVI.SpillVNI)) {
>> - DEBUG(dbgs() << "Stale value: " << PrintReg(SVI.SpillReg) << '\n');
>> - SibValues.erase(I);
>> + unsigned SrcReg = CopyMI.getOperand(1).getReg();
>> + LiveInterval &SrcLI = LIS.getInterval(SrcReg);
>> + VNInfo *SrcVNI = SrcLI.getVNInfoAt(Idx);
>> + LiveQueryResult SrcQ = SrcLI.Query(Idx);
>> + MachineBasicBlock *DefMBB = LIS.getMBBFromIndex(SrcVNI->def);
>> + if (DefMBB != CopyMI.getParent() || !SrcQ.isKill())
>> return false;
>> - }
>>
>> // Conservatively extend the stack slot range to the range of the
>> original
>> // value. We may be able to do better with stack slot coloring by
>> being more
>> @@ -719,35 +365,29 @@ bool InlineSpiller::hoistSpill(LiveInter
>> DEBUG(dbgs() << "\tmerged orig valno " << OrigVNI->id << ": "
>> << *StackInt << '\n');
>>
>> - // Already spilled everywhere.
>> - if (SVI.AllDefsAreReloads) {
>> - DEBUG(dbgs() << "\tno spill needed: " << SVI);
>> - ++NumOmitReloadSpill;
>> - return true;
>> - }
>> - // We are going to spill SVI.SpillVNI immediately after its def, so
>> clear out
>> + // We are going to spill SrcVNI immediately after its def, so clear out
>> // any later spills of the same value.
>> - eliminateRedundantSpills(SibLI, SVI.SpillVNI);
>> + eliminateRedundantSpills(SrcLI, SrcVNI);
>>
>> - MachineBasicBlock *MBB = LIS.getMBBFromIndex(SVI.SpillVNI->def);
>> + MachineBasicBlock *MBB = LIS.getMBBFromIndex(SrcVNI->def);
>> MachineBasicBlock::iterator MII;
>> - if (SVI.SpillVNI->isPHIDef())
>> + if (SrcVNI->isPHIDef())
>> MII = MBB->SkipPHIsAndLabels(MBB->begin());
>> else {
>> - MachineInstr *DefMI = LIS.getInstructionFromIndex(SVI.SpillVNI->def);
>> + MachineInstr *DefMI = LIS.getInstructionFromIndex(SrcVNI->def);
>> assert(DefMI && "Defining instruction disappeared");
>> MII = DefMI;
>> ++MII;
>> }
>> // Insert spill without kill flag immediately after def.
>> - TII.storeRegToStackSlot(*MBB, MII, SVI.SpillReg, false, StackSlot,
>> - MRI.getRegClass(SVI.SpillReg), &TRI);
>> + TII.storeRegToStackSlot(*MBB, MII, SrcReg, false, StackSlot,
>> + MRI.getRegClass(SrcReg), &TRI);
>> --MII; // Point to store instruction.
>> LIS.InsertMachineInstrInMaps(*MII);
>> - DEBUG(dbgs() << "\thoisted: " << SVI.SpillVNI->def << '\t' << *MII);
>> + DEBUG(dbgs() << "\thoisted: " << SrcVNI->def << '\t' << *MII);
>>
>> + HSpiller.addToMergeableSpills(&(*MII), StackSlot, Original);
>> ++NumSpills;
>> - ++NumHoists;
>> return true;
>> }
>>
>> @@ -805,7 +445,8 @@ void InlineSpiller::eliminateRedundantSp
>> MI->setDesc(TII.get(TargetOpcode::KILL));
>> DeadDefs.push_back(MI);
>> ++NumSpillsRemoved;
>> - --NumSpills;
>> + if (HSpiller.rmFromMergeableSpills(MI, StackSlot))
>> + --NumSpills;
>> }
>> }
>> } while (!WorkList.empty());
>> @@ -876,12 +517,12 @@ bool InlineSpiller::reMaterializeFor(Liv
>> if (SnippetCopies.count(&MI))
>> return false;
>>
>> - // Use an OrigVNI from traceSiblingValue when ParentVNI is a sibling
>> copy.
>> + LiveInterval &OrigLI = LIS.getInterval(Original);
>> + VNInfo *OrigVNI = OrigLI.getVNInfoAt(UseIdx);
>> LiveRangeEdit::Remat RM(ParentVNI);
>> - SibValueMap::const_iterator SibI = SibValues.find(ParentVNI);
>> - if (SibI != SibValues.end())
>> - RM.OrigMI = SibI->second.DefMI;
>> - if (!Edit->canRematerializeAt(RM, UseIdx, false)) {
>> + RM.OrigMI = LIS.getInstructionFromIndex(OrigVNI->def);
>> +
>> + if (!Edit->canRematerializeAt(RM, OrigVNI, UseIdx, false)) {
>> markValueUsed(&VirtReg, ParentVNI);
>> DEBUG(dbgs() << "\tcannot remat for " << UseIdx << '\t' << MI);
>> return false;
>> @@ -931,7 +572,6 @@ bool InlineSpiller::reMaterializeFor(Liv
>> /// reMaterializeAll - Try to rematerialize as many uses as possible,
>> /// and trim the live ranges after.
>> void InlineSpiller::reMaterializeAll() {
>> - // analyzeSiblingValues has already tested all relevant defining
>> instructions.
>> if (!Edit->anyRematerializable(AA))
>> return;
>>
>> @@ -1017,6 +657,9 @@ bool InlineSpiller::coalesceStackAccess(
>> if (InstrReg != Reg || FI != StackSlot)
>> return false;
>>
>> + if (!IsLoad)
>> + HSpiller.rmFromMergeableSpills(MI, StackSlot);
>> +
>> DEBUG(dbgs() << "Coalescing stack access: " << *MI);
>> LIS.RemoveMachineInstrFromMaps(*MI);
>> MI->eraseFromParent();
>> @@ -1141,6 +784,9 @@ foldMemoryOperand(ArrayRef<std::pair<Mac
>> LIS.removePhysRegDefAt(Reg, Idx);
>> }
>>
>> + int FI;
>> + if (TII.isStoreToStackSlot(MI, FI) &&
>> HSpiller.rmFromMergeableSpills(MI, FI))
>> + --NumSpills;
>> LIS.ReplaceMachineInstrInMaps(*MI, *FoldMI);
>> MI->eraseFromParent();
>>
>> @@ -1166,9 +812,10 @@ foldMemoryOperand(ArrayRef<std::pair<Mac
>>
>> if (!WasCopy)
>> ++NumFolded;
>> - else if (Ops.front().second == 0)
>> + else if (Ops.front().second == 0) {
>> ++NumSpills;
>> - else
>> + HSpiller.addToMergeableSpills(FoldMI, StackSlot, Original);
>> + } else
>> ++NumReloads;
>> return true;
>> }
>> @@ -1203,6 +850,7 @@ void InlineSpiller::insertSpill(unsigned
>> DEBUG(dumpMachineInstrRangeWithSlotIndex(std::next(MI), MIS.end(),
>> LIS,
>> "spill"));
>> ++NumSpills;
>> + HSpiller.addToMergeableSpills(std::next(MI), StackSlot, Original);
>> }
>>
>> /// spillAroundUses - insert spill code around each use of Reg.
>> @@ -1266,8 +914,7 @@ void InlineSpiller::spillAroundUses(unsi
>> continue;
>> }
>> if (RI.Writes) {
>> - // Hoist the spill of a sib-reg copy.
>> - if (hoistSpill(OldLI, *MI)) {
>> + if (hoistSpillInsideBB(OldLI, *MI)) {
>> // This COPY is now dead, the value is already in the stack
>> slot.
>> MI->getOperand(0).setIsDead();
>> DeadDefs.push_back(MI);
>> @@ -1380,7 +1027,6 @@ void InlineSpiller::spill(LiveRangeEdit
>> assert(DeadDefs.empty() && "Previous spill didn't remove dead defs");
>>
>> collectRegsToSpill();
>> - analyzeSiblingValues();
>> reMaterializeAll();
>>
>> // Remat may handle everything.
>> @@ -1389,3 +1035,393 @@ void InlineSpiller::spill(LiveRangeEdit
>>
>> Edit->calculateRegClassAndHint(MF, Loops, MBFI);
>> }
>> +
>> +/// Optimizations after all the reg selections and spills are done.
>> +///
>> +void InlineSpiller::postOptimization() {
>> + SmallVector<unsigned, 4> NewVRegs;
>> + LiveRangeEdit LRE(nullptr, NewVRegs, MF, LIS, &VRM, nullptr);
>> + HSpiller.hoistAllSpills(LRE);
>> + assert(NewVRegs.size() == 0 &&
>> + "No new vregs should be generated in hoistAllSpills");
>> +}
>> +
>> +/// When a spill is inserted, add the spill to MergeableSpills map.
>> +///
>> +void HoistSpillHelper::addToMergeableSpills(MachineInstr *Spill, int
>> StackSlot,
>> + unsigned Original) {
>> + StackSlotToReg[StackSlot] = Original;
>> + SlotIndex Idx = LIS.getInstructionIndex(*Spill);
>> + VNInfo *OrigVNI =
>> LIS.getInterval(Original).getVNInfoAt(Idx.getRegSlot());
>> + std::pair<int, VNInfo *> MIdx = std::make_pair(StackSlot, OrigVNI);
>> + MergeableSpills[MIdx].insert(Spill);
>> +}
>> +
>> +/// When a spill is removed, remove the spill from MergeableSpills map.
>> +/// Return true if the spill is removed successfully.
>> +///
>> +bool HoistSpillHelper::rmFromMergeableSpills(MachineInstr *Spill,
>> + int StackSlot) {
>> + int Original = StackSlotToReg[StackSlot];
>> + if (!Original)
>> + return false;
>> + SlotIndex Idx = LIS.getInstructionIndex(*Spill);
>> + VNInfo *OrigVNI =
>> LIS.getInterval(Original).getVNInfoAt(Idx.getRegSlot());
>> + std::pair<int, VNInfo *> MIdx = std::make_pair(StackSlot, OrigVNI);
>> + return MergeableSpills[MIdx].erase(Spill);
>> +}
>> +
>> +/// Check BB to see if it is a possible target BB to place a hoisted
>> spill,
>> +/// i.e., there should be a living sibling of OrigReg at the insert
>> point.
>> +///
>> +bool HoistSpillHelper::isSpillCandBB(unsigned OrigReg, VNInfo &OrigVNI,
>> + MachineBasicBlock &BB, unsigned
>> &LiveReg) {
>> + SlotIndex Idx;
>> + MachineBasicBlock::iterator MI = BB.getFirstTerminator();
>> + if (MI != BB.end())
>> + Idx = LIS.getInstructionIndex(*MI);
>> + else
>> + Idx = LIS.getMBBEndIdx(&BB).getPrevSlot();
>> + SmallSetVector<unsigned, 16> &Siblings = Virt2SiblingsMap[OrigReg];
>> + assert((LIS.getInterval(OrigReg)).getVNInfoAt(Idx) == &OrigVNI &&
>> + "Unexpected VNI");
>> +
>> + for (auto const SibReg : Siblings) {
>> + LiveInterval &LI = LIS.getInterval(SibReg);
>> + VNInfo *VNI = LI.getVNInfoAt(Idx);
>> + if (VNI) {
>> + LiveReg = SibReg;
>> + return true;
>> + }
>> + }
>> + return false;
>> +}
>> +
>> +/// Remove redundent spills in the same BB. Save those redundent spills
>> in
>> +/// SpillsToRm, and save the spill to keep and its BB in SpillBBToSpill
>> map.
>> +///
>> +void HoistSpillHelper::rmRedundantSpills(
>> + SmallPtrSet<MachineInstr *, 16> &Spills,
>> + SmallVectorImpl<MachineInstr *> &SpillsToRm,
>> + DenseMap<MachineDomTreeNode *, MachineInstr *> &SpillBBToSpill) {
>> + // For each spill saw, check SpillBBToSpill[] and see if its BB
>> already has
>> + // another spill inside. If a BB contains more than one spill, only
>> keep the
>> + // earlier spill with smaller SlotIndex.
>> + for (const auto CurrentSpill : Spills) {
>> + MachineBasicBlock *Block = CurrentSpill->getParent();
>> + MachineDomTreeNode *Node = MDT.DT->getNode(Block);
>> + MachineInstr *PrevSpill = SpillBBToSpill[Node];
>> + if (PrevSpill) {
>> + SlotIndex PIdx = LIS.getInstructionIndex(*PrevSpill);
>> + SlotIndex CIdx = LIS.getInstructionIndex(*CurrentSpill);
>> + MachineInstr *SpillToRm = (CIdx > PIdx) ? CurrentSpill : PrevSpill;
>> + MachineInstr *SpillToKeep = (CIdx > PIdx) ? PrevSpill :
>> CurrentSpill;
>> + SpillsToRm.push_back(SpillToRm);
>> + SpillBBToSpill[MDT.DT->getNode(Block)] = SpillToKeep;
>> + } else {
>> + SpillBBToSpill[MDT.DT->getNode(Block)] = CurrentSpill;
>> + }
>> + }
>> + for (const auto SpillToRm : SpillsToRm)
>> + Spills.erase(SpillToRm);
>> +}
>> +
>> +/// Starting from \p Root find a top-down traversal order of the
>> dominator
>> +/// tree to visit all basic blocks containing the elements of \p Spills.
>> +/// Redundant spills will be found and put into \p SpillsToRm at the same
>> +/// time. \p SpillBBToSpill will be populated as part of the process and
>> +/// maps a basic block to the first store occurring in the basic block.
>> +/// \post SpillsToRm.union(Spills at post) == Spills at pre
>> +///
>> +void HoistSpillHelper::getVisitOrders(
>> + MachineBasicBlock *Root, SmallPtrSet<MachineInstr *, 16> &Spills,
>> + SmallVectorImpl<MachineDomTreeNode *> &Orders,
>> + SmallVectorImpl<MachineInstr *> &SpillsToRm,
>> + DenseMap<MachineDomTreeNode *, unsigned> &SpillsToKeep,
>> + DenseMap<MachineDomTreeNode *, MachineInstr *> &SpillBBToSpill) {
>> + // The set contains all the possible BB nodes to which we may hoist
>> + // original spills.
>> + SmallPtrSet<MachineDomTreeNode *, 8> WorkSet;
>> + // Save the BB nodes on the path from the first BB node containing
>> + // non-redundent spill to the Root node.
>> + SmallPtrSet<MachineDomTreeNode *, 8> NodesOnPath;
>> + // All the spills to be hoisted must originate from a single def
>> instruction
>> + // to the OrigReg. It means the def instruction should dominate all
>> the spills
>> + // to be hoisted. We choose the BB where the def instruction is
>> located as
>> + // the Root.
>> + MachineDomTreeNode *RootIDomNode = MDT[Root]->getIDom();
>> + // For every node on the dominator tree with spill, walk up on the
>> dominator
>> + // tree towards the Root node until it is reached. If there is other
>> node
>> + // containing spill in the middle of the path, the previous spill saw
>> will
>> + // be redundent and the node containing it will be removed. All the
>> nodes on
>> + // the path starting from the first node with non-redundent spill to
>> the Root
>> + // node will be added to the WorkSet, which will contain all the
>> possible
>> + // locations where spills may be hoisted to after the loop below is
>> done.
>> + for (const auto Spill : Spills) {
>> + MachineBasicBlock *Block = Spill->getParent();
>> + MachineDomTreeNode *Node = MDT[Block];
>> + MachineInstr *SpillToRm = nullptr;
>> + while (Node != RootIDomNode) {
>> + // If Node dominates Block, and it already contains a spill, the
>> spill in
>> + // Block will be redundent.
>> + if (Node != MDT[Block] && SpillBBToSpill[Node]) {
>> + SpillToRm = SpillBBToSpill[MDT[Block]];
>> + break;
>> + /// If we see the Node already in WorkSet, the path from the
>> Node to
>> + /// the Root node must already be traversed by another spill.
>> + /// Then no need to repeat.
>> + } else if (WorkSet.count(Node)) {
>> + break;
>> + } else {
>> + NodesOnPath.insert(Node);
>> + }
>> + Node = Node->getIDom();
>> + }
>> + if (SpillToRm) {
>> + SpillsToRm.push_back(SpillToRm);
>> + } else {
>> + // Add a BB containing the original spills to SpillsToKeep -- i.e.,
>> + // set the initial status before hoisting start. The value of BBs
>> + // containing original spills is set to 0, in order to descriminate
>> + // with BBs containing hoisted spills which will be inserted to
>> + // SpillsToKeep later during hoisting.
>> + SpillsToKeep[MDT[Block]] = 0;
>> + WorkSet.insert(NodesOnPath.begin(), NodesOnPath.end());
>> + }
>> + NodesOnPath.clear();
>> + }
>> +
>> + // Sort the nodes in WorkSet in top-down order and save the nodes
>> + // in Orders. Orders will be used for hoisting in runHoistSpills.
>> + unsigned idx = 0;
>> + Orders.push_back(MDT.DT->getNode(Root));
>> + do {
>> + MachineDomTreeNode *Node = Orders[idx++];
>> + const std::vector<MachineDomTreeNode *> &Children =
>> Node->getChildren();
>> + unsigned NumChildren = Children.size();
>> + for (unsigned i = 0; i != NumChildren; ++i) {
>> + MachineDomTreeNode *Child = Children[i];
>> + if (WorkSet.count(Child))
>> + Orders.push_back(Child);
>> + }
>> + } while (idx != Orders.size());
>> + assert(Orders.size() == WorkSet.size() &&
>> + "Orders have different size with WorkSet");
>> +
>> +#ifndef NDEBUG
>> + DEBUG(dbgs() << "Orders size is " << Orders.size() << "\n");
>> + SmallVector<MachineDomTreeNode *, 32>::reverse_iterator RIt =
>> Orders.rbegin();
>> + for (; RIt != Orders.rend(); RIt++)
>> + DEBUG(dbgs() << "BB" << (*RIt)->getBlock()->getNumber() << ",");
>> + DEBUG(dbgs() << "\n");
>> +#endif
>> +}
>> +
>> +/// Try to hoist spills according to BB hotness. The spills to removed
>> will
>> +/// be saved in \p SpillsToRm. The spills to be inserted will be saved in
>> +/// \p SpillsToIns.
>> +///
>> +void HoistSpillHelper::runHoistSpills(
>> + unsigned OrigReg, VNInfo &OrigVNI, SmallPtrSet<MachineInstr *, 16>
>> &Spills,
>> + SmallVectorImpl<MachineInstr *> &SpillsToRm,
>> + DenseMap<MachineBasicBlock *, unsigned> &SpillsToIns) {
>> + // Visit order of dominator tree nodes.
>> + SmallVector<MachineDomTreeNode *, 32> Orders;
>> + // SpillsToKeep contains all the nodes where spills are to be inserted
>> + // during hoisting. If the spill to be inserted is an original spill
>> + // (not a hoisted one), the value of the map entry is 0. If the spill
>> + // is a hoisted spill, the value of the map entry is the VReg to be
>> used
>> + // as the source of the spill.
>> + DenseMap<MachineDomTreeNode *, unsigned> SpillsToKeep;
>> + // Map from BB to the first spill inside of it.
>> + DenseMap<MachineDomTreeNode *, MachineInstr *> SpillBBToSpill;
>> +
>> + rmRedundantSpills(Spills, SpillsToRm, SpillBBToSpill);
>> +
>> + MachineBasicBlock *Root = LIS.getMBBFromIndex(OrigVNI.def);
>> + getVisitOrders(Root, Spills, Orders, SpillsToRm, SpillsToKeep,
>> + SpillBBToSpill);
>> +
>> + // SpillsInSubTreeMap keeps the map from a dom tree node to a pair of
>> + // nodes set and the cost of all the spills inside those nodes.
>> + // The nodes set are the locations where spills are to be inserted
>> + // in the subtree of current node.
>> + typedef std::pair<SmallPtrSet<MachineDomTreeNode *, 16>,
>> BlockFrequency>
>> + NodesCostPair;
>> + DenseMap<MachineDomTreeNode *, NodesCostPair> SpillsInSubTreeMap;
>> + // Iterate Orders set in reverse order, which will be a bottom-up order
>> + // in the dominator tree. Once we visit a dom tree node, we know its
>> + // children have already been visited and the spill locations in the
>> + // subtrees of all the children have been determined.
>> + SmallVector<MachineDomTreeNode *, 32>::reverse_iterator RIt =
>> Orders.rbegin();
>> + for (; RIt != Orders.rend(); RIt++) {
>> + MachineBasicBlock *Block = (*RIt)->getBlock();
>> +
>> + // If Block contains an original spill, simply continue.
>> + if (SpillsToKeep.find(*RIt) != SpillsToKeep.end() &&
>> !SpillsToKeep[*RIt]) {
>> + SpillsInSubTreeMap[*RIt].first.insert(*RIt);
>> + // SpillsInSubTreeMap[*RIt].second contains the cost of spill.
>> + SpillsInSubTreeMap[*RIt].second = MBFI.getBlockFreq(Block);
>> + continue;
>> + }
>> +
>> + // Collect spills in subtree of current node (*RIt) to
>> + // SpillsInSubTreeMap[*RIt].first.
>> + const std::vector<MachineDomTreeNode *> &Children =
>> (*RIt)->getChildren();
>> + unsigned NumChildren = Children.size();
>> + for (unsigned i = 0; i != NumChildren; ++i) {
>> + MachineDomTreeNode *Child = Children[i];
>> + auto BI = SpillsInSubTreeMap[Child].first.begin();
>> + auto EI = SpillsInSubTreeMap[Child].first.end();
>> + SpillsInSubTreeMap[*RIt].first.insert(BI, EI);
>> + SpillsInSubTreeMap[*RIt].second +=
>> SpillsInSubTreeMap[Child].second;
>> + SpillsInSubTreeMap.erase(Child);
>> + }
>> +
>> + // No spills in subtree, simply continue.
>> + if (SpillsInSubTreeMap[*RIt].first.empty())
>> + continue;
>> +
>> + // Check whether Block is a possible candidate to insert spill.
>> + unsigned LiveReg = 0;
>> + if (!isSpillCandBB(OrigReg, OrigVNI, *Block, LiveReg))
>> + continue;
>> +
>> + // If there are multiple spills that could be merged, bias a little
>> + // to hoist the spill.
>> + BranchProbability MarginProb =
>> (SpillsInSubTreeMap[*RIt].first.size() > 1)
>> + ? BranchProbability(9, 10)
>> + : BranchProbability(1, 1);
>> + if (SpillsInSubTreeMap[*RIt].second >
>> + MBFI.getBlockFreq(Block) * MarginProb) {
>> + // Hoist: Move spills to current Block.
>> + for (const auto SpillBB : SpillsInSubTreeMap[*RIt].first) {
>> + // When SpillBB is a BB contains original spill, insert the spill
>> + // to SpillsToRm.
>> + if (SpillsToKeep.find(SpillBB) != SpillsToKeep.end() &&
>> + !SpillsToKeep[SpillBB]) {
>> + MachineInstr *SpillToRm = SpillBBToSpill[SpillBB];
>> + SpillsToRm.push_back(SpillToRm);
>> + }
>> + // SpillBB will not contain spill anymore, remove it from
>> SpillsToKeep.
>> + SpillsToKeep.erase(SpillBB);
>> + }
>> + // Current Block is the BB containing the new hoisted spill. Add
>> it to
>> + // SpillsToKeep. LiveReg is the source of the new spill.
>> + SpillsToKeep[*RIt] = LiveReg;
>> + DEBUG({
>> + dbgs() << "spills in BB: ";
>> + for (const auto Rspill : SpillsInSubTreeMap[*RIt].first)
>> + dbgs() << Rspill->getBlock()->getNumber() << " ";
>> + dbgs() << "were promoted to BB" <<
>> (*RIt)->getBlock()->getNumber()
>> + << "\n";
>> + });
>> + SpillsInSubTreeMap[*RIt].first.clear();
>> + SpillsInSubTreeMap[*RIt].first.insert(*RIt);
>> + SpillsInSubTreeMap[*RIt].second = MBFI.getBlockFreq(Block);
>> + }
>> + }
>> + // For spills in SpillsToKeep with LiveReg set (i.e., not original
>> spill),
>> + // save them to SpillsToIns.
>> + for (const auto Ent : SpillsToKeep) {
>> + if (Ent.second)
>> + SpillsToIns[Ent.first->getBlock()] = Ent.second;
>> + }
>> +}
>> +
>> +/// For spills with equal values, remove redundent spills and hoist the
>> left
>> +/// to less hot spots.
>> +///
>> +/// Spills with equal values will be collected into the same set in
>> +/// MergeableSpills when spill is inserted. These equal spills are
>> originated
>> +/// from the same define instruction and are dominated by the
>> instruction.
>> +/// Before hoisting all the equal spills, redundent spills inside in the
>> same
>> +/// BB is first marked to be deleted. Then starting from spills left,
>> walk up
>> +/// on the dominator tree towards the Root node where the define
>> instruction
>> +/// is located, mark the dominated spills to be deleted along the way and
>> +/// collect the BB nodes on the path from non-dominated spills to the
>> define
>> +/// instruction into a WorkSet. The nodes in WorkSet are the candidate
>> places
>> +/// where we consider to hoist the spills. We iterate the WorkSet in
>> bottom-up
>> +/// order, and for each node, we will decide whether to hoist spills
>> inside
>> +/// its subtree to that node. In this way, we can get benefit locally
>> even if
>> +/// hoisting all the equal spills to one cold place is impossible.
>> +///
>> +void HoistSpillHelper::hoistAllSpills(LiveRangeEdit &Edit) {
>> + // Save the mapping between stackslot and its original reg.
>> + DenseMap<int, unsigned> SlotToOrigReg;
>> + for (unsigned i = 0, e = MRI.getNumVirtRegs(); i != e; ++i) {
>> + unsigned Reg = TargetRegisterInfo::index2VirtReg(i);
>> + int Slot = VRM.getStackSlot(Reg);
>> + if (Slot != VirtRegMap::NO_STACK_SLOT)
>> + SlotToOrigReg[Slot] = VRM.getOriginal(Reg);
>> + unsigned Original = VRM.getPreSplitReg(Reg);
>> + if (!MRI.def_empty(Reg))
>> + Virt2SiblingsMap[Original].insert(Reg);
>> + }
>> +
>> + // Each entry in MergeableSpills contains a spill set with equal
>> values.
>> + for (auto &Ent : MergeableSpills) {
>> + int Slot = Ent.first.first;
>> + unsigned OrigReg = SlotToOrigReg[Slot];
>> + VNInfo *OrigVNI = Ent.first.second;
>> + SmallPtrSet<MachineInstr *, 16> &EqValSpills = Ent.second;
>> + if (Ent.second.empty())
>> + continue;
>> +
>> + DEBUG({
>> + dbgs() << "\nFor Slot" << Slot << " and VN" << OrigVNI->id << ":\n"
>> + << "Equal spills in BB: ";
>> + for (const auto spill : EqValSpills)
>> + dbgs() << spill->getParent()->getNumber() << " ";
>> + dbgs() << "\n";
>> + });
>> +
>> + // SpillsToRm is the spill set to be removed from EqValSpills.
>> + SmallVector<MachineInstr *, 16> SpillsToRm;
>> + // SpillsToIns is the spill set to be newly inserted after hoisting.
>> + DenseMap<MachineBasicBlock *, unsigned> SpillsToIns;
>> +
>> + runHoistSpills(OrigReg, *OrigVNI, EqValSpills, SpillsToRm,
>> SpillsToIns);
>> +
>> + DEBUG({
>> + dbgs() << "Finally inserted spills in BB: ";
>> + for (const auto Ispill : SpillsToIns)
>> + dbgs() << Ispill.first->getNumber() << " ";
>> + dbgs() << "\nFinally removed spills in BB: ";
>> + for (const auto Rspill : SpillsToRm)
>> + dbgs() << Rspill->getParent()->getNumber() << " ";
>> + dbgs() << "\n";
>> + });
>> +
>> + // Stack live range update.
>> + LiveInterval &StackIntvl = LSS.getInterval(Slot);
>> + if (!SpillsToIns.empty() || !SpillsToRm.empty()) {
>> + LiveInterval &OrigLI = LIS.getInterval(OrigReg);
>> + StackIntvl.MergeValueInAsValue(OrigLI, OrigVNI,
>> + StackIntvl.getValNumInfo(0));
>> + }
>> +
>> + // Insert hoisted spills.
>> + for (auto const Insert : SpillsToIns) {
>> + MachineBasicBlock *BB = Insert.first;
>> + unsigned LiveReg = Insert.second;
>> + MachineBasicBlock::iterator MI = BB->getFirstTerminator();
>> + TII.storeRegToStackSlot(*BB, MI, LiveReg, false, Slot,
>> + MRI.getRegClass(LiveReg), &TRI);
>> + LIS.InsertMachineInstrRangeInMaps(std::prev(MI), MI);
>> + ++NumSpills;
>> + }
>> +
>> + // Remove redundent spills or change them to dead instructions.
>> + NumSpills -= SpillsToRm.size();
>> + for (auto const RMEnt : SpillsToRm) {
>> + RMEnt->setDesc(TII.get(TargetOpcode::KILL));
>> + for (unsigned i = RMEnt->getNumOperands(); i; --i) {
>> + MachineOperand &MO = RMEnt->getOperand(i - 1);
>> + if (MO.isReg() && MO.isImplicit() && MO.isDef() && !MO.isDead())
>> + RMEnt->RemoveOperand(i - 1);
>> + }
>> + }
>> + Edit.eliminateDeadDefs(SpillsToRm, None, true);
>> + }
>> +}
>>
>> Modified: llvm/trunk/lib/CodeGen/LiveRangeEdit.cpp
>> URL:
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/LiveRangeEdit.cpp?rev=265547&r1=265546&r2=265547&view=diff
>>
>> ==============================================================================
>> --- llvm/trunk/lib/CodeGen/LiveRangeEdit.cpp (original)
>> +++ llvm/trunk/lib/CodeGen/LiveRangeEdit.cpp Wed Apr 6 10:41:07 2016
>> @@ -63,10 +63,13 @@ void LiveRangeEdit::scanRemattable(Alias
>> for (VNInfo *VNI : getParent().valnos) {
>> if (VNI->isUnused())
>> continue;
>> - MachineInstr *DefMI = LIS.getInstructionFromIndex(VNI->def);
>> + unsigned Original = VRM->getOriginal(getReg());
>> + LiveInterval &OrigLI = LIS.getInterval(Original);
>> + VNInfo *OrigVNI = OrigLI.getVNInfoAt(VNI->def);
>> + MachineInstr *DefMI = LIS.getInstructionFromIndex(OrigVNI->def);
>> if (!DefMI)
>> continue;
>> - checkRematerializable(VNI, DefMI, aa);
>> + checkRematerializable(OrigVNI, DefMI, aa);
>> }
>> ScannedRemattable = true;
>> }
>> @@ -113,24 +116,18 @@ bool LiveRangeEdit::allUsesAvailableAt(c
>> return true;
>> }
>>
>> -bool LiveRangeEdit::canRematerializeAt(Remat &RM,
>> - SlotIndex UseIdx,
>> - bool cheapAsAMove) {
>> +bool LiveRangeEdit::canRematerializeAt(Remat &RM, VNInfo *OrigVNI,
>> + SlotIndex UseIdx, bool
>> cheapAsAMove) {
>> assert(ScannedRemattable && "Call anyRematerializable first");
>>
>> // Use scanRemattable info.
>> - if (!Remattable.count(RM.ParentVNI))
>> + if (!Remattable.count(OrigVNI))
>> return false;
>>
>> // No defining instruction provided.
>> SlotIndex DefIdx;
>> - if (RM.OrigMI)
>> - DefIdx = LIS.getInstructionIndex(*RM.OrigMI);
>> - else {
>> - DefIdx = RM.ParentVNI->def;
>> - RM.OrigMI = LIS.getInstructionFromIndex(DefIdx);
>> - assert(RM.OrigMI && "No defining instruction for remattable value");
>> - }
>> + assert(RM.OrigMI && "No defining instruction for remattable value");
>> + DefIdx = LIS.getInstructionIndex(*RM.OrigMI);
>>
>> // If only cheap remats were requested, bail out early.
>> if (cheapAsAMove && !TII.isAsCheapAsAMove(RM.OrigMI))
>> @@ -261,6 +258,15 @@ void LiveRangeEdit::eliminateDeadDef(Mac
>> // Collect virtual registers to be erased after MI is gone.
>> SmallVector<unsigned, 8> RegsToErase;
>> bool ReadsPhysRegs = false;
>> + bool isOrigDef = false;
>> + unsigned Dest;
>> + if (VRM && MI->getOperand(0).isReg()) {
>> + Dest = MI->getOperand(0).getReg();
>> + unsigned Original = VRM->getOriginal(Dest);
>> + LiveInterval &OrigLI = LIS.getInterval(Original);
>> + VNInfo *OrigVNI = OrigLI.getVNInfoAt(Idx);
>> + isOrigDef = SlotIndex::isSameInstr(OrigVNI->def, Idx);
>> + }
>>
>> // Check for live intervals that may shrink
>> for (MachineInstr::mop_iterator MOI = MI->operands_begin(),
>> @@ -314,11 +320,24 @@ void LiveRangeEdit::eliminateDeadDef(Mac
>> }
>> DEBUG(dbgs() << "Converted physregs to:\t" << *MI);
>> } else {
>> - if (TheDelegate)
>> - TheDelegate->LRE_WillEraseInstruction(MI);
>> - LIS.RemoveMachineInstrFromMaps(*MI);
>> - MI->eraseFromParent();
>> - ++NumDCEDeleted;
>> + // If the dest of MI is an original reg, don't delete the inst.
>> Replace
>> + // the dest with a new reg, keep the inst for remat of other
>> siblings.
>> + // The inst is saved in LiveRangeEdit::DeadRemats and will be deleted
>> + // after all the allocations of the func are done.
>> + if (isOrigDef) {
>> + unsigned NewDest = createFrom(Dest);
>> + pop_back();
>> + markDeadRemat(MI);
>> + const TargetRegisterInfo &TRI = *MRI.getTargetRegisterInfo();
>> + MI->substituteRegister(Dest, NewDest, 0, TRI);
>> + MI->getOperand(0).setIsDead(false);
>> + } else {
>> + if (TheDelegate)
>> + TheDelegate->LRE_WillEraseInstruction(MI);
>> + LIS.RemoveMachineInstrFromMaps(*MI);
>> + MI->eraseFromParent();
>> + ++NumDCEDeleted;
>> + }
>> }
>>
>> // Erase any virtregs that are now empty and unused. There may be
>> <undef>
>> @@ -332,8 +351,9 @@ void LiveRangeEdit::eliminateDeadDef(Mac
>> }
>> }
>>
>> -void LiveRangeEdit::eliminateDeadDefs(SmallVectorImpl<MachineInstr*>
>> &Dead,
>> - ArrayRef<unsigned>
>> RegsBeingSpilled) {
>> +void LiveRangeEdit::eliminateDeadDefs(SmallVectorImpl<MachineInstr *>
>> &Dead,
>> + ArrayRef<unsigned>
>> RegsBeingSpilled,
>> + bool NoSplit) {
>> ToShrinkSet ToShrink;
>>
>> for (;;) {
>> @@ -355,6 +375,9 @@ void LiveRangeEdit::eliminateDeadDefs(Sm
>> if (!LIS.shrinkToUses(LI, &Dead))
>> continue;
>>
>> + if (NoSplit)
>> + continue;
>> +
>> // Don't create new intervals for a register being spilled.
>> // The new intervals would have to be spilled anyway so its not
>> worth it.
>> // Also they currently aren't spilled so creating them and not
>> spilling
>>
>> Modified: llvm/trunk/lib/CodeGen/RegAllocBase.cpp
>> URL:
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/RegAllocBase.cpp?rev=265547&r1=265546&r2=265547&view=diff
>>
>> ==============================================================================
>> --- llvm/trunk/lib/CodeGen/RegAllocBase.cpp (original)
>> +++ llvm/trunk/lib/CodeGen/RegAllocBase.cpp Wed Apr 6 10:41:07 2016
>> @@ -153,3 +153,12 @@ void RegAllocBase::allocatePhysRegs() {
>> }
>> }
>> }
>> +
>> +void RegAllocBase::postOptimization() {
>> + spiller().postOptimization();
>> + for (auto DeadInst : DeadRemats) {
>> + LIS->RemoveMachineInstrFromMaps(*DeadInst);
>> + DeadInst->eraseFromParent();
>> + }
>> + DeadRemats.clear();
>> +}
>>
>> Modified: llvm/trunk/lib/CodeGen/RegAllocBase.h
>> URL:
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/RegAllocBase.h?rev=265547&r1=265546&r2=265547&view=diff
>>
>> ==============================================================================
>> --- llvm/trunk/lib/CodeGen/RegAllocBase.h (original)
>> +++ llvm/trunk/lib/CodeGen/RegAllocBase.h Wed Apr 6 10:41:07 2016
>> @@ -65,6 +65,12 @@ protected:
>> LiveRegMatrix *Matrix;
>> RegisterClassInfo RegClassInfo;
>>
>> + /// Inst which is a def of an original reg and whose defs are already
>> all
>> + /// dead after remat is saved in DeadRemats. The deletion of such inst
>> is
>> + /// postponed till all the allocations are done, so its remat expr is
>> + /// always available for the remat of all the siblings of the original
>> reg.
>> + SmallPtrSet<MachineInstr *, 32> DeadRemats;
>> +
>> RegAllocBase()
>> : TRI(nullptr), MRI(nullptr), VRM(nullptr), LIS(nullptr),
>> Matrix(nullptr) {}
>>
>> @@ -77,6 +83,10 @@ protected:
>> // physical register assignments.
>> void allocatePhysRegs();
>>
>> + // Include spiller post optimization and removing dead defs left
>> because of
>> + // rematerialization.
>> + virtual void postOptimization();
>> +
>> // Get a temporary reference to a Spiller instance.
>> virtual Spiller &spiller() = 0;
>>
>>
>> Modified: llvm/trunk/lib/CodeGen/RegAllocBasic.cpp
>> URL:
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/RegAllocBasic.cpp?rev=265547&r1=265546&r2=265547&view=diff
>>
>> ==============================================================================
>> --- llvm/trunk/lib/CodeGen/RegAllocBasic.cpp (original)
>> +++ llvm/trunk/lib/CodeGen/RegAllocBasic.cpp Wed Apr 6 10:41:07 2016
>> @@ -199,7 +199,7 @@ bool RABasic::spillInterferences(LiveInt
>> Matrix->unassign(Spill);
>>
>> // Spill the extracted interval.
>> - LiveRangeEdit LRE(&Spill, SplitVRegs, *MF, *LIS, VRM);
>> + LiveRangeEdit LRE(&Spill, SplitVRegs, *MF, *LIS, VRM, nullptr,
>> &DeadRemats);
>> spiller().spill(LRE);
>> }
>> return true;
>> @@ -258,7 +258,7 @@ unsigned RABasic::selectOrSplit(LiveInte
>> DEBUG(dbgs() << "spilling: " << VirtReg << '\n');
>> if (!VirtReg.isSpillable())
>> return ~0u;
>> - LiveRangeEdit LRE(&VirtReg, SplitVRegs, *MF, *LIS, VRM);
>> + LiveRangeEdit LRE(&VirtReg, SplitVRegs, *MF, *LIS, VRM, nullptr,
>> &DeadRemats);
>> spiller().spill(LRE);
>>
>> // The live virtual register requesting allocation was spilled, so
>> tell
>> @@ -283,6 +283,7 @@ bool RABasic::runOnMachineFunction(Machi
>> SpillerInstance.reset(createInlineSpiller(*this, *MF, *VRM));
>>
>> allocatePhysRegs();
>> + postOptimization();
>>
>> // Diagnostic output before rewriting
>> DEBUG(dbgs() << "Post alloc VirtRegMap:\n" << *VRM << "\n");
>>
>> Modified: llvm/trunk/lib/CodeGen/RegAllocGreedy.cpp
>> URL:
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/RegAllocGreedy.cpp?rev=265547&r1=265546&r2=265547&view=diff
>>
>> ==============================================================================
>> --- llvm/trunk/lib/CodeGen/RegAllocGreedy.cpp (original)
>> +++ llvm/trunk/lib/CodeGen/RegAllocGreedy.cpp Wed Apr 6 10:41:07 2016
>> @@ -12,7 +12,6 @@
>> //
>>
>> //===----------------------------------------------------------------------===//
>>
>> -#include "llvm/CodeGen/Passes.h"
>> #include "AllocationOrder.h"
>> #include "InterferenceCache.h"
>> #include "LiveDebugVariables.h"
>> @@ -33,6 +32,7 @@
>> #include "llvm/CodeGen/MachineFunctionPass.h"
>> #include "llvm/CodeGen/MachineLoopInfo.h"
>> #include "llvm/CodeGen/MachineRegisterInfo.h"
>> +#include "llvm/CodeGen/Passes.h"
>> #include "llvm/CodeGen/RegAllocRegistry.h"
>> #include "llvm/CodeGen/RegisterClassInfo.h"
>> #include "llvm/CodeGen/VirtRegMap.h"
>> @@ -44,6 +44,7 @@
>> #include "llvm/Support/ErrorHandling.h"
>> #include "llvm/Support/Timer.h"
>> #include "llvm/Support/raw_ostream.h"
>> +#include "llvm/Target/TargetInstrInfo.h"
>> #include "llvm/Target/TargetSubtargetInfo.h"
>> #include <queue>
>>
>> @@ -55,14 +56,14 @@ STATISTIC(NumGlobalSplits, "Number of sp
>> STATISTIC(NumLocalSplits, "Number of split local live ranges");
>> STATISTIC(NumEvicted, "Number of interferences evicted");
>>
>> -static cl::opt<SplitEditor::ComplementSpillMode>
>> -SplitSpillMode("split-spill-mode", cl::Hidden,
>> - cl::desc("Spill mode for splitting live ranges"),
>> - cl::values(clEnumValN(SplitEditor::SM_Partition, "default", "Default"),
>> - clEnumValN(SplitEditor::SM_Size, "size", "Optimize for
>> size"),
>> - clEnumValN(SplitEditor::SM_Speed, "speed", "Optimize for
>> speed"),
>> - clEnumValEnd),
>> - cl::init(SplitEditor::SM_Partition));
>> +static cl::opt<SplitEditor::ComplementSpillMode> SplitSpillMode(
>> + "split-spill-mode", cl::Hidden,
>> + cl::desc("Spill mode for splitting live ranges"),
>> + cl::values(clEnumValN(SplitEditor::SM_Partition, "default",
>> "Default"),
>> + clEnumValN(SplitEditor::SM_Size, "size", "Optimize for
>> size"),
>> + clEnumValN(SplitEditor::SM_Speed, "speed", "Optimize for
>> speed"),
>> + clEnumValEnd),
>> + cl::init(SplitEditor::SM_Speed));
>>
>> static cl::opt<unsigned>
>> LastChanceRecoloringMaxDepth("lcr-max-depth", cl::Hidden,
>> @@ -1465,7 +1466,7 @@ unsigned RAGreedy::doRegionSplit(LiveInt
>> SmallVectorImpl<unsigned> &NewVRegs) {
>> SmallVector<unsigned, 8> UsedCands;
>> // Prepare split editor.
>> - LiveRangeEdit LREdit(&VirtReg, NewVRegs, *MF, *LIS, VRM, this);
>> + LiveRangeEdit LREdit(&VirtReg, NewVRegs, *MF, *LIS, VRM, this,
>> &DeadRemats);
>> SE->reset(LREdit, SplitSpillMode);
>>
>> // Assign all edge bundles to the preferred candidate, or NoCand.
>> @@ -1513,7 +1514,7 @@ unsigned RAGreedy::tryBlockSplit(LiveInt
>> assert(&SA->getParent() == &VirtReg && "Live range wasn't analyzed");
>> unsigned Reg = VirtReg.reg;
>> bool SingleInstrs =
>> RegClassInfo.isProperSubClass(MRI->getRegClass(Reg));
>> - LiveRangeEdit LREdit(&VirtReg, NewVRegs, *MF, *LIS, VRM, this);
>> + LiveRangeEdit LREdit(&VirtReg, NewVRegs, *MF, *LIS, VRM, this,
>> &DeadRemats);
>> SE->reset(LREdit, SplitSpillMode);
>> ArrayRef<SplitAnalysis::BlockInfo> UseBlocks = SA->getUseBlocks();
>> for (unsigned i = 0; i != UseBlocks.size(); ++i) {
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20160407/d98dc6bb/attachment-0001.html>
More information about the llvm-commits
mailing list