<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">2015-05-27 10:15 GMT-07:00 Duncan P. N. Exon Smith <span dir="ltr"><<a href="mailto:dexonsmith@apple.com" target="_blank">dexonsmith@apple.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5"><br>
> On 2015 May 27, at 09:49, Alex Lorenz <<a href="mailto:arphaman@gmail.com">arphaman@gmail.com</a>> wrote:<br>
><br>
> Hi dexonsmith, bogner, bob.wilson,<br>
><br>
> This patch is related to my proposal for separating machine IR from CodeGen: <a href="http://lists.cs.uiuc.edu/pipermail/llvmdev/2015-May/086063.html" target="_blank">http://lists.cs.uiuc.edu/pipermail/llvmdev/2015-May/086063.html</a>.<br>
><br>
> This patch moves the SplitCriticalEdge method out of MachineBasicBlock class so that MachineBasicBlock.cpp won't have to include several header<br>
> files that declare passes that won't be moved from CodeGen to the new Machine IR library.<br>
><br>
> REPOSITORY<br>
> rL LLVM<br>
><br>
> <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__reviews.llvm.org_D10064&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=mQ4LZ2PUj9hpadE3cDHZnIdEwhEBrbAstXeMaFoB9tg&m=Mm7Xeak_hHY5PXahP67YuQ4kfo8h5NTbhp8HqALfIkw&s=FbgLLn6BD77AJ847QNYpPclP01MMXR2nmmC6bcyMEBc&e=" target="_blank">http://reviews.llvm.org/D10064</a><br>
><br>
> Files:<br>
> include/llvm/CodeGen/MachineBasicBlock.h<br>
> include/llvm/CodeGen/MachineIRUtils.h<br>
> lib/CodeGen/CMakeLists.txt<br>
> lib/CodeGen/MachineBasicBlock.cpp<br>
> lib/CodeGen/MachineIRUtils.cpp<br>
> lib/CodeGen/MachineLICM.cpp<br>
> lib/CodeGen/MachineSink.cpp<br>
> lib/CodeGen/PHIElimination.cpp<br>
><br>
> EMAIL PREFERENCES<br>
> <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__reviews.llvm.org_settings_panel_emailpreferences_&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=mQ4LZ2PUj9hpadE3cDHZnIdEwhEBrbAstXeMaFoB9tg&m=Mm7Xeak_hHY5PXahP67YuQ4kfo8h5NTbhp8HqALfIkw&s=Qy01AZZ5shA7q2WBaO5_fk2D5LYPaaqxycXdDfJWcLo&e=" target="_blank">http://reviews.llvm.org/settings/panel/emailpreferences/</a><br>
</div></div>> <D10064.26608.patch><br>
<br>
> Index: include/llvm/CodeGen/MachineBasicBlock.h<br>
> ===================================================================<br>
> --- include/llvm/CodeGen/MachineBasicBlock.h<br>
> +++ include/llvm/CodeGen/MachineBasicBlock.h<br>
> @@ -21,7 +21,6 @@<br>
><br>
> namespace llvm {<br>
><br>
> -class Pass;<br>
> class BasicBlock;<br>
> class MachineFunction;<br>
> class MCSymbol;<br>
> @@ -472,14 +471,6 @@<br>
> iterator getLastNonDebugInstr();<br>
> const_iterator getLastNonDebugInstr() const;<br>
><br>
> - /// SplitCriticalEdge - Split the critical edge from this block to the<br>
> - /// given successor block, and return the newly created block, or null<br>
> - /// if splitting is not possible.<br>
> - ///<br>
> - /// This function updates LiveVariables, MachineDominatorTree, and<br>
> - /// MachineLoopInfo, as applicable.<br>
> - MachineBasicBlock *SplitCriticalEdge(MachineBasicBlock *Succ, Pass *P);<br>
> -<br>
> void pop_front() { Insts.pop_front(); }<br>
> void pop_back() { Insts.pop_back(); }<br>
> void push_back(MachineInstr *MI) { Insts.push_back(MI); }<br>
> Index: include/llvm/CodeGen/MachineIRUtils.h<br>
> ===================================================================<br>
> --- /dev/null<br>
> +++ include/llvm/CodeGen/MachineIRUtils.h<br>
<br>
This name is pretty generic. Is there a list of things you're planning<br>
to add here? If not, `SplitCriticalEdge.h` seems like a better name.<br>
If so, it'd be good to get a high-level view of all the code motion<br>
you're going to do up front.<br></blockquote><div><br></div><div>You're right about this. I wasn't sure about what will have to be moved</div><div>and decoupled when I was working on this patch, but now I have a </div><div>good idea of those things, so I think the generic name isn't required here.</div><div><br></div><div>I renamed the file in the updated patch.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
> @@ -0,0 +1,38 @@<br>
> +//===-- llvm/CodeGen/MachineIRUtils.h ---------------------------*- C++ -*-===//<br>
> +//<br>
> +// The LLVM Compiler Infrastructure<br>
> +//<br>
> +// This file is distributed under the University of Illinois Open Source<br>
> +// License. See LICENSE.TXT for details.<br>
> +//<br>
> +//===----------------------------------------------------------------------===//<br>
> +//<br>
> +// This file contains the declarations of functions that implement machine IR<br>
> +// utility algorithms.<br>
> +//<br>
> +//===----------------------------------------------------------------------===//<br>
> +<br>
> +#ifndef LLVM_CODEGEN_MACHINEIRUTILS_H<br>
> +#define LLVM_CODEGEN_MACHINEIRUTILS_H<br>
> +<br>
> +namespace llvm {<br>
> +<br>
> +class Pass;<br>
> +class MachineBasicBlock;<br>
> +<br>
> +namespace MachineIRUtils {<br>
<br>
Do we need the extra namespace?<br>
<br>
> +<br>
> +/// SplitCriticalEdge - Split the critical edge from this block to the<br>
<br>
Since you're moving this anyway, please fix up the old-style comment<br>
(stop repeating the function name).<br>
<br>
Also, the comment talks about "this" block, but should probably refer<br>
to "BB".<br>
<br>
> +/// given successor block, and return the newly created block, or null<br>
> +/// if splitting is not possible.<br>
> +///<br>
> +/// This function updates LiveVariables, MachineDominatorTree, and<br>
> +/// MachineLoopInfo, as applicable.<br>
> +MachineBasicBlock *SplitCriticalEdge(MachineBasicBlock *BB,<br>
> + MachineBasicBlock *Succ, Pass *P);<br>
<br>
Since you're moving this and updating all the callers anyway, please<br>
take the opportunity to fix the function name. It should be<br>
`splitCriticalEdge()`.<br>
<br>
> +<br>
> +} // End namespace MachineIRUtils<br>
> +<br>
> +} // End namespace llvm<br>
> +<br>
> +#endif<br>
> Index: lib/CodeGen/CMakeLists.txt<br>
> ===================================================================<br>
> --- lib/CodeGen/CMakeLists.txt<br>
> +++ lib/CodeGen/CMakeLists.txt<br>
> @@ -58,6 +58,7 @@<br>
> MachineFunctionPrinterPass.cpp<br>
> MachineInstr.cpp<br>
> MachineInstrBundle.cpp<br>
> + MachineIRUtils.cpp<br>
> MachineLICM.cpp<br>
> MachineLoopInfo.cpp<br>
> MachineModuleInfo.cpp<br>
> Index: lib/CodeGen/MachineBasicBlock.cpp<br>
> ===================================================================<br>
> --- lib/CodeGen/MachineBasicBlock.cpp<br>
> +++ lib/CodeGen/MachineBasicBlock.cpp<br>
> @@ -14,12 +14,8 @@<br>
> #include "llvm/CodeGen/MachineBasicBlock.h"<br>
> #include "llvm/ADT/SmallPtrSet.h"<br>
> #include "llvm/ADT/SmallString.h"<br>
> -#include "llvm/CodeGen/LiveIntervalAnalysis.h"<br>
> -#include "llvm/CodeGen/LiveVariables.h"<br>
> -#include "llvm/CodeGen/MachineDominators.h"<br>
> #include "llvm/CodeGen/MachineFunction.h"<br>
> #include "llvm/CodeGen/MachineInstrBuilder.h"<br>
> -#include "llvm/CodeGen/MachineLoopInfo.h"<br>
> #include "llvm/CodeGen/MachineRegisterInfo.h"<br>
> #include "llvm/CodeGen/SlotIndexes.h"<br>
> #include "llvm/IR/BasicBlock.h"<br>
> @@ -664,270 +660,6 @@<br>
> return FBB == nullptr;<br>
> }<br>
><br>
> -MachineBasicBlock *<br>
> -MachineBasicBlock::SplitCriticalEdge(MachineBasicBlock *Succ, Pass *P) {<br>
> - // Splitting the critical edge to a landing pad block is non-trivial. Don't do<br>
> - // it in this generic function.<br>
> - if (Succ->isLandingPad())<br>
> - return nullptr;<br>
> -<br>
> - MachineFunction *MF = getParent();<br>
> - DebugLoc dl; // FIXME: this is nowhere<br>
> -<br>
> - // Performance might be harmed on HW that implements branching using exec mask<br>
> - // where both sides of the branches are always executed.<br>
> - if (MF->getTarget().requiresStructuredCFG())<br>
> - return nullptr;<br>
> -<br>
> - // We may need to update this's terminator, but we can't do that if<br>
> - // AnalyzeBranch fails. If this uses a jump table, we won't touch it.<br>
> - const TargetInstrInfo *TII = MF->getSubtarget().getInstrInfo();<br>
> - MachineBasicBlock *TBB = nullptr, *FBB = nullptr;<br>
> - SmallVector<MachineOperand, 4> Cond;<br>
> - if (TII->AnalyzeBranch(*this, TBB, FBB, Cond))<br>
> - return nullptr;<br>
> -<br>
> - // Avoid bugpoint weirdness: A block may end with a conditional branch but<br>
> - // jumps to the same MBB is either case. We have duplicate CFG edges in that<br>
> - // case that we can't handle. Since this never happens in properly optimized<br>
> - // code, just skip those edges.<br>
> - if (TBB && TBB == FBB) {<br>
> - DEBUG(dbgs() << "Won't split critical edge after degenerate BB#"<br>
> - << getNumber() << '\n');<br>
> - return nullptr;<br>
> - }<br>
> -<br>
> - MachineBasicBlock *NMBB = MF->CreateMachineBasicBlock();<br>
> - MF->insert(std::next(MachineFunction::iterator(this)), NMBB);<br>
> - DEBUG(dbgs() << "Splitting critical edge:"<br>
> - " BB#" << getNumber()<br>
> - << " -- BB#" << NMBB->getNumber()<br>
> - << " -- BB#" << Succ->getNumber() << '\n');<br>
> -<br>
> - LiveIntervals *LIS = P->getAnalysisIfAvailable<LiveIntervals>();<br>
> - SlotIndexes *Indexes = P->getAnalysisIfAvailable<SlotIndexes>();<br>
> - if (LIS)<br>
> - LIS->insertMBBInMaps(NMBB);<br>
> - else if (Indexes)<br>
> - Indexes->insertMBBInMaps(NMBB);<br>
> -<br>
> - // On some targets like Mips, branches may kill virtual registers. Make sure<br>
> - // that LiveVariables is properly updated after updateTerminator replaces the<br>
> - // terminators.<br>
> - LiveVariables *LV = P->getAnalysisIfAvailable<LiveVariables>();<br>
> -<br>
> - // Collect a list of virtual registers killed by the terminators.<br>
> - SmallVector<unsigned, 4> KilledRegs;<br>
> - if (LV)<br>
> - for (instr_iterator I = getFirstInstrTerminator(), E = instr_end();<br>
> - I != E; ++I) {<br>
> - MachineInstr *MI = I;<br>
> - for (MachineInstr::mop_iterator OI = MI->operands_begin(),<br>
> - OE = MI->operands_end(); OI != OE; ++OI) {<br>
> - if (!OI->isReg() || OI->getReg() == 0 ||<br>
> - !OI->isUse() || !OI->isKill() || OI->isUndef())<br>
> - continue;<br>
> - unsigned Reg = OI->getReg();<br>
> - if (TargetRegisterInfo::isPhysicalRegister(Reg) ||<br>
> - LV->getVarInfo(Reg).removeKill(MI)) {<br>
> - KilledRegs.push_back(Reg);<br>
> - DEBUG(dbgs() << "Removing terminator kill: " << *MI);<br>
> - OI->setIsKill(false);<br>
> - }<br>
> - }<br>
> - }<br>
> -<br>
> - SmallVector<unsigned, 4> UsedRegs;<br>
> - if (LIS) {<br>
> - for (instr_iterator I = getFirstInstrTerminator(), E = instr_end();<br>
> - I != E; ++I) {<br>
> - MachineInstr *MI = I;<br>
> -<br>
> - for (MachineInstr::mop_iterator OI = MI->operands_begin(),<br>
> - OE = MI->operands_end(); OI != OE; ++OI) {<br>
> - if (!OI->isReg() || OI->getReg() == 0)<br>
> - continue;<br>
> -<br>
> - unsigned Reg = OI->getReg();<br>
> - if (std::find(UsedRegs.begin(), UsedRegs.end(), Reg) == UsedRegs.end())<br>
> - UsedRegs.push_back(Reg);<br>
> - }<br>
> - }<br>
> - }<br>
> -<br>
> - ReplaceUsesOfBlockWith(Succ, NMBB);<br>
> -<br>
> - // If updateTerminator() removes instructions, we need to remove them from<br>
> - // SlotIndexes.<br>
> - SmallVector<MachineInstr*, 4> Terminators;<br>
> - if (Indexes) {<br>
> - for (instr_iterator I = getFirstInstrTerminator(), E = instr_end();<br>
> - I != E; ++I)<br>
> - Terminators.push_back(I);<br>
> - }<br>
> -<br>
> - updateTerminator();<br>
> -<br>
> - if (Indexes) {<br>
> - SmallVector<MachineInstr*, 4> NewTerminators;<br>
> - for (instr_iterator I = getFirstInstrTerminator(), E = instr_end();<br>
> - I != E; ++I)<br>
> - NewTerminators.push_back(I);<br>
> -<br>
> - for (SmallVectorImpl<MachineInstr*>::iterator I = Terminators.begin(),<br>
> - E = Terminators.end(); I != E; ++I) {<br>
> - if (std::find(NewTerminators.begin(), NewTerminators.end(), *I) ==<br>
> - NewTerminators.end())<br>
> - Indexes->removeMachineInstrFromMaps(*I);<br>
> - }<br>
> - }<br>
> -<br>
> - // Insert unconditional "jump Succ" instruction in NMBB if necessary.<br>
> - NMBB->addSuccessor(Succ);<br>
> - if (!NMBB->isLayoutSuccessor(Succ)) {<br>
> - Cond.clear();<br>
> - MF->getSubtarget().getInstrInfo()->InsertBranch(*NMBB, Succ, nullptr, Cond,<br>
> - dl);<br>
> -<br>
> - if (Indexes) {<br>
> - for (instr_iterator I = NMBB->instr_begin(), E = NMBB->instr_end();<br>
> - I != E; ++I) {<br>
> - // Some instructions may have been moved to NMBB by updateTerminator(),<br>
> - // so we first remove any instruction that already has an index.<br>
> - if (Indexes->hasIndex(I))<br>
> - Indexes->removeMachineInstrFromMaps(I);<br>
> - Indexes->insertMachineInstrInMaps(I);<br>
> - }<br>
> - }<br>
> - }<br>
> -<br>
> - // Fix PHI nodes in Succ so they refer to NMBB instead of this<br>
> - for (MachineBasicBlock::instr_iterator<br>
> - i = Succ->instr_begin(),e = Succ->instr_end();<br>
> - i != e && i->isPHI(); ++i)<br>
> - for (unsigned ni = 1, ne = i->getNumOperands(); ni != ne; ni += 2)<br>
> - if (i->getOperand(ni+1).getMBB() == this)<br>
> - i->getOperand(ni+1).setMBB(NMBB);<br>
> -<br>
> - // Inherit live-ins from the successor<br>
> - for (MachineBasicBlock::livein_iterator I = Succ->livein_begin(),<br>
> - E = Succ->livein_end(); I != E; ++I)<br>
> - NMBB->addLiveIn(*I);<br>
> -<br>
> - // Update LiveVariables.<br>
> - const TargetRegisterInfo *TRI = MF->getSubtarget().getRegisterInfo();<br>
> - if (LV) {<br>
> - // Restore kills of virtual registers that were killed by the terminators.<br>
> - while (!KilledRegs.empty()) {<br>
> - unsigned Reg = KilledRegs.pop_back_val();<br>
> - for (instr_iterator I = instr_end(), E = instr_begin(); I != E;) {<br>
> - if (!(--I)->addRegisterKilled(Reg, TRI, /* addIfNotFound= */ false))<br>
> - continue;<br>
> - if (TargetRegisterInfo::isVirtualRegister(Reg))<br>
> - LV->getVarInfo(Reg).Kills.push_back(I);<br>
> - DEBUG(dbgs() << "Restored terminator kill: " << *I);<br>
> - break;<br>
> - }<br>
> - }<br>
> - // Update relevant live-through information.<br>
> - LV->addNewBlock(NMBB, this, Succ);<br>
> - }<br>
> -<br>
> - if (LIS) {<br>
> - // After splitting the edge and updating SlotIndexes, live intervals may be<br>
> - // in one of two situations, depending on whether this block was the last in<br>
> - // the function. If the original block was the last in the function, all live<br>
> - // intervals will end prior to the beginning of the new split block. If the<br>
> - // original block was not at the end of the function, all live intervals will<br>
> - // extend to the end of the new split block.<br>
> -<br>
> - bool isLastMBB =<br>
> - std::next(MachineFunction::iterator(NMBB)) == getParent()->end();<br>
> -<br>
> - SlotIndex StartIndex = Indexes->getMBBEndIdx(this);<br>
> - SlotIndex PrevIndex = StartIndex.getPrevSlot();<br>
> - SlotIndex EndIndex = Indexes->getMBBEndIdx(NMBB);<br>
> -<br>
> - // Find the registers used from NMBB in PHIs in Succ.<br>
> - SmallSet<unsigned, 8> PHISrcRegs;<br>
> - for (MachineBasicBlock::instr_iterator<br>
> - I = Succ->instr_begin(), E = Succ->instr_end();<br>
> - I != E && I->isPHI(); ++I) {<br>
> - for (unsigned ni = 1, ne = I->getNumOperands(); ni != ne; ni += 2) {<br>
> - if (I->getOperand(ni+1).getMBB() == NMBB) {<br>
> - MachineOperand &MO = I->getOperand(ni);<br>
> - unsigned Reg = MO.getReg();<br>
> - PHISrcRegs.insert(Reg);<br>
> - if (MO.isUndef())<br>
> - continue;<br>
> -<br>
> - LiveInterval &LI = LIS->getInterval(Reg);<br>
> - VNInfo *VNI = LI.getVNInfoAt(PrevIndex);<br>
> - assert(VNI && "PHI sources should be live out of their predecessors.");<br>
> - LI.addSegment(LiveInterval::Segment(StartIndex, EndIndex, VNI));<br>
> - }<br>
> - }<br>
> - }<br>
> -<br>
> - MachineRegisterInfo *MRI = &getParent()->getRegInfo();<br>
> - for (unsigned i = 0, e = MRI->getNumVirtRegs(); i != e; ++i) {<br>
> - unsigned Reg = TargetRegisterInfo::index2VirtReg(i);<br>
> - if (PHISrcRegs.count(Reg) || !LIS->hasInterval(Reg))<br>
> - continue;<br>
> -<br>
> - LiveInterval &LI = LIS->getInterval(Reg);<br>
> - if (!LI.liveAt(PrevIndex))<br>
> - continue;<br>
> -<br>
> - bool isLiveOut = LI.liveAt(LIS->getMBBStartIdx(Succ));<br>
> - if (isLiveOut && isLastMBB) {<br>
> - VNInfo *VNI = LI.getVNInfoAt(PrevIndex);<br>
> - assert(VNI && "LiveInterval should have VNInfo where it is live.");<br>
> - LI.addSegment(LiveInterval::Segment(StartIndex, EndIndex, VNI));<br>
> - } else if (!isLiveOut && !isLastMBB) {<br>
> - LI.removeSegment(StartIndex, EndIndex);<br>
> - }<br>
> - }<br>
> -<br>
> - // Update all intervals for registers whose uses may have been modified by<br>
> - // updateTerminator().<br>
> - LIS->repairIntervalsInRange(this, getFirstTerminator(), end(), UsedRegs);<br>
> - }<br>
> -<br>
> - if (MachineDominatorTree *MDT =<br>
> - P->getAnalysisIfAvailable<MachineDominatorTree>())<br>
> - MDT->recordSplitCriticalEdge(this, Succ, NMBB);<br>
> -<br>
> - if (MachineLoopInfo *MLI = P->getAnalysisIfAvailable<MachineLoopInfo>())<br>
> - if (MachineLoop *TIL = MLI->getLoopFor(this)) {<br>
> - // If one or the other blocks were not in a loop, the new block is not<br>
> - // either, and thus LI doesn't need to be updated.<br>
> - if (MachineLoop *DestLoop = MLI->getLoopFor(Succ)) {<br>
> - if (TIL == DestLoop) {<br>
> - // Both in the same loop, the NMBB joins loop.<br>
> - DestLoop->addBasicBlockToLoop(NMBB, MLI->getBase());<br>
> - } else if (TIL->contains(DestLoop)) {<br>
> - // Edge from an outer loop to an inner loop. Add to the outer loop.<br>
> - TIL->addBasicBlockToLoop(NMBB, MLI->getBase());<br>
> - } else if (DestLoop->contains(TIL)) {<br>
> - // Edge from an inner loop to an outer loop. Add to the outer loop.<br>
> - DestLoop->addBasicBlockToLoop(NMBB, MLI->getBase());<br>
> - } else {<br>
> - // Edge from two loops with no containment relation. Because these<br>
> - // are natural loops, we know that the destination block must be the<br>
> - // header of its loop (adding a branch into a loop elsewhere would<br>
> - // create an irreducible loop).<br>
> - assert(DestLoop->getHeader() == Succ &&<br>
> - "Should not create irreducible loops!");<br>
> - if (MachineLoop *P = DestLoop->getParentLoop())<br>
> - P->addBasicBlockToLoop(NMBB, MLI->getBase());<br>
> - }<br>
> - }<br>
> - }<br>
> -<br>
> - return NMBB;<br>
> -}<br>
> -<br>
> /// Prepare MI to be removed from its bundle. This fixes bundle flags on MI's<br>
> /// neighboring instructions so the bundle won't be broken by removing MI.<br>
> static void unbundleSingleMI(MachineInstr *MI) {<br>
> Index: lib/CodeGen/MachineIRUtils.cpp<br>
> ===================================================================<br>
> --- /dev/null<br>
> +++ lib/CodeGen/MachineIRUtils.cpp<br>
> @@ -0,0 +1,308 @@<br>
> +//===-- llvm/CodeGen/MachineIRUtils.cpp -------------------------*- C++ -*-===//<br>
> +//<br>
> +// The LLVM Compiler Infrastructure<br>
> +//<br>
> +// This file is distributed under the University of Illinois Open Source<br>
> +// License. See LICENSE.TXT for details.<br>
> +//<br>
> +//===----------------------------------------------------------------------===//<br>
> +//<br>
> +// This file implements the implement machine IR utility algorithms.<br>
> +//<br>
> +//===----------------------------------------------------------------------===//<br>
> +<br>
> +#include "llvm/CodeGen/MachineIRUtils.h"<br>
> +#include "llvm/CodeGen/LiveIntervalAnalysis.h"<br>
> +#include "llvm/CodeGen/LiveVariables.h"<br>
> +#include "llvm/CodeGen/MachineDominators.h"<br>
> +#include "llvm/CodeGen/MachineFunction.h"<br>
> +#include "llvm/CodeGen/MachineLoopInfo.h"<br>
> +#include "llvm/CodeGen/MachineRegisterInfo.h"<br>
> +#include "llvm/CodeGen/SlotIndexes.h"<br>
> +#include "llvm/Support/Debug.h"<br>
> +#include "llvm/Support/raw_ostream.h"<br>
> +#include "llvm/Target/TargetInstrInfo.h"<br>
> +#include "llvm/Target/TargetMachine.h"<br>
> +#include "llvm/Target/TargetRegisterInfo.h"<br>
> +#include <algorithm><br>
> +using namespace llvm;<br>
> +<br>
> +#define DEBUG_TYPE "codegen"<br>
> +<br>
> +MachineBasicBlock *MachineIRUtils::SplitCriticalEdge(MachineBasicBlock *BB,<br>
> + MachineBasicBlock *Succ,<br>
> + Pass *P) {<br>
> + // Splitting the critical edge to a landing pad block is non-trivial. Don't do<br>
> + // it in this generic function.<br>
> + if (Succ->isLandingPad())<br>
> + return nullptr;<br>
> +<br>
> + MachineFunction *MF = BB->getParent();<br>
> + DebugLoc dl; // FIXME: this is nowhere<br>
> +<br>
> + // Performance might be harmed on HW that implements branching using exec mask<br>
> + // where both sides of the branches are always executed.<br>
> + if (MF->getTarget().requiresStructuredCFG())<br>
> + return nullptr;<br>
> +<br>
> + // We may need to update this's terminator, but we can't do that if<br>
> + // AnalyzeBranch fails. If this uses a jump table, we won't touch it.<br>
> + const TargetInstrInfo *TII = MF->getSubtarget().getInstrInfo();<br>
> + MachineBasicBlock *TBB = nullptr, *FBB = nullptr;<br>
> + SmallVector<MachineOperand, 4> Cond;<br>
> + if (TII->AnalyzeBranch(*BB, TBB, FBB, Cond))<br>
> + return nullptr;<br>
> +<br>
> + // Avoid bugpoint weirdness: A block may end with a conditional branch but<br>
> + // jumps to the same MBB is either case. We have duplicate CFG edges in that<br>
> + // case that we can't handle. Since this never happens in properly optimized<br>
> + // code, just skip those edges.<br>
> + if (TBB && TBB == FBB) {<br>
> + DEBUG(dbgs() << "Won't split critical edge after degenerate BB#"<br>
> + << BB->getNumber() << '\n');<br>
> + return nullptr;<br>
> + }<br>
> +<br>
> + MachineBasicBlock *NMBB = MF->CreateMachineBasicBlock();<br>
> + MF->insert(std::next(MachineFunction::iterator(BB)), NMBB);<br>
> + DEBUG(dbgs() << "Splitting critical edge:"<br>
> + " BB#" << BB->getNumber() << " -- BB#" << NMBB->getNumber()<br>
> + << " -- BB#" << Succ->getNumber() << '\n');<br>
> +<br>
> + LiveIntervals *LIS = P->getAnalysisIfAvailable<LiveIntervals>();<br>
> + SlotIndexes *Indexes = P->getAnalysisIfAvailable<SlotIndexes>();<br>
> + if (LIS)<br>
> + LIS->insertMBBInMaps(NMBB);<br>
> + else if (Indexes)<br>
> + Indexes->insertMBBInMaps(NMBB);<br>
> +<br>
> + // On some targets like Mips, branches may kill virtual registers. Make sure<br>
> + // that LiveVariables is properly updated after updateTerminator replaces the<br>
> + // terminators.<br>
> + LiveVariables *LV = P->getAnalysisIfAvailable<LiveVariables>();<br>
> +<br>
> + // Collect a list of virtual registers killed by the terminators.<br>
> + SmallVector<unsigned, 4> KilledRegs;<br>
> + if (LV)<br>
> + for (MachineBasicBlock::instr_iterator I = BB->getFirstInstrTerminator(),<br>
> + E = BB->instr_end();<br>
> + I != E; ++I) {<br>
> + MachineInstr *MI = I;<br>
> + for (MachineInstr::mop_iterator OI = MI->operands_begin(),<br>
> + OE = MI->operands_end();<br>
> + OI != OE; ++OI) {<br>
> + if (!OI->isReg() || OI->getReg() == 0 || !OI->isUse() ||<br>
> + !OI->isKill() || OI->isUndef())<br>
> + continue;<br>
> + unsigned Reg = OI->getReg();<br>
> + if (TargetRegisterInfo::isPhysicalRegister(Reg) ||<br>
> + LV->getVarInfo(Reg).removeKill(MI)) {<br>
> + KilledRegs.push_back(Reg);<br>
> + DEBUG(dbgs() << "Removing terminator kill: " << *MI);<br>
> + OI->setIsKill(false);<br>
> + }<br>
> + }<br>
> + }<br>
> +<br>
> + SmallVector<unsigned, 4> UsedRegs;<br>
> + if (LIS) {<br>
> + for (MachineBasicBlock::instr_iterator I = BB->getFirstInstrTerminator(),<br>
> + E = BB->instr_end();<br>
> + I != E; ++I) {<br>
> + MachineInstr *MI = I;<br>
> +<br>
> + for (MachineInstr::mop_iterator OI = MI->operands_begin(),<br>
> + OE = MI->operands_end();<br>
> + OI != OE; ++OI) {<br>
> + if (!OI->isReg() || OI->getReg() == 0)<br>
> + continue;<br>
> +<br>
> + unsigned Reg = OI->getReg();<br>
> + if (std::find(UsedRegs.begin(), UsedRegs.end(), Reg) == UsedRegs.end())<br>
> + UsedRegs.push_back(Reg);<br>
> + }<br>
> + }<br>
> + }<br>
> +<br>
> + BB->ReplaceUsesOfBlockWith(Succ, NMBB);<br>
> +<br>
> + // If updateTerminator() removes instructions, we need to remove them from<br>
> + // SlotIndexes.<br>
> + SmallVector<MachineInstr *, 4> Terminators;<br>
> + if (Indexes) {<br>
> + for (MachineBasicBlock::instr_iterator I = BB->getFirstInstrTerminator(),<br>
> + E = BB->instr_end();<br>
> + I != E; ++I)<br>
> + Terminators.push_back(I);<br>
> + }<br>
> +<br>
> + BB->updateTerminator();<br>
> +<br>
> + if (Indexes) {<br>
> + SmallVector<MachineInstr *, 4> NewTerminators;<br>
> + for (MachineBasicBlock::instr_iterator I = BB->getFirstInstrTerminator(),<br>
> + E = BB->instr_end();<br>
> + I != E; ++I)<br>
> + NewTerminators.push_back(I);<br>
> +<br>
> + for (SmallVectorImpl<MachineInstr *>::iterator I = Terminators.begin(),<br>
> + E = Terminators.end();<br>
> + I != E; ++I) {<br>
> + if (std::find(NewTerminators.begin(), NewTerminators.end(), *I) ==<br>
> + NewTerminators.end())<br>
> + Indexes->removeMachineInstrFromMaps(*I);<br>
> + }<br>
> + }<br>
> +<br>
> + // Insert unconditional "jump Succ" instruction in NMBB if necessary.<br>
> + NMBB->addSuccessor(Succ);<br>
> + if (!NMBB->isLayoutSuccessor(Succ)) {<br>
> + Cond.clear();<br>
> + MF->getSubtarget().getInstrInfo()->InsertBranch(*NMBB, Succ, nullptr, Cond,<br>
> + dl);<br>
> +<br>
> + if (Indexes) {<br>
> + for (MachineBasicBlock::instr_iterator I = NMBB->instr_begin(),<br>
> + E = NMBB->instr_end();<br>
> + I != E; ++I) {<br>
> + // Some instructions may have been moved to NMBB by updateTerminator(),<br>
> + // so we first remove any instruction that already has an index.<br>
> + if (Indexes->hasIndex(I))<br>
> + Indexes->removeMachineInstrFromMaps(I);<br>
> + Indexes->insertMachineInstrInMaps(I);<br>
> + }<br>
> + }<br>
> + }<br>
> +<br>
> + // Fix PHI nodes in Succ so they refer to NMBB instead of this<br>
> + for (MachineBasicBlock::instr_iterator i = Succ->instr_begin(),<br>
> + e = Succ->instr_end();<br>
> + i != e && i->isPHI(); ++i)<br>
> + for (unsigned ni = 1, ne = i->getNumOperands(); ni != ne; ni += 2)<br>
> + if (i->getOperand(ni + 1).getMBB() == BB)<br>
> + i->getOperand(ni + 1).setMBB(NMBB);<br>
> +<br>
> + // Inherit live-ins from the successor<br>
> + for (MachineBasicBlock::livein_iterator I = Succ->livein_begin(),<br>
> + E = Succ->livein_end();<br>
> + I != E; ++I)<br>
> + NMBB->addLiveIn(*I);<br>
> +<br>
> + // Update LiveVariables.<br>
> + const TargetRegisterInfo *TRI = MF->getSubtarget().getRegisterInfo();<br>
> + if (LV) {<br>
> + // Restore kills of virtual registers that were killed by the terminators.<br>
> + while (!KilledRegs.empty()) {<br>
> + unsigned Reg = KilledRegs.pop_back_val();<br>
> + for (MachineBasicBlock::instr_iterator I = BB->instr_end(),<br>
> + E = BB->instr_begin();<br>
> + I != E;) {<br>
> + if (!(--I)->addRegisterKilled(Reg, TRI, /* addIfNotFound= */ false))<br>
> + continue;<br>
> + if (TargetRegisterInfo::isVirtualRegister(Reg))<br>
> + LV->getVarInfo(Reg).Kills.push_back(I);<br>
> + DEBUG(dbgs() << "Restored terminator kill: " << *I);<br>
> + break;<br>
> + }<br>
> + }<br>
> + // Update relevant live-through information.<br>
> + LV->addNewBlock(NMBB, BB, Succ);<br>
> + }<br>
> +<br>
> + if (LIS) {<br>
> + // After splitting the edge and updating SlotIndexes, live intervals may be<br>
> + // in one of two situations, depending on whether this block was the last in<br>
> + // the function. If the original block was the last in the function, all<br>
> + // live<br>
> + // intervals will end prior to the beginning of the new split block. If the<br>
> + // original block was not at the end of the function, all live intervals<br>
> + // will<br>
> + // extend to the end of the new split block.<br>
> +<br>
> + bool isLastMBB = std::next(MachineFunction::iterator(NMBB)) == MF->end();<br>
> +<br>
> + SlotIndex StartIndex = Indexes->getMBBEndIdx(BB);<br>
> + SlotIndex PrevIndex = StartIndex.getPrevSlot();<br>
> + SlotIndex EndIndex = Indexes->getMBBEndIdx(NMBB);<br>
> +<br>
> + // Find the registers used from NMBB in PHIs in Succ.<br>
> + SmallSet<unsigned, 8> PHISrcRegs;<br>
> + for (MachineBasicBlock::instr_iterator I = Succ->instr_begin(),<br>
> + E = Succ->instr_end();<br>
> + I != E && I->isPHI(); ++I) {<br>
> + for (unsigned ni = 1, ne = I->getNumOperands(); ni != ne; ni += 2) {<br>
> + if (I->getOperand(ni + 1).getMBB() == NMBB) {<br>
> + MachineOperand &MO = I->getOperand(ni);<br>
> + unsigned Reg = MO.getReg();<br>
> + PHISrcRegs.insert(Reg);<br>
> + if (MO.isUndef())<br>
> + continue;<br>
> +<br>
> + LiveInterval &LI = LIS->getInterval(Reg);<br>
> + VNInfo *VNI = LI.getVNInfoAt(PrevIndex);<br>
> + assert(VNI &&<br>
> + "PHI sources should be live out of their predecessors.");<br>
> + LI.addSegment(LiveInterval::Segment(StartIndex, EndIndex, VNI));<br>
> + }<br>
> + }<br>
> + }<br>
> +<br>
> + MachineRegisterInfo *MRI = &MF->getRegInfo();<br>
> + for (unsigned i = 0, e = MRI->getNumVirtRegs(); i != e; ++i) {<br>
> + unsigned Reg = TargetRegisterInfo::index2VirtReg(i);<br>
> + if (PHISrcRegs.count(Reg) || !LIS->hasInterval(Reg))<br>
> + continue;<br>
> +<br>
> + LiveInterval &LI = LIS->getInterval(Reg);<br>
> + if (!LI.liveAt(PrevIndex))<br>
> + continue;<br>
> +<br>
> + bool isLiveOut = LI.liveAt(LIS->getMBBStartIdx(Succ));<br>
> + if (isLiveOut && isLastMBB) {<br>
> + VNInfo *VNI = LI.getVNInfoAt(PrevIndex);<br>
> + assert(VNI && "LiveInterval should have VNInfo where it is live.");<br>
> + LI.addSegment(LiveInterval::Segment(StartIndex, EndIndex, VNI));<br>
> + } else if (!isLiveOut && !isLastMBB) {<br>
> + LI.removeSegment(StartIndex, EndIndex);<br>
> + }<br>
> + }<br>
> +<br>
> + // Update all intervals for registers whose uses may have been modified by<br>
> + // updateTerminator().<br>
> + LIS->repairIntervalsInRange(BB, BB->getFirstTerminator(), BB->end(),<br>
> + UsedRegs);<br>
> + }<br>
> +<br>
> + if (MachineDominatorTree *MDT =<br>
> + P->getAnalysisIfAvailable<MachineDominatorTree>())<br>
> + MDT->recordSplitCriticalEdge(BB, Succ, NMBB);<br>
> +<br>
> + if (MachineLoopInfo *MLI = P->getAnalysisIfAvailable<MachineLoopInfo>())<br>
> + if (MachineLoop *TIL = MLI->getLoopFor(BB)) {<br>
> + // If one or the other blocks were not in a loop, the new block is not<br>
> + // either, and thus LI doesn't need to be updated.<br>
> + if (MachineLoop *DestLoop = MLI->getLoopFor(Succ)) {<br>
> + if (TIL == DestLoop) {<br>
> + // Both in the same loop, the NMBB joins loop.<br>
> + DestLoop->addBasicBlockToLoop(NMBB, MLI->getBase());<br>
> + } else if (TIL->contains(DestLoop)) {<br>
> + // Edge from an outer loop to an inner loop. Add to the outer loop.<br>
> + TIL->addBasicBlockToLoop(NMBB, MLI->getBase());<br>
> + } else if (DestLoop->contains(TIL)) {<br>
> + // Edge from an inner loop to an outer loop. Add to the outer loop.<br>
> + DestLoop->addBasicBlockToLoop(NMBB, MLI->getBase());<br>
> + } else {<br>
> + // Edge from two loops with no containment relation. Because these<br>
> + // are natural loops, we know that the destination block must be the<br>
> + // header of its loop (adding a branch into a loop elsewhere would<br>
> + // create an irreducible loop).<br>
> + assert(DestLoop->getHeader() == Succ &&<br>
> + "Should not create irreducible loops!");<br>
> + if (MachineLoop *P = DestLoop->getParentLoop())<br>
> + P->addBasicBlockToLoop(NMBB, MLI->getBase());<br>
> + }<br>
> + }<br>
> + }<br>
> +<br>
> + return NMBB;<br>
> +}<br>
> Index: lib/CodeGen/MachineLICM.cpp<br>
> ===================================================================<br>
> --- lib/CodeGen/MachineLICM.cpp<br>
> +++ lib/CodeGen/MachineLICM.cpp<br>
> @@ -23,6 +23,7 @@<br>
> #include "llvm/Analysis/AliasAnalysis.h"<br>
> #include "llvm/CodeGen/MachineDominators.h"<br>
> #include "llvm/CodeGen/MachineFrameInfo.h"<br>
> +#include "llvm/CodeGen/MachineIRUtils.h"<br>
> #include "llvm/CodeGen/MachineLoopInfo.h"<br>
> #include "llvm/CodeGen/MachineMemOperand.h"<br>
> #include "llvm/CodeGen/MachineRegisterInfo.h"<br>
> @@ -1459,7 +1460,8 @@<br>
> return nullptr;<br>
> }<br>
><br>
> - CurPreheader = Pred->SplitCriticalEdge(CurLoop->getHeader(), this);<br>
> + CurPreheader =<br>
> + MachineIRUtils::SplitCriticalEdge(Pred, CurLoop->getHeader(), this);<br>
> if (!CurPreheader) {<br>
> CurPreheader = reinterpret_cast<MachineBasicBlock *>(-1);<br>
> return nullptr;<br>
> Index: lib/CodeGen/MachineSink.cpp<br>
> ===================================================================<br>
> --- lib/CodeGen/MachineSink.cpp<br>
> +++ lib/CodeGen/MachineSink.cpp<br>
> @@ -24,6 +24,7 @@<br>
> #include "llvm/Analysis/AliasAnalysis.h"<br>
> #include "llvm/CodeGen/MachineBlockFrequencyInfo.h"<br>
> #include "llvm/CodeGen/MachineDominators.h"<br>
> +#include "llvm/CodeGen/MachineIRUtils.h"<br>
> #include "llvm/CodeGen/MachineLoopInfo.h"<br>
> #include "llvm/CodeGen/MachinePostDominators.h"<br>
> #include "llvm/CodeGen/MachineRegisterInfo.h"<br>
> @@ -275,7 +276,8 @@<br>
><br>
> // If we have anything we marked as toSplit, split it now.<br>
> for (auto &Pair : ToSplit) {<br>
> - auto NewSucc = Pair.first->SplitCriticalEdge(Pair.second, this);<br>
> + auto NewSucc =<br>
> + MachineIRUtils::SplitCriticalEdge(Pair.first, Pair.second, this);<br>
> if (NewSucc != nullptr) {<br>
> DEBUG(dbgs() << " *** Splitting critical edge:"<br>
> " BB#" << Pair.first->getNumber()<br>
> Index: lib/CodeGen/PHIElimination.cpp<br>
> ===================================================================<br>
> --- lib/CodeGen/PHIElimination.cpp<br>
> +++ lib/CodeGen/PHIElimination.cpp<br>
> @@ -23,6 +23,7 @@<br>
> #include "llvm/CodeGen/MachineDominators.h"<br>
> #include "llvm/CodeGen/MachineInstr.h"<br>
> #include "llvm/CodeGen/MachineInstrBuilder.h"<br>
> +#include "llvm/CodeGen/MachineIRUtils.h"<br>
> #include "llvm/CodeGen/MachineLoopInfo.h"<br>
> #include "llvm/CodeGen/MachineRegisterInfo.h"<br>
> #include "llvm/IR/Function.h"<br>
> @@ -612,7 +613,7 @@<br>
> }<br>
> if (!ShouldSplit && !SplitAllCriticalEdges)<br>
> continue;<br>
> - if (!PreMBB->SplitCriticalEdge(&MBB, this)) {<br>
> + if (!MachineIRUtils::SplitCriticalEdge(PreMBB, &MBB, this)) {<br>
> DEBUG(dbgs() << "Failed to split critical edge.\n");<br>
> continue;<br>
> }<br>
><br>
<br>
</blockquote></div><br></div></div>