Just FYI, as I forgot to mention this in the commit log, I am working on a set of test cases for this. I'm just trying to get this into the tree sooner so others can see it and hack on it if desired; its still experimental and behind flags.<br>
<br><div class="gmail_quote">On Thu, Oct 20, 2011 at 11:46 PM, Chandler Carruth <span dir="ltr"><<a href="mailto:chandlerc@gmail.com">chandlerc@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Author: chandlerc<br>
Date: Fri Oct 21 01:46:38 2011<br>
New Revision: 142641<br>
<br>
URL: <a href="http://llvm.org/viewvc/llvm-project?rev=142641&view=rev" target="_blank">http://llvm.org/viewvc/llvm-project?rev=142641&view=rev</a><br>
Log:<br>
Implement a block placement pass based on the branch probability and<br>
block frequency analyses. This differs substantially from the existing<br>
block-placement pass in LLVM:<br>
<br>
1) It operates on the Machine-IR in the CodeGen layer. This exposes much<br>
more (and more precise) information and opportunities. Also, the<br>
results are more stable due to fewer transforms ocurring after the<br>
pass runs.<br>
2) It uses the generalized probability and frequency analyses. These can<br>
model static heuristics, code annotation derived heuristics as well<br>
as eventual profile loading. By basing the optimization on the<br>
analysis interface it can work from any (or a combination) of these<br>
inputs.<br>
3) It uses a more aggressive algorithm, both building chains from tho<br>
bottom up to maximize benefit, and using an SCC-based walk to layout<br>
chains of blocks in a profitable ordering without O(N^2) iterations<br>
which the old pass involves.<br>
<br>
The pass is currently gated behind a flag, and not enabled by default<br>
because it still needs to grow some important features. Most notably, it<br>
needs to support loop aligning and careful layout of loop structures<br>
much as done by hand currently in CodePlacementOpt. Once it supports<br>
these, and has sufficient testing and quality tuning, it should replace<br>
both of these passes.<br>
<br>
Thanks to Nick Lewycky and Richard Smith for help authoring & debugging<br>
this, and to Jakob, Andy, Eric, Jim, and probably a few others I'm<br>
forgetting for reviewing and answering all my questions. Writing<br>
a backend pass is *sooo* much better now than it used to be. =D<br>
<br>
Added:<br>
llvm/trunk/lib/CodeGen/MachineBlockPlacement.cpp<br>
Modified:<br>
llvm/trunk/include/llvm/CodeGen/Passes.h<br>
llvm/trunk/include/llvm/InitializePasses.h<br>
llvm/trunk/lib/CodeGen/CMakeLists.txt<br>
llvm/trunk/lib/CodeGen/CodeGen.cpp<br>
llvm/trunk/lib/CodeGen/LLVMTargetMachine.cpp<br>
<br>
Modified: llvm/trunk/include/llvm/CodeGen/Passes.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/Passes.h?rev=142641&r1=142640&r2=142641&view=diff" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/Passes.h?rev=142641&r1=142640&r2=142641&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/include/llvm/CodeGen/Passes.h (original)<br>
+++ llvm/trunk/include/llvm/CodeGen/Passes.h Fri Oct 21 01:46:38 2011<br>
@@ -155,6 +155,10 @@<br>
/// IfConverter Pass - This pass performs machine code if conversion.<br>
FunctionPass *createIfConverterPass();<br>
<br>
+ /// MachineBlockPlacement Pass - This pass places basic blocks based on branch<br>
+ /// probabilities.<br>
+ FunctionPass *createMachineBlockPlacementPass();<br>
+<br>
/// Code Placement Pass - This pass optimize code placement and aligns loop<br>
/// headers to target specific alignment boundary.<br>
FunctionPass *createCodePlacementOptPass();<br>
<br>
Modified: llvm/trunk/include/llvm/InitializePasses.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/InitializePasses.h?rev=142641&r1=142640&r2=142641&view=diff" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/InitializePasses.h?rev=142641&r1=142640&r2=142641&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/include/llvm/InitializePasses.h (original)<br>
+++ llvm/trunk/include/llvm/InitializePasses.h Fri Oct 21 01:46:38 2011<br>
@@ -146,6 +146,7 @@<br>
void initializeLowerInvokePass(PassRegistry&);<br>
void initializeLowerSwitchPass(PassRegistry&);<br>
void initializeMachineBlockFrequencyInfoPass(PassRegistry&);<br>
+void initializeMachineBlockPlacementPass(PassRegistry&);<br>
void initializeMachineBranchProbabilityInfoPass(PassRegistry&);<br>
void initializeMachineCSEPass(PassRegistry&);<br>
void initializeMachineDominatorTreePass(PassRegistry&);<br>
<br>
Modified: llvm/trunk/lib/CodeGen/CMakeLists.txt<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/CMakeLists.txt?rev=142641&r1=142640&r2=142641&view=diff" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/CMakeLists.txt?rev=142641&r1=142640&r2=142641&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/lib/CodeGen/CMakeLists.txt (original)<br>
+++ llvm/trunk/lib/CodeGen/CMakeLists.txt Fri Oct 21 01:46:38 2011<br>
@@ -37,6 +37,7 @@<br>
LocalStackSlotAllocation.cpp<br>
MachineBasicBlock.cpp<br>
MachineBlockFrequencyInfo.cpp<br>
+ MachineBlockPlacement.cpp<br>
MachineBranchProbabilityInfo.cpp<br>
MachineCSE.cpp<br>
MachineDominators.cpp<br>
<br>
Modified: llvm/trunk/lib/CodeGen/CodeGen.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/CodeGen.cpp?rev=142641&r1=142640&r2=142641&view=diff" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/CodeGen.cpp?rev=142641&r1=142640&r2=142641&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/lib/CodeGen/CodeGen.cpp (original)<br>
+++ llvm/trunk/lib/CodeGen/CodeGen.cpp Fri Oct 21 01:46:38 2011<br>
@@ -28,6 +28,7 @@<br>
initializeLiveStacksPass(Registry);<br>
initializeLiveVariablesPass(Registry);<br>
initializeMachineBlockFrequencyInfoPass(Registry);<br>
+ initializeMachineBlockPlacementPass(Registry);<br>
initializeMachineCSEPass(Registry);<br>
initializeMachineDominatorTreePass(Registry);<br>
initializeMachineLICMPass(Registry);<br>
<br>
Modified: llvm/trunk/lib/CodeGen/LLVMTargetMachine.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/LLVMTargetMachine.cpp?rev=142641&r1=142640&r2=142641&view=diff" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/LLVMTargetMachine.cpp?rev=142641&r1=142640&r2=142641&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/lib/CodeGen/LLVMTargetMachine.cpp (original)<br>
+++ llvm/trunk/lib/CodeGen/LLVMTargetMachine.cpp Fri Oct 21 01:46:38 2011<br>
@@ -53,6 +53,8 @@<br>
cl::desc("Disable tail duplication"));<br>
static cl::opt<bool> DisableEarlyTailDup("disable-early-taildup", cl::Hidden,<br>
cl::desc("Disable pre-register allocation tail duplication"));<br>
+static cl::opt<bool> EnableBlockPlacement("enable-block-placement",<br>
+ cl::Hidden, cl::desc("Enable probability-driven block placement"));<br>
static cl::opt<bool> DisableCodePlace("disable-code-place", cl::Hidden,<br>
cl::desc("Disable code placement"));<br>
static cl::opt<bool> DisableSSC("disable-ssc", cl::Hidden,<br>
@@ -486,8 +488,16 @@<br>
PM.add(createGCInfoPrinter(dbgs()));<br>
<br>
if (OptLevel != CodeGenOpt::None && !DisableCodePlace) {<br>
- PM.add(createCodePlacementOptPass());<br>
- printNoVerify(PM, "After CodePlacementOpt");<br>
+ if (EnableBlockPlacement) {<br>
+ // MachineBlockPlacement is an experimental pass which is disabled by<br>
+ // default currently. Eventually it should subsume CodePlacementOpt, so<br>
+ // when enabled, the other is disabled.<br>
+ PM.add(createMachineBlockPlacementPass());<br>
+ printNoVerify(PM, "After MachineBlockPlacement");<br>
+ } else {<br>
+ PM.add(createCodePlacementOptPass());<br>
+ printNoVerify(PM, "After CodePlacementOpt");<br>
+ }<br>
}<br>
<br>
if (addPreEmitPass(PM, OptLevel))<br>
<br>
Added: llvm/trunk/lib/CodeGen/MachineBlockPlacement.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/MachineBlockPlacement.cpp?rev=142641&view=auto" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/MachineBlockPlacement.cpp?rev=142641&view=auto</a><br>
==============================================================================<br>
--- llvm/trunk/lib/CodeGen/MachineBlockPlacement.cpp (added)<br>
+++ llvm/trunk/lib/CodeGen/MachineBlockPlacement.cpp Fri Oct 21 01:46:38 2011<br>
@@ -0,0 +1,624 @@<br>
+//===-- MachineBlockPlacement.cpp - Basic Block Code Layout optimization --===//<br>
+//<br>
+// The LLVM Compiler Infrastructure<br>
+//<br>
+// This file is distributed under the University of Illinois Open Source<br>
+// License. See LICENSE.TXT for details.<br>
+//<br>
+//===----------------------------------------------------------------------===//<br>
+//<br>
+// This file implements basic block placement transformations using branch<br>
+// probability estimates. It is based around "Algo2" from Profile Guided Code<br>
+// Positioning [<a href="http://portal.acm.org/citation.cfm?id=989433" target="_blank">http://portal.acm.org/citation.cfm?id=989433</a>].<br>
+//<br>
+// We combine the BlockFrequencyInfo with BranchProbabilityInfo to simulate<br>
+// measured edge-weights. The BlockFrequencyInfo effectively summarizes the<br>
+// probability of starting from any particular block, and the<br>
+// BranchProbabilityInfo the probability of exiting the block via a particular<br>
+// edge. Combined they form a function-wide ordering of the edges.<br>
+//<br>
+//===----------------------------------------------------------------------===//<br>
+<br>
+#define DEBUG_TYPE "block-placement2"<br>
+#include "llvm/CodeGen/Passes.h"<br>
+#include "llvm/CodeGen/MachineModuleInfo.h"<br>
+#include "llvm/CodeGen/MachineBlockFrequencyInfo.h"<br>
+#include "llvm/CodeGen/MachineBranchProbabilityInfo.h"<br>
+#include "llvm/CodeGen/MachineFunction.h"<br>
+#include "llvm/CodeGen/MachineBasicBlock.h"<br>
+#include "llvm/CodeGen/MachineFunctionPass.h"<br>
+#include "llvm/Support/Allocator.h"<br>
+#include "llvm/Support/ErrorHandling.h"<br>
+#include "llvm/ADT/DenseMap.h"<br>
+#include "llvm/ADT/SCCIterator.h"<br>
+#include "llvm/ADT/SmallPtrSet.h"<br>
+#include "llvm/ADT/SmallVector.h"<br>
+#include "llvm/ADT/Statistic.h"<br>
+#include "llvm/Target/TargetInstrInfo.h"<br>
+#include <algorithm><br>
+using namespace llvm;<br>
+<br>
+namespace {<br>
+/// \brief A structure for storing a weighted edge.<br>
+///<br>
+/// This stores an edge and its weight, computed as the product of the<br>
+/// frequency that the starting block is entered with the probability of<br>
+/// a particular exit block.<br>
+struct WeightedEdge {<br>
+ BlockFrequency EdgeFrequency;<br>
+ MachineBasicBlock *From, *To;<br>
+<br>
+ bool operator<(const WeightedEdge &RHS) const {<br>
+ return EdgeFrequency < RHS.EdgeFrequency;<br>
+ }<br>
+};<br>
+}<br>
+<br>
+namespace {<br>
+struct BlockChain;<br>
+/// \brief Type for our function-wide basic block -> block chain mapping.<br>
+typedef DenseMap<MachineBasicBlock *, BlockChain *> BlockToChainMapType;<br>
+}<br>
+<br>
+namespace {<br>
+/// \brief A chain of blocks which will be laid out contiguously.<br>
+///<br>
+/// This is the datastructure representing a chain of consecutive blocks that<br>
+/// are profitable to layout together in order to maximize fallthrough<br>
+/// probabilities. We also can use a block chain to represent a sequence of<br>
+/// basic blocks which have some external (correctness) requirement for<br>
+/// sequential layout.<br>
+///<br>
+/// Eventually, the block chains will form a directed graph over the function.<br>
+/// We provide an SCC-supporting-iterator in order to quicky build and walk the<br>
+/// SCCs of block chains within a function.<br>
+///<br>
+/// The block chains also have support for calculating and caching probability<br>
+/// information related to the chain itself versus other chains. This is used<br>
+/// for ranking during the final layout of block chains.<br>
+struct BlockChain {<br>
+ class SuccIterator;<br>
+<br>
+ /// \brief The first and last basic block that from this chain.<br>
+ ///<br>
+ /// The chain is stored within the existing function ilist of basic blocks.<br>
+ /// When merging chains or otherwise manipulating them, we splice the blocks<br>
+ /// within this ilist, giving us very cheap storage here and constant time<br>
+ /// merge operations.<br>
+ ///<br>
+ /// It is extremely important to note that LastBB is the iterator pointing<br>
+ /// *at* the last basic block in the chain. That is, the chain consists of<br>
+ /// the *closed* range [FirstBB, LastBB]. We cannot use half-open ranges<br>
+ /// because the next basic block may get relocated to a different part of the<br>
+ /// function at any time during the run of this pass.<br>
+ MachineFunction::iterator FirstBB, LastBB;<br>
+<br>
+ /// \brief A handle to the function-wide basic block to block chain mapping.<br>
+ ///<br>
+ /// This is retained in each block chain to simplify the computation of child<br>
+ /// block chains for SCC-formation and iteration. We store the edges to child<br>
+ /// basic blocks, and map them back to their associated chains using this<br>
+ /// structure.<br>
+ BlockToChainMapType &BlockToChain;<br>
+<br>
+ /// \brief The weight used to rank two block chains in the same SCC.<br>
+ ///<br>
+ /// This is used during SCC layout of block chains to cache and rank the<br>
+ /// chains. It is supposed to represent the expected frequency with which<br>
+ /// control reaches a block within this chain, has the option of branching to<br>
+ /// a block in some other chain participating in the SCC, but instead<br>
+ /// continues within this chain. The higher this is, the more costly we<br>
+ /// expect mis-predicted branches between this chain and other chains within<br>
+ /// the SCC to be. Thus, since we expect branches between chains to be<br>
+ /// predicted when backwards and not predicted when forwards, the higher this<br>
+ /// is the more important that this chain is laid out first among those<br>
+ /// chains in the same SCC as it.<br>
+ BlockFrequency InChainEdgeFrequency;<br>
+<br>
+ /// \brief Construct a new BlockChain.<br>
+ ///<br>
+ /// This builds a new block chain representing a single basic block in the<br>
+ /// function. It also registers itself as the chain that block participates<br>
+ /// in with the BlockToChain mapping.<br>
+ BlockChain(BlockToChainMapType &BlockToChain, MachineBasicBlock *BB)<br>
+ : FirstBB(BB), LastBB(BB), BlockToChain(BlockToChain) {<br>
+ assert(BB && "Cannot create a chain with a null basic block");<br>
+ BlockToChain[BB] = this;<br>
+ }<br>
+<br>
+ /// \brief Merge another block chain into this one.<br>
+ ///<br>
+ /// This routine merges a block chain into this one. It takes care of forming<br>
+ /// a contiguous sequence of basic blocks, updating the edge list, and<br>
+ /// updating the block -> chain mapping. It does not free or tear down the<br>
+ /// old chain, but the old chain's block list is no longer valid.<br>
+ void merge(BlockChain *Chain) {<br>
+ assert(Chain && "Cannot merge a null chain");<br>
+ MachineFunction::iterator EndBB = llvm::next(LastBB);<br>
+ MachineFunction::iterator ChainEndBB = llvm::next(Chain->LastBB);<br>
+<br>
+ // Update the incoming blocks to point to this chain.<br>
+ for (MachineFunction::iterator BI = Chain->FirstBB, BE = ChainEndBB;<br>
+ BI != BE; ++BI) {<br>
+ assert(BlockToChain[BI] == Chain && "Incoming blocks not in chain");<br>
+ BlockToChain[BI] = this;<br>
+ }<br>
+<br>
+ // We splice the blocks together within the function (unless they already<br>
+ // are adjacent) so we can represent the new chain with a pair of pointers<br>
+ // to basic blocks within the function. This is also useful as each chain<br>
+ // of blocks will end up being laid out contiguously within the function.<br>
+ if (EndBB != Chain->FirstBB)<br>
+ FirstBB->getParent()->splice(EndBB, Chain->FirstBB, ChainEndBB);<br>
+ LastBB = Chain->LastBB;<br>
+ }<br>
+};<br>
+}<br>
+<br>
+namespace {<br>
+/// \brief Successor iterator for BlockChains.<br>
+///<br>
+/// This is an iterator that walks over the successor block chains by looking<br>
+/// through its blocks successors and mapping those back to block chains. This<br>
+/// iterator is not a fully-functioning iterator, it is designed specifically<br>
+/// to support the interface required by SCCIterator when forming and walking<br>
+/// SCCs of BlockChains.<br>
+///<br>
+/// Note that this iterator cannot be used while the chains are still being<br>
+/// formed and/or merged. Unlike the chains themselves, it does store end<br>
+/// iterators which could be moved if the chains are re-ordered. Once we begin<br>
+/// forming and iterating over an SCC of chains, the order of blocks within the<br>
+/// function must not change until we finish using the SCC iterators.<br>
+class BlockChain::SuccIterator<br>
+ : public std::iterator<std::forward_iterator_tag,<br>
+ BlockChain *, ptrdiff_t> {<br>
+ BlockChain *Chain;<br>
+ MachineFunction::iterator BI, BE;<br>
+ MachineBasicBlock::succ_iterator SI;<br>
+<br>
+public:<br>
+ explicit SuccIterator(BlockChain *Chain)<br>
+ : Chain(Chain), BI(Chain->FirstBB), BE(llvm::next(Chain->LastBB)),<br>
+ SI(BI->succ_begin()) {<br>
+ while (BI != BE && BI->succ_begin() == BI->succ_end())<br>
+ ++BI;<br>
+ if (BI != BE)<br>
+ SI = BI->succ_begin();<br>
+ }<br>
+<br>
+ /// \brief Helper function to create an end iterator for a particular chain.<br>
+ ///<br>
+ /// The "end" state is extremely arbitrary. We chose to have BI == BE, and SI<br>
+ /// == Chain->FirstBB->succ_begin(). The value of SI doesn't really make any<br>
+ /// sense, but rather than try to rationalize SI and our increment, when we<br>
+ /// detect an "end" state, we just immediately call this function to build<br>
+ /// the canonical end iterator.<br>
+ static SuccIterator CreateEnd(BlockChain *Chain) {<br>
+ SuccIterator It(Chain);<br>
+ It.BI = It.BE;<br>
+ return It;<br>
+ }<br>
+<br>
+ bool operator==(const SuccIterator &RHS) const {<br>
+ return (Chain == RHS.Chain && BI == <a href="http://RHS.BI" target="_blank">RHS.BI</a> && SI == <a href="http://RHS.SI" target="_blank">RHS.SI</a>);<br>
+ }<br>
+ bool operator!=(const SuccIterator &RHS) const {<br>
+ return !operator==(RHS);<br>
+ }<br>
+<br>
+ SuccIterator& operator++() {<br>
+ assert(*this != CreateEnd(Chain) && "Cannot increment the end iterator");<br>
+ // There may be null successor pointers, skip over them.<br>
+ // FIXME: I don't understand *why* there are null successor pointers.<br>
+ do {<br>
+ ++SI;<br>
+ if (SI != BI->succ_end() && *SI)<br>
+ return *this;<br>
+<br>
+ // There may be a basic block without successors. Skip over them.<br>
+ do {<br>
+ ++BI;<br>
+ if (BI == BE)<br>
+ return *this = CreateEnd(Chain);<br>
+ } while (BI->succ_begin() == BI->succ_end());<br>
+ SI = BI->succ_begin();<br>
+ } while (!*SI);<br>
+ return *this;<br>
+ }<br>
+ SuccIterator operator++(int) {<br>
+ SuccIterator tmp = *this;<br>
+ ++*this;<br>
+ return tmp;<br>
+ }<br>
+<br>
+ BlockChain *operator*() const {<br>
+ assert(Chain->BlockToChain.lookup(*SI) && "Missing chain");<br>
+ return Chain->BlockToChain.lookup(*SI);<br>
+ }<br>
+};<br>
+}<br>
+<br>
+namespace {<br>
+/// \brief Sorter used with containers of BlockChain pointers.<br>
+///<br>
+/// Sorts based on the \see BlockChain::InChainEdgeFrequency -- see its<br>
+/// comments for details on what this ordering represents.<br>
+struct ChainPtrPrioritySorter {<br>
+ bool operator()(const BlockChain *LHS, const BlockChain *RHS) const {<br>
+ assert(LHS && RHS && "Null chain entry");<br>
+ return LHS->InChainEdgeFrequency < RHS->InChainEdgeFrequency;<br>
+ }<br>
+};<br>
+}<br>
+<br>
+namespace {<br>
+class MachineBlockPlacement : public MachineFunctionPass {<br>
+ /// \brief A handle to the branch probability pass.<br>
+ const MachineBranchProbabilityInfo *MBPI;<br>
+<br>
+ /// \brief A handle to the function-wide block frequency pass.<br>
+ const MachineBlockFrequencyInfo *MBFI;<br>
+<br>
+ /// \brief A handle to the target's instruction info.<br>
+ const TargetInstrInfo *TII;<br>
+<br>
+ /// \brief A prioritized list of edges in the BB-graph.<br>
+ ///<br>
+ /// For each function, we insert all control flow edges between BBs, along<br>
+ /// with their "global" frequency. The Frequency of an edge being taken is<br>
+ /// defined as the frequency of entering the source BB (from MBFI) times the<br>
+ /// probability of taking a particular branch out of that block (from MBPI).<br>
+ ///<br>
+ /// Once built, this list is sorted in ascending frequency, making the last<br>
+ /// edge the hottest one in the function.<br>
+ SmallVector<WeightedEdge, 64> Edges;<br>
+<br>
+ /// \brief Allocator and owner of BlockChain structures.<br>
+ ///<br>
+ /// We build BlockChains lazily by merging together high probability BB<br>
+ /// sequences acording to the "Algo2" in the paper mentioned at the top of<br>
+ /// the file. To reduce malloc traffic, we allocate them using this slab-like<br>
+ /// allocator, and destroy them after the pass completes.<br>
+ SpecificBumpPtrAllocator<BlockChain> ChainAllocator;<br>
+<br>
+ /// \brief Function wide BasicBlock to BlockChain mapping.<br>
+ ///<br>
+ /// This mapping allows efficiently moving from any given basic block to the<br>
+ /// BlockChain it participates in, if any. We use it to, among other things,<br>
+ /// allow implicitly defining edges between chains as the existing edges<br>
+ /// between basic blocks.<br>
+ DenseMap<MachineBasicBlock *, BlockChain *> BlockToChain;<br>
+<br>
+ /// \brief A prioritized sequence of chains.<br>
+ ///<br>
+ /// We build up the ideal sequence of basic block chains in reverse order<br>
+ /// here, and then walk backwards to arrange the final function ordering.<br>
+ SmallVector<BlockChain *, 16> PChains;<br>
+<br>
+#ifndef NDEBUG<br>
+ /// \brief A set of active chains used to sanity-check the pass algorithm.<br>
+ ///<br>
+ /// All operations on this member should be wrapped in an assert or NDEBUG.<br>
+ SmallPtrSet<BlockChain *, 16> ActiveChains;<br>
+#endif<br>
+<br>
+ BlockChain *CreateChain(MachineBasicBlock *BB);<br>
+ void PrioritizeEdges(MachineFunction &F);<br>
+ void BuildBlockChains();<br>
+ void PrioritizeChains(MachineFunction &F);<br>
+ void PlaceBlockChains(MachineFunction &F);<br>
+<br>
+public:<br>
+ static char ID; // Pass identification, replacement for typeid<br>
+ MachineBlockPlacement() : MachineFunctionPass(ID) {<br>
+ initializeMachineBlockPlacementPass(*PassRegistry::getPassRegistry());<br>
+ }<br>
+<br>
+ bool runOnMachineFunction(MachineFunction &F);<br>
+<br>
+ void getAnalysisUsage(AnalysisUsage &AU) const {<br>
+ AU.addRequired<MachineBranchProbabilityInfo>();<br>
+ AU.addRequired<MachineBlockFrequencyInfo>();<br>
+ MachineFunctionPass::getAnalysisUsage(AU);<br>
+ }<br>
+<br>
+ const char *getPassName() const { return "Block Placement"; }<br>
+};<br>
+}<br>
+<br>
+char MachineBlockPlacement::ID = 0;<br>
+INITIALIZE_PASS_BEGIN(MachineBlockPlacement, "block-placement2",<br>
+ "Branch Probability Basic Block Placement", false, false)<br>
+INITIALIZE_PASS_DEPENDENCY(MachineBranchProbabilityInfo)<br>
+INITIALIZE_PASS_DEPENDENCY(MachineBlockFrequencyInfo)<br>
+INITIALIZE_PASS_END(MachineBlockPlacement, "block-placement2",<br>
+ "Branch Probability Basic Block Placement", false, false)<br>
+<br>
+FunctionPass *llvm::createMachineBlockPlacementPass() {<br>
+ return new MachineBlockPlacement();<br>
+}<br>
+<br>
+namespace llvm {<br>
+/// \brief GraphTraits specialization for our BlockChain graph.<br>
+template <> struct GraphTraits<BlockChain *> {<br>
+ typedef BlockChain NodeType;<br>
+ typedef BlockChain::SuccIterator ChildIteratorType;<br>
+<br>
+ static NodeType *getEntryNode(NodeType *N) { return N; }<br>
+ static BlockChain::SuccIterator child_begin(NodeType *N) {<br>
+ return BlockChain::SuccIterator(N);<br>
+ }<br>
+ static BlockChain::SuccIterator child_end(NodeType *N) {<br>
+ return BlockChain::SuccIterator::CreateEnd(N);<br>
+ }<br>
+};<br>
+}<br>
+<br>
+/// \brief Helper to create a new chain for a single BB.<br>
+///<br>
+/// Takes care of growing the Chains, setting up the BlockChain object, and any<br>
+/// debug checking logic.<br>
+/// \returns A pointer to the new BlockChain.<br>
+BlockChain *MachineBlockPlacement::CreateChain(MachineBasicBlock *BB) {<br>
+ BlockChain *Chain =<br>
+ new (ChainAllocator.Allocate()) BlockChain(BlockToChain, BB);<br>
+ assert(ActiveChains.insert(Chain));<br>
+ return Chain;<br>
+}<br>
+<br>
+/// \brief Build a prioritized list of edges.<br>
+///<br>
+/// The priority is determined by the product of the block frequency (how<br>
+/// likely it is to arrive at a particular block) times the probability of<br>
+/// taking this particular edge out of the block. This provides a function-wide<br>
+/// ordering of the edges.<br>
+void MachineBlockPlacement::PrioritizeEdges(MachineFunction &F) {<br>
+ assert(Edges.empty() && "Already have an edge list");<br>
+ SmallVector<MachineOperand, 4> Cond; // For AnalyzeBranch.<br>
+ BlockChain *RequiredChain = 0;<br>
+ for (MachineFunction::iterator FI = F.begin(), FE = F.end(); FI != FE; ++FI) {<br>
+ MachineBasicBlock *From = &*FI;<br>
+ // We only consider MBBs with analyzable branches. Even if the analysis<br>
+ // fails, if there is no fallthrough, we can still work with the MBB.<br>
+ MachineBasicBlock *TBB = 0, *FBB = 0;<br>
+ Cond.clear();<br>
+ if (TII->AnalyzeBranch(*From, TBB, FBB, Cond) && From->canFallThrough()) {<br>
+ // We push all unanalyzed blocks onto a chain eagerly to prevent them<br>
+ // from being split later. Create the chain if needed, otherwise just<br>
+ // keep track that these blocks reside on it.<br>
+ if (!RequiredChain)<br>
+ RequiredChain = CreateChain(From);<br>
+ else<br>
+ BlockToChain[From] = RequiredChain;<br>
+ } else {<br>
+ // As soon as we find an analyzable branch, add that block to and<br>
+ // finalize any required chain that has been started. The required chain<br>
+ // is only modeling potentially inexplicable fallthrough, so the first<br>
+ // block to have analyzable fallthrough is a known-safe stopping point.<br>
+ if (RequiredChain) {<br>
+ BlockToChain[From] = RequiredChain;<br>
+ RequiredChain->LastBB = FI;<br>
+ RequiredChain = 0;<br>
+ }<br>
+ }<br>
+<br>
+ BlockFrequency BaseFrequency = MBFI->getBlockFreq(From);<br>
+ for (MachineBasicBlock::succ_iterator SI = From->succ_begin(),<br>
+ SE = From->succ_end();<br>
+ SI != SE; ++SI) {<br>
+ MachineBasicBlock *To = *SI;<br>
+ WeightedEdge WE = { BaseFrequency * MBPI->getEdgeProbability(From, To),<br>
+ From, To };<br>
+ Edges.push_back(WE);<br>
+ }<br>
+ }<br>
+ assert(!RequiredChain && "Never found a terminator for a required chain");<br>
+ std::stable_sort(Edges.begin(), Edges.end());<br>
+}<br>
+<br>
+/// \brief Build chains of basic blocks along hot paths.<br>
+///<br>
+/// Build chains by trying to merge each pair of blocks from the mostly costly<br>
+/// edge first. This is essentially "Algo2" from the Profile Guided Code<br>
+/// Placement paper. While each node is considered a chain of one block, this<br>
+/// routine lazily build the chain objects themselves so that when possible it<br>
+/// can just merge a block into an existing chain.<br>
+void MachineBlockPlacement::BuildBlockChains() {<br>
+ for (SmallVectorImpl<WeightedEdge>::reverse_iterator EI = Edges.rbegin(),<br>
+ EE = Edges.rend();<br>
+ EI != EE; ++EI) {<br>
+ MachineBasicBlock *SourceB = EI->From, *DestB = EI->To;<br>
+ if (SourceB == DestB) continue;<br>
+<br>
+ BlockChain *SourceChain = BlockToChain.lookup(SourceB);<br>
+ if (!SourceChain) SourceChain = CreateChain(SourceB);<br>
+ BlockChain *DestChain = BlockToChain.lookup(DestB);<br>
+ if (!DestChain) DestChain = CreateChain(DestB);<br>
+ if (SourceChain == DestChain)<br>
+ continue;<br>
+<br>
+ bool IsSourceTail =<br>
+ SourceChain->LastBB == MachineFunction::iterator(SourceB);<br>
+ bool IsDestHead =<br>
+ DestChain->FirstBB == MachineFunction::iterator(DestB);<br>
+<br>
+ if (!IsSourceTail || !IsDestHead)<br>
+ continue;<br>
+<br>
+ SourceChain->merge(DestChain);<br>
+ assert(ActiveChains.erase(DestChain));<br>
+ }<br>
+}<br>
+<br>
+/// \brief Prioritize the chains to minimize back-edges between chains.<br>
+///<br>
+/// This is the trickiest part of the placement algorithm. Each chain is<br>
+/// a hot-path through a sequence of basic blocks, but there are conditional<br>
+/// branches away from this hot path, and to some other chain. Hardware branch<br>
+/// predictors favor back edges over forward edges, and so it is desirable to<br>
+/// arrange the targets of branches away from a hot path and to some other<br>
+/// chain to come later in the function, making them forward branches, and<br>
+/// helping the branch predictor to predict fallthrough.<br>
+///<br>
+/// In some cases, this is easy. simply topologically walking from the entry<br>
+/// chain through its successors in order would work if there were no cycles<br>
+/// between the chains of blocks, but often there are. In such a case, we first<br>
+/// need to identify the participants in the cycle, and then rank them so that<br>
+/// the linearizing of the chains has the lowest *probability* of causing<br>
+/// a mispredicted branch. To compute the correct rank for a chain, we take the<br>
+/// complement of the branch probability for each branch leading away from the<br>
+/// chain and multiply it by the frequency of the source block for that branch.<br>
+/// This gives us the probability of that particular branch *not* being taken<br>
+/// in this function. The sum of these probabilities for each chain is used as<br>
+/// a rank, so that we order the chain with the highest such sum first.<br>
+/// FIXME: This seems like a good approximation, but there is probably a known<br>
+/// technique for ordering of an SCC given edge weights. It would be good to<br>
+/// use that, or even use its code if possible.<br>
+///<br>
+/// Also notable is that we prioritize the chains from the bottom up, and so<br>
+/// all of the "first" and "before" relationships end up inverted in the code.<br>
+void MachineBlockPlacement::PrioritizeChains(MachineFunction &F) {<br>
+ MachineBasicBlock *EntryB = &F.front();<br>
+ BlockChain *EntryChain = BlockToChain[EntryB];<br>
+ assert(EntryChain && "Missing chain for entry block");<br>
+ assert(EntryChain->FirstBB == F.begin() &&<br>
+ "Entry block is not the head of the entry block chain");<br>
+<br>
+ // Form an SCC and walk it from the bottom up.<br>
+ SmallPtrSet<BlockChain *, 4> IsInSCC;<br>
+ for (scc_iterator<BlockChain *> I = scc_begin(EntryChain);<br>
+ !I.isAtEnd(); ++I) {<br>
+ const std::vector<BlockChain *> &SCC = *I;<br>
+ PChains.insert(PChains.end(), SCC.begin(), SCC.end());<br>
+<br>
+ // If there is only one chain in the SCC, it's trivially sorted so just<br>
+ // bail out early. Sorting the SCC is expensive.<br>
+ if (SCC.size() == 1)<br>
+ continue;<br>
+<br>
+ // We work strictly on the PChains range from here on out to maximize<br>
+ // locality.<br>
+ SmallVectorImpl<BlockChain *>::iterator SCCEnd = PChains.end(),<br>
+ SCCBegin = SCCEnd - SCC.size();<br>
+ IsInSCC.clear();<br>
+ IsInSCC.insert(SCCBegin, SCCEnd);<br>
+<br>
+ // Compute the edge frequency of staying in a chain, despite the existency<br>
+ // of an edge to some other chain within this SCC.<br>
+ for (SmallVectorImpl<BlockChain *>::iterator SCCI = SCCBegin;<br>
+ SCCI != SCCEnd; ++SCCI) {<br>
+ BlockChain *Chain = *SCCI;<br>
+<br>
+ // Special case the entry chain. Regardless of the weights of other<br>
+ // chains, the entry chain *must* come first, so move it to the end, and<br>
+ // avoid processing that chain at all.<br>
+ if (Chain == EntryChain) {<br>
+ --SCCEnd;<br>
+ if (SCCI == SCCEnd) break;<br>
+ Chain = *SCCI = *SCCEnd;<br>
+ *SCCEnd = EntryChain;<br>
+ }<br>
+<br>
+ // Walk over every block in this chain looking for out-bound edges to<br>
+ // other chains in this SCC.<br>
+ for (MachineFunction::iterator BI = Chain->FirstBB,<br>
+ BE = llvm::next(Chain->LastBB);<br>
+ BI != BE; ++BI) {<br>
+ MachineBasicBlock *From = &*BI;<br>
+ for (MachineBasicBlock::succ_iterator SI = BI->succ_begin(),<br>
+ SE = BI->succ_end();<br>
+ SI != SE; ++SI) {<br>
+ MachineBasicBlock *To = *SI;<br>
+ if (!To || !IsInSCC.count(BlockToChain[To]))<br>
+ continue;<br>
+ BranchProbability ComplEdgeProb =<br>
+ MBPI->getEdgeProbability(From, To).getCompl();<br>
+ Chain->InChainEdgeFrequency +=<br>
+ MBFI->getBlockFreq(From) * ComplEdgeProb;<br>
+ }<br>
+ }<br>
+ }<br>
+<br>
+ // Sort the chains within the SCC according to their edge frequencies,<br>
+ // which should make the least costly chain of blocks to mis-place be<br>
+ // ordered first in the prioritized sequence.<br>
+ std::stable_sort(SCCBegin, SCCEnd, ChainPtrPrioritySorter());<br>
+ }<br>
+}<br>
+<br>
+/// \brief Splice the function blocks together based on the chain priorities.<br>
+///<br>
+/// Each chain is already represented as a contiguous range of blocks in the<br>
+/// function. Simply walk backwards down the prioritized chains and splice in<br>
+/// any chains out of order. Note that the first chain we visit is necessarily<br>
+/// the entry chain. It has no predecessors and so must be the top of the SCC.<br>
+/// Also, we cannot splice any chain prior to the entry chain as we can't<br>
+/// splice any blocks prior to the entry block.<br>
+void MachineBlockPlacement::PlaceBlockChains(MachineFunction &F) {<br>
+ assert(!PChains.empty() && "No chains were prioritized");<br>
+ assert(PChains.back() == BlockToChain[&F.front()] &&<br>
+ "The entry chain must always be the final chain");<br>
+<br>
+ MachineFunction::iterator InsertPos = F.begin();<br>
+ for (SmallVectorImpl<BlockChain *>::reverse_iterator CI = PChains.rbegin(),<br>
+ CE = PChains.rend();<br>
+ CI != CE; ++CI) {<br>
+ BlockChain *Chain = *CI;<br>
+ // Check that we process this chain only once for debugging.<br>
+ assert(ActiveChains.erase(Chain) && "Processed a chain twice");<br>
+<br>
+ // If this chain is already in the right position, just skip past it.<br>
+ // Otherwise, splice it into position.<br>
+ if (InsertPos == Chain->FirstBB)<br>
+ InsertPos = llvm::next(Chain->LastBB);<br>
+ else<br>
+ F.splice(InsertPos, Chain->FirstBB, llvm::next(Chain->LastBB));<br>
+ }<br>
+<br>
+ // Note that we can't assert this is empty as there may be unreachable blocks<br>
+ // in the function.<br>
+#ifndef NDEBUG<br>
+ ActiveChains.clear();<br>
+#endif<br>
+<br>
+ // Now that every block is in its final position, update all of the<br>
+ // terminators.<br>
+ SmallVector<MachineOperand, 4> Cond; // For AnalyzeBranch.<br>
+ for (MachineFunction::iterator FI = F.begin(), FE = F.end(); FI != FE; ++FI) {<br>
+ // FIXME: It would be awesome of updateTerminator would just return rather<br>
+ // than assert when the branch cannot be analyzed in order to remove this<br>
+ // boiler plate.<br>
+ Cond.clear();<br>
+ MachineBasicBlock *TBB = 0, *FBB = 0; // For AnalyzeBranch.<br>
+ if (!TII->AnalyzeBranch(*FI, TBB, FBB, Cond))<br>
+ FI->updateTerminator();<br>
+ }<br>
+}<br>
+<br>
+bool MachineBlockPlacement::runOnMachineFunction(MachineFunction &F) {<br>
+ // Check for single-block functions and skip them.<br>
+ if (llvm::next(F.begin()) == F.end())<br>
+ return false;<br>
+<br>
+ MBPI = &getAnalysis<MachineBranchProbabilityInfo>();<br>
+ MBFI = &getAnalysis<MachineBlockFrequencyInfo>();<br>
+ TII = F.getTarget().getInstrInfo();<br>
+ assert(Edges.empty());<br>
+ assert(BlockToChain.empty());<br>
+ assert(PChains.empty());<br>
+ assert(ActiveChains.empty());<br>
+<br>
+ PrioritizeEdges(F);<br>
+ BuildBlockChains();<br>
+ PrioritizeChains(F);<br>
+ PlaceBlockChains(F);<br>
+<br>
+ Edges.clear();<br>
+ BlockToChain.clear();<br>
+ PChains.clear();<br>
+ ChainAllocator.DestroyAll();<br>
+<br>
+ // We always return true as we have no way to track whether the final order<br>
+ // differs from the original order.<br>
+ return true;<br>
+}<br>
<br>
<br>
_______________________________________________<br>
llvm-commits mailing list<br>
<a href="mailto:llvm-commits@cs.uiuc.edu">llvm-commits@cs.uiuc.edu</a><br>
<a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a><br>
</blockquote></div><br>