<html><head><meta http-equiv="Content-Type" content="text/html charset=iso-8859-1"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Thanks, Chad! Fixed the space and posted the option code for review.<div><br></div><div>Cheers</div><div>Gerolf</div><div><br><div><div>On Sep 4, 2014, at 11:50 AM, Chad Rosier <<a href="mailto:mcrosier@codeaurora.org">mcrosier@codeaurora.org</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="font-size: 18px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">Hi Gerolf,<br><br><blockquote type="cite">Author: ghoflehner<br>Date: Fri Jul 18 14:13:09 2014<br>New Revision: 213396<br><br>URL: <a href="http://llvm.org/viewvc/llvm-project?rev=213396&view=rev">http://llvm.org/viewvc/llvm-project?rev=213396&view=rev</a><br>Log:<br>MergedLoadStoreMotion pass<br><br>Merges equivalent loads on both sides of a hammock/diamond<br>and hoists into into the header.<br>Merges equivalent stores on both sides of a hammock/diamond<br>and sinks it to the footer.<br>Can enable if conversion and tolerate better load misses<br>and store operand latencies.<br><br><br>Added:<br> llvm/trunk/lib/Transforms/Scalar/MergedLoadStoreMotion.cpp<br> llvm/trunk/test/Transforms/InstMerge/<br> llvm/trunk/test/Transforms/InstMerge/ld_hoist_st_sink.ll<br>Modified:<br> llvm/trunk/bindings/ocaml/transforms/scalar/scalar_opts_ocaml.c<br> llvm/trunk/include/llvm-c/Transforms/Scalar.h<br> llvm/trunk/include/llvm/InitializePasses.h<br> llvm/trunk/include/llvm/LinkAllPasses.h<br> llvm/trunk/include/llvm/Transforms/Scalar.h<br> llvm/trunk/lib/LTO/LTOCodeGenerator.cpp<br> llvm/trunk/lib/Transforms/IPO/PassManagerBuilder.cpp<br> llvm/trunk/lib/Transforms/Scalar/CMakeLists.txt<br> llvm/trunk/lib/Transforms/Scalar/Scalar.cpp<br><br>Modified: llvm/trunk/bindings/ocaml/transforms/scalar/scalar_opts_ocaml.c<br>URL:<br><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/bindings/ocaml/transforms/scalar/scalar_opts_ocaml.c?rev=213396&r1=213395&r2=213396&view=diff">http://llvm.org/viewvc/llvm-project/llvm/trunk/bindings/ocaml/transforms/scalar/scalar_opts_ocaml.c?rev=213396&r1=213395&r2=213396&view=diff</a><br>==============================================================================<br>--- llvm/trunk/bindings/ocaml/transforms/scalar/scalar_opts_ocaml.c<br>(original)<br>+++ llvm/trunk/bindings/ocaml/transforms/scalar/scalar_opts_ocaml.c Fri<br>Jul 18 14:13:09 2014<br>@@ -135,6 +135,12 @@ CAMLprim value llvm_add_tail_call_elimin<br>}<br><br>/* [<Llvm.PassManager.any] Llvm.PassManager.t -> unit */<br>+CAMLprim value llvm_add_merged_load_store_motion(LLVMPassManagerRef PM) {<br>+ LLVMAddMergedLoadStoreMotionPass(PM);<br>+ return Val_unit;<br>+}<br>+<br>+/* [<Llvm.PassManager.any] Llvm.PassManager.t -> unit */<br>CAMLprim value llvm_add_gvn(LLVMPassManagerRef PM) {<br> LLVMAddGVNPass(PM);<br> return Val_unit;<br><br>Modified: llvm/trunk/include/llvm-c/Transforms/Scalar.h<br>URL:<br>http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm-c/Transforms/Scalar.h?rev=213396&r1=213395&r2=213396&view=diff<br>==============================================================================<br>--- llvm/trunk/include/llvm-c/Transforms/Scalar.h (original)<br>+++ llvm/trunk/include/llvm-c/Transforms/Scalar.h Fri Jul 18 14:13:09 2014<br>@@ -44,6 +44,9 @@ void LLVMAddDeadStoreEliminationPass(LLV<br>/** See llvm::createScalarizerPass function. */<br>void LLVMAddScalarizerPass(LLVMPassManagerRef PM);<br><br>+/** See llvm::createMergedLoadStoreMotionPass function. */<br>+void LLVMAddMergedLoadStoreMotionPass(LLVMPassManagerRef PM);<br>+<br>/** See llvm::createGVNPass function. */<br>void LLVMAddGVNPass(LLVMPassManagerRef PM);<br><br><br>Modified: llvm/trunk/include/llvm/InitializePasses.h<br>URL:<br>http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/InitializePasses.h?rev=213396&r1=213395&r2=213396&view=diff<br>==============================================================================<br>--- llvm/trunk/include/llvm/InitializePasses.h (original)<br>+++ llvm/trunk/include/llvm/InitializePasses.h Fri Jul 18 14:13:09 2014<br>@@ -196,6 +196,7 @@ void initializeMachineVerifierPassPass(P<br>void initializeMemCpyOptPass(PassRegistry&);<br>void initializeMemDepPrinterPass(PassRegistry&);<br>void initializeMemoryDependenceAnalysisPass(PassRegistry&);<br>+void initializeMergedLoadStoreMotionPass(PassRegistry &);<br>void initializeMetaRenamerPass(PassRegistry&);<br>void initializeMergeFunctionsPass(PassRegistry&);<br>void initializeModuleDebugInfoPrinterPass(PassRegistry&);<br><br>Modified: llvm/trunk/include/llvm/LinkAllPasses.h<br>URL:<br>http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/LinkAllPasses.h?rev=213396&r1=213395&r2=213396&view=diff<br>==============================================================================<br>--- llvm/trunk/include/llvm/LinkAllPasses.h (original)<br>+++ llvm/trunk/include/llvm/LinkAllPasses.h Fri Jul 18 14:13:09 2014<br>@@ -134,6 +134,7 @@ namespace {<br> (void) llvm::createConstantHoistingPass();<br> (void) llvm::createCodeGenPreparePass();<br> (void) llvm::createEarlyCSEPass();<br>+ (void)llvm::createMergedLoadStoreMotionPass();<br></blockquote><br>Need a space here.<br><br><blockquote type="cite"> (void) llvm::createGVNPass();<br> (void) llvm::createMemCpyOptPass();<br> (void) llvm::createLoopDeletionPass();<br><br>Modified: llvm/trunk/include/llvm/Transforms/Scalar.h<br>URL:<br><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Transforms/Scalar.h?rev=213396&r1=213395&r2=213396&view=diff">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Transforms/Scalar.h?rev=213396&r1=213395&r2=213396&view=diff</a><br>==============================================================================<br>--- llvm/trunk/include/llvm/Transforms/Scalar.h (original)<br>+++ llvm/trunk/include/llvm/Transforms/Scalar.h Fri Jul 18 14:13:09 2014<br>@@ -288,6 +288,13 @@ FunctionPass *createEarlyCSEPass();<br><br>//===----------------------------------------------------------------------===//<br>//<br>+// MergedLoadStoreMotion - This pass merges loads and stores in diamonds.<br>Loads<br>+// are hoisted into the header, while stores sink into the footer.<br>+//<br>+FunctionPass *createMergedLoadStoreMotionPass();<br>+<br>+//===----------------------------------------------------------------------===//<br>+//<br>// GVN - This pass performs global value numbering and redundant load<br>// elimination cotemporaneously.<br>//<br><br>Modified: llvm/trunk/lib/LTO/LTOCodeGenerator.cpp<br>URL:<br>http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/LTO/LTOCodeGenerator.cpp?rev=213396&r1=213395&r2=213396&view=diff<br>==============================================================================<br>--- llvm/trunk/lib/LTO/LTOCodeGenerator.cpp (original)<br>+++ llvm/trunk/lib/LTO/LTOCodeGenerator.cpp Fri Jul 18 14:13:09 2014<br>@@ -107,6 +107,7 @@ void LTOCodeGenerator::initializeLTOPass<br> initializeFunctionAttrsPass(R);<br> initializeGlobalsModRefPass(R);<br> initializeLICMPass(R);<br>+ initializeMergedLoadStoreMotionPass(R);<br> initializeGVNPass(R);<br> initializeMemCpyOptPass(R);<br> initializeDCEPass(R);<br><br>Modified: llvm/trunk/lib/Transforms/IPO/PassManagerBuilder.cpp<br>URL:<br>http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/IPO/PassManagerBuilder.cpp?rev=213396&r1=213395&r2=213396&view=diff<br>==============================================================================<br>--- llvm/trunk/lib/Transforms/IPO/PassManagerBuilder.cpp (original)<br>+++ llvm/trunk/lib/Transforms/IPO/PassManagerBuilder.cpp Fri Jul 18<br>14:13:09 2014<br>@@ -207,8 +207,10 @@ void PassManagerBuilder::populateModuleP<br> MPM.add(createSimpleLoopUnrollPass()); // Unroll small loops<br> addExtensionsToPM(EP_LoopOptimizerEnd, MPM);<br><br>- if (OptLevel > 1)<br>+ if (OptLevel > 1) {<br>+ MPM.add(createMergedLoadStoreMotionPass()); // Merge load/stores in<br>diamond<br> MPM.add(createGVNPass()); // Remove redundancies<br>+ }<br> MPM.add(createMemCpyOptPass()); // Remove memcpy / form<br>memset<br> MPM.add(createSCCPPass()); // Constant prop with SCCP<br><br>@@ -346,6 +348,7 @@ void PassManagerBuilder::populateLTOPass<br> PM.add(createGlobalsModRefPass()); // IP alias analysis.<br><br> PM.add(createLICMPass()); // Hoist loop invariants.<br>+ PM.add(createMergedLoadStoreMotionPass()); // Merge load/stores in<br>diamonds<br> PM.add(createGVNPass(DisableGVNLoadPRE)); // Remove redundancies.<br> PM.add(createMemCpyOptPass()); // Remove dead memcpys.<br><br><br>Modified: llvm/trunk/lib/Transforms/Scalar/CMakeLists.txt<br>URL:<br>http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Scalar/CMakeLists.txt?rev=213396&r1=213395&r2=213396&view=diff<br>==============================================================================<br>--- llvm/trunk/lib/Transforms/Scalar/CMakeLists.txt (original)<br>+++ llvm/trunk/lib/Transforms/Scalar/CMakeLists.txt Fri Jul 18 14:13:09<br>2014<br>@@ -22,6 +22,7 @@ add_llvm_library(LLVMScalarOpts<br> LoopUnswitch.cpp<br> LowerAtomic.cpp<br> MemCpyOptimizer.cpp<br>+ MergedLoadStoreMotion.cpp<br> PartiallyInlineLibCalls.cpp<br> Reassociate.cpp<br> Reg2Mem.cpp<br><br>Added: llvm/trunk/lib/Transforms/Scalar/MergedLoadStoreMotion.cpp<br>URL:<br>http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Scalar/MergedLoadStoreMotion.cpp?rev=213396&view=auto<br>==============================================================================<br>--- llvm/trunk/lib/Transforms/Scalar/MergedLoadStoreMotion.cpp (added)<br>+++ llvm/trunk/lib/Transforms/Scalar/MergedLoadStoreMotion.cpp Fri Jul 18<br>14:13:09 2014<br>@@ -0,0 +1,623 @@<br>+//===- MergedLoadStoreMotion.cpp - merge and hoist/sink load/stores<br>-------===//<br>+//<br>+// The LLVM Compiler Infrastructure<br>+//<br>+// This file is distributed under the University of Illinois Open Source<br>+// License. See LICENSE.TXT for details.<br>+//<br>+//===----------------------------------------------------------------------===//<br>+//<br>+//! \file<br>+//! \brief This pass performs merges of loads and stores on both sides of<br>a<br>+// diamond (hammock). It hoists the loads and sinks the stores.<br>+//<br>+// The algorithm iteratively hoists two loads to the same address out of<br>a<br>+// diamond (hammock) and merges them into a single load in the header.<br>Similar<br>+// it sinks and merges two stores to the tail block (footer). The<br>algorithm<br>+// iterates over the instructions of one side of the diamond and attempts<br>to<br>+// find a matching load/store on the other side. It hoists / sinks when<br>it<br>+// thinks it safe to do so. This optimization helps with eg. hiding load<br>+// latencies, triggering if-conversion, and reducing static code size.<br>+//<br>+//===----------------------------------------------------------------------===//<br>+//<br>+//<br>+// Example:<br>+// Diamond shaped code before merge:<br>+//<br>+// header:<br>+// br %cond, label %if.then, label %if.else<br>+// / \<br>+// / \<br>+// / \<br>+// if.then: if.else:<br>+// %lt = load %addr_l %le = load %addr_l<br>+// <use %lt> <use %le><br>+// <...> <...><br>+// store %st, %addr_s store %se, %addr_s<br>+// br label %if.end br label %if.end<br>+// \ /<br>+// \ /<br>+// \ /<br>+// if.end ("footer"):<br>+// <...><br>+//<br>+// Diamond shaped code after merge:<br>+//<br>+// header:<br>+// %l = load %addr_l<br>+// br %cond, label %if.then, label %if.else<br>+// / \<br>+// / \<br>+// / \<br>+// if.then: if.else:<br>+// <use %l> <use %l><br>+// <...> <...><br>+// br label %if.end br label %if.end<br>+// \ /<br>+// \ /<br>+// \ /<br>+// if.end ("footer"):<br>+// %s.sink = phi [%st, if.then], [%se, if.else]<br>+// <...><br>+// store %s.sink, %addr_s<br>+// <...><br>+//<br>+//<br>+//===----------------------- TODO<br>-----------------------------------------===//<br>+//<br>+// 1) Generalize to regions other than diamonds<br>+// 2) Be more aggressive merging memory operations<br>+// Note that both changes require register pressure control<br>+//<br>+//===----------------------------------------------------------------------===//<br>+<br>+#include "llvm/Transforms/Scalar.h"<br>+#include "llvm/ADT/SetVector.h"<br>+#include "llvm/ADT/SmallPtrSet.h"<br>+#include "llvm/ADT/Statistic.h"<br>+#include "llvm/Analysis/AliasAnalysis.h"<br>+#include "llvm/Analysis/CFG.h"<br>+#include "llvm/Analysis/Loads.h"<br>+#include "llvm/Analysis/MemoryBuiltins.h"<br>+#include "llvm/Analysis/MemoryDependenceAnalysis.h"<br>+#include "llvm/IR/Metadata.h"<br>+#include "llvm/IR/PatternMatch.h"<br>+#include "llvm/Support/Allocator.h"<br>+#include "llvm/Support/CommandLine.h"<br>+#include "llvm/Support/Debug.h"<br>+#include "llvm/Target/TargetLibraryInfo.h"<br>+#include "llvm/Transforms/Utils/BasicBlockUtils.h"<br>+#include "llvm/Transforms/Utils/SSAUpdater.h"<br>+#include <vector><br>+using namespace llvm;<br>+<br>+#define DEBUG_TYPE "mldst-motion"<br>+<br>+//===----------------------------------------------------------------------===//<br>+// MergedLoadStoreMotion Pass<br>+//===----------------------------------------------------------------------===//<br>+static cl::opt<bool><br>+EnableMLSM("mlsm", cl::desc("Enable motion of merged load and store"),<br>+ cl::init(true));<br></blockquote><br>Command-line option should be in the PassManagerBuilder. See additional<br>comments below.<br><br><blockquote type="cite">+<br>+namespace {<br>+class MergedLoadStoreMotion : public FunctionPass {<br>+ AliasAnalysis *AA;<br>+ MemoryDependenceAnalysis *MD;<br>+<br>+public:<br>+ static char ID; // Pass identification, replacement for typeid<br>+ explicit MergedLoadStoreMotion(void) : FunctionPass(ID), MD(nullptr) {<br>+<br>initializeMergedLoadStoreMotionPass(*PassRegistry::getPassRegistry());<br>+ }<br>+<br>+ bool runOnFunction(Function &F) override;<br>+<br>+private:<br>+ // This transformation requires dominator postdominator info<br>+ void getAnalysisUsage(AnalysisUsage &AU) const override {<br>+ AU.addRequired<TargetLibraryInfo>();<br>+ AU.addRequired<MemoryDependenceAnalysis>();<br>+ AU.addRequired<AliasAnalysis>();<br>+ AU.addPreserved<AliasAnalysis>();<br>+ }<br>+<br>+ // Helper routines<br>+<br>+ ///<br>+ /// \brief Remove instruction from parent and update memory dependence<br>+ /// analysis.<br>+ ///<br>+ void removeInstruction(Instruction *Inst);<br>+ BasicBlock *getDiamondTail(BasicBlock *BB);<br>+ bool isDiamondHead(BasicBlock *BB);<br>+ // Routines for hoisting loads<br>+ bool isLoadHoistBarrier(Instruction *Inst);<br>+ LoadInst *canHoistFromBlock(BasicBlock *BB, LoadInst *LI);<br>+ void hoistInstruction(BasicBlock *BB, Instruction *HoistCand,<br>+ Instruction *ElseInst);<br>+ bool isSafeToHoist(Instruction *I) const;<br>+ bool hoistLoad(BasicBlock *BB, LoadInst *HoistCand, LoadInst<br>*ElseInst);<br>+ bool mergeLoads(BasicBlock *BB);<br>+ // Routines for sinking stores<br>+ StoreInst *canSinkFromBlock(BasicBlock *BB, StoreInst *SI);<br>+ PHINode *getPHIOperand(BasicBlock *BB, StoreInst *S0, StoreInst *S1);<br>+ bool isStoreSinkBarrier(Instruction *Inst);<br>+ bool sinkStore(BasicBlock *BB, StoreInst *SinkCand, StoreInst<br>*ElseInst);<br>+ bool mergeStores(BasicBlock *BB);<br>+ // The mergeLoad/Store algorithms could have Size0 * Size1 complexity,<br>+ // where Size0 and Size1 are the #instructions on the two sides of<br>+ // the diamond. The constant chosen here is arbitrary. Compiler Time<br>+ // Control is enforced by the check Size0 * Size1 <<br>MagicCompileTimeControl.<br>+ const int MagicCompileTimeControl = 250;<br>+};<br>+<br>+char MergedLoadStoreMotion::ID = 0;<br>+}<br>+<br>+///<br>+/// \brief createMergedLoadStoreMotionPass - The public interface to this<br>file.<br>+///<br>+FunctionPass *llvm::createMergedLoadStoreMotionPass() {<br>+ return new MergedLoadStoreMotion();<br>+}<br>+<br>+INITIALIZE_PASS_BEGIN(MergedLoadStoreMotion, "mldst-motion",<br>+ "MergedLoadStoreMotion", false, false)<br>+INITIALIZE_PASS_DEPENDENCY(MemoryDependenceAnalysis)<br>+INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfo)<br>+INITIALIZE_AG_DEPENDENCY(AliasAnalysis)<br>+INITIALIZE_PASS_END(MergedLoadStoreMotion, "mldst-motion",<br>+ "MergedLoadStoreMotion", false, false)<br>+<br>+///<br>+/// \brief Remove instruction from parent and update memory dependence<br>analysis.<br>+///<br>+void MergedLoadStoreMotion::removeInstruction(Instruction *Inst) {<br>+ // Notify the memory dependence analysis.<br>+ if (MD) {<br>+ MD->removeInstruction(Inst);<br>+ if (LoadInst *LI = dyn_cast<LoadInst>(Inst))<br>+ MD->invalidateCachedPointerInfo(LI->getPointerOperand());<br>+ if (Inst->getType()->getScalarType()->isPointerTy()) {<br>+ MD->invalidateCachedPointerInfo(Inst);<br>+ }<br>+ }<br>+ Inst->eraseFromParent();<br>+}<br>+<br>+///<br>+/// \brief Return tail block of a diamond.<br>+///<br>+BasicBlock *MergedLoadStoreMotion::getDiamondTail(BasicBlock *BB) {<br>+ assert(isDiamondHead(BB) && "Basic block is not head of a diamond");<br>+ BranchInst *BI = (BranchInst *)(BB->getTerminator());<br>+ BasicBlock *Succ0 = BI->getSuccessor(0);<br>+ BasicBlock *Tail = Succ0->getTerminator()->getSuccessor(0);<br>+ return Tail;<br>+}<br>+<br>+///<br>+/// \brief True when BB is the head of a diamond (hammock)<br>+///<br>+bool MergedLoadStoreMotion::isDiamondHead(BasicBlock *BB) {<br>+ if (!BB)<br>+ return false;<br>+ if (!isa<BranchInst>(BB->getTerminator()))<br>+ return false;<br>+ if (BB->getTerminator()->getNumSuccessors() != 2)<br>+ return false;<br>+<br>+ BranchInst *BI = (BranchInst *)(BB->getTerminator());<br>+ BasicBlock *Succ0 = BI->getSuccessor(0);<br>+ BasicBlock *Succ1 = BI->getSuccessor(1);<br>+<br>+ if (!Succ0->getSinglePredecessor() ||<br>+ Succ0->getTerminator()->getNumSuccessors() != 1)<br>+ return false;<br>+ if (!Succ1->getSinglePredecessor() ||<br>+ Succ1->getTerminator()->getNumSuccessors() != 1)<br>+ return false;<br>+<br>+ BasicBlock *Tail = Succ0->getTerminator()->getSuccessor(0);<br>+ // Ignore triangles.<br>+ if (Succ1->getTerminator()->getSuccessor(0) != Tail)<br>+ return false;<br>+ return true;<br>+}<br>+<br>+///<br>+/// \brief True when instruction is a hoist barrier for a load<br>+///<br>+/// Whenever an instruction could possibly modify the value<br>+/// being loaded or protect against the load from happening<br>+/// it is considered a hoist barrier.<br>+///<br>+bool MergedLoadStoreMotion::isLoadHoistBarrier(Instruction *Inst) {<br>+ // FIXME: A call with no side effects should not be a barrier.<br>+ // Aren't all such calls covered by mayHaveSideEffects() below?<br>+ // Then this check can be removed.<br>+ if (isa<CallInst>(Inst))<br>+ return true;<br>+ if (isa<TerminatorInst>(Inst))<br>+ return true;<br>+ // Note: mayHaveSideEffects covers all instructions that could<br>+ // trigger a change to state. Eg. in-flight stores have to be executed<br>+ // before ordered loads or fences, calls could invoke functions that<br>store<br>+ // data to memory etc.<br>+ if (Inst->mayHaveSideEffects()) {<br>+ return true;<br>+ }<br>+ DEBUG(dbgs() << "No Hoist Barrier\n");<br>+ return false;<br>+}<br>+<br>+///<br>+/// \brief Decide if a load can be hoisted<br>+///<br>+/// When there is a load in \p BB to the same address as \p LI<br>+/// and it can be hoisted from \p BB, return that load.<br>+/// Otherwise return Null.<br>+///<br>+LoadInst *MergedLoadStoreMotion::canHoistFromBlock(BasicBlock *BB,<br>+ LoadInst *LI) {<br>+ LoadInst *I = nullptr;<br>+ assert(isa<LoadInst>(LI));<br>+ if (LI->isUsedOutsideOfBlock(LI->getParent()))<br>+ return nullptr;<br>+<br>+ for (BasicBlock::iterator BBI = BB->begin(), BBE = BB->end(); BBI !=<br>BBE;<br>+ ++BBI) {<br>+ Instruction *Inst = BBI;<br>+<br>+ // Only merge and hoist loads when their result in used only in BB<br>+ if (isLoadHoistBarrier(Inst))<br>+ break;<br>+ if (!isa<LoadInst>(Inst))<br>+ continue;<br>+ if (Inst->isUsedOutsideOfBlock(Inst->getParent()))<br>+ continue;<br>+<br>+ AliasAnalysis::Location LocLI = AA->getLocation(LI);<br>+ AliasAnalysis::Location LocInst = AA->getLocation((LoadInst *)Inst);<br>+ if (AA->isMustAlias(LocLI, LocInst) && LI->getType() ==<br>Inst->getType()) {<br>+ I = (LoadInst *)Inst;<br>+ break;<br>+ }<br>+ }<br>+ return I;<br>+}<br>+<br>+///<br>+/// \brief Merge two equivalent instructions \p HoistCand and \p ElseInst<br>into<br>+/// \p BB<br>+///<br>+/// BB is the head of a diamond<br>+///<br>+void MergedLoadStoreMotion::hoistInstruction(BasicBlock *BB,<br>+ Instruction *HoistCand,<br>+ Instruction *ElseInst) {<br>+ DEBUG(dbgs() << " Hoist Instruction into BB \n"; BB->dump();<br>+ dbgs() << "Instruction Left\n"; HoistCand->dump(); dbgs() <<<br>"\n";<br>+ dbgs() << "Instruction Right\n"; ElseInst->dump(); dbgs() <<<br>"\n");<br>+ // Hoist the instruction.<br>+ assert(HoistCand->getParent() != BB);<br>+<br>+ // Intersect optional metadata.<br>+ HoistCand->intersectOptionalDataWith(ElseInst);<br>+ HoistCand->dropUnknownMetadata();<br>+<br>+ // Prepend point for instruction insert<br>+ Instruction *HoistPt = BB->getTerminator();<br>+<br>+ // Merged instruction<br>+ Instruction *HoistedInst = HoistCand->clone();<br>+<br>+ // Notify AA of the new value.<br>+ if (isa<LoadInst>(HoistCand))<br>+ AA->copyValue(HoistCand, HoistedInst);<br>+<br>+ // Hoist instruction.<br>+ HoistedInst->insertBefore(HoistPt);<br>+<br>+ HoistCand->replaceAllUsesWith(HoistedInst);<br>+ removeInstruction(HoistCand);<br>+ // Replace the else block instruction.<br>+ ElseInst->replaceAllUsesWith(HoistedInst);<br>+ removeInstruction(ElseInst);<br>+}<br>+<br>+///<br>+/// \brief Return true if no operand of \p I is defined in I's parent<br>block<br>+///<br>+bool MergedLoadStoreMotion::isSafeToHoist(Instruction *I) const {<br>+ BasicBlock *Parent = I->getParent();<br>+ for (unsigned i = 0, e = I->getNumOperands(); i != e; ++i) {<br>+ Instruction *Instr = dyn_cast<Instruction>(I->getOperand(i));<br>+ if (Instr && Instr->getParent() == Parent)<br>+ return false;<br>+ }<br>+ return true;<br>+}<br>+<br>+///<br>+/// \brief Merge two equivalent loads and GEPs and hoist into diamond<br>head<br>+///<br>+bool MergedLoadStoreMotion::hoistLoad(BasicBlock *BB, LoadInst *L0,<br>+ LoadInst *L1) {<br>+ // Only one definition?<br>+ Instruction *A0 = dyn_cast<Instruction>(L0->getPointerOperand());<br>+ Instruction *A1 = dyn_cast<Instruction>(L1->getPointerOperand());<br>+ if (A0 && A1 && A0->isIdenticalTo(A1) && isSafeToHoist(A0) &&<br>+ A0->hasOneUse() && (A0->getParent() == L0->getParent()) &&<br>+ A1->hasOneUse() && (A1->getParent() == L1->getParent()) &&<br>+ isa<GetElementPtrInst>(A0)) {<br>+ DEBUG(dbgs() << "Hoist Instruction into BB \n"; BB->dump();<br>+ dbgs() << "Instruction Left\n"; L0->dump(); dbgs() << "\n";<br>+ dbgs() << "Instruction Right\n"; L1->dump(); dbgs() << "\n");<br>+ hoistInstruction(BB, A0, A1);<br>+ hoistInstruction(BB, L0, L1);<br>+ return true;<br>+ } else<br>+ return false;<br>+}<br>+<br>+///<br>+/// \brief Try to hoist two loads to same address into diamond header<br>+///<br>+/// Starting from a diamond head block, iterate over the instructions in<br>one<br>+/// successor block and try to match a load in the second successor.<br>+///<br>+bool MergedLoadStoreMotion::mergeLoads(BasicBlock *BB) {<br>+ bool MergedLoads = false;<br>+ assert(isDiamondHead(BB));<br>+ BranchInst *BI = dyn_cast<BranchInst>(BB->getTerminator());<br>+ BasicBlock *Succ0 = BI->getSuccessor(0);<br>+ BasicBlock *Succ1 = BI->getSuccessor(1);<br>+ // #Instructions in Succ1 for Compile Time Control<br>+ int Size1 = Succ1->size();<br>+ int NLoads = 0;<br>+ for (BasicBlock::iterator BBI = Succ0->begin(), BBE = Succ0->end();<br>+ BBI != BBE;) {<br>+<br>+ Instruction *I = BBI;<br>+ ++BBI;<br>+ if (isLoadHoistBarrier(I))<br>+ break;<br>+<br>+ // Only move non-simple (atomic, volatile) loads.<br>+ if (!isa<LoadInst>(I))<br>+ continue;<br>+<br>+ LoadInst *L0 = (LoadInst *)I;<br>+ if (!L0->isSimple())<br>+ continue;<br>+<br>+ ++NLoads;<br>+ if (NLoads * Size1 >= MagicCompileTimeControl)<br>+ break;<br>+ if (LoadInst *L1 = canHoistFromBlock(Succ1, L0)) {<br>+ bool Res = hoistLoad(BB, L0, L1);<br>+ MergedLoads |= Res;<br>+ // Don't attempt to hoist above loads that had not been hoisted.<br>+ if (!Res)<br>+ break;<br>+ }<br>+ }<br>+ return MergedLoads;<br>+}<br>+<br>+///<br>+/// \brief True when instruction is sink barrier for a store<br>+///<br>+bool MergedLoadStoreMotion::isStoreSinkBarrier(Instruction *Inst) {<br>+ if (isa<CallInst>(Inst))<br>+ return true;<br>+ if (isa<TerminatorInst>(Inst) && !isa<BranchInst>(Inst))<br>+ return true;<br>+ // Note: mayHaveSideEffects covers all instructions that could<br>+ // trigger a change to state. Eg. in-flight stores have to be executed<br>+ // before ordered loads or fences, calls could invoke functions that<br>store<br>+ // data to memory etc.<br>+ if (!isa<StoreInst>(Inst) && Inst->mayHaveSideEffects()) {<br>+ return true;<br>+ }<br>+ DEBUG(dbgs() << "No Sink Barrier\n");<br>+ return false;<br>+}<br>+<br>+///<br>+/// \brief Check if \p BB contains a store to the same address as \p SI<br>+///<br>+/// \return The store in \p when it is safe to sink. Otherwise return<br>Null.<br>+///<br>+StoreInst *MergedLoadStoreMotion::canSinkFromBlock(BasicBlock *BB,<br>+ StoreInst *SI) {<br>+ StoreInst *I = 0;<br>+ DEBUG(dbgs() << "can Sink? : "; SI->dump(); dbgs() << "\n");<br>+ for (BasicBlock::reverse_iterator RBI = BB->rbegin(), RBE = BB->rend();<br>+ RBI != RBE; ++RBI) {<br>+ Instruction *Inst = &*RBI;<br>+<br>+ // Only move loads if they are used in the block.<br>+ if (isStoreSinkBarrier(Inst))<br>+ break;<br>+ if (isa<StoreInst>(Inst)) {<br>+ AliasAnalysis::Location LocSI = AA->getLocation(SI);<br>+ AliasAnalysis::Location LocInst = AA->getLocation((StoreInst<br>*)Inst);<br>+ if (AA->isMustAlias(LocSI, LocInst)) {<br>+ I = (StoreInst *)Inst;<br>+ break;<br>+ }<br>+ }<br>+ }<br>+ return I;<br>+}<br>+<br>+///<br>+/// \brief Create a PHI node in BB for the operands of S0 and S1<br>+///<br>+PHINode *MergedLoadStoreMotion::getPHIOperand(BasicBlock *BB, StoreInst<br>*S0,<br>+ StoreInst *S1) {<br>+ // Create a phi if the values mismatch.<br>+ PHINode *NewPN = 0;<br>+ Value *Opd1 = S0->getValueOperand();<br>+ Value *Opd2 = S1->getValueOperand();<br>+ if (Opd1 != Opd2) {<br>+ NewPN = PHINode::Create(Opd1->getType(), 2, Opd2->getName() +<br>".sink",<br>+ BB->begin());<br>+ NewPN->addIncoming(Opd1, S0->getParent());<br>+ NewPN->addIncoming(Opd2, S1->getParent());<br>+ if (NewPN->getType()->getScalarType()->isPointerTy()) {<br>+ // Notify AA of the new value.<br>+ AA->copyValue(Opd1, NewPN);<br>+ AA->copyValue(Opd2, NewPN);<br>+ // AA needs to be informed when a PHI-use of the pointer value is<br>added<br>+ for (unsigned I = 0, E = NewPN->getNumIncomingValues(); I != E;<br>++I) {<br>+ unsigned J = PHINode::getOperandNumForIncomingValue(I);<br>+ AA->addEscapingUse(NewPN->getOperandUse(J));<br>+ }<br>+ if (MD)<br>+ MD->invalidateCachedPointerInfo(NewPN);<br>+ }<br>+ }<br>+ return NewPN;<br>+}<br>+<br>+///<br>+/// \brief Merge two stores to same address and sink into \p BB<br>+///<br>+/// Also sinks GEP instruction computing the store address<br>+///<br>+bool MergedLoadStoreMotion::sinkStore(BasicBlock *BB, StoreInst *S0,<br>+ StoreInst *S1) {<br>+ // Only one definition?<br>+ Instruction *A0 = dyn_cast<Instruction>(S0->getPointerOperand());<br>+ Instruction *A1 = dyn_cast<Instruction>(S1->getPointerOperand());<br>+ if (A0 && A1 && A0->isIdenticalTo(A1) && A0->hasOneUse() &&<br>+ (A0->getParent() == S0->getParent()) && A1->hasOneUse() &&<br>+ (A1->getParent() == S1->getParent()) && isa<GetElementPtrInst>(A0))<br>{<br>+ DEBUG(dbgs() << "Sink Instruction into BB \n"; BB->dump();<br>+ dbgs() << "Instruction Left\n"; S0->dump(); dbgs() << "\n";<br>+ dbgs() << "Instruction Right\n"; S1->dump(); dbgs() << "\n");<br>+ // Hoist the instruction.<br>+ BasicBlock::iterator InsertPt = BB->getFirstInsertionPt();<br>+ // Intersect optional metadata.<br>+ S0->intersectOptionalDataWith(S1);<br>+ S0->dropUnknownMetadata();<br>+<br>+ // Create the new store to be inserted at the join point.<br>+ StoreInst *SNew = (StoreInst *)(S0->clone());<br>+ Instruction *ANew = A0->clone();<br>+ AA->copyValue(S0, SNew);<br>+ SNew->insertBefore(InsertPt);<br>+ ANew->insertBefore(SNew);<br>+<br>+ assert(S0->getParent() == A0->getParent());<br>+ assert(S1->getParent() == A1->getParent());<br>+<br>+ PHINode *NewPN = getPHIOperand(BB, S0, S1);<br>+ // New PHI operand? Use it.<br>+ if (NewPN)<br>+ SNew->setOperand(0, NewPN);<br>+ removeInstruction(S0);<br>+ removeInstruction(S1);<br>+ A0->replaceAllUsesWith(ANew);<br>+ removeInstruction(A0);<br>+ A1->replaceAllUsesWith(ANew);<br>+ removeInstruction(A1);<br>+ return true;<br>+ }<br>+ return false;<br>+}<br>+<br>+///<br>+/// \brief True when two stores are equivalent and can sink into the<br>footer<br>+///<br>+/// Starting from a diamond tail block, iterate over the instructions in<br>one<br>+/// predecessor block and try to match a store in the second predecessor.<br>+///<br>+bool MergedLoadStoreMotion::mergeStores(BasicBlock *T) {<br>+<br>+ bool MergedStores = false;<br>+ assert(T && "Footer of a diamond cannot be empty");<br>+<br>+ pred_iterator PI = pred_begin(T), E = pred_end(T);<br>+ assert(PI != E);<br>+ BasicBlock *Pred0 = *PI;<br>+ ++PI;<br>+ BasicBlock *Pred1 = *PI;<br>+ ++PI;<br>+ // tail block of a diamond/hammock?<br>+ if (Pred0 == Pred1)<br>+ return false; // No.<br>+ if (PI != E)<br>+ return false; // No. More than 2 predecessors.<br>+<br>+ // #Instructions in Succ1 for Compile Time Control<br>+ int Size1 = Pred1->size();<br>+ int NStores = 0;<br>+<br>+ for (BasicBlock::reverse_iterator RBI = Pred0->rbegin(), RBE =<br>Pred0->rend();<br>+ RBI != RBE;) {<br>+<br>+ Instruction *I = &*RBI;<br>+ ++RBI;<br>+ if (isStoreSinkBarrier(I))<br>+ break;<br>+ // Sink move non-simple (atomic, volatile) stores<br>+ if (!isa<StoreInst>(I))<br>+ continue;<br>+ StoreInst *S0 = (StoreInst *)I;<br>+ if (!S0->isSimple())<br>+ continue;<br>+<br>+ ++NStores;<br>+ if (NStores * Size1 >= MagicCompileTimeControl)<br>+ break;<br>+ if (StoreInst *S1 = canSinkFromBlock(Pred1, S0)) {<br>+ bool Res = sinkStore(T, S0, S1);<br>+ MergedStores |= Res;<br>+ // Don't attempt to sink below stores that had to stick around<br>+ // But after removal of a store and some of its feeding<br>+ // instruction search again from the beginning since the iterator<br>+ // is likely stale at this point.<br>+ if (!Res)<br>+ break;<br>+ else {<br>+ RBI = Pred0->rbegin();<br>+ RBE = Pred0->rend();<br>+ DEBUG(dbgs() << "Search again\n"; Instruction *I = &*RBI;<br>I->dump());<br>+ }<br>+ }<br>+ }<br>+ return MergedStores;<br>+}<br>+///<br>+/// \brief Run the transformation for each function<br>+///<br>+bool MergedLoadStoreMotion::runOnFunction(Function &F) {<br>+ MD = &getAnalysis<MemoryDependenceAnalysis>();<br>+ AA = &getAnalysis<AliasAnalysis>();<br>+<br>+ bool Changed = false;<br>+ if (!EnableMLSM)<br>+ return false;<br></blockquote><br>This check should be in the PassManagerBuilder. It avoids generating the<br>analysis when the pass is disabled as well as produces the expected output<br>when -mllvm -debug-pass=Structure is run.<br><br><blockquote type="cite">+ DEBUG(dbgs() << "Instruction Merger\n");<br>+<br>+ // Merge unconditional branches, allowing PRE to catch more<br>+ // optimization opportunities.<br>+ for (Function::iterator FI = F.begin(), FE = F.end(); FI != FE;) {<br>+ BasicBlock *BB = FI++;<br>+<br>+ // Hoist equivalent loads and sink stores<br>+ // outside diamonds when possible<br>+ // Run outside core GVN<br>+ if (isDiamondHead(BB)) {<br>+ Changed |= mergeLoads(BB);<br>+ Changed |= mergeStores(getDiamondTail(BB));<br>+ }<br>+ }<br>+ return Changed;<br>+}<br><br>Modified: llvm/trunk/lib/Transforms/Scalar/Scalar.cpp<br>URL:<br><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Scalar/Scalar.cpp?rev=213396&r1=213395&r2=213396&view=diff">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Scalar/Scalar.cpp?rev=213396&r1=213395&r2=213396&view=diff</a><br>==============================================================================<br>--- llvm/trunk/lib/Transforms/Scalar/Scalar.cpp (original)<br>+++ llvm/trunk/lib/Transforms/Scalar/Scalar.cpp Fri Jul 18 14:13:09 2014<br>@@ -52,6 +52,7 @@ void llvm::initializeScalarOpts(PassRegi<br> initializeLowerAtomicPass(Registry);<br> initializeLowerExpectIntrinsicPass(Registry);<br> initializeMemCpyOptPass(Registry);<br>+ initializeMergedLoadStoreMotionPass(Registry);<br> initializePartiallyInlineLibCallsPass(Registry);<br> initializeReassociatePass(Registry);<br> initializeRegToMemPass(Registry);<br>@@ -92,6 +93,10 @@ void LLVMAddGVNPass(LLVMPassManagerRef P<br> unwrap(PM)->add(createGVNPass());<br>}<br><br>+void LLVMAddMergedLoadStoreMotionPass(LLVMPassManagerRef PM) {<br>+ unwrap(PM)->add(createMergedLoadStoreMotionPass());<br>+}<br>+<br>void LLVMAddIndVarSimplifyPass(LLVMPassManagerRef PM) {<br> unwrap(PM)->add(createIndVarSimplifyPass());<br>}<br><br>Added: llvm/trunk/test/Transforms/InstMerge/ld_hoist_st_sink.ll<br>URL:<br>http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/InstMerge/ld_hoist_st_sink.ll?rev=213396&view=auto<br>==============================================================================<br>--- llvm/trunk/test/Transforms/InstMerge/ld_hoist_st_sink.ll (added)<br>+++ llvm/trunk/test/Transforms/InstMerge/ld_hoist_st_sink.ll Fri Jul 18<br>14:13:09 2014<br>@@ -0,0 +1,84 @@<br>+; Tests to make sure that loads and stores in a diamond get merged<br>+; Loads are hoisted into the header. Stores sunks into the footer.<br>+; RUN: opt -basicaa -memdep -mldst-motion -S < %s | FileCheck %s<br>+target datalayout = "e-m:o-i64:64-i128:128-n32:64-S128"<br>+<br>+%struct.node = type { i64, %struct.node*, %struct.node*, %struct.node*,<br>i64, %struct.arc*, i64, i64, i64 }<br>+%struct.arc = type { i64, i64, i64 }<br>+<br>+define i64 @foo(%struct.node* nocapture readonly %r) nounwind {<br>+entry:<br>+ %node.0.in16 = getelementptr inbounds %struct.node* %r, i64 0, i32 2<br>+ %node.017 = load %struct.node** %node.0.in16, align 8<br>+ %tobool18 = icmp eq %struct.node* %node.017, null<br>+ br i1 %tobool18, label %while.end, label %while.body.preheader<br>+<br>+; CHECK-LABEL: while.body.preheader<br>+while.body.preheader: ; preds = %entry<br>+; CHECK: load<br>+ br label %while.body<br>+<br>+while.body: ; preds =<br>%while.body.preheader, %if.end<br>+ %node.020 = phi %struct.node* [ %node.0, %if.end ], [ %node.017,<br>%while.body.preheader ]<br>+ %sum.019 = phi i64 [ %inc, %if.end ], [ 0, %while.body.preheader ]<br>+ %orientation = getelementptr inbounds %struct.node* %node.020, i64 0,<br>i32 4<br>+ %0 = load i64* %orientation, align 8<br>+ %cmp = icmp eq i64 %0, 1<br>+ br i1 %cmp, label %if.then, label %if.else<br>+; CHECK: if.then<br>+if.then: ; preds = %while.body<br>+ %a = getelementptr inbounds %struct.node* %node.020, i64 0, i32 5<br>+; CHECK-NOT: load %struct.arc<br>+ %1 = load %struct.arc** %a, align 8<br>+ %cost = getelementptr inbounds %struct.arc* %1, i64 0, i32 0<br>+; CHECK-NOT: load i64*<br>+ %2 = load i64* %cost, align 8<br>+ %pred = getelementptr inbounds %struct.node* %node.020, i64 0, i32 1<br>+; CHECK-NOT: load %struct.node**<br>+ %3 = load %struct.node** %pred, align 8<br>+ %p = getelementptr inbounds %struct.node* %3, i64 0, i32 6<br>+; CHECK-NOT: load i64*<br>+ %4 = load i64* %p, align 8<br>+ %add = add nsw i64 %4, %2<br>+ %p1 = getelementptr inbounds %struct.node* %node.020, i64 0, i32 6<br>+; CHECK-NOT: store i64<br>+ store i64 %add, i64* %p1, align 8<br>+ br label %if.end<br>+<br>+; CHECK: if.else<br>+if.else: ; preds = %while.body<br>+ %pred2 = getelementptr inbounds %struct.node* %node.020, i64 0, i32 1<br>+; CHECK-NOT: load %struct.node**<br>+ %5 = load %struct.node** %pred2, align 8<br>+ %p3 = getelementptr inbounds %struct.node* %5, i64 0, i32 6<br>+; CHECK-NOT: load i64*<br>+ %6 = load i64* %p3, align 8<br>+ %a4 = getelementptr inbounds %struct.node* %node.020, i64 0, i32 5<br>+; CHECK-NOT: load %struct.arc**<br>+ %7 = load %struct.arc** %a4, align 8<br>+ %cost5 = getelementptr inbounds %struct.arc* %7, i64 0, i32 0<br>+; CHECK-NOT: load i64*<br>+ %8 = load i64* %cost5, align 8<br>+ %sub = sub nsw i64 %6, %8<br>+ %p6 = getelementptr inbounds %struct.node* %node.020, i64 0, i32 6<br>+; CHECK-NOT: store i64<br>+ store i64 %sub, i64* %p6, align 8<br>+ br label %if.end<br>+<br>+; CHECK: if.end<br>+if.end: ; preds = %if.else,<br>%if.then<br>+; CHECK: store<br>+ %inc = add nsw i64 %sum.019, 1<br>+ %node.0.in = getelementptr inbounds %struct.node* %node.020, i64 0, i32<br>2<br>+ %node.0 = load %struct.node** %node.0.in, align 8<br>+ %tobool = icmp eq %struct.node* %node.0, null<br>+ br i1 %tobool, label %while.end.loopexit, label %while.body<br>+<br>+while.end.loopexit: ; preds = %if.end<br>+ %inc.lcssa = phi i64 [ %inc, %if.end ]<br>+ br label %while.end<br>+<br>+while.end: ; preds =<br>%while.end.loopexit, %entry<br>+ %sum.0.lcssa = phi i64 [ 0, %entry ], [ %inc.lcssa, %while.end.loopexit<br>]<br>+ ret i64 %sum.0.lcssa<br>+}<br><br><br>_______________________________________________<br>llvm-commits mailing list<br>llvm-commits@cs.uiuc.edu<br>http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</blockquote></div></blockquote></div><br></div></body></html>