<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div apple-content-edited="true">Hi Juergen,
</div><div apple-content-edited="true"><br></div><div apple-content-edited="true">Sorry for the late comments on this.</div><div apple-content-edited="true">1. I have one comment on the added test case, see inline comments.</div><div apple-content-edited="true">2. When you create a displacement from one constant to another, you are creating data dependencies. I know that you did not measure any performance impact for this pass, but is the scheduler able to break these dependency it is increase the critical path?</div><div apple-content-edited="true">What I have in mind in something like this:</div><div apple-content-edited="true"><br></div><div apple-content-edited="true">Before:</div><div apple-content-edited="true">op1 BigCst</div><div apple-content-edited="true">op2 BigCst+MaxDisplacement</div><div apple-content-edited="true">op3 BigCst+2*MaxDisplacement</div><div apple-content-edited="true">=></div><div apple-content-edited="true"><br></div><div apple-content-edited="true">After:</div><div apple-content-edited="true">regCst = bitcast BigCst</div><div apple-content-edited="true">op1 regCst</div><div apple-content-edited="true">regCst2 = add regCst, MaxDisplacement</div><div apple-content-edited="true">op2 regCst2</div><div apple-content-edited="true"><div apple-content-edited="true">regCst3 = add regCst2, MaxDisplacement</div><div apple-content-edited="true">op3 regCst3</div><div><br></div><div>Before has 3 independent computations.</div><div>After has one big chain of computation.</div><div><br></div><div>That said, I believe that wouldn’t happen often, just curious :).</div><div><br></div></div><div apple-content-edited="true">Thanks,</div><div apple-content-edited="true">-Quentin</div><div apple-content-edited="true"><br></div>
<br><div><div>On Jan 24, 2014, at 10:23 AM, Juergen Ributzka <<a href="mailto:juergen@apple.com">juergen@apple.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">Author: ributzka<br>Date: Fri Jan 24 12:23:08 2014<br>New Revision: 200022<br><br>URL: <a href="http://llvm.org/viewvc/llvm-project?rev=200022&view=rev">http://llvm.org/viewvc/llvm-project?rev=200022&view=rev</a><br>Log:<br>Add Constant Hoisting Pass<br><br>This pass identifies expensive constants to hoist and coalesces them to<br>better prepare it for SelectionDAG-based code generation. This works around the<br>limitations of the basic-block-at-a-time approach.<br><br>First it scans all instructions for integer constants and calculates its<br>cost. If the constant can be folded into the instruction (the cost is<br>TCC_Free) or the cost is just a simple operation (TCC_BASIC), then we don't<br>consider it expensive and leave it alone. This is the default behavior and<br>the default implementation of getIntImmCost will always return TCC_Free.<br><br>If the cost is more than TCC_BASIC, then the integer constant can't be folded<br>into the instruction and it might be beneficial to hoist the constant.<br>Similar constants are coalesced to reduce register pressure and<br>materialization code.<br><br>When a constant is hoisted, it is also hidden behind a bitcast to force it to<br>be live-out of the basic block. Otherwise the constant would be just<br>duplicated and each basic block would have its own copy in the SelectionDAG.<br>The SelectionDAG recognizes such constants as opaque and doesn't perform<br>certain transformations on them, which would create a new expensive constant.<br><br>This optimization is only applied to integer constants in instructions and<br>simple (this means not nested) constant cast experessions. For example:<br>%0 = load i64* inttoptr (i64 big_constant to i64*)<br><br>Reviewed by Eric<br><br>Added:<br> llvm/trunk/lib/Transforms/Scalar/ConstantHoisting.cpp<br> llvm/trunk/test/CodeGen/X86/large-constants.ll<br>Modified:<br> llvm/trunk/include/llvm/Analysis/TargetTransformInfo.h<br> llvm/trunk/include/llvm/CodeGen/SelectionDAG.h<br> llvm/trunk/include/llvm/CodeGen/SelectionDAGNodes.h<br> llvm/trunk/include/llvm/InitializePasses.h<br> llvm/trunk/include/llvm/LinkAllPasses.h<br> llvm/trunk/include/llvm/Transforms/Scalar.h<br> llvm/trunk/lib/Analysis/TargetTransformInfo.cpp<br> llvm/trunk/lib/CodeGen/Passes.cpp<br> llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp<br> llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAG.cpp<br> llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp<br> llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp<br> llvm/trunk/lib/CodeGen/SelectionDAG/TargetLowering.cpp<br> llvm/trunk/lib/Target/X86/X86TargetTransformInfo.cpp<br> llvm/trunk/lib/Transforms/Scalar/CMakeLists.txt<br> llvm/trunk/lib/Transforms/Scalar/CodeGenPrepare.cpp<br> llvm/trunk/lib/Transforms/Scalar/Scalar.cpp<br> llvm/trunk/test/CodeGen/ARM/memcpy-inline.ll<br><br>Modified: llvm/trunk/include/llvm/Analysis/TargetTransformInfo.h<br>URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Analysis/TargetTransformInfo.h?rev=200022&r1=200021&r2=200022&view=diff">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Analysis/TargetTransformInfo.h?rev=200022&r1=200021&r2=200022&view=diff</a><br>==============================================================================<br>--- llvm/trunk/include/llvm/Analysis/TargetTransformInfo.h (original)<br>+++ llvm/trunk/include/llvm/Analysis/TargetTransformInfo.h Fri Jan 24 12:23:08 2014<br>@@ -92,6 +92,7 @@ public:<br> enum TargetCostConstants {<br> TCC_Free = 0, ///< Expected to fold away in lowering.<br> TCC_Basic = 1, ///< The cost of a typical 'add' instruction.<br>+ TCC_Load = 3,<br> TCC_Expensive = 4 ///< The cost of a 'div' instruction on x86.<br> };<br><br>@@ -299,6 +300,13 @@ public:<br> /// immediate of the specified type.<br> virtual unsigned getIntImmCost(const APInt &Imm, Type *Ty) const;<br><br>+ /// \brief Return the expected cost of materialization for the given integer<br>+ /// immediate of the specified type for a given instruction. The cost can be<br>+ /// zero if the immediate can be folded into the specified instruction.<br>+ virtual unsigned getIntImmCost(unsigned Opcode, const APInt &Imm,<br>+ Type *Ty) const;<br>+ virtual unsigned getIntImmCost(Intrinsic::ID IID, const APInt &Imm,<br>+ Type *Ty) const;<br> /// @}<br><br> /// \name Vector Target Information<br><br>Modified: llvm/trunk/include/llvm/CodeGen/SelectionDAG.h<br>URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/SelectionDAG.h?rev=200022&r1=200021&r2=200022&view=diff">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/SelectionDAG.h?rev=200022&r1=200021&r2=200022&view=diff</a><br>==============================================================================<br>--- llvm/trunk/include/llvm/CodeGen/SelectionDAG.h (original)<br>+++ llvm/trunk/include/llvm/CodeGen/SelectionDAG.h Fri Jan 24 12:23:08 2014<br>@@ -401,18 +401,22 @@ public:<br> //===--------------------------------------------------------------------===//<br> // Node creation methods.<br> //<br>- SDValue getConstant(uint64_t Val, EVT VT, bool isTarget = false);<br>- SDValue getConstant(const APInt &Val, EVT VT, bool isTarget = false);<br>- SDValue getConstant(const ConstantInt &Val, EVT VT, bool isTarget = false);<br>+ SDValue getConstant(uint64_t Val, EVT VT, bool isTarget = false,<br>+ bool isOpaque = false);<br>+ SDValue getConstant(const APInt &Val, EVT VT, bool isTarget = false,<br>+ bool isOpaque = false);<br>+ SDValue getConstant(const ConstantInt &Val, EVT VT, bool isTarget = false,<br>+ bool isOpaque = false);<br> SDValue getIntPtrConstant(uint64_t Val, bool isTarget = false);<br>- SDValue getTargetConstant(uint64_t Val, EVT VT) {<br>- return getConstant(Val, VT, true);<br>+ SDValue getTargetConstant(uint64_t Val, EVT VT, bool isOpaque = false) {<br>+ return getConstant(Val, VT, true, isOpaque);<br> }<br>- SDValue getTargetConstant(const APInt &Val, EVT VT) {<br>- return getConstant(Val, VT, true);<br>+ SDValue getTargetConstant(const APInt &Val, EVT VT, bool isOpaque = false) {<br>+ return getConstant(Val, VT, true, isOpaque);<br> }<br>- SDValue getTargetConstant(const ConstantInt &Val, EVT VT) {<br>- return getConstant(Val, VT, true);<br>+ SDValue getTargetConstant(const ConstantInt &Val, EVT VT,<br>+ bool isOpaque = false) {<br>+ return getConstant(Val, VT, true, isOpaque);<br> }<br> // The forms below that take a double should only be used for simple<br> // constants that can be exactly represented in VT. No checks are made.<br><br>Modified: llvm/trunk/include/llvm/CodeGen/SelectionDAGNodes.h<br>URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/SelectionDAGNodes.h?rev=200022&r1=200021&r2=200022&view=diff">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/SelectionDAGNodes.h?rev=200022&r1=200021&r2=200022&view=diff</a><br>==============================================================================<br>--- llvm/trunk/include/llvm/CodeGen/SelectionDAGNodes.h (original)<br>+++ llvm/trunk/include/llvm/CodeGen/SelectionDAGNodes.h Fri Jan 24 12:23:08 2014<br>@@ -1250,9 +1250,10 @@ public:<br> class ConstantSDNode : public SDNode {<br> const ConstantInt *Value;<br> friend class SelectionDAG;<br>- ConstantSDNode(bool isTarget, const ConstantInt *val, EVT VT)<br>+ ConstantSDNode(bool isTarget, bool isOpaque, const ConstantInt *val, EVT VT)<br> : SDNode(isTarget ? ISD::TargetConstant : ISD::Constant,<br> 0, DebugLoc(), getSDVTList(VT)), Value(val) {<br>+ SubclassData |= isOpaque;<br> }<br> public:<br><br>@@ -1265,6 +1266,8 @@ public:<br> bool isNullValue() const { return Value->isNullValue(); }<br> bool isAllOnesValue() const { return Value->isAllOnesValue(); }<br><br>+ bool isOpaque() const { return SubclassData & 1; }<br>+<br> static bool classof(const SDNode *N) {<br> return N->getOpcode() == ISD::Constant ||<br> N->getOpcode() == ISD::TargetConstant;<br><br>Modified: llvm/trunk/include/llvm/InitializePasses.h<br>URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/InitializePasses.h?rev=200022&r1=200021&r2=200022&view=diff">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/InitializePasses.h?rev=200022&r1=200021&r2=200022&view=diff</a><br>==============================================================================<br>--- llvm/trunk/include/llvm/InitializePasses.h (original)<br>+++ llvm/trunk/include/llvm/InitializePasses.h Fri Jan 24 12:23:08 2014<br>@@ -90,6 +90,7 @@ void initializeCFGSimplifyPassPass(PassR<br> void initializeFlattenCFGPassPass(PassRegistry&);<br> void initializeStructurizeCFGPass(PassRegistry&);<br> void initializeCFGViewerPass(PassRegistry&);<br>+void initializeConstantHoistingPass(PassRegistry&);<br> void initializeCodeGenPreparePass(PassRegistry&);<br> void initializeConstantMergePass(PassRegistry&);<br> void initializeConstantPropagationPass(PassRegistry&);<br><br>Modified: llvm/trunk/include/llvm/LinkAllPasses.h<br>URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/LinkAllPasses.h?rev=200022&r1=200021&r2=200022&view=diff">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/LinkAllPasses.h?rev=200022&r1=200021&r2=200022&view=diff</a><br>==============================================================================<br>--- llvm/trunk/include/llvm/LinkAllPasses.h (original)<br>+++ llvm/trunk/include/llvm/LinkAllPasses.h Fri Jan 24 12:23:08 2014<br>@@ -129,6 +129,7 @@ namespace {<br> (void) llvm::createJumpThreadingPass();<br> (void) llvm::createUnifyFunctionExitNodesPass();<br> (void) llvm::createInstCountPass();<br>+ (void) llvm::createConstantHoistingPass();<br> (void) llvm::createCodeGenPreparePass();<br> (void) llvm::createEarlyCSEPass();<br> (void) llvm::createGVNPass();<br><br>Modified: llvm/trunk/include/llvm/Transforms/Scalar.h<br>URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Transforms/Scalar.h?rev=200022&r1=200021&r2=200022&view=diff">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Transforms/Scalar.h?rev=200022&r1=200021&r2=200022&view=diff</a><br>==============================================================================<br>--- llvm/trunk/include/llvm/Transforms/Scalar.h (original)<br>+++ llvm/trunk/include/llvm/Transforms/Scalar.h Fri Jan 24 12:23:08 2014<br>@@ -312,6 +312,12 @@ Pass *createLoopDeletionPass();<br><br> //===----------------------------------------------------------------------===//<br> //<br>+// ConstantHoisting - This pass prepares a function for expensive constants.<br>+//<br>+FunctionPass *createConstantHoistingPass();<br>+<br>+//===----------------------------------------------------------------------===//<br>+//<br> // CodeGenPrepare - This pass prepares a function for instruction selection.<br> //<br> FunctionPass *createCodeGenPreparePass(const TargetMachine *TM = 0);<br><br>Modified: llvm/trunk/lib/Analysis/TargetTransformInfo.cpp<br>URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Analysis/TargetTransformInfo.cpp?rev=200022&r1=200021&r2=200022&view=diff">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Analysis/TargetTransformInfo.cpp?rev=200022&r1=200021&r2=200022&view=diff</a><br>==============================================================================<br>--- llvm/trunk/lib/Analysis/TargetTransformInfo.cpp (original)<br>+++ llvm/trunk/lib/Analysis/TargetTransformInfo.cpp Fri Jan 24 12:23:08 2014<br>@@ -158,6 +158,16 @@ unsigned TargetTransformInfo::getIntImmC<br> return PrevTTI->getIntImmCost(Imm, Ty);<br> }<br><br>+unsigned TargetTransformInfo::getIntImmCost(unsigned Opcode, const APInt &Imm,<br>+ Type *Ty) const {<br>+ return PrevTTI->getIntImmCost(Opcode, Imm, Ty);<br>+}<br>+<br>+unsigned TargetTransformInfo::getIntImmCost(Intrinsic::ID IID, const APInt &Imm,<br>+ Type *Ty) const {<br>+ return PrevTTI->getIntImmCost(IID, Imm, Ty);<br>+}<br>+<br> unsigned TargetTransformInfo::getNumberOfRegisters(bool Vector) const {<br> return PrevTTI->getNumberOfRegisters(Vector);<br> }<br>@@ -541,7 +551,17 @@ struct NoTTI LLVM_FINAL : ImmutablePass,<br> }<br><br> unsigned getIntImmCost(const APInt &Imm, Type *Ty) const LLVM_OVERRIDE {<br>- return 1;<br>+ return TCC_Basic;<br>+ }<br>+<br>+ unsigned getIntImmCost(unsigned Opcode, const APInt &Imm,<br>+ Type *Ty) const LLVM_OVERRIDE {<br>+ return TCC_Free;<br>+ }<br>+<br>+ unsigned getIntImmCost(Intrinsic::ID IID, const APInt &Imm,<br>+ Type *Ty) const LLVM_OVERRIDE {<br>+ return TCC_Free;<br> }<br><br> unsigned getNumberOfRegisters(bool Vector) const LLVM_OVERRIDE {<br><br>Modified: llvm/trunk/lib/CodeGen/Passes.cpp<br>URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/Passes.cpp?rev=200022&r1=200021&r2=200022&view=diff">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/Passes.cpp?rev=200022&r1=200021&r2=200022&view=diff</a><br>==============================================================================<br>--- llvm/trunk/lib/CodeGen/Passes.cpp (original)<br>+++ llvm/trunk/lib/CodeGen/Passes.cpp Fri Jan 24 12:23:08 2014<br>@@ -70,6 +70,8 @@ static cl::opt<bool> DisableMachineSink(<br> cl::desc("Disable Machine Sinking"));<br> static cl::opt<bool> DisableLSR("disable-lsr", cl::Hidden,<br> cl::desc("Disable Loop Strength Reduction Pass"));<br>+static cl::opt<bool> DisableConstantHoisting("disable-constant-hoisting",<br>+ cl::Hidden, cl::desc("Disable ConstantHoisting"));<br> static cl::opt<bool> DisableCGP("disable-cgp", cl::Hidden,<br> cl::desc("Disable Codegen Prepare"));<br> static cl::opt<bool> DisableCopyProp("disable-copyprop", cl::Hidden,<br>@@ -396,6 +398,10 @@ void TargetPassConfig::addIRPasses() {<br><br> // Make sure that no unreachable blocks are instruction selected.<br> addPass(createUnreachableBlockEliminationPass());<br>+<br>+ // Prepare expensive constants for SelectionDAG.<br>+ if (getOptLevel() != CodeGenOpt::None && !DisableConstantHoisting)<br>+ addPass(createConstantHoistingPass());<br> }<br><br> /// Turn exception handling constructs into something the code generators can<br><br>Modified: llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp<br>URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp?rev=200022&r1=200021&r2=200022&view=diff">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp?rev=200022&r1=200021&r2=200022&view=diff</a><br>==============================================================================<br>--- llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp (original)<br>+++ llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp Fri Jan 24 12:23:08 2014<br>@@ -3212,11 +3212,14 @@ SDValue DAGCombiner::visitOR(SDNode *N)<br> if (N1C && N0.getOpcode() == ISD::AND && N0.getNode()->hasOneUse() &&<br> isa<ConstantSDNode>(N0.getOperand(1))) {<br> ConstantSDNode *C1 = cast<ConstantSDNode>(N0.getOperand(1));<br>- if ((C1->getAPIntValue() & N1C->getAPIntValue()) != 0)<br>+ if ((C1->getAPIntValue() & N1C->getAPIntValue()) != 0) {<br>+ SDValue COR = DAG.FoldConstantArithmetic(ISD::OR, VT, N1C, C1);<br>+ if (!COR.getNode())<br>+ return SDValue();<br> return DAG.getNode(ISD::AND, SDLoc(N), VT,<br> DAG.getNode(ISD::OR, SDLoc(N0), VT,<br>- N0.getOperand(0), N1),<br>- DAG.FoldConstantArithmetic(ISD::OR, VT, N1C, C1));<br>+ N0.getOperand(0), N1), COR);<br>+ }<br> }<br> // fold (or (setcc x), (setcc y)) -> (setcc (or x, y))<br> if (isSetCCEquivalent(N0, LL, LR, CC0) && isSetCCEquivalent(N1, RL, RR, CC1)){<br><br>Modified: llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAG.cpp<br>URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAG.cpp?rev=200022&r1=200021&r2=200022&view=diff">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAG.cpp?rev=200022&r1=200021&r2=200022&view=diff</a><br>==============================================================================<br>--- llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAG.cpp (original)<br>+++ llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAG.cpp Fri Jan 24 12:23:08 2014<br>@@ -384,9 +384,12 @@ static void AddNodeIDCustom(FoldingSetNo<br> llvm_unreachable("Should only be used on nodes with operands");<br> default: break; // Normal nodes don't need extra info.<br> case ISD::TargetConstant:<br>- case ISD::Constant:<br>- ID.AddPointer(cast<ConstantSDNode>(N)->getConstantIntValue());<br>+ case ISD::Constant: {<br>+ const ConstantSDNode *C = cast<ConstantSDNode>(N);<br>+ ID.AddPointer(C->getConstantIntValue());<br>+ ID.AddBoolean(C->isOpaque());<br> break;<br>+ }<br> case ISD::TargetConstantFP:<br> case ISD::ConstantFP: {<br> ID.AddPointer(cast<ConstantFPSDNode>(N)->getConstantFPValue());<br>@@ -971,19 +974,21 @@ SDValue SelectionDAG::getNOT(SDLoc DL, S<br> return getNode(ISD::XOR, DL, VT, Val, NegOne);<br> }<br><br>-SDValue SelectionDAG::getConstant(uint64_t Val, EVT VT, bool isT) {<br>+SDValue SelectionDAG::getConstant(uint64_t Val, EVT VT, bool isT, bool isO) {<br> EVT EltVT = VT.getScalarType();<br> assert((EltVT.getSizeInBits() >= 64 ||<br> (uint64_t)((int64_t)Val >> EltVT.getSizeInBits()) + 1 < 2) &&<br> "getConstant with a uint64_t value that doesn't fit in the type!");<br>- return getConstant(APInt(EltVT.getSizeInBits(), Val), VT, isT);<br>+ return getConstant(APInt(EltVT.getSizeInBits(), Val), VT, isT, isO);<br> }<br><br>-SDValue SelectionDAG::getConstant(const APInt &Val, EVT VT, bool isT) {<br>- return getConstant(*ConstantInt::get(*Context, Val), VT, isT);<br>+SDValue SelectionDAG::getConstant(const APInt &Val, EVT VT, bool isT, bool isO)<br>+{<br>+ return getConstant(*ConstantInt::get(*Context, Val), VT, isT, isO);<br> }<br><br>-SDValue SelectionDAG::getConstant(const ConstantInt &Val, EVT VT, bool isT) {<br>+SDValue SelectionDAG::getConstant(const ConstantInt &Val, EVT VT, bool isT,<br>+ bool isO) {<br> assert(VT.isInteger() && "Cannot create FP integer constant!");<br><br> EVT EltVT = VT.getScalarType();<br>@@ -1025,7 +1030,7 @@ SDValue SelectionDAG::getConstant(const<br> for (unsigned i = 0; i < ViaVecNumElts / VT.getVectorNumElements(); ++i) {<br> EltParts.push_back(getConstant(NewVal.lshr(i * ViaEltSizeInBits)<br> .trunc(ViaEltSizeInBits),<br>- ViaEltVT, isT));<br>+ ViaEltVT, isT, isO));<br> }<br><br> // EltParts is currently in little endian order. If we actually want<br>@@ -1056,6 +1061,7 @@ SDValue SelectionDAG::getConstant(const<br> FoldingSetNodeID ID;<br> AddNodeIDNode(ID, Opc, getVTList(EltVT), 0, 0);<br> ID.AddPointer(Elt);<br>+ ID.AddBoolean(isO);<br> void *IP = 0;<br> SDNode *N = NULL;<br> if ((N = CSEMap.FindNodeOrInsertPos(ID, IP)))<br>@@ -1063,7 +1069,7 @@ SDValue SelectionDAG::getConstant(const<br> return SDValue(N, 0);<br><br> if (!N) {<br>- N = new (NodeAllocator) ConstantSDNode(isT, Elt, EltVT);<br>+ N = new (NodeAllocator) ConstantSDNode(isT, isO, Elt, EltVT);<br> CSEMap.InsertNode(N, IP);<br> AllNodes.push_back(N);<br> }<br>@@ -2789,10 +2795,13 @@ SDValue SelectionDAG::FoldConstantArithm<br><br> ConstantSDNode *Scalar1 = dyn_cast<ConstantSDNode>(Cst1);<br> ConstantSDNode *Scalar2 = dyn_cast<ConstantSDNode>(Cst2);<br>- if (Scalar1 && Scalar2) {<br>+ if (Scalar1 && Scalar2 && (Scalar1->isOpaque() || Scalar2->isOpaque()))<br>+ return SDValue();<br>+<br>+ if (Scalar1 && Scalar2)<br> // Scalar instruction.<br> Inputs.push_back(std::make_pair(Scalar1, Scalar2));<br>- } else {<br>+ else {<br> // For vectors extract each constant element into Inputs so we can constant<br> // fold them individually.<br> BuildVectorSDNode *BV1 = dyn_cast<BuildVectorSDNode>(Cst1);<br>@@ -2808,6 +2817,9 @@ SDValue SelectionDAG::FoldConstantArithm<br> if (!V1 || !V2) // Not a constant, bail.<br> return SDValue();<br><br>+ if (V1->isOpaque() || V2->isOpaque())<br>+ return SDValue();<br>+<br> // Avoid BUILD_VECTOR nodes that perform implicit truncation.<br> // FIXME: This is valid and could be handled by truncating the APInts.<br> if (V1->getValueType(0) != SVT || V2->getValueType(0) != SVT)<br>@@ -3561,10 +3573,11 @@ static SDValue getMemsetStringVal(EVT VT<br> Val |= (uint64_t)(unsigned char)Str[i] << (NumVTBytes-i-1)*8;<br> }<br><br>- // If the "cost" of materializing the integer immediate is 1 or free, then<br>- // it is cost effective to turn the load into the immediate.<br>+ // If the "cost" of materializing the integer immediate is less than the cost<br>+ // of a load, then it is cost effective to turn the load into the immediate.<br> const TargetTransformInfo *TTI = DAG.getTargetTransformInfo();<br>- if (TTI->getIntImmCost(Val, VT.getTypeForEVT(*DAG.getContext())) < 2)<br>+ if (TTI->getIntImmCost(Val, VT.getTypeForEVT(*DAG.getContext())) <<br>+ TargetTransformInfo::TCC_Load)<br> return DAG.getConstant(Val, VT);<br> return SDValue(0, 0);<br> }<br><br>Modified: llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp<br>URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp?rev=200022&r1=200021&r2=200022&view=diff">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp?rev=200022&r1=200021&r2=200022&view=diff</a><br>==============================================================================<br>--- llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp (original)<br>+++ llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp Fri Jan 24 12:23:08 2014<br>@@ -2945,6 +2945,9 @@ void SelectionDAGBuilder::visitBitCast(c<br> if (DestVT != N.getValueType())<br> setValue(&I, DAG.getNode(ISD::BITCAST, getCurSDLoc(),<br> DestVT, N)); // convert types.<br>+ else if(ConstantSDNode *C = dyn_cast<ConstantSDNode>(N))<br>+ setValue(&I, DAG.getConstant(C->getAPIntValue(), C->getValueType(0),<br>+ /*isTarget=*/false, /*isOpaque*/true));<br> else<br> setValue(&I, N); // noop cast.<br> }<br><br>Modified: llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp<br>URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp?rev=200022&r1=200021&r2=200022&view=diff">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp?rev=200022&r1=200021&r2=200022&view=diff</a><br>==============================================================================<br>--- llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp (original)<br>+++ llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp Fri Jan 24 12:23:08 2014<br>@@ -81,7 +81,10 @@ std::string SDNode::getOperationName(con<br> case ISD::VALUETYPE: return "ValueType";<br> case ISD::Register: return "Register";<br> case ISD::RegisterMask: return "RegisterMask";<br>- case ISD::Constant: return "Constant";<br>+ case ISD::Constant:<br>+ if (cast<ConstantSDNode>(this)->isOpaque())<br>+ return "OpaqueConstant";<br>+ return "Constant";<br> case ISD::ConstantFP: return "ConstantFP";<br> case ISD::GlobalAddress: return "GlobalAddress";<br> case ISD::GlobalTLSAddress: return "GlobalTLSAddress";<br>@@ -111,7 +114,10 @@ std::string SDNode::getOperationName(con<br> }<br><br> case ISD::BUILD_VECTOR: return "BUILD_VECTOR";<br>- case ISD::TargetConstant: return "TargetConstant";<br>+ case ISD::TargetConstant:<br>+ if (cast<ConstantSDNode>(this)->isOpaque())<br>+ return "OpaqueTargetConstant";<br>+ return "TargetConstant";<br> case ISD::TargetConstantFP: return "TargetConstantFP";<br> case ISD::TargetGlobalAddress: return "TargetGlobalAddress";<br> case ISD::TargetGlobalTLSAddress: return "TargetGlobalTLSAddress";<br><br>Modified: llvm/trunk/lib/CodeGen/SelectionDAG/TargetLowering.cpp<br>URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/TargetLowering.cpp?rev=200022&r1=200021&r2=200022&view=diff">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/TargetLowering.cpp?rev=200022&r1=200021&r2=200022&view=diff</a><br>==============================================================================<br>--- llvm/trunk/lib/CodeGen/SelectionDAG/TargetLowering.cpp (original)<br>+++ llvm/trunk/lib/CodeGen/SelectionDAG/TargetLowering.cpp Fri Jan 24 12:23:08 2014<br>@@ -1470,17 +1470,23 @@ TargetLowering::SimplifySetCC(EVT VT, SD<br> if (Cond == ISD::SETGE || Cond == ISD::SETUGE) {<br> if (C1 == MinVal) return DAG.getConstant(1, VT); // X >= MIN --> true<br> // X >= C0 --> X > (C0-1)<br>- return DAG.getSetCC(dl, VT, N0,<br>- DAG.getConstant(C1-1, N1.getValueType()),<br>- (Cond == ISD::SETGE) ? ISD::SETGT : ISD::SETUGT);<br>+ APInt C = C1-1;<br>+ if (!N1C->isOpaque() || (N1C->isOpaque() && C.getBitWidth() <= 64 &&<br>+ isLegalICmpImmediate(C.getSExtValue())))<br>+ return DAG.getSetCC(dl, VT, N0,<br>+ DAG.getConstant(C, N1.getValueType()),<br>+ (Cond == ISD::SETGE) ? ISD::SETGT : ISD::SETUGT);<br> }<br><br> if (Cond == ISD::SETLE || Cond == ISD::SETULE) {<br> if (C1 == MaxVal) return DAG.getConstant(1, VT); // X <= MAX --> true<br> // X <= C0 --> X < (C0+1)<br>- return DAG.getSetCC(dl, VT, N0,<br>- DAG.getConstant(C1+1, N1.getValueType()),<br>- (Cond == ISD::SETLE) ? ISD::SETLT : ISD::SETULT);<br>+ APInt C = C1+1;<br>+ if (!N1C->isOpaque() || (N1C->isOpaque() && C.getBitWidth() <= 64 &&<br>+ isLegalICmpImmediate(C.getSExtValue())))<br>+ return DAG.getSetCC(dl, VT, N0,<br>+ DAG.getConstant(C, N1.getValueType()),<br>+ (Cond == ISD::SETLE) ? ISD::SETLT : ISD::SETULT);<br> }<br><br> if ((Cond == ISD::SETLT || Cond == ISD::SETULT) && C1 == MinVal)<br><br>Modified: llvm/trunk/lib/Target/X86/X86TargetTransformInfo.cpp<br>URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/X86/X86TargetTransformInfo.cpp?rev=200022&r1=200021&r2=200022&view=diff">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/X86/X86TargetTransformInfo.cpp?rev=200022&r1=200021&r2=200022&view=diff</a><br>==============================================================================<br>--- llvm/trunk/lib/Target/X86/X86TargetTransformInfo.cpp (original)<br>+++ llvm/trunk/lib/Target/X86/X86TargetTransformInfo.cpp Fri Jan 24 12:23:08 2014<br>@@ -18,6 +18,7 @@<br> #include "X86.h"<br> #include "X86TargetMachine.h"<br> #include "llvm/Analysis/TargetTransformInfo.h"<br>+#include "llvm/IR/IntrinsicInst.h"<br> #include "llvm/Support/Debug.h"<br> #include "llvm/Target/CostTable.h"<br> #include "llvm/Target/TargetLowering.h"<br>@@ -107,6 +108,14 @@ public:<br> virtual unsigned getReductionCost(unsigned Opcode, Type *Ty,<br> bool IsPairwiseForm) const LLVM_OVERRIDE;<br><br>+ virtual unsigned getIntImmCost(const APInt &Imm,<br>+ Type *Ty) const LLVM_OVERRIDE;<br>+<br>+ virtual unsigned getIntImmCost(unsigned Opcode, const APInt &Imm,<br>+ Type *Ty) const LLVM_OVERRIDE;<br>+ virtual unsigned getIntImmCost(Intrinsic::ID IID, const APInt &Imm,<br>+ Type *Ty) const LLVM_OVERRIDE;<br>+<br> /// @}<br> };<br><br>@@ -694,3 +703,89 @@ unsigned X86TTI::getReductionCost(unsign<br> return TargetTransformInfo::getReductionCost(Opcode, ValTy, IsPairwise);<br> }<br><br>+unsigned X86TTI::getIntImmCost(const APInt &Imm, Type *Ty) const {<br>+ assert(Ty->isIntegerTy());<br>+<br>+ unsigned BitSize = Ty->getPrimitiveSizeInBits();<br>+ if (BitSize == 0)<br>+ return ~0U;<br>+<br>+ if (Imm.getBitWidth() <= 64 &&<br>+ (isInt<32>(Imm.getSExtValue()) || isUInt<32>(Imm.getZExtValue())))<br>+ return TCC_Basic;<br>+ else<br>+ return 2 * TCC_Basic;<br>+}<br>+<br>+unsigned X86TTI::getIntImmCost(unsigned Opcode, const APInt &Imm,<br>+ Type *Ty) const {<br>+ assert(Ty->isIntegerTy());<br>+<br>+ unsigned BitSize = Ty->getPrimitiveSizeInBits();<br>+ if (BitSize == 0)<br>+ return ~0U;<br>+<br>+ switch (Opcode) {<br>+ case Instruction::Add:<br>+ case Instruction::Sub:<br>+ case Instruction::Mul:<br>+ case Instruction::UDiv:<br>+ case Instruction::SDiv:<br>+ case Instruction::URem:<br>+ case Instruction::SRem:<br>+ case Instruction::Shl:<br>+ case Instruction::LShr:<br>+ case Instruction::AShr:<br>+ case Instruction::And:<br>+ case Instruction::Or:<br>+ case Instruction::Xor:<br>+ case Instruction::ICmp:<br>+ if (Imm.getBitWidth() <= 64 && isInt<32>(Imm.getSExtValue()))<br>+ return TCC_Free;<br>+ else<br>+ return X86TTI::getIntImmCost(Imm, Ty);<br>+ case Instruction::Trunc:<br>+ case Instruction::ZExt:<br>+ case Instruction::SExt:<br>+ case Instruction::IntToPtr:<br>+ case Instruction::PtrToInt:<br>+ case Instruction::BitCast:<br>+ case Instruction::Call:<br>+ case Instruction::Select:<br>+ case Instruction::Ret:<br>+ case Instruction::Load:<br>+ case Instruction::Store:<br>+ return X86TTI::getIntImmCost(Imm, Ty);<br>+ }<br>+ return TargetTransformInfo::getIntImmCost(Opcode, Imm, Ty);<br>+}<br>+<br>+unsigned X86TTI::getIntImmCost(Intrinsic::ID IID, const APInt &Imm,<br>+ Type *Ty) const {<br>+ assert(Ty->isIntegerTy());<br>+<br>+ unsigned BitSize = Ty->getPrimitiveSizeInBits();<br>+ if (BitSize == 0)<br>+ return ~0U;<br>+<br>+ switch (IID) {<br>+ default: return TargetTransformInfo::getIntImmCost(IID, Imm, Ty);<br>+ case Intrinsic::sadd_with_overflow:<br>+ case Intrinsic::uadd_with_overflow:<br>+ case Intrinsic::ssub_with_overflow:<br>+ case Intrinsic::usub_with_overflow:<br>+ case Intrinsic::smul_with_overflow:<br>+ case Intrinsic::umul_with_overflow:<br>+ if (Imm.getBitWidth() <= 64 && isInt<32>(Imm.getSExtValue()))<br>+ return TCC_Free;<br>+ else<br>+ return X86TTI::getIntImmCost(Imm, Ty);<br>+ case Intrinsic::experimental_stackmap:<br>+ case Intrinsic::experimental_patchpoint_void:<br>+ case Intrinsic::experimental_patchpoint_i64:<br>+ if (Imm.getBitWidth() <= 64 && isInt<64>(Imm.getSExtValue()))<br>+ return TCC_Free;<br>+ else<br>+ return X86TTI::getIntImmCost(Imm, Ty);<br>+ }<br>+}<br><br>Modified: llvm/trunk/lib/Transforms/Scalar/CMakeLists.txt<br>URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Scalar/CMakeLists.txt?rev=200022&r1=200021&r2=200022&view=diff">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Scalar/CMakeLists.txt?rev=200022&r1=200021&r2=200022&view=diff</a><br>==============================================================================<br>--- llvm/trunk/lib/Transforms/Scalar/CMakeLists.txt (original)<br>+++ llvm/trunk/lib/Transforms/Scalar/CMakeLists.txt Fri Jan 24 12:23:08 2014<br>@@ -1,6 +1,7 @@<br> add_llvm_library(LLVMScalarOpts<br> ADCE.cpp<br> CodeGenPrepare.cpp<br>+ ConstantHoisting.cpp<br> ConstantProp.cpp<br> CorrelatedValuePropagation.cpp<br> DCE.cpp<br><br>Modified: llvm/trunk/lib/Transforms/Scalar/CodeGenPrepare.cpp<br>URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Scalar/CodeGenPrepare.cpp?rev=200022&r1=200021&r2=200022&view=diff">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Scalar/CodeGenPrepare.cpp?rev=200022&r1=200021&r2=200022&view=diff</a><br>==============================================================================<br>--- llvm/trunk/lib/Transforms/Scalar/CodeGenPrepare.cpp (original)<br>+++ llvm/trunk/lib/Transforms/Scalar/CodeGenPrepare.cpp Fri Jan 24 12:23:08 2014<br>@@ -240,7 +240,7 @@ bool CodeGenPrepare::runOnFunction(Funct<br> bool CodeGenPrepare::EliminateFallThrough(Function &F) {<br> bool Changed = false;<br> // Scan all of the blocks in the function, except for the entry block.<br>- for (Function::iterator I = ++F.begin(), E = F.end(); I != E; ) {<br>+ for (Function::iterator I = llvm::next(F.begin()), E = F.end(); I != E; ) {<br> BasicBlock *BB = I++;<br> // If the destination block has a single pred, then this is a trivial<br> // edge, just collapse it.<br>@@ -276,7 +276,7 @@ bool CodeGenPrepare::EliminateFallThroug<br> bool CodeGenPrepare::EliminateMostlyEmptyBlocks(Function &F) {<br> bool MadeChange = false;<br> // Note that this intentionally skips the entry block.<br>- for (Function::iterator I = ++F.begin(), E = F.end(); I != E; ) {<br>+ for (Function::iterator I = llvm::next(F.begin()), E = F.end(); I != E; ) {<br> BasicBlock *BB = I++;<br><br> // If this block doesn't end with an uncond branch, ignore it.<br><br>Added: llvm/trunk/lib/Transforms/Scalar/ConstantHoisting.cpp<br>URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Scalar/ConstantHoisting.cpp?rev=200022&view=auto">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Scalar/ConstantHoisting.cpp?rev=200022&view=auto</a><br>==============================================================================<br>--- llvm/trunk/lib/Transforms/Scalar/ConstantHoisting.cpp (added)<br>+++ llvm/trunk/lib/Transforms/Scalar/ConstantHoisting.cpp Fri Jan 24 12:23:08 2014<br>@@ -0,0 +1,429 @@<br>+//===- ConstantHoisting.cpp - Prepare code for expensive constants --------===//<br>+//<br>+// The LLVM Compiler Infrastructure<br>+//<br>+// This file is distributed under the University of Illinois Open Source<br>+// License. See LICENSE.TXT for details.<br>+//<br>+//===----------------------------------------------------------------------===//<br>+//<br>+// This pass identifies expensive constants to hoist and coalesces them to<br>+// better prepare it for SelectionDAG-based code generation. This works around<br>+// the limitations of the basic-block-at-a-time approach.<br>+//<br>+// First it scans all instructions for integer constants and calculates its<br>+// cost. If the constant can be folded into the instruction (the cost is<br>+// TCC_Free) or the cost is just a simple operation (TCC_BASIC), then we don't<br>+// consider it expensive and leave it alone. This is the default behavior and<br>+// the default implementation of getIntImmCost will always return TCC_Free.<br>+//<br>+// If the cost is more than TCC_BASIC, then the integer constant can't be folded<br>+// into the instruction and it might be beneficial to hoist the constant.<br>+// Similar constants are coalesced to reduce register pressure and<br>+// materialization code.<br>+//<br>+// When a constant is hoisted, it is also hidden behind a bitcast to force it to<br>+// be live-out of the basic block. Otherwise the constant would be just<br>+// duplicated and each basic block would have its own copy in the SelectionDAG.<br>+// The SelectionDAG recognizes such constants as opaque and doesn't perform<br>+// certain transformations on them, which would create a new expensive constant.<br>+//<br>+// This optimization is only applied to integer constants in instructions and<br>+// simple (this means not nested) constant cast experessions. For example:<br>+// %0 = load i64* inttoptr (i64 big_constant to i64*)<br>+//===----------------------------------------------------------------------===//<br>+<br>+#define DEBUG_TYPE "consthoist"<br>+#include "llvm/Transforms/Scalar.h"<br>+#include "llvm/ADT/MapVector.h"<br>+#include "llvm/ADT/SmallSet.h"<br>+#include "llvm/ADT/Statistic.h"<br>+#include "llvm/Analysis/TargetTransformInfo.h"<br>+#include "llvm/IR/Constants.h"<br>+#include "llvm/IR/Dominators.h"<br>+#include "llvm/IR/IntrinsicInst.h"<br>+#include "llvm/Pass.h"<br>+#include "llvm/Support/CommandLine.h"<br>+#include "llvm/Support/Debug.h"<br>+<br>+using namespace llvm;<br>+<br>+STATISTIC(NumConstantsHoisted, "Number of constants hoisted");<br>+STATISTIC(NumConstantsRebased, "Number of constants rebased");<br>+<br>+<br>+namespace {<br>+typedef SmallVector<User *, 4> ConstantUseListType;<br>+struct ConstantCandidate {<br>+ unsigned CumulativeCost;<br>+ ConstantUseListType Uses;<br>+};<br>+<br>+struct ConstantInfo {<br>+ ConstantInt *BaseConstant;<br>+ struct RebasedConstantInfo {<br>+ ConstantInt *OriginalConstant;<br>+ Constant *Offset;<br>+ ConstantUseListType Uses;<br>+ };<br>+ typedef SmallVector<RebasedConstantInfo, 4> RebasedConstantListType;<br>+ RebasedConstantListType RebasedConstants;<br>+};<br>+<br>+class ConstantHoisting : public FunctionPass {<br>+ const TargetTransformInfo *TTI;<br>+ DominatorTree *DT;<br>+<br>+ /// Keeps track of expensive constants found in the function.<br>+ typedef MapVector<ConstantInt *, ConstantCandidate> ConstantMapType;<br>+ ConstantMapType ConstantMap;<br>+<br>+ /// These are the final constants we decided to hoist.<br>+ SmallVector<ConstantInfo, 4> Constants;<br>+public:<br>+ static char ID; // Pass identification, replacement for typeid<br>+ ConstantHoisting() : FunctionPass(ID), TTI(0) {<br>+ initializeConstantHoistingPass(*PassRegistry::getPassRegistry());<br>+ }<br>+<br>+ bool runOnFunction(Function &F);<br>+<br>+ const char *getPassName() const { return "Constant Hoisting"; }<br>+<br>+ virtual void getAnalysisUsage(AnalysisUsage &AU) const {<br>+ AU.setPreservesCFG();<br>+ AU.addRequired<DominatorTreeWrapperPass>();<br>+ AU.addRequired<TargetTransformInfo>();<br>+ }<br>+<br>+private:<br>+ void CollectConstant(User *U, unsigned Opcode, Intrinsic::ID IID,<br>+ ConstantInt *C);<br>+ void CollectConstants(Instruction *I);<br>+ void CollectConstants(Function &F);<br>+ void FindAndMakeBaseConstant(ConstantMapType::iterator S,<br>+ ConstantMapType::iterator E);<br>+ void FindBaseConstants();<br>+ Instruction *FindConstantInsertionPoint(Function &F,<br>+ const ConstantInfo &CI) const;<br>+ void EmitBaseConstants(Function &F, User *U, Instruction *Base,<br>+ Constant *Offset, ConstantInt *OriginalConstant);<br>+ bool EmitBaseConstants(Function &F);<br>+ bool OptimizeConstants(Function &F);<br>+};<br>+}<br>+<br>+char ConstantHoisting::ID = 0;<br>+INITIALIZE_PASS_BEGIN(ConstantHoisting, "consthoist", "Constant Hoisting",<br>+ false, false)<br>+INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)<br>+INITIALIZE_AG_DEPENDENCY(TargetTransformInfo)<br>+INITIALIZE_PASS_END(ConstantHoisting, "consthoist", "Constant Hoisting",<br>+ false, false)<br>+<br>+FunctionPass *llvm::createConstantHoistingPass() {<br>+ return new ConstantHoisting();<br>+}<br>+<br>+/// \brief Perform the constant hoisting optimization for the given function.<br>+bool ConstantHoisting::runOnFunction(Function &F) {<br>+ DEBUG(dbgs() << "********** Constant Hoisting **********\n");<br>+ DEBUG(dbgs() << "********** Function: " << F.getName() << '\n');<br>+<br>+ DT = &getAnalysis<DominatorTreeWrapperPass>().getDomTree();<br>+ TTI = &getAnalysis<TargetTransformInfo>();<br>+<br>+ return OptimizeConstants(F);<br>+}<br>+<br>+void ConstantHoisting::CollectConstant(User * U, unsigned Opcode,<br>+ Intrinsic::ID IID, ConstantInt *C) {<br>+ unsigned Cost;<br>+ if (Opcode)<br>+ Cost = TTI->getIntImmCost(Opcode, C->getValue(), C->getType());<br>+ else<br>+ Cost = TTI->getIntImmCost(IID, C->getValue(), C->getType());<br>+<br>+ if (Cost > TargetTransformInfo::TCC_Basic) {<br>+ ConstantCandidate &CC = ConstantMap[C];<br>+ CC.CumulativeCost += Cost;<br>+ CC.Uses.push_back(U);<br>+ }<br>+}<br>+<br>+/// \brief Scan the instruction or constant expression for expensive integer<br>+/// constants and record them in the constant map.<br>+void ConstantHoisting::CollectConstants(Instruction *I) {<br>+ unsigned Opcode = 0;<br>+ Intrinsic::ID IID = Intrinsic::not_intrinsic;<br>+ if (IntrinsicInst *II = dyn_cast<IntrinsicInst>(I))<br>+ IID = II->getIntrinsicID();<br>+ else<br>+ Opcode = I->getOpcode();<br>+<br>+ // Scan all operands.<br>+ for (User::op_iterator O = I->op_begin(), E = I->op_end(); O != E; ++O) {<br>+ if (ConstantInt *C = dyn_cast<ConstantInt>(O)) {<br>+ CollectConstant(I, Opcode, IID, C);<br>+ continue;<br>+ }<br>+ if (ConstantExpr *CE = dyn_cast<ConstantExpr>(O)) {<br>+ // We only handle constant cast expressions.<br>+ if (!CE->isCast())<br>+ continue;<br>+<br>+ if (ConstantInt *C = dyn_cast<ConstantInt>(CE->getOperand(0))) {<br>+ // Ignore the cast expression and use the opcode of the instruction.<br>+ CollectConstant(CE, Opcode, IID, C);<br>+ continue;<br>+ }<br>+ }<br>+ }<br>+}<br>+<br>+/// \brief Collect all integer constants in the function that cannot be folded<br>+/// into an instruction itself.<br>+void ConstantHoisting::CollectConstants(Function &F) {<br>+ for (Function::iterator BB = F.begin(), E = F.end(); BB != E; ++BB)<br>+ for (BasicBlock::iterator I = BB->begin(), E = BB->end(); I != E; ++I)<br>+ CollectConstants(I);<br>+}<br>+<br>+/// \brief Compare function for sorting integer constants by type and by value<br>+/// within a type in ConstantMaps.<br>+static bool<br>+ConstantMapLessThan(const std::pair<ConstantInt *, ConstantCandidate> &LHS,<br>+ const std::pair<ConstantInt *, ConstantCandidate> &RHS) {<br>+ if (LHS.first->getType() == RHS.first->getType())<br>+ return LHS.first->getValue().ult(RHS.first->getValue());<br>+ else<br>+ return LHS.first->getType()->getBitWidth() <<br>+ RHS.first->getType()->getBitWidth();<br>+}<br>+<br>+/// \brief Find the base constant within the given range and rebase all other<br>+/// constants with respect to the base constant.<br>+void ConstantHoisting::FindAndMakeBaseConstant(ConstantMapType::iterator S,<br>+ ConstantMapType::iterator E) {<br>+ ConstantMapType::iterator MaxCostItr = S;<br>+ unsigned NumUses = 0;<br>+ // Use the constant that has the maximum cost as base constant.<br>+ for (ConstantMapType::iterator I = S; I != E; ++I) {<br>+ NumUses += I->second.Uses.size();<br>+ if (I->second.CumulativeCost > MaxCostItr->second.CumulativeCost)<br>+ MaxCostItr = I;<br>+ }<br>+<br>+ // Don't hoist constants that have only one use.<br>+ if (NumUses <= 1)<br>+ return;<br>+<br>+ ConstantInfo CI;<br>+ CI.BaseConstant = MaxCostItr->first;<br>+ Type *Ty = CI.BaseConstant->getType();<br>+ // Rebase the constants with respect to the base constant.<br>+ for (ConstantMapType::iterator I = S; I != E; ++I) {<br>+ APInt Diff = I->first->getValue() - CI.BaseConstant->getValue();<br>+ ConstantInfo::RebasedConstantInfo RCI;<br>+ RCI.OriginalConstant = I->first;<br>+ RCI.Offset = ConstantInt::get(Ty, Diff);<br>+ RCI.Uses = llvm_move(I->second.Uses);<br>+ CI.RebasedConstants.push_back(RCI);<br>+ }<br>+ Constants.push_back(CI);<br>+}<br>+<br>+/// \brief Finds and combines constants that can be easily rematerialized with<br>+/// an add from a common base constant.<br>+void ConstantHoisting::FindBaseConstants() {<br>+ // Sort the constants by value and type. This invalidates the mapping.<br>+ std::sort(ConstantMap.begin(), ConstantMap.end(), ConstantMapLessThan);<br>+<br>+ // Simple linear scan through the sorted constant map for viable merge<br>+ // candidates.<br>+ ConstantMapType::iterator MinValItr = ConstantMap.begin();<br>+ for (ConstantMapType::iterator I = llvm::next(ConstantMap.begin()),<br>+ E = ConstantMap.end(); I != E; ++I) {<br>+ if (MinValItr->first->getType() == I->first->getType()) {<br>+ // Check if the constant is in range of an add with immediate.<br>+ APInt Diff = I->first->getValue() - MinValItr->first->getValue();<br>+ if ((Diff.getBitWidth() <= 64) &&<br>+ TTI->isLegalAddImmediate(Diff.getSExtValue()))<br>+ continue;<br>+ }<br>+ // We either have now a different constant type or the constant is not in<br>+ // range of an add with immediate anymore.<br>+ FindAndMakeBaseConstant(MinValItr, I);<br>+ // Start a new base constant search.<br>+ MinValItr = I;<br>+ }<br>+ // Finalize the last base constant search.<br>+ FindAndMakeBaseConstant(MinValItr, ConstantMap.end());<br>+}<br>+<br>+/// \brief Records the basic block of the instruction or all basic blocks of the<br>+/// users of the constant expression.<br>+static void CollectBasicBlocks(SmallPtrSet<BasicBlock *, 4> &BBs, User *U) {<br>+ if (Instruction *I = dyn_cast<Instruction>(U))<br>+ BBs.insert(I->getParent());<br>+ else if (ConstantExpr *CE = dyn_cast<ConstantExpr>(U))<br>+ // Find all users of this constant expression.<br>+ for (Value::use_iterator UU = CE->use_begin(), E = CE->use_end();<br>+ UU != E; ++UU)<br>+ // Only record users that are instructions. We don't want to go down a<br>+ // nested constant expression chain.<br>+ if (Instruction *I = dyn_cast<Instruction>(*UU))<br>+ BBs.insert(I->getParent());<br>+}<br>+<br>+/// \brief Find an insertion point that dominates all uses.<br>+Instruction *ConstantHoisting::<br>+FindConstantInsertionPoint(Function &F, const ConstantInfo &CI) const {<br>+ BasicBlock *Entry = &F.getEntryBlock();<br>+<br>+ // Collect all basic blocks.<br>+ SmallPtrSet<BasicBlock *, 4> BBs;<br>+ ConstantInfo::RebasedConstantListType::const_iterator RCI, RCE;<br>+ for (RCI = CI.RebasedConstants.begin(), RCE = CI.RebasedConstants.end();<br>+ RCI != RCE; ++RCI)<br>+ for (SmallVectorImpl<User *>::const_iterator U = RCI->Uses.begin(),<br>+ E = RCI->Uses.end(); U != E; ++U)<br>+ CollectBasicBlocks(BBs, *U);<br>+<br>+ if (BBs.count(Entry))<br>+ return Entry->getFirstInsertionPt();<br>+<br>+ while (BBs.size() >= 2) {<br>+ BasicBlock *BB, *BB1, *BB2;<br>+ BB1 = *BBs.begin();<br>+ BB2 = *llvm::next(BBs.begin());<br>+ BB = DT->findNearestCommonDominator(BB1, BB2);<br>+ if (BB == Entry)<br>+ return Entry->getFirstInsertionPt();<br>+ BBs.erase(BB1);<br>+ BBs.erase(BB2);<br>+ BBs.insert(BB);<br>+ }<br>+ assert((BBs.size() == 1) && "Expected only one element.");<br>+ return (*BBs.begin())->getFirstInsertionPt();<br>+}<br>+<br>+/// \brief Emit materialization code for all rebased constants and update their<br>+/// users.<br>+void ConstantHoisting::EmitBaseConstants(Function &F, User *U,<br>+ Instruction *Base, Constant *Offset,<br>+ ConstantInt *OriginalConstant) {<br>+ if (Instruction *I = dyn_cast<Instruction>(U)) {<br>+ Instruction *Mat = Base;<br>+ if (!Offset->isNullValue()) {<br>+ Mat = BinaryOperator::Create(Instruction::Add, Base, Offset,<br>+ "const_mat", I);<br>+<br>+ // Use the same debug location as the instruction we are about to update.<br>+ Mat->setDebugLoc(I->getDebugLoc());<br>+<br>+ DEBUG(dbgs() << "Materialize constant (" << *Base->getOperand(0)<br>+ << " + " << *Offset << ") in BB "<br>+ << I->getParent()->getName() << '\n' << *Mat << '\n');<br>+ }<br>+ DEBUG(dbgs() << "Update: " << *I << '\n');<br>+ I->replaceUsesOfWith(OriginalConstant, Mat);<br>+ DEBUG(dbgs() << "To: " << *I << '\n');<br>+ return;<br>+ }<br>+ assert(isa<ConstantExpr>(U) && "Expected a ConstantExpr.");<br>+ ConstantExpr *CE = cast<ConstantExpr>(U);<br>+ for (Value::use_iterator UU = CE->use_begin(), E = CE->use_end();<br>+ UU != E; ++UU) {<br>+ // We only handel instructions here and won't walk down a ConstantExpr chain<br>+ // to replace all ConstExpr with instructions.<br>+ if (Instruction *I = dyn_cast<Instruction>(*UU)) {<br>+ Instruction *Mat = Base;<br>+ if (!Offset->isNullValue()) {<br>+ Mat = BinaryOperator::Create(Instruction::Add, Base, Offset,<br>+ "const_mat", I);<br>+<br>+ // Use the same debug location as the instruction we are about to<br>+ // update.<br>+ Mat->setDebugLoc(I->getDebugLoc());<br>+<br>+ DEBUG(dbgs() << "Materialize constant (" << *Base->getOperand(0)<br>+ << " + " << *Offset << ") in BB "<br>+ << I->getParent()->getName() << '\n' << *Mat << '\n');<br>+ }<br>+ Instruction *ICE = CE->getAsInstruction();<br>+ ICE->replaceUsesOfWith(OriginalConstant, Mat);<br>+ ICE->insertBefore(I);<br>+<br>+ // Use the same debug location as the instruction we are about to update.<br>+ ICE->setDebugLoc(I->getDebugLoc());<br>+<br>+ DEBUG(dbgs() << "Create instruction: " << *ICE << '\n');<br>+ DEBUG(dbgs() << "Update: " << *I << '\n');<br>+ I->replaceUsesOfWith(CE, ICE);<br>+ DEBUG(dbgs() << "To: " << *I << '\n');<br>+ }<br>+ }<br>+}<br>+<br>+/// \brief Hoist and hide the base constant behind a bitcast and emit<br>+/// materialization code for derived constants.<br>+bool ConstantHoisting::EmitBaseConstants(Function &F) {<br>+ bool MadeChange = false;<br>+ SmallVectorImpl<ConstantInfo>::iterator CI, CE;<br>+ for (CI = Constants.begin(), CE = Constants.end(); CI != CE; ++CI) {<br>+ // Hoist and hide the base constant behind a bitcast.<br>+ Instruction *IP = FindConstantInsertionPoint(F, *CI);<br>+ IntegerType *Ty = CI->BaseConstant->getType();<br>+ Instruction *Base = new BitCastInst(CI->BaseConstant, Ty, "const", IP);<br>+ DEBUG(dbgs() << "Hoist constant (" << *CI->BaseConstant << ") to BB "<br>+ << IP->getParent()->getName() << '\n');<br>+ NumConstantsHoisted++;<br>+<br>+ // Emit materialization code for all rebased constants.<br>+ ConstantInfo::RebasedConstantListType::iterator RCI, RCE;<br>+ for (RCI = CI->RebasedConstants.begin(), RCE = CI->RebasedConstants.end();<br>+ RCI != RCE; ++RCI) {<br>+ NumConstantsRebased++;<br>+ for (SmallVectorImpl<User *>::iterator U = RCI->Uses.begin(),<br>+ E = RCI->Uses.end(); U != E; ++U)<br>+ EmitBaseConstants(F, *U, Base, RCI->Offset, RCI->OriginalConstant);<br>+ }<br>+<br>+ // Use the same debug location as the last user of the constant.<br>+ assert(!Base->use_empty() && "The use list is empty!?");<br>+ assert(isa<Instruction>(Base->use_back()) &&<br>+ "All uses should be instructions.");<br>+ Base->setDebugLoc(cast<Instruction>(Base->use_back())->getDebugLoc());<br>+<br>+ // Correct for base constant, which we counted above too.<br>+ NumConstantsRebased--;<br>+ MadeChange = true;<br>+ }<br>+ return MadeChange;<br>+}<br>+<br>+/// \brief Optimize expensive integer constants in the given function.<br>+bool ConstantHoisting::OptimizeConstants(Function &F) {<br>+ bool MadeChange = false;<br>+<br>+ // Collect all constant candidates.<br>+ CollectConstants(F);<br>+<br>+ // There are no constants to worry about.<br>+ if (ConstantMap.empty())<br>+ return MadeChange;<br>+<br>+ // Combine constants that can be easily materialized with an add from a common<br>+ // base constant.<br>+ FindBaseConstants();<br>+<br>+ // Finaly hoist the base constant and emit materializating code for dependent<br>+ // constants.<br>+ MadeChange |= EmitBaseConstants(F);<br>+<br>+ ConstantMap.clear();<br>+ Constants.clear();<br>+<br>+ return MadeChange;<br>+}<br><br>Modified: llvm/trunk/lib/Transforms/Scalar/Scalar.cpp<br>URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Scalar/Scalar.cpp?rev=200022&r1=200021&r2=200022&view=diff">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Scalar/Scalar.cpp?rev=200022&r1=200021&r2=200022&view=diff</a><br>==============================================================================<br>--- llvm/trunk/lib/Transforms/Scalar/Scalar.cpp (original)<br>+++ llvm/trunk/lib/Transforms/Scalar/Scalar.cpp Fri Jan 24 12:23:08 2014<br>@@ -30,6 +30,7 @@ void llvm::initializeScalarOpts(PassRegi<br> initializeADCEPass(Registry);<br> initializeSampleProfileLoaderPass(Registry);<br> initializeCodeGenPreparePass(Registry);<br>+ initializeConstantHoistingPass(Registry);<br> initializeConstantPropagationPass(Registry);<br> initializeCorrelatedValuePropagationPass(Registry);<br> initializeDCEPass(Registry);<br><br>Modified: llvm/trunk/test/CodeGen/ARM/memcpy-inline.ll<br>URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/test/CodeGen/ARM/memcpy-inline.ll?rev=200022&r1=200021&r2=200022&view=diff">http://llvm.org/viewvc/llvm-project/llvm/trunk/test/CodeGen/ARM/memcpy-inline.ll?rev=200022&r1=200021&r2=200022&view=diff</a><br>==============================================================================<br>--- llvm/trunk/test/CodeGen/ARM/memcpy-inline.ll (original)<br>+++ llvm/trunk/test/CodeGen/ARM/memcpy-inline.ll Fri Jan 24 12:23:08 2014<br>@@ -38,7 +38,8 @@ entry:<br> define void @t2(i8* nocapture %C) nounwind {<br> entry:<br> ; CHECK-LABEL: t2:<br>-; CHECK: ldr [[REG2:r[0-9]+]], [r1, #32]<br>+; CHECK: movw [[REG2:r[0-9]+]], #16716<br>+; CHECK: movt [[REG2:r[0-9]+]], #72<br> ; CHECK: str [[REG2]], [r0, #32]<br> ; CHECK: vld1.8 {d{{[0-9]+}}, d{{[0-9]+}}}, [r1]<br> ; CHECK: vst1.8 {d{{[0-9]+}}, d{{[0-9]+}}}, [r0]<br>@@ -79,7 +80,8 @@ entry:<br> ; CHECK: strb [[REG5]], [r0, #6]<br> ; CHECK: movw [[REG6:r[0-9]+]], #21587<br> ; CHECK: strh [[REG6]], [r0, #4]<br>-; CHECK: ldr [[REG7:r[0-9]+]], <br>+; CHECK: movw [[REG7:r[0-9]+]], #18500<br>+; CHECK: movt [[REG7:r[0-9]+]], #22866<br> ; CHECK: str [[REG7]]<br> tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %C, i8* getelementptr inbounds ([7 x i8]* @.str5, i64 0, i64 0), i64 7, i32 1, i1 false)<br> ret void<br><br>Added: llvm/trunk/test/CodeGen/X86/large-constants.ll<br>URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/test/CodeGen/X86/large-constants.ll?rev=200022&view=auto">http://llvm.org/viewvc/llvm-project/llvm/trunk/test/CodeGen/X86/large-constants.ll?rev=200022&view=auto</a><br>==============================================================================<br>--- llvm/trunk/test/CodeGen/X86/large-constants.ll (added)<br>+++ llvm/trunk/test/CodeGen/X86/large-constants.ll Fri Jan 24 12:23:08 2014<br>@@ -0,0 +1,53 @@<br>+; RUN: llc < %s -mtriple=x86_64-darwin -mcpu=corei7 | grep movabsq | count 2<br></blockquote><div>Could you update your test to use FileCheck instead?</div><div><br></div><div>Also, for readability purposes, I would have expected this test to use opt. It would also ease writing the CHECK lines.</div><div>You can achieve that by doing something like this:</div><div><div style="margin: 0px; font-size: 11px; font-family: Menlo;">RUN: opt -S -consthoist -mtriple=x86_64-apple-macosx < %s -o -</div></div><div><br></div>Now, if you really want to check the produced assembly, you can use two RUN, one with llc, one with opt :).<br><blockquote type="cite">+<br>+define i64 @constant_hoisting(i64 %o0, i64 %o1, i64 %o2, i64 %o3, i64 %o4, i64 %o5) {<br>+entry:<br>+ %l0 = and i64 %o0, -281474976710654<br>+ %c0 = icmp ne i64 %l0, 0<br>+ br i1 %c0, label %fail, label %bb1<br>+<br>+bb1:<br>+ %l1 = and i64 %o1, -281474976710654<br>+ %c1 = icmp ne i64 %l1, 0<br>+ br i1 %c1, label %fail, label %bb2<br>+<br>+bb2:<br>+ %l2 = and i64 %o2, -281474976710654<br>+ %c2 = icmp ne i64 %l2, 0<br>+ br i1 %c2, label %fail, label %bb3<br>+<br>+bb3:<br>+ %l3 = and i64 %o3, -281474976710654<br>+ %c3 = icmp ne i64 %l3, 0<br>+ br i1 %c3, label %fail, label %bb4<br>+<br>+bb4:<br>+ %l4 = and i64 %o4, -281474976710653<br>+ %c4 = icmp ne i64 %l4, 0<br>+ br i1 %c4, label %fail, label %bb5<br>+<br>+bb5:<br>+ %l5 = and i64 %o5, -281474976710652<br>+ %c5 = icmp ne i64 %l5, 0<br>+ br i1 %c5, label %fail, label %bb6<br>+<br>+bb6:<br>+ ret i64 %l5<br>+<br>+fail:<br>+ ret i64 -1<br>+}<br>+<br>+define void @constant_expressions() {<br>+entry:<br>+ %0 = load i64* inttoptr (i64 add (i64 51250129900, i64 0) to i64*)<br>+ %1 = load i64* inttoptr (i64 add (i64 51250129900, i64 8) to i64*)<br>+ %2 = load i64* inttoptr (i64 add (i64 51250129900, i64 16) to i64*)<br>+ %3 = load i64* inttoptr (i64 add (i64 51250129900, i64 24) to i64*)<br>+ %4 = add i64 %0, %1<br>+ %5 = add i64 %2, %3<br>+ %6 = add i64 %4, %5<br>+ store i64 %6, i64* inttoptr (i64 add (i64 51250129900, i64 0) to i64*)<br>+ ret void<br>+}<br>+<br><br><br>_______________________________________________<br>llvm-commits mailing list<br><a href="mailto:llvm-commits@cs.uiuc.edu">llvm-commits@cs.uiuc.edu</a><br>http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits<br></blockquote></div><br></body></html>