[llvm] r238276 - [inliner] Fix the early-exit of the inline cost analysis to correctly
Chandler Carruth
chandlerc at gmail.com
Tue May 26 19:49:06 PDT 2015
Author: chandlerc
Date: Tue May 26 21:49:05 2015
New Revision: 238276
URL: http://llvm.org/viewvc/llvm-project?rev=238276&view=rev
Log:
[inliner] Fix the early-exit of the inline cost analysis to correctly
model the dense vector instruction bonuses.
Previously, this code really didn't effectively compute the density of
inlined vector instructions and apply the intended inliner bonus. It
would try to compute it repeatedly while analyzing the function and
didn't handle the case where future vector instructions would tip the
scales back towards the bonus.
Instead, speculatively apply all possible bonuses to the threshold
initially. Once we *know* that a certain bonus can not be applied,
subtract it. This should delay early bailout enough to get much more
consistent results without actually causing us to analyze huge swaths of
code. I expect some (hopefully mild) compile time hit here, and some
swings in performance, but this was definitely the intended behavior of
these bonuses.
This also dramatically simplifies the computation of the bonuses to not
interact with each other in confusing ways. The previous code didn't do
a good job of this and the values for bonuses may be surprising but are
at least now clearly written in the code.
Finally, fix code to be in line with comments and use zero as the
bailout condition.
Patch by Easwaran Raman, with some comment tweaks by me to try and
further clarify what is going on with this code.
http://reviews.llvm.org/D8267
Added:
llvm/trunk/test/Transforms/Inline/vector-bonus.ll
Modified:
llvm/trunk/lib/Analysis/IPA/InlineCost.cpp
Modified: llvm/trunk/lib/Analysis/IPA/InlineCost.cpp
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Analysis/IPA/InlineCost.cpp?rev=238276&r1=238275&r2=238276&view=diff
==============================================================================
--- llvm/trunk/lib/Analysis/IPA/InlineCost.cpp (original)
+++ llvm/trunk/lib/Analysis/IPA/InlineCost.cpp Tue May 26 21:49:05 2015
@@ -955,16 +955,9 @@ bool CallAnalyzer::analyzeBlock(BasicBlo
AllocatedSize > InlineConstants::TotalAllocaSizeRecursiveCaller)
return false;
- if (NumVectorInstructions > NumInstructions/2)
- VectorBonus = FiftyPercentVectorBonus;
- else if (NumVectorInstructions > NumInstructions/10)
- VectorBonus = TenPercentVectorBonus;
- else
- VectorBonus = 0;
-
- // Check if we've past the threshold so we don't spin in huge basic
- // blocks that will never inline.
- if (Cost > (Threshold + VectorBonus))
+ // Check if we've past the maximum possible threshold so we don't spin in
+ // huge basic blocks that will never inline.
+ if (Cost > Threshold)
return false;
}
@@ -1020,25 +1013,33 @@ ConstantInt *CallAnalyzer::stripAndCompu
bool CallAnalyzer::analyzeCall(CallSite CS) {
++NumCallsAnalyzed;
- // Track whether the post-inlining function would have more than one basic
- // block. A single basic block is often intended for inlining. Balloon the
- // threshold by 50% until we pass the single-BB phase.
- bool SingleBB = true;
- int SingleBBBonus = Threshold / 2;
- Threshold += SingleBBBonus;
-
// Perform some tweaks to the cost and threshold based on the direct
// callsite information.
// We want to more aggressively inline vector-dense kernels, so up the
// threshold, and we'll lower it if the % of vector instructions gets too
- // low.
+ // low. Note that these bonuses are some what arbitrary and evolved over time
+ // by accident as much as because they are principled bonuses.
+ //
+ // FIXME: It would be nice to remove all such bonuses. At least it would be
+ // nice to base the bonus values on something more scientific.
assert(NumInstructions == 0);
assert(NumVectorInstructions == 0);
- FiftyPercentVectorBonus = Threshold;
- TenPercentVectorBonus = Threshold / 2;
+ FiftyPercentVectorBonus = 3 * Threshold / 2;
+ TenPercentVectorBonus = 3 * Threshold / 4;
const DataLayout &DL = F.getParent()->getDataLayout();
+ // Track whether the post-inlining function would have more than one basic
+ // block. A single basic block is often intended for inlining. Balloon the
+ // threshold by 50% until we pass the single-BB phase.
+ bool SingleBB = true;
+ int SingleBBBonus = Threshold / 2;
+
+ // Speculatively apply all possible bonuses to Threshold. If cost exceeds
+ // this Threshold any time, and cost cannot decrease, we can stop processing
+ // the rest of the function body.
+ Threshold += (SingleBBBonus + FiftyPercentVectorBonus);
+
// Give out bonuses per argument, as the instructions setting them up will
// be gone after inlining.
for (unsigned I = 0, E = CS.arg_size(); I != E; ++I) {
@@ -1081,9 +1082,9 @@ bool CallAnalyzer::analyzeCall(CallSite
Instruction *Instr = CS.getInstruction();
if (InvokeInst *II = dyn_cast<InvokeInst>(Instr)) {
if (isa<UnreachableInst>(II->getNormalDest()->begin()))
- Threshold = 1;
+ Threshold = 0;
} else if (isa<UnreachableInst>(++BasicBlock::iterator(Instr)))
- Threshold = 1;
+ Threshold = 0;
// If this function uses the coldcc calling convention, prefer not to inline
// it.
@@ -1155,7 +1156,7 @@ bool CallAnalyzer::analyzeCall(CallSite
for (unsigned Idx = 0; Idx != BBWorklist.size(); ++Idx) {
// Bail out the moment we cross the threshold. This means we'll under-count
// the cost, but only when undercounting doesn't matter.
- if (Cost > (Threshold + VectorBonus))
+ if (Cost > Threshold)
break;
BasicBlock *BB = BBWorklist[Idx];
@@ -1233,7 +1234,13 @@ bool CallAnalyzer::analyzeCall(CallSite
if (!OnlyOneCallAndLocalLinkage && ContainsNoDuplicateCall)
return false;
- Threshold += VectorBonus;
+ // We applied the maximum possible vector bonus at the beginning. Now,
+ // subtract the excess bonus, if any, from the Threshold before
+ // comparing against Cost.
+ if (NumVectorInstructions <= NumInstructions / 10)
+ Threshold -= FiftyPercentVectorBonus;
+ else if (NumVectorInstructions <= NumInstructions / 2)
+ Threshold -= (FiftyPercentVectorBonus - TenPercentVectorBonus);
return Cost < Threshold;
}
@@ -1248,12 +1255,12 @@ void CallAnalyzer::dump() {
DEBUG_PRINT_STAT(NumConstantPtrCmps);
DEBUG_PRINT_STAT(NumConstantPtrDiffs);
DEBUG_PRINT_STAT(NumInstructionsSimplified);
+ DEBUG_PRINT_STAT(NumInstructions);
DEBUG_PRINT_STAT(SROACostSavings);
DEBUG_PRINT_STAT(SROACostSavingsLost);
DEBUG_PRINT_STAT(ContainsNoDuplicateCall);
DEBUG_PRINT_STAT(Cost);
DEBUG_PRINT_STAT(Threshold);
- DEBUG_PRINT_STAT(VectorBonus);
#undef DEBUG_PRINT_STAT
}
#endif
Added: llvm/trunk/test/Transforms/Inline/vector-bonus.ll
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/Inline/vector-bonus.ll?rev=238276&view=auto
==============================================================================
--- llvm/trunk/test/Transforms/Inline/vector-bonus.ll (added)
+++ llvm/trunk/test/Transforms/Inline/vector-bonus.ll Tue May 26 21:49:05 2015
@@ -0,0 +1,37 @@
+; RUN: opt < %s -inline -inline-threshold=35 -S | FileCheck %s
+
+define i32 @bar(<4 x i32> %v, i32 %i) #0 {
+entry:
+ %cmp = icmp sgt i32 %i, 4
+ br i1 %cmp, label %if.then, label %if.else
+
+if.then: ; preds = %entry
+ %mul1 = mul nsw i32 %i, %i
+ br label %return
+
+if.else: ; preds = %entry
+ %add1 = add nsw i32 %i, %i
+ %add2 = add nsw i32 %i, %i
+ %add3 = add nsw i32 %i, %i
+ %add4 = add nsw i32 %i, %i
+ %add5 = add nsw i32 %i, %i
+ %add6 = add nsw i32 %i, %i
+ %vecext = extractelement <4 x i32> %v, i32 0
+ %vecext7 = extractelement <4 x i32> %v, i32 1
+ %add7 = add nsw i32 %vecext, %vecext7
+ br label %return
+
+return: ; preds = %if.else, %if.then
+ %retval.0 = phi i32 [ %mul1, %if.then ], [ %add7, %if.else ]
+ ret i32 %retval.0
+}
+
+define i32 @foo(<4 x i32> %v, i32 %a) #1 {
+; CHECK-LABEL: @foo(
+; CHECK-NOT: call i32 @bar
+; CHECK: ret
+entry:
+ %call = call i32 @bar(<4 x i32> %v, i32 %a)
+ ret i32 %call
+}
+
More information about the llvm-commits
mailing list