[llvm] r237156 - Reimplement heuristic for estimating complete-unroll optimization effects.
Chandler Carruth
chandlerc at google.com
Thu May 21 00:01:45 PDT 2015
That'll work....
Honestly, I'd find it easier to read the test as a single run with 4
function with 4 different loops hitting differently set thresholds. But
hey, that's a minor point.
On Wed, May 20, 2015 at 11:20 PM Michael Zolotukhin <mzolotukhin at apple.com>
wrote:
> On May 20, 2015, at 10:38 PM, Chandler Carruth <chandlerc at google.com>
> wrote:
>
> If they cover the same different use cases, most notably the *specific*
> reason for using a percentage based threshold, that's what I was looking
> for.
>
> Hi Chandler,
>
> It is basically a single test, but it’s invoked with 4 different sets of
> thresholds:
> 1) Check that we unroll smallest loops (size < unroll-threshold)
> independently on the new heuristics - it’s basically a loop that we’d
> unroll even without the heuristic.
> 2) Check that we don’t unroll huge loops (size >
> absolute-unroll-threshold).
> 3) Check that we unroll mid-sized loops (unroll-threshold < size <
> absolute-unroll-threshold) for which we predict a certain part of
> instructions to be removed (percentage of predicted-to-be-removed
> instructions > min-percent-to-unroll).
> 4) Check that we don’t unroll mid-sized loops (unroll-threshold < size <
> absolute-unroll-threshold) for which we predict less than a certain part of
> instructions to be removed (percentage of predicted-to-be-removed
> instructions < min-percent-to-unroll).
>
> Is it what you were looking for?
>
> Thanks,
> Michael
>
>
> On Thu, May 14, 2015 at 5:16 PM Michael Zolotukhin <mzolotukhin at apple.com>
> wrote:
>
>> Hi Chandler,
>>
>> We agreed that I’ll follow-up on this with adding examples from our
>> discussion as tests. However, the original tests
>> (test/Transforms/LoopUnroll/full-unroll-heuristics.ll) are essentially
>> covering the same cases. Did you mean some other tests, or these are
>> sufficient?
>>
>> Thanks,
>> Michael
>>
>> > On May 12, 2015, at 10:20 AM, Michael Zolotukhin <mzolotukhin at apple.com>
>> wrote:
>> >
>> > Author: mzolotukhin
>> > Date: Tue May 12 12:20:03 2015
>> > New Revision: 237156
>> >
>> > URL: http://llvm.org/viewvc/llvm-project?rev=237156&view=rev
>> > Log:
>> > Reimplement heuristic for estimating complete-unroll optimization
>> effects.
>> >
>> > Summary:
>> > This patch reimplements heuristic that tries to estimate optimization
>> beneftis
>> > from complete loop unrolling.
>> >
>> > In this patch I kept the minimal changes - e.g. I removed code handling
>> > branches and folding compares. That's a promising area, but now there
>> > are too many questions to discuss before we can enable it.
>> >
>> > Test Plan: Tests are included in the patch.
>> >
>> > Reviewers: hfinkel, chandlerc
>> >
>> > Subscribers: llvm-commits
>> >
>> > Differential Revision: http://reviews.llvm.org/D8816
>> >
>> > Added:
>> > llvm/trunk/test/Transforms/LoopUnroll/full-unroll-bad-geps.ll
>> > Modified:
>> > llvm/trunk/lib/Transforms/Scalar/LoopUnrollPass.cpp
>> > llvm/trunk/test/Transforms/LoopUnroll/full-unroll-heuristics.ll
>> >
>> > Modified: llvm/trunk/lib/Transforms/Scalar/LoopUnrollPass.cpp
>> > URL:
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Scalar/LoopUnrollPass.cpp?rev=237156&r1=237155&r2=237156&view=diff
>> >
>> ==============================================================================
>> > --- llvm/trunk/lib/Transforms/Scalar/LoopUnrollPass.cpp (original)
>> > +++ llvm/trunk/lib/Transforms/Scalar/LoopUnrollPass.cpp Tue May 12
>> 12:20:03 2015
>> > @@ -186,33 +186,21 @@ namespace {
>> > void selectThresholds(const Loop *L, bool HasPragma,
>> > const
>> TargetTransformInfo::UnrollingPreferences &UP,
>> > unsigned &Threshold, unsigned
>> &PartialThreshold,
>> > - unsigned NumberOfOptimizedInstructions) {
>> > + unsigned &AbsoluteThreshold,
>> > + unsigned
>> &PercentOfOptimizedForCompleteUnroll) {
>> > // Determine the current unrolling threshold. While this is
>> > // normally set from UnrollThreshold, it is overridden to a
>> > // smaller value if the current function is marked as
>> > // optimize-for-size, and the unroll threshold was not user
>> > // specified.
>> > Threshold = UserThreshold ? CurrentThreshold : UP.Threshold;
>> > -
>> > - // If we are allowed to completely unroll if we can remove M% of
>> > - // instructions, and we know that with complete unrolling we'll
>> be able
>> > - // to kill N instructions, then we can afford to completely
>> unroll loops
>> > - // with unrolled size up to N*100/M.
>> > - // Adjust the threshold according to that:
>> > - unsigned PercentOfOptimizedForCompleteUnroll =
>> > - UserPercentOfOptimized ? CurrentMinPercentOfOptimized
>> > - : UP.MinPercentOfOptimized;
>> > - unsigned AbsoluteThreshold = UserAbsoluteThreshold
>> > - ? CurrentAbsoluteThreshold
>> > - : UP.AbsoluteThreshold;
>> > - if (PercentOfOptimizedForCompleteUnroll)
>> > - Threshold = std::max<unsigned>(Threshold,
>> > - NumberOfOptimizedInstructions *
>> 100 /
>> > -
>> PercentOfOptimizedForCompleteUnroll);
>> > - // But don't allow unrolling loops bigger than absolute
>> threshold.
>> > - Threshold = std::min<unsigned>(Threshold, AbsoluteThreshold);
>> > -
>> > PartialThreshold = UserThreshold ? CurrentThreshold :
>> UP.PartialThreshold;
>> > + AbsoluteThreshold = UserAbsoluteThreshold ?
>> CurrentAbsoluteThreshold
>> > + : UP.AbsoluteThreshold;
>> > + PercentOfOptimizedForCompleteUnroll = UserPercentOfOptimized
>> > + ?
>> CurrentMinPercentOfOptimized
>> > + :
>> UP.MinPercentOfOptimized;
>> > +
>> > if (!UserThreshold &&
>> > L->getHeader()->getParent()->hasFnAttribute(
>> > Attribute::OptimizeForSize)) {
>> > @@ -231,6 +219,10 @@ namespace {
>> > std::max<unsigned>(PartialThreshold,
>> PragmaUnrollThreshold);
>> > }
>> > }
>> > + bool canUnrollCompletely(Loop *L, unsigned Threshold,
>> > + unsigned AbsoluteThreshold, uint64_t
>> UnrolledSize,
>> > + unsigned NumberOfOptimizedInstructions,
>> > + unsigned
>> PercentOfOptimizedForCompleteUnroll);
>> > };
>> > }
>> >
>> > @@ -253,57 +245,75 @@ Pass *llvm::createSimpleLoopUnrollPass()
>> > return llvm::createLoopUnrollPass(-1, -1, 0, 0);
>> > }
>> >
>> > -static bool isLoadFromConstantInitializer(Value *V) {
>> > - if (GlobalVariable *GV = dyn_cast<GlobalVariable>(V))
>> > - if (GV->isConstant() && GV->hasDefinitiveInitializer())
>> > - return GV->getInitializer();
>> > - return false;
>> > -}
>> > -
>> > namespace {
>> > +/// \brief SCEV expressions visitor used for finding expressions that
>> would
>> > +/// become constants if the loop L is unrolled.
>> > struct FindConstantPointers {
>> > - bool LoadCanBeConstantFolded;
>> > + /// \brief Shows whether the expression is ConstAddress+Constant or
>> not.
>> > bool IndexIsConstant;
>> > - APInt Step;
>> > - APInt StartValue;
>> > +
>> > + /// \brief Used for filtering out SCEV expressions with two or more
>> AddRec
>> > + /// subexpressions.
>> > + ///
>> > + /// Used to filter out complicated SCEV expressions, having several
>> AddRec
>> > + /// sub-expressions. We don't handle them, because unrolling one loop
>> > + /// would help to replace only one of these inductions with a
>> constant, and
>> > + /// consequently, the expression would remain non-constant.
>> > + bool HaveSeenAR;
>> > +
>> > + /// \brief If the SCEV expression becomes ConstAddress+Constant,
>> this value
>> > + /// holds ConstAddress. Otherwise, it's nullptr.
>> > Value *BaseAddress;
>> > +
>> > + /// \brief The loop, which we try to completely unroll.
>> > const Loop *L;
>> > +
>> > ScalarEvolution &SE;
>> > - FindConstantPointers(const Loop *loop, ScalarEvolution &SE)
>> > - : LoadCanBeConstantFolded(true), IndexIsConstant(true), L(loop),
>> SE(SE) {}
>> >
>> > + FindConstantPointers(const Loop *L, ScalarEvolution &SE)
>> > + : IndexIsConstant(true), HaveSeenAR(false), BaseAddress(nullptr),
>> > + L(L), SE(SE) {}
>> > +
>> > + /// Examine the given expression S and figure out, if it can be a
>> part of an
>> > + /// expression, that could become a constant after the loop is
>> unrolled.
>> > + /// The routine sets IndexIsConstant and HaveSeenAR according to the
>> analysis
>> > + /// results.
>> > + /// \returns true if we need to examine subexpressions, and false
>> otherwise.
>> > bool follow(const SCEV *S) {
>> > if (const SCEVUnknown *SC = dyn_cast<SCEVUnknown>(S)) {
>> > // We've reached the leaf node of SCEV, it's most probably just a
>> > - // variable. Now it's time to see if it corresponds to a global
>> constant
>> > - // global (in which case we can eliminate the load), or not.
>> > + // variable.
>> > + // If it's the only one SCEV-subexpression, then it might be a
>> base
>> > + // address of an index expression.
>> > + // If we've already recorded base address, then just give up on
>> this SCEV
>> > + // - it's too complicated.
>> > + if (BaseAddress) {
>> > + IndexIsConstant = false;
>> > + return false;
>> > + }
>> > BaseAddress = SC->getValue();
>> > - LoadCanBeConstantFolded =
>> > - IndexIsConstant &&
>> isLoadFromConstantInitializer(BaseAddress);
>> > return false;
>> > }
>> > if (isa<SCEVConstant>(S))
>> > - return true;
>> > + return false;
>> > if (const SCEVAddRecExpr *AR = dyn_cast<SCEVAddRecExpr>(S)) {
>> > // If the current SCEV expression is AddRec, and its loop isn't
>> the loop
>> > // we are about to unroll, then we won't get a constant address
>> after
>> > // unrolling, and thus, won't be able to eliminate the load.
>> > - if (AR->getLoop() != L)
>> > - return IndexIsConstant = false;
>> > - // If the step isn't constant, we won't get constant addresses
>> in unrolled
>> > - // version. Bail out.
>> > - if (const SCEVConstant *StepSE =
>> > - dyn_cast<SCEVConstant>(AR->getStepRecurrence(SE)))
>> > - Step = StepSE->getValue()->getValue();
>> > - else
>> > - return IndexIsConstant = false;
>> > -
>> > - return IndexIsConstant;
>> > + if (AR->getLoop() != L) {
>> > + IndexIsConstant = false;
>> > + return false;
>> > + }
>> > + // We don't handle multiple AddRecs here, so give up in this
>> case.
>> > + if (HaveSeenAR) {
>> > + IndexIsConstant = false;
>> > + return false;
>> > + }
>> > + HaveSeenAR = true;
>> > }
>> > - // If Result is true, continue traversal.
>> > - // Otherwise, we have found something that prevents us from
>> (possible) load
>> > - // elimination.
>> > - return IndexIsConstant;
>> > +
>> > + // Continue traversal.
>> > + return true;
>> > }
>> > bool isDone() const { return !IndexIsConstant; }
>> > };
>> > @@ -328,27 +338,54 @@ class UnrollAnalyzer : public InstVisito
>> > typedef InstVisitor<UnrollAnalyzer, bool> Base;
>> > friend class InstVisitor<UnrollAnalyzer, bool>;
>> >
>> > + struct SCEVGEPDescriptor {
>> > + Value *BaseAddr;
>> > + APInt Start;
>> > + APInt Step;
>> > + };
>> > +
>> > + /// \brief The loop we're going to analyze.
>> > const Loop *L;
>> > +
>> > + /// \brief TripCount of the given loop.
>> > unsigned TripCount;
>> > +
>> > ScalarEvolution &SE;
>> > +
>> > const TargetTransformInfo &TTI;
>> >
>> > + // While we walk the loop instructions, we we build up and maintain
>> a mapping
>> > + // of simplified values specific to this iteration. The idea is to
>> propagate
>> > + // any special information we have about loads that can be replaced
>> with
>> > + // constants after complete unrolling, and account for likely
>> simplifications
>> > + // post-unrolling.
>> > DenseMap<Value *, Constant *> SimplifiedValues;
>> > - DenseMap<LoadInst *, Value *> LoadBaseAddresses;
>> > - SmallPtrSet<Instruction *, 32> CountedInstructions;
>> >
>> > - /// \brief Count the number of optimized instructions.
>> > - unsigned NumberOfOptimizedInstructions;
>> > + // To avoid requesting SCEV info on every iteration, request it
>> once, and
>> > + // for each value that would become ConstAddress+Constant after loop
>> > + // unrolling, save the corresponding data.
>> > + SmallDenseMap<Value *, SCEVGEPDescriptor> SCEVCache;
>> > +
>> > + /// \brief Number of currently simulated iteration.
>> > + ///
>> > + /// If an expression is ConstAddress+Constant, then the Constant is
>> > + /// Start + Iteration*Step, where Start and Step could be obtained
>> from
>> > + /// SCEVCache.
>> > + unsigned Iteration;
>> > +
>> > + /// \brief Upper threshold for complete unrolling.
>> > + unsigned MaxUnrolledLoopSize;
>> >
>> > - // Provide base case for our instruction visit.
>> > + /// Base case for the instruction visitor.
>> > bool visitInstruction(Instruction &I) { return false; };
>> > - // TODO: We should also visit ICmp, FCmp, GetElementPtr, Trunc,
>> ZExt, SExt,
>> > - // FPTrunc, FPExt, FPToUI, FPToSI, UIToFP, SIToFP, BitCast, Select,
>> > - // ExtractElement, InsertElement, ShuffleVector, ExtractValue,
>> InsertValue.
>> > - //
>> > - // Probaly it's worth to hoist the code for estimating the
>> simplifications
>> > - // effects to a separate class, since we have a very similar code in
>> > - // InlineCost already.
>> > +
>> > + /// TODO: Add visitors for other instruction types, e.g. ZExt, SExt.
>> > +
>> > + /// Try to simplify binary operator I.
>> > + ///
>> > + /// TODO: Probaly it's worth to hoist the code for estimating the
>> > + /// simplifications effects to a separate class, since we have a
>> very similar
>> > + /// code in InlineCost already.
>> > bool visitBinaryOperator(BinaryOperator &I) {
>> > Value *LHS = I.getOperand(0), *RHS = I.getOperand(1);
>> > if (!isa<Constant>(LHS))
>> > @@ -365,7 +402,7 @@ class UnrollAnalyzer : public InstVisito
>> > else
>> > SimpleV = SimplifyBinOp(I.getOpcode(), LHS, RHS, DL);
>> >
>> > - if (SimpleV && CountedInstructions.insert(&I).second)
>> > + if (SimpleV)
>> > NumberOfOptimizedInstructions += TTI.getUserCost(&I);
>> >
>> > if (Constant *C = dyn_cast_or_null<Constant>(SimpleV)) {
>> > @@ -375,207 +412,172 @@ class UnrollAnalyzer : public InstVisito
>> > return false;
>> > }
>> >
>> > - Constant *computeLoadValue(LoadInst *LI, unsigned Iteration) {
>> > - if (!LI)
>> > - return nullptr;
>> > - Value *BaseAddr = LoadBaseAddresses[LI];
>> > - if (!BaseAddr)
>> > - return nullptr;
>> > -
>> > - auto GV = dyn_cast<GlobalVariable>(BaseAddr);
>> > - if (!GV)
>> > - return nullptr;
>> > + /// Try to fold load I.
>> > + bool visitLoad(LoadInst &I) {
>> > + Value *AddrOp = I.getPointerOperand();
>> > + if (!isa<Constant>(AddrOp))
>> > + if (Constant *SimplifiedAddrOp = SimplifiedValues.lookup(AddrOp))
>> > + AddrOp = SimplifiedAddrOp;
>> > +
>> > + auto It = SCEVCache.find(AddrOp);
>> > + if (It == SCEVCache.end())
>> > + return false;
>> > + SCEVGEPDescriptor GEPDesc = It->second;
>> > +
>> > + auto GV = dyn_cast<GlobalVariable>(GEPDesc.BaseAddr);
>> > + // We're only interested in loads that can be completely folded to
>> a
>> > + // constant.
>> > + if (!GV || !GV->hasInitializer())
>> > + return false;
>> >
>> > ConstantDataSequential *CDS =
>> > dyn_cast<ConstantDataSequential>(GV->getInitializer());
>> > if (!CDS)
>> > - return nullptr;
>> > -
>> > - const SCEV *BaseAddrSE = SE.getSCEV(BaseAddr);
>> > - const SCEV *S = SE.getSCEV(LI->getPointerOperand());
>> > - const SCEV *OffSE = SE.getMinusSCEV(S, BaseAddrSE);
>> > -
>> > - APInt StepC, StartC;
>> > - const SCEVAddRecExpr *AR = dyn_cast<SCEVAddRecExpr>(OffSE);
>> > - if (!AR)
>> > - return nullptr;
>> > -
>> > - if (const SCEVConstant *StepSE =
>> > - dyn_cast<SCEVConstant>(AR->getStepRecurrence(SE)))
>> > - StepC = StepSE->getValue()->getValue();
>> > - else
>> > - return nullptr;
>> > -
>> > - if (const SCEVConstant *StartSE =
>> dyn_cast<SCEVConstant>(AR->getStart()))
>> > - StartC = StartSE->getValue()->getValue();
>> > - else
>> > - return nullptr;
>> > + return false;
>> >
>> > + // Check possible overflow.
>> > + if (GEPDesc.Start.getActiveBits() > 32 ||
>> GEPDesc.Step.getActiveBits() > 32)
>> > + return false;
>> > unsigned ElemSize = CDS->getElementType()->getPrimitiveSizeInBits()
>> / 8U;
>> > - unsigned Start = StartC.getLimitedValue();
>> > - unsigned Step = StepC.getLimitedValue();
>> > -
>> > - unsigned Index = (Start + Step * Iteration) / ElemSize;
>> > - if (Index >= CDS->getNumElements())
>> > - return nullptr;
>> > + uint64_t Index = (GEPDesc.Start.getLimitedValue() +
>> > + GEPDesc.Step.getLimitedValue() * Iteration) /
>> > + ElemSize;
>> > + if (Index >= CDS->getNumElements()) {
>> > + // FIXME: For now we conservatively ignore out of bound
>> accesses, but
>> > + // we're allowed to perform the optimization in this case.
>> > + return false;
>> > + }
>> >
>> > Constant *CV = CDS->getElementAsConstant(Index);
>> > + assert(CV && "Constant expected.");
>> > + SimplifiedValues[&I] = CV;
>> >
>> > - return CV;
>> > + NumberOfOptimizedInstructions += TTI.getUserCost(&I);
>> > + return true;
>> > }
>> >
>> > -public:
>> > - UnrollAnalyzer(const Loop *L, unsigned TripCount, ScalarEvolution
>> &SE,
>> > - const TargetTransformInfo &TTI)
>> > - : L(L), TripCount(TripCount), SE(SE), TTI(TTI),
>> > - NumberOfOptimizedInstructions(0) {}
>> > -
>> > - // Visit all loads the loop L, and for those that, after complete
>> loop
>> > - // unrolling, would have a constant address and it will point to a
>> known
>> > - // constant initializer, record its base address for future use. It
>> is used
>> > - // when we estimate number of potentially simplified instructions.
>> > - void findConstFoldableLoads() {
>> > + /// Visit all GEPs in the loop and find those which after complete
>> loop
>> > + /// unrolling would become a constant, or BaseAddress+Constant.
>> > + ///
>> > + /// Such GEPs could allow to evaluate a load to a constant later -
>> for now we
>> > + /// just store the corresponding BaseAddress and StartValue with
>> StepValue in
>> > + /// the SCEVCache.
>> > + void cacheSCEVResults() {
>> > for (auto BB : L->getBlocks()) {
>> > - for (BasicBlock::iterator I = BB->begin(), E = BB->end(); I !=
>> E; ++I) {
>> > - if (LoadInst *LI = dyn_cast<LoadInst>(I)) {
>> > - if (!LI->isSimple())
>> > - continue;
>> > - Value *AddrOp = LI->getPointerOperand();
>> > - const SCEV *S = SE.getSCEV(AddrOp);
>> > + for (Instruction &I : *BB) {
>> > + if (GetElementPtrInst *GEP = dyn_cast<GetElementPtrInst>(&I)) {
>> > + Value *V = cast<Value>(GEP);
>> > + if (!SE.isSCEVable(V->getType()))
>> > + continue;
>> > + const SCEV *S = SE.getSCEV(V);
>> > + // FIXME: Hoist the initialization out of the loop.
>> > FindConstantPointers Visitor(L, SE);
>> > SCEVTraversal<FindConstantPointers> T(Visitor);
>> > + // Try to find (BaseAddress+Step+Offset) tuple.
>> > + // If succeeded, save it to the cache - it might help in
>> folding
>> > + // loads.
>> > T.visitAll(S);
>> > - if (Visitor.IndexIsConstant &&
>> Visitor.LoadCanBeConstantFolded) {
>> > - LoadBaseAddresses[LI] = Visitor.BaseAddress;
>> > - }
>> > + if (!Visitor.IndexIsConstant || !Visitor.BaseAddress)
>> > + continue;
>> > +
>> > + const SCEV *BaseAddrSE = SE.getSCEV(Visitor.BaseAddress);
>> > + if (BaseAddrSE->getType() != S->getType())
>> > + continue;
>> > + const SCEV *OffSE = SE.getMinusSCEV(S, BaseAddrSE);
>> > + const SCEVAddRecExpr *AR = dyn_cast<SCEVAddRecExpr>(OffSE);
>> > +
>> > + if (!AR)
>> > + continue;
>> > +
>> > + const SCEVConstant *StepSE =
>> > + dyn_cast<SCEVConstant>(AR->getStepRecurrence(SE));
>> > + const SCEVConstant *StartSE =
>> dyn_cast<SCEVConstant>(AR->getStart());
>> > + if (!StepSE || !StartSE)
>> > + continue;
>> > +
>> > + SCEVCache[V] = {Visitor.BaseAddress,
>> StartSE->getValue()->getValue(),
>> > + StepSE->getValue()->getValue()};
>> > }
>> > }
>> > }
>> > }
>> >
>> > - // Given a list of loads that could be constant-folded
>> (LoadBaseAddresses),
>> > - // estimate number of optimized instructions after substituting the
>> concrete
>> > - // values for the given Iteration. Also track how many instructions
>> become
>> > - // dead through this process.
>> > - unsigned estimateNumberOfOptimizedInstructions(unsigned Iteration) {
>> > - // We keep a set vector for the worklist so that we don't wast
>> space in the
>> > - // worklist queuing up the same instruction repeatedly. This can
>> happen due
>> > - // to multiple operands being the same instruction or due to the
>> same
>> > - // instruction being an operand of lots of things that end up dead
>> or
>> > - // simplified.
>> > - SmallSetVector<Instruction *, 8> Worklist;
>> > -
>> > - // Clear the simplified values and counts for this iteration.
>> > - SimplifiedValues.clear();
>> > - CountedInstructions.clear();
>> > - NumberOfOptimizedInstructions = 0;
>> > -
>> > - // We start by adding all loads to the worklist.
>> > - for (auto &LoadDescr : LoadBaseAddresses) {
>> > - LoadInst *LI = LoadDescr.first;
>> > - SimplifiedValues[LI] = computeLoadValue(LI, Iteration);
>> > - if (CountedInstructions.insert(LI).second)
>> > - NumberOfOptimizedInstructions += TTI.getUserCost(LI);
>> > +public:
>> > + UnrollAnalyzer(const Loop *L, unsigned TripCount, ScalarEvolution
>> &SE,
>> > + const TargetTransformInfo &TTI, unsigned
>> MaxUnrolledLoopSize)
>> > + : L(L), TripCount(TripCount), SE(SE), TTI(TTI),
>> > + MaxUnrolledLoopSize(MaxUnrolledLoopSize),
>> > + NumberOfOptimizedInstructions(0), UnrolledLoopSize(0) {}
>> >
>> > - for (User *U : LI->users())
>> > - Worklist.insert(cast<Instruction>(U));
>> > - }
>> > + /// \brief Count the number of optimized instructions.
>> > + unsigned NumberOfOptimizedInstructions;
>> >
>> > - // And then we try to simplify every user of every instruction
>> from the
>> > - // worklist. If we do simplify a user, add it to the worklist to
>> process
>> > - // its users as well.
>> > - while (!Worklist.empty()) {
>> > - Instruction *I = Worklist.pop_back_val();
>> > - if (!L->contains(I))
>> > - continue;
>> > - if (!visit(I))
>> > - continue;
>> > - for (User *U : I->users())
>> > - Worklist.insert(cast<Instruction>(U));
>> > - }
>> > + /// \brief Count the total number of instructions.
>> > + unsigned UnrolledLoopSize;
>> >
>> > - // Now that we know the potentially simplifed instructions,
>> estimate number
>> > - // of instructions that would become dead if we do perform the
>> > - // simplification.
>> > -
>> > - // The dead instructions are held in a separate set. This is used
>> to
>> > - // prevent us from re-examining instructions and make sure we only
>> count
>> > - // the benifit once. The worklist's internal set handles insertion
>> > - // deduplication.
>> > - SmallPtrSet<Instruction *, 16> DeadInstructions;
>> > -
>> > - // Lambda to enque operands onto the worklist.
>> > - auto EnqueueOperands = [&](Instruction &I) {
>> > - for (auto *Op : I.operand_values())
>> > - if (auto *OpI = dyn_cast<Instruction>(Op))
>> > - if (!OpI->use_empty())
>> > - Worklist.insert(OpI);
>> > - };
>> > -
>> > - // Start by initializing worklist with simplified instructions.
>> > - for (auto &FoldedKeyValue : SimplifiedValues)
>> > - if (auto *FoldedInst =
>> dyn_cast<Instruction>(FoldedKeyValue.first)) {
>> > - DeadInstructions.insert(FoldedInst);
>> > -
>> > - // Add each instruction operand of this dead instruction to the
>> > - // worklist.
>> > - EnqueueOperands(*FoldedInst);
>> > - }
>> > + /// \brief Figure out if the loop is worth full unrolling.
>> > + ///
>> > + /// Complete loop unrolling can make some loads constant, and we
>> need to know
>> > + /// if that would expose any further optimization opportunities.
>> This routine
>> > + /// estimates this optimization. It assigns computed number of
>> instructions,
>> > + /// that potentially might be optimized away, to
>> > + /// NumberOfOptimizedInstructions, and total number of instructions
>> to
>> > + /// UnrolledLoopSize (not counting blocks that won't be reached, if
>> we were
>> > + /// able to compute the condition).
>> > + /// \returns false if we can't analyze the loop, or if we discovered
>> that
>> > + /// unrolling won't give anything. Otherwise, returns true.
>> > + bool analyzeLoop() {
>> > + SmallSetVector<BasicBlock *, 16> BBWorklist;
>> > +
>> > + // Don't simulate loops with a big or unknown tripcount
>> > + if (!UnrollMaxIterationsCountToAnalyze || !TripCount ||
>> > + TripCount > UnrollMaxIterationsCountToAnalyze)
>> > + return false;
>> >
>> > - // If a definition of an insn is only used by simplified or dead
>> > - // instructions, it's also dead. Check defs of all instructions
>> from the
>> > - // worklist.
>> > - while (!Worklist.empty()) {
>> > - Instruction *I = Worklist.pop_back_val();
>> > - if (!L->contains(I))
>> > - continue;
>> > - if (DeadInstructions.count(I))
>> > - continue;
>> > -
>> > - if (std::all_of(I->user_begin(), I->user_end(), [&](User *U) {
>> > - return DeadInstructions.count(cast<Instruction>(U));
>> > - })) {
>> > - NumberOfOptimizedInstructions += TTI.getUserCost(I);
>> > - DeadInstructions.insert(I);
>> > - EnqueueOperands(*I);
>> > + // To avoid compute SCEV-expressions on every iteration, compute
>> them once
>> > + // and store interesting to us in SCEVCache.
>> > + cacheSCEVResults();
>> > +
>> > + // Simulate execution of each iteration of the loop counting
>> instructions,
>> > + // which would be simplified.
>> > + // Since the same load will take different values on different
>> iterations,
>> > + // we literally have to go through all loop's iterations.
>> > + for (Iteration = 0; Iteration < TripCount; ++Iteration) {
>> > + SimplifiedValues.clear();
>> > + BBWorklist.clear();
>> > + BBWorklist.insert(L->getHeader());
>> > + // Note that we *must not* cache the size, this loop grows the
>> worklist.
>> > + for (unsigned Idx = 0; Idx != BBWorklist.size(); ++Idx) {
>> > + BasicBlock *BB = BBWorklist[Idx];
>> > +
>> > + // Visit all instructions in the given basic block and try to
>> simplify
>> > + // it. We don't change the actual IR, just count optimization
>> > + // opportunities.
>> > + for (Instruction &I : *BB) {
>> > + UnrolledLoopSize += TTI.getUserCost(&I);
>> > + Base::visit(I);
>> > + // If unrolled body turns out to be too big, bail out.
>> > + if (UnrolledLoopSize - NumberOfOptimizedInstructions >
>> > + MaxUnrolledLoopSize)
>> > + return false;
>> > + }
>> > +
>> > + // Add BB's successors to the worklist.
>> > + for (BasicBlock *Succ : successors(BB))
>> > + if (L->contains(Succ))
>> > + BBWorklist.insert(Succ);
>> > }
>> > +
>> > + // If we found no optimization opportunities on the first
>> iteration, we
>> > + // won't find them on later ones too.
>> > + if (!NumberOfOptimizedInstructions)
>> > + return false;
>> > }
>> > - return NumberOfOptimizedInstructions;
>> > + return true;
>> > }
>> > };
>> > } // namespace
>> >
>> > -// Complete loop unrolling can make some loads constant, and we need
>> to know if
>> > -// that would expose any further optimization opportunities.
>> > -// This routine estimates this optimization effect and returns the
>> number of
>> > -// instructions, that potentially might be optimized away.
>> > -static unsigned
>> > -approximateNumberOfOptimizedInstructions(const Loop *L,
>> ScalarEvolution &SE,
>> > - unsigned TripCount,
>> > - const TargetTransformInfo
>> &TTI) {
>> > - if (!TripCount || !UnrollMaxIterationsCountToAnalyze)
>> > - return 0;
>> > -
>> > - UnrollAnalyzer UA(L, TripCount, SE, TTI);
>> > - UA.findConstFoldableLoads();
>> > -
>> > - // Estimate number of instructions, that could be simplified if we
>> replace a
>> > - // load with the corresponding constant. Since the same load will
>> take
>> > - // different values on different iterations, we have to go through
>> all loop's
>> > - // iterations here. To limit ourselves here, we check only first N
>> > - // iterations, and then scale the found number, if necessary.
>> > - unsigned IterationsNumberForEstimate =
>> > - std::min<unsigned>(UnrollMaxIterationsCountToAnalyze, TripCount);
>> > - unsigned NumberOfOptimizedInstructions = 0;
>> > - for (unsigned i = 0; i < IterationsNumberForEstimate; ++i)
>> > - NumberOfOptimizedInstructions +=
>> > - UA.estimateNumberOfOptimizedInstructions(i);
>> > -
>> > - NumberOfOptimizedInstructions *= TripCount /
>> IterationsNumberForEstimate;
>> > -
>> > - return NumberOfOptimizedInstructions;
>> > -}
>> > -
>> > /// ApproximateLoopSize - Approximate the size of the loop.
>> > static unsigned ApproximateLoopSize(const Loop *L, unsigned &NumCalls,
>> > bool &NotDuplicatable,
>> > @@ -679,6 +681,49 @@ static void SetLoopAlreadyUnrolled(Loop
>> > L->setLoopID(NewLoopID);
>> > }
>> >
>> > +bool LoopUnroll::canUnrollCompletely(
>> > + Loop *L, unsigned Threshold, unsigned AbsoluteThreshold,
>> > + uint64_t UnrolledSize, unsigned NumberOfOptimizedInstructions,
>> > + unsigned PercentOfOptimizedForCompleteUnroll) {
>> > +
>> > + if (Threshold == NoThreshold) {
>> > + DEBUG(dbgs() << " Can fully unroll, because no threshold is
>> set.\n");
>> > + return true;
>> > + }
>> > +
>> > + if (UnrolledSize <= Threshold) {
>> > + DEBUG(dbgs() << " Can fully unroll, because unrolled size: "
>> > + << UnrolledSize << "<" << Threshold << "\n");
>> > + return true;
>> > + }
>> > +
>> > + assert(UnrolledSize && "UnrolledSize can't be 0 at this point.");
>> > + unsigned PercentOfOptimizedInstructions =
>> > + (uint64_t)NumberOfOptimizedInstructions * 100ull / UnrolledSize;
>> > +
>> > + if (UnrolledSize <= AbsoluteThreshold &&
>> > + PercentOfOptimizedInstructions >=
>> PercentOfOptimizedForCompleteUnroll) {
>> > + DEBUG(dbgs() << " Can fully unroll, because unrolling will help
>> removing "
>> > + << PercentOfOptimizedInstructions
>> > + << "% instructions (threshold: "
>> > + << PercentOfOptimizedForCompleteUnroll << "%)\n");
>> > + DEBUG(dbgs() << " Unrolled size (" << UnrolledSize
>> > + << ") is less than the threshold (" <<
>> AbsoluteThreshold
>> > + << ").\n");
>> > + return true;
>> > + }
>> > +
>> > + DEBUG(dbgs() << " Too large to fully unroll:\n");
>> > + DEBUG(dbgs() << " Unrolled size: " << UnrolledSize << "\n");
>> > + DEBUG(dbgs() << " Estimated number of optimized instructions: "
>> > + << NumberOfOptimizedInstructions << "\n");
>> > + DEBUG(dbgs() << " Absolute threshold: " << AbsoluteThreshold <<
>> "\n");
>> > + DEBUG(dbgs() << " Minimum percent of removed instructions: "
>> > + << PercentOfOptimizedForCompleteUnroll << "\n");
>> > + DEBUG(dbgs() << " Threshold for small loops: " << Threshold <<
>> "\n");
>> > + return false;
>> > +}
>> > +
>> > unsigned LoopUnroll::selectUnrollCount(
>> > const Loop *L, unsigned TripCount, bool PragmaFullUnroll,
>> > unsigned PragmaCount, const
>> TargetTransformInfo::UnrollingPreferences &UP,
>> > @@ -785,27 +830,34 @@ bool LoopUnroll::runOnLoop(Loop *L, LPPa
>> > return false;
>> > }
>> >
>> > - unsigned NumberOfOptimizedInstructions =
>> > - approximateNumberOfOptimizedInstructions(L, *SE, TripCount, TTI);
>> > - DEBUG(dbgs() << " Complete unrolling could save: "
>> > - << NumberOfOptimizedInstructions << "\n");
>> > -
>> > unsigned Threshold, PartialThreshold;
>> > + unsigned AbsoluteThreshold, PercentOfOptimizedForCompleteUnroll;
>> > selectThresholds(L, HasPragma, UP, Threshold, PartialThreshold,
>> > - NumberOfOptimizedInstructions);
>> > + AbsoluteThreshold,
>> PercentOfOptimizedForCompleteUnroll);
>> >
>> > // Given Count, TripCount and thresholds determine the type of
>> > // unrolling which is to be performed.
>> > enum { Full = 0, Partial = 1, Runtime = 2 };
>> > int Unrolling;
>> > if (TripCount && Count == TripCount) {
>> > - if (Threshold != NoThreshold && UnrolledSize > Threshold) {
>> > - DEBUG(dbgs() << " Too large to fully unroll with count: " <<
>> Count
>> > - << " because size: " << UnrolledSize << ">" <<
>> Threshold
>> > - << "\n");
>> > - Unrolling = Partial;
>> > - } else {
>> > + Unrolling = Partial;
>> > + // If the loop is really small, we don't need to run an expensive
>> analysis.
>> > + if (canUnrollCompletely(
>> > + L, Threshold, AbsoluteThreshold,
>> > + UnrolledSize, 0, 100)) {
>> > Unrolling = Full;
>> > + } else {
>> > + // The loop isn't that small, but we still can fully unroll it
>> if that
>> > + // helps to remove a significant number of instructions.
>> > + // To check that, run additional analysis on the loop.
>> > + UnrollAnalyzer UA(L, TripCount, *SE, TTI, AbsoluteThreshold);
>> > + if (UA.analyzeLoop() &&
>> > + canUnrollCompletely(L, Threshold, AbsoluteThreshold,
>> > + UA.UnrolledLoopSize,
>> > + UA.NumberOfOptimizedInstructions,
>> > + PercentOfOptimizedForCompleteUnroll)) {
>> > + Unrolling = Full;
>> > + }
>> > }
>> > } else if (TripCount && Count < TripCount) {
>> > Unrolling = Partial;
>> >
>> > Added: llvm/trunk/test/Transforms/LoopUnroll/full-unroll-bad-geps.ll
>> > URL:
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/LoopUnroll/full-unroll-bad-geps.ll?rev=237156&view=auto
>> >
>> ==============================================================================
>> > --- llvm/trunk/test/Transforms/LoopUnroll/full-unroll-bad-geps.ll
>> (added)
>> > +++ llvm/trunk/test/Transforms/LoopUnroll/full-unroll-bad-geps.ll Tue
>> May 12 12:20:03 2015
>> > @@ -0,0 +1,34 @@
>> > +; Check that we don't crash on corner cases.
>> > +; RUN: opt < %s -S -loop-unroll
>> -unroll-max-iteration-count-to-analyze=1000 -unroll-absolute-threshold=10
>> -unroll-threshold=10 -unroll-percent-of-optimized-for-complete-unroll=20
>> -o /dev/null
>> > +target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
>> > +
>> > +define void @foo1() {
>> > +entry:
>> > + br label %for.body
>> > +
>> > +for.body:
>> > + %phi = phi i64 [ 0, %entry ], [ %inc, %for.body ]
>> > + %idx = zext i32 undef to i64
>> > + %add.ptr = getelementptr inbounds i64, i64* null, i64 %idx
>> > + %inc = add nuw nsw i64 %phi, 1
>> > + %cmp = icmp ult i64 %inc, 999
>> > + br i1 %cmp, label %for.body, label %for.exit
>> > +
>> > +for.exit:
>> > + ret void
>> > +}
>> > +
>> > +define void @foo2() {
>> > +entry:
>> > + br label %for.body
>> > +
>> > +for.body:
>> > + %phi = phi i64 [ 0, %entry ], [ %inc, %for.body ]
>> > + %x = getelementptr i32, <4 x i32*> undef, <4 x i32> <i32 1, i32 1,
>> i32 1, i32 1>
>> > + %inc = add nuw nsw i64 %phi, 1
>> > + %cmp = icmp ult i64 %inc, 999
>> > + br i1 %cmp, label %for.body, label %for.exit
>> > +
>> > +for.exit:
>> > + ret void
>> > +}
>> >
>> > Modified:
>> llvm/trunk/test/Transforms/LoopUnroll/full-unroll-heuristics.ll
>> > URL:
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/LoopUnroll/full-unroll-heuristics.ll?rev=237156&r1=237155&r2=237156&view=diff
>> >
>> ==============================================================================
>> > --- llvm/trunk/test/Transforms/LoopUnroll/full-unroll-heuristics.ll
>> (original)
>> > +++ llvm/trunk/test/Transforms/LoopUnroll/full-unroll-heuristics.ll Tue
>> May 12 12:20:03 2015
>> > @@ -17,8 +17,8 @@
>> > ; optimizations to remove ~55% of the instructions, the loop body size
>> is 9,
>> > ; and unrolled size is 65.
>> >
>> > -; RUN: opt < %s -S -loop-unroll
>> -unroll-max-iteration-count-to-analyze=1000 -unroll-absolute-threshold=10
>> -unroll-threshold=10 -unroll-percent-of-optimized-for-complete-unroll=30 |
>> FileCheck %s -check-prefix=TEST1
>> > -; RUN: opt < %s -S -loop-unroll
>> -unroll-max-iteration-count-to-analyze=1000 -unroll-absolute-threshold=100
>> -unroll-threshold=10 -unroll-percent-of-optimized-for-complete-unroll=30 |
>> FileCheck %s -check-prefix=TEST2
>> > +; RUN: opt < %s -S -loop-unroll
>> -unroll-max-iteration-count-to-analyze=1000 -unroll-absolute-threshold=10
>> -unroll-threshold=10 -unroll-percent-of-optimized-for-complete-unroll=20 |
>> FileCheck %s -check-prefix=TEST1
>> > +; RUN: opt < %s -S -loop-unroll
>> -unroll-max-iteration-count-to-analyze=1000 -unroll-absolute-threshold=100
>> -unroll-threshold=10 -unroll-percent-of-optimized-for-complete-unroll=20 |
>> FileCheck %s -check-prefix=TEST2
>> > ; RUN: opt < %s -S -loop-unroll
>> -unroll-max-iteration-count-to-analyze=1000 -unroll-absolute-threshold=100
>> -unroll-threshold=10 -unroll-percent-of-optimized-for-complete-unroll=80 |
>> FileCheck %s -check-prefix=TEST3
>> > ; RUN: opt < %s -S -loop-unroll
>> -unroll-max-iteration-count-to-analyze=1000 -unroll-absolute-threshold=100
>> -unroll-threshold=100 -unroll-percent-of-optimized-for-complete-unroll=80 |
>> FileCheck %s -check-prefix=TEST4
>> >
>> >
>> >
>> > _______________________________________________
>> > llvm-commits mailing list
>> > llvm-commits at cs.uiuc.edu
>> > http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20150521/865dc536/attachment.html>
More information about the llvm-commits
mailing list