[llvm] r265388 - Adds the ability to use an epilog remainder loop during loop unrolling and makes
Michael Zolotukhin via llvm-commits
llvm-commits at lists.llvm.org
Wed Jul 20 13:06:21 PDT 2016
Hi Evgeny,
Thanks for the detailed update! After a second thought - can we please switch the default to the previous value while you’re working on the patch? Generally, it’s bad to have such a big regression even though your original patch just exposed an issue in another place.
Thanks,
Michael
> On Jul 11, 2016, at 9:08 PM, Evgeny Stupachenko <evstupac at gmail.com> wrote:
>
> Hi Michael,
>
> The patch is under internal review. It changes a lot. I plan to send
> it here this month or in August.
> The patch fixes addressing, but not the regression.
> We looked into the regression itself on simulator. It appears that the
> big issue is branch misprediction.
>
> Regarding addresses.
> When we have prolog (with unroll on 2):
>
> for_body
> i = phi [i.next, i0]; //i0 is 0 or 1 depending on the prolog execution
> x = *(array + i);
> ...
> cmp i, N
> br for_body
>
> LSR is trying to simplify ind vars to one.
> lsr.i = phi [lsr.i.next, 0];
> or
> lsr.i = phi [lsr.i.next, -N];
> That way array access will have base: “array + i0” or “array + i0 - N”
> which is not static even if array is static.
>
> x = *((array + i0) + lsr.i);
>
> So it will require a register inside loop. Right now number of
> registers has the highest priority in LSR when it chooses a solution.
>
> When we have epilog, the same access is without “i0”
>
> x = *(array + lsr.i);
>
> If array is static we do not need an additional register.
> In the test where regression occurred the array is static and N is constant.
>
> However if we have enough registers we can use register instead of
> constant address. The patch should address this.
>
> Thanks,
> Evgeny
>
> On Mon, Jul 11, 2016 at 6:04 PM, Michael Zolotukhin
> <mzolotukhin at apple.com> wrote:
>>
>>> On Apr 22, 2016, at 11:35 AM, Michael Zolotukhin <mzolotukhin at apple.com> wrote:
>>>
>>>
>>>> On Apr 20, 2016, at 4:43 PM, Evgeny Stupachenko <evstupac at gmail.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> Yes if the test is that important. However I believe that the
>>>> regression caused randomly, but not by epilog unrolling.
>>>> On my Corei7 "I was able to reproduce it only in 32 bit mode, 64 bit
>>>> mode epilog is ~30% faster.".
>>>> The analysis showed no critical changes in the hottest loop.
>>>>
>>>> As for LSR issue.
>>>> Sorry for missing your request for posting PR. I'm preparing LSR patch
>>>> fixing the issue. And issue itself is related to already filed PR23384
>>>> (on an inefficient x86 LSR transformation).
>> Hi Evgeny,
>>
>> Is there any progress with the LSR patch? Or is there something blocking you?
>>
>> Thanks,
>> Michael
>>> Hi Evgeny,
>>>
>>> It’s fine with me to keep it on, I just want to make sure we address the regression. So, if you’re working on the patch, there are no concerns from my side.
>>>
>>> Thanks,
>>> Michael
>>>
>>>>
>>>> Thanks,
>>>> Evgeny
>>>>
>>>> On Wed, Apr 20, 2016 at 11:41 AM, Michael Zolotukhin
>>>> <mzolotukhin at apple.com> wrote:
>>>>> Hi,
>>>>>
>>>>> Could we change the default value to the original one until this 90%
>>>>> regression is addressed?
>>>>>
>>>>> Thanks,
>>>>> Michael
>>>>>
>>>>> On Apr 15, 2016, at 12:37 PM, Mikhail Zolotukhin <mzolotukhin at apple.com>
>>>>> wrote:
>>>>>
>>>>>
>>>>> On Apr 8, 2016, at 5:39 PM, Evgeny Stupachenko <evstupac at gmail.com> wrote:
>>>>>
>>>>> Investigation showed that the regression is most likely LSR related,
>>>>> but not loop unroll.
>>>>> I was able to reproduce it only in 32 bit mode, 64 bit mode epilog is
>>>>> ~30% faster.
>>>>>
>>>>> Going back to the changes in the test. Epilog variant has 1 move more
>>>>> in the hottest loop (which is redundant and should be deleted) and use
>>>>> constant address in moves inside the loop. However manual deleting of
>>>>> the move does not give any significant performance gain.
>>>>> The different behavior of LSR comes from additional inductive variable
>>>>> in unrolled loop. For Epilog case we add "k" (supposed to be
>>>>> eliminated after LSR):
>>>>>
>>>>> k = n - n % (unroll factor));
>>>>> ...
>>>>> access a[i];
>>>>> i++;
>>>>> k--;
>>>>> if (!k) break;
>>>>>
>>>>> Moving code of the loop modifying "palign" before loop showed up to 2
>>>>> times perf difference. So most likely epilog unroll itself does not
>>>>> influence on performance here.
>>>>> However, the fact that LSR leave constant address inside the loop
>>>>> moves (making them much longer), should be addressed to LSR. I'll
>>>>> submit a bug report on this.
>>>>>
>>>>> Hi Evgeny,
>>>>>
>>>>> Could you please post the PR number here as well?
>>>>>
>>>>> Michael
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Apr 8, 2016 at 2:09 PM, Evgeny Stupachenko <evstupac at gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi Michael,
>>>>>
>>>>> Yes I do have the data.
>>>>> There were ~10% improvements on some EEMBC tests; spec2000 performance
>>>>> was almost flat (within 2%).
>>>>>
>>>>> Let me look into BubbleSort case and come back with analysis and
>>>>> possible improvement.
>>>>>
>>>>> Thanks,
>>>>> Evgeny
>>>>>
>>>>>
>>>>> On Fri, Apr 8, 2016 at 1:23 PM, Michael Zolotukhin via llvm-commits
>>>>> <llvm-commits at lists.llvm.org> wrote:
>>>>>
>>>>> Hi Evgeny,
>>>>>
>>>>> We’ve found several performance regressions on LLVM testsuite caused by this
>>>>> patch. Do you have a data from your performance experiments to back-up the
>>>>> decision to make epilogues the default strategy?
>>>>>
>>>>> One of the biggest regressions we see is 89% on
>>>>> SingleSource/Benchmarks/Stanford/Bubblesort (on x86), while the biggest gain
>>>>> is only 30%.
>>>>>
>>>>> Thanks,
>>>>> Michael
>>>>>
>>>>>
>>>>> On Apr 5, 2016, at 5:19 AM, David L Kreitzer via llvm-commits
>>>>> <llvm-commits at lists.llvm.org> wrote:
>>>>>
>>>>> Author: dlkreitz
>>>>> Date: Tue Apr 5 07:19:35 2016
>>>>> New Revision: 265388
>>>>>
>>>>> URL: http://llvm.org/viewvc/llvm-project?rev=265388&view=rev
>>>>> Log:
>>>>> Adds the ability to use an epilog remainder loop during loop unrolling and
>>>>> makes
>>>>> this the default behavior.
>>>>>
>>>>> Patch by Evgeny Stupachenko (evstupac at gmail.com).
>>>>>
>>>>> Differential Revision: http://reviews.llvm.org/D18158
>>>>>
>>>>> Modified:
>>>>> llvm/trunk/include/llvm/Transforms/Utils/UnrollLoop.h
>>>>> llvm/trunk/lib/Transforms/Utils/LoopUnroll.cpp
>>>>> llvm/trunk/lib/Transforms/Utils/LoopUnrollRuntime.cpp
>>>>> llvm/trunk/test/Transforms/LoopUnroll/AArch64/runtime-loop.ll
>>>>> llvm/trunk/test/Transforms/LoopUnroll/PowerPC/a2-unrolling.ll
>>>>> llvm/trunk/test/Transforms/LoopUnroll/X86/mmx.ll
>>>>> llvm/trunk/test/Transforms/LoopUnroll/high-cost-trip-count-computation.ll
>>>>> llvm/trunk/test/Transforms/LoopUnroll/runtime-loop.ll
>>>>> llvm/trunk/test/Transforms/LoopUnroll/runtime-loop1.ll
>>>>> llvm/trunk/test/Transforms/LoopUnroll/runtime-loop2.ll
>>>>> llvm/trunk/test/Transforms/LoopUnroll/runtime-loop4.ll
>>>>> llvm/trunk/test/Transforms/LoopUnroll/runtime-loop5.ll
>>>>> llvm/trunk/test/Transforms/LoopUnroll/tripcount-overflow.ll
>>>>> llvm/trunk/test/Transforms/LoopUnroll/unroll-cleanup.ll
>>>>> llvm/trunk/test/Transforms/LoopUnroll/unroll-pragmas.ll
>>>>>
>>>>> Modified: llvm/trunk/include/llvm/Transforms/Utils/UnrollLoop.h
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Transforms/Utils/UnrollLoop.h?rev=265388&r1=265387&r2=265388&view=diff
>>>>> ==============================================================================
>>>>> --- llvm/trunk/include/llvm/Transforms/Utils/UnrollLoop.h (original)
>>>>> +++ llvm/trunk/include/llvm/Transforms/Utils/UnrollLoop.h Tue Apr 5
>>>>> 07:19:35 2016
>>>>> @@ -34,10 +34,11 @@ bool UnrollLoop(Loop *L, unsigned Count,
>>>>> LoopInfo *LI, ScalarEvolution *SE, DominatorTree *DT,
>>>>> AssumptionCache *AC, bool PreserveLCSSA);
>>>>>
>>>>> -bool UnrollRuntimeLoopProlog(Loop *L, unsigned Count,
>>>>> - bool AllowExpensiveTripCount, LoopInfo *LI,
>>>>> - ScalarEvolution *SE, DominatorTree *DT,
>>>>> - bool PreserveLCSSA);
>>>>> +bool UnrollRuntimeLoopRemainder(Loop *L, unsigned Count,
>>>>> + bool AllowExpensiveTripCount,
>>>>> + bool UseEpilogRemainder, LoopInfo *LI,
>>>>> + ScalarEvolution *SE, DominatorTree *DT,
>>>>> + bool PreserveLCSSA);
>>>>>
>>>>> MDNode *GetUnrollMetadata(MDNode *LoopID, StringRef Name);
>>>>> }
>>>>>
>>>>> Modified: llvm/trunk/lib/Transforms/Utils/LoopUnroll.cpp
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Utils/LoopUnroll.cpp?rev=265388&r1=265387&r2=265388&view=diff
>>>>> ==============================================================================
>>>>> --- llvm/trunk/lib/Transforms/Utils/LoopUnroll.cpp (original)
>>>>> +++ llvm/trunk/lib/Transforms/Utils/LoopUnroll.cpp Tue Apr 5 07:19:35 2016
>>>>> @@ -44,6 +44,11 @@ using namespace llvm;
>>>>> STATISTIC(NumCompletelyUnrolled, "Number of loops completely unrolled");
>>>>> STATISTIC(NumUnrolled, "Number of loops unrolled (completely or
>>>>> otherwise)");
>>>>>
>>>>> +static cl::opt<bool>
>>>>> +UnrollRuntimeEpilog("unroll-runtime-epilog", cl::init(true), cl::Hidden,
>>>>> + cl::desc("Allow runtime unrolled loops to be unrolled "
>>>>> + "with epilog instead of prolog."));
>>>>> +
>>>>> /// Convert the instruction operands from referencing the current values
>>>>> into
>>>>> /// those specified by VMap.
>>>>> static inline void remapInstruction(Instruction *I,
>>>>> @@ -288,12 +293,13 @@ bool llvm::UnrollLoop(Loop *L, unsigned
>>>>> "convergent "
>>>>> "operation.");
>>>>> });
>>>>> - // Don't output the runtime loop prolog if Count is a multiple of
>>>>> - // TripMultiple. Such a prolog is never needed, and is unsafe if the
>>>>> loop
>>>>> + // Don't output the runtime loop remainder if Count is a multiple of
>>>>> + // TripMultiple. Such a remainder is never needed, and is unsafe if the
>>>>> loop
>>>>> // contains a convergent instruction.
>>>>> if (RuntimeTripCount && TripMultiple % Count != 0 &&
>>>>> - !UnrollRuntimeLoopProlog(L, Count, AllowExpensiveTripCount, LI, SE,
>>>>> DT,
>>>>> - PreserveLCSSA))
>>>>> + !UnrollRuntimeLoopRemainder(L, Count, AllowExpensiveTripCount,
>>>>> + UnrollRuntimeEpilog, LI, SE, DT,
>>>>> + PreserveLCSSA))
>>>>> return false;
>>>>>
>>>>> // Notify ScalarEvolution that the loop will be substantially changed,
>>>>>
>>>>> Modified: llvm/trunk/lib/Transforms/Utils/LoopUnrollRuntime.cpp
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Utils/LoopUnrollRuntime.cpp?rev=265388&r1=265387&r2=265388&view=diff
>>>>> ==============================================================================
>>>>> --- llvm/trunk/lib/Transforms/Utils/LoopUnrollRuntime.cpp (original)
>>>>> +++ llvm/trunk/lib/Transforms/Utils/LoopUnrollRuntime.cpp Tue Apr 5
>>>>> 07:19:35 2016
>>>>> @@ -16,8 +16,8 @@
>>>>> // case, we need to generate code to execute these 'left over' iterations.
>>>>> //
>>>>> // The current strategy generates an if-then-else sequence prior to the
>>>>> -// unrolled loop to execute the 'left over' iterations. Other strategies
>>>>> -// include generate a loop before or after the unrolled loop.
>>>>> +// unrolled loop to execute the 'left over' iterations before or after the
>>>>> +// unrolled loop.
>>>>> //
>>>>> //===----------------------------------------------------------------------===//
>>>>>
>>>>> @@ -60,33 +60,35 @@ STATISTIC(NumRuntimeUnrolled,
>>>>> /// than the unroll factor.
>>>>> ///
>>>>> static void ConnectProlog(Loop *L, Value *BECount, unsigned Count,
>>>>> - BasicBlock *LastPrologBB, BasicBlock *PrologEnd,
>>>>> - BasicBlock *OrigPH, BasicBlock *NewPH,
>>>>> - ValueToValueMapTy &VMap, DominatorTree *DT,
>>>>> - LoopInfo *LI, bool PreserveLCSSA) {
>>>>> + BasicBlock *PrologExit, BasicBlock *PreHeader,
>>>>> + BasicBlock *NewPreHeader, ValueToValueMapTy
>>>>> &VMap,
>>>>> + DominatorTree *DT, LoopInfo *LI, bool
>>>>> PreserveLCSSA) {
>>>>> BasicBlock *Latch = L->getLoopLatch();
>>>>> assert(Latch && "Loop must have a latch");
>>>>> + BasicBlock *PrologLatch = cast<BasicBlock>(VMap[Latch]);
>>>>>
>>>>> // Create a PHI node for each outgoing value from the original loop
>>>>> // (which means it is an outgoing value from the prolog code too).
>>>>> // The new PHI node is inserted in the prolog end basic block.
>>>>> - // The new PHI name is added as an operand of a PHI node in either
>>>>> + // The new PHI node value is added as an operand of a PHI node in either
>>>>> // the loop header or the loop exit block.
>>>>> - for (succ_iterator SBI = succ_begin(Latch), SBE = succ_end(Latch);
>>>>> - SBI != SBE; ++SBI) {
>>>>> - for (BasicBlock::iterator BBI = (*SBI)->begin();
>>>>> - PHINode *PN = dyn_cast<PHINode>(BBI); ++BBI) {
>>>>> -
>>>>> + for (BasicBlock *Succ : successors(Latch)) {
>>>>> + for (Instruction &BBI : *Succ) {
>>>>> + PHINode *PN = dyn_cast<PHINode>(&BBI);
>>>>> + // Exit when we passed all PHI nodes.
>>>>> + if (!PN)
>>>>> + break;
>>>>> // Add a new PHI node to the prolog end block and add the
>>>>> // appropriate incoming values.
>>>>> - PHINode *NewPN = PHINode::Create(PN->getType(), 2,
>>>>> PN->getName()+".unr",
>>>>> - PrologEnd->getTerminator());
>>>>> + PHINode *NewPN = PHINode::Create(PN->getType(), 2, PN->getName() +
>>>>> ".unr",
>>>>> + PrologExit->getFirstNonPHI());
>>>>> // Adding a value to the new PHI node from the original loop preheader.
>>>>> // This is the value that skips all the prolog code.
>>>>> if (L->contains(PN)) {
>>>>> - NewPN->addIncoming(PN->getIncomingValueForBlock(NewPH), OrigPH);
>>>>> + NewPN->addIncoming(PN->getIncomingValueForBlock(NewPreHeader),
>>>>> + PreHeader);
>>>>> } else {
>>>>> - NewPN->addIncoming(UndefValue::get(PN->getType()), OrigPH);
>>>>> + NewPN->addIncoming(UndefValue::get(PN->getType()), PreHeader);
>>>>> }
>>>>>
>>>>> Value *V = PN->getIncomingValueForBlock(Latch);
>>>>> @@ -97,22 +99,22 @@ static void ConnectProlog(Loop *L, Value
>>>>> }
>>>>> // Adding a value to the new PHI node from the last prolog block
>>>>> // that was created.
>>>>> - NewPN->addIncoming(V, LastPrologBB);
>>>>> + NewPN->addIncoming(V, PrologLatch);
>>>>>
>>>>> // Update the existing PHI node operand with the value from the
>>>>> // new PHI node. How this is done depends on if the existing
>>>>> // PHI node is in the original loop block, or the exit block.
>>>>> if (L->contains(PN)) {
>>>>> - PN->setIncomingValue(PN->getBasicBlockIndex(NewPH), NewPN);
>>>>> + PN->setIncomingValue(PN->getBasicBlockIndex(NewPreHeader), NewPN);
>>>>> } else {
>>>>> - PN->addIncoming(NewPN, PrologEnd);
>>>>> + PN->addIncoming(NewPN, PrologExit);
>>>>> }
>>>>> }
>>>>> }
>>>>>
>>>>> // Create a branch around the original loop, which is taken if there are no
>>>>> // iterations remaining to be executed after running the prologue.
>>>>> - Instruction *InsertPt = PrologEnd->getTerminator();
>>>>> + Instruction *InsertPt = PrologExit->getTerminator();
>>>>> IRBuilder<> B(InsertPt);
>>>>>
>>>>> assert(Count != 0 && "nonsensical Count!");
>>>>> @@ -126,25 +128,152 @@ static void ConnectProlog(Loop *L, Value
>>>>> BasicBlock *Exit = L->getUniqueExitBlock();
>>>>> assert(Exit && "Loop must have a single exit block only");
>>>>> // Split the exit to maintain loop canonicalization guarantees
>>>>> - SmallVector<BasicBlock*, 4> Preds(pred_begin(Exit), pred_end(Exit));
>>>>> + SmallVector<BasicBlock*, 4> Preds(predecessors(Exit));
>>>>> SplitBlockPredecessors(Exit, Preds, ".unr-lcssa", DT, LI,
>>>>> PreserveLCSSA);
>>>>> // Add the branch to the exit block (around the unrolled loop)
>>>>> - B.CreateCondBr(BrLoopExit, Exit, NewPH);
>>>>> + B.CreateCondBr(BrLoopExit, Exit, NewPreHeader);
>>>>> + InsertPt->eraseFromParent();
>>>>> +}
>>>>> +
>>>>> +/// Connect the unrolling epilog code to the original loop.
>>>>> +/// The unrolling epilog code contains code to execute the
>>>>> +/// 'extra' iterations if the run-time trip count modulo the
>>>>> +/// unroll count is non-zero.
>>>>> +///
>>>>> +/// This function performs the following:
>>>>> +/// - Update PHI nodes at the unrolling loop exit and epilog loop exit
>>>>> +/// - Create PHI nodes at the unrolling loop exit to combine
>>>>> +/// values that exit the unrolling loop code and jump around it.
>>>>> +/// - Update PHI operands in the epilog loop by the new PHI nodes
>>>>> +/// - Branch around the epilog loop if extra iters (ModVal) is zero.
>>>>> +///
>>>>> +static void ConnectEpilog(Loop *L, Value *ModVal, BasicBlock *NewExit,
>>>>> + BasicBlock *Exit, BasicBlock *PreHeader,
>>>>> + BasicBlock *EpilogPreHeader, BasicBlock
>>>>> *NewPreHeader,
>>>>> + ValueToValueMapTy &VMap, DominatorTree *DT,
>>>>> + LoopInfo *LI, bool PreserveLCSSA) {
>>>>> + BasicBlock *Latch = L->getLoopLatch();
>>>>> + assert(Latch && "Loop must have a latch");
>>>>> + BasicBlock *EpilogLatch = cast<BasicBlock>(VMap[Latch]);
>>>>> +
>>>>> + // Loop structure should be the following:
>>>>> + //
>>>>> + // PreHeader
>>>>> + // NewPreHeader
>>>>> + // Header
>>>>> + // ...
>>>>> + // Latch
>>>>> + // NewExit (PN)
>>>>> + // EpilogPreHeader
>>>>> + // EpilogHeader
>>>>> + // ...
>>>>> + // EpilogLatch
>>>>> + // Exit (EpilogPN)
>>>>> +
>>>>> + // Update PHI nodes at NewExit and Exit.
>>>>> + for (Instruction &BBI : *NewExit) {
>>>>> + PHINode *PN = dyn_cast<PHINode>(&BBI);
>>>>> + // Exit when we passed all PHI nodes.
>>>>> + if (!PN)
>>>>> + break;
>>>>> + // PN should be used in another PHI located in Exit block as
>>>>> + // Exit was split by SplitBlockPredecessors into Exit and NewExit
>>>>> + // Basicaly it should look like:
>>>>> + // NewExit:
>>>>> + // PN = PHI [I, Latch]
>>>>> + // ...
>>>>> + // Exit:
>>>>> + // EpilogPN = PHI [PN, EpilogPreHeader]
>>>>> + //
>>>>> + // There is EpilogPreHeader incoming block instead of NewExit as
>>>>> + // NewExit was spilt 1 more time to get EpilogPreHeader.
>>>>> + assert(PN->hasOneUse() && "The phi should have 1 use");
>>>>> + PHINode *EpilogPN = cast<PHINode> (PN->use_begin()->getUser());
>>>>> + assert(EpilogPN->getParent() == Exit && "EpilogPN should be in Exit
>>>>> block");
>>>>> +
>>>>> + // Add incoming PreHeader from branch around the Loop
>>>>> + PN->addIncoming(UndefValue::get(PN->getType()), PreHeader);
>>>>> +
>>>>> + Value *V = PN->getIncomingValueForBlock(Latch);
>>>>> + Instruction *I = dyn_cast<Instruction>(V);
>>>>> + if (I && L->contains(I))
>>>>> + // If value comes from an instruction in the loop add VMap value.
>>>>> + V = VMap[I];
>>>>> + // For the instruction out of the loop, constant or undefined value
>>>>> + // insert value itself.
>>>>> + EpilogPN->addIncoming(V, EpilogLatch);
>>>>> +
>>>>> + assert(EpilogPN->getBasicBlockIndex(EpilogPreHeader) >= 0 &&
>>>>> + "EpilogPN should have EpilogPreHeader incoming block");
>>>>> + // Change EpilogPreHeader incoming block to NewExit.
>>>>> +
>>>>> EpilogPN->setIncomingBlock(EpilogPN->getBasicBlockIndex(EpilogPreHeader),
>>>>> + NewExit);
>>>>> + // Now PHIs should look like:
>>>>> + // NewExit:
>>>>> + // PN = PHI [I, Latch], [undef, PreHeader]
>>>>> + // ...
>>>>> + // Exit:
>>>>> + // EpilogPN = PHI [PN, NewExit], [VMap[I], EpilogLatch]
>>>>> + }
>>>>> +
>>>>> + // Create PHI nodes at NewExit (from the unrolling loop Latch and
>>>>> PreHeader).
>>>>> + // Update corresponding PHI nodes in epilog loop.
>>>>> + for (BasicBlock *Succ : successors(Latch)) {
>>>>> + // Skip this as we already updated phis in exit blocks.
>>>>> + if (!L->contains(Succ))
>>>>> + continue;
>>>>> + for (Instruction &BBI : *Succ) {
>>>>> + PHINode *PN = dyn_cast<PHINode>(&BBI);
>>>>> + // Exit when we passed all PHI nodes.
>>>>> + if (!PN)
>>>>> + break;
>>>>> + // Add new PHI nodes to the loop exit block and update epilog
>>>>> + // PHIs with the new PHI values.
>>>>> + PHINode *NewPN = PHINode::Create(PN->getType(), 2, PN->getName() +
>>>>> ".unr",
>>>>> + NewExit->getFirstNonPHI());
>>>>> + // Adding a value to the new PHI node from the unrolling loop
>>>>> preheader.
>>>>> + NewPN->addIncoming(PN->getIncomingValueForBlock(NewPreHeader),
>>>>> PreHeader);
>>>>> + // Adding a value to the new PHI node from the unrolling loop latch.
>>>>> + NewPN->addIncoming(PN->getIncomingValueForBlock(Latch), Latch);
>>>>> +
>>>>> + // Update the existing PHI node operand with the value from the new
>>>>> PHI
>>>>> + // node. Corresponding instruction in epilog loop should be PHI.
>>>>> + PHINode *VPN = cast<PHINode>(VMap[&BBI]);
>>>>> + VPN->setIncomingValue(VPN->getBasicBlockIndex(EpilogPreHeader),
>>>>> NewPN);
>>>>> + }
>>>>> + }
>>>>> +
>>>>> + Instruction *InsertPt = NewExit->getTerminator();
>>>>> + IRBuilder<> B(InsertPt);
>>>>> + Value *BrLoopExit = B.CreateIsNotNull(ModVal);
>>>>> + assert(Exit && "Loop must have a single exit block only");
>>>>> + // Split the exit to maintain loop canonicalization guarantees
>>>>> + SmallVector<BasicBlock*, 4> Preds(predecessors(Exit));
>>>>> + SplitBlockPredecessors(Exit, Preds, ".epilog-lcssa", DT, LI,
>>>>> + PreserveLCSSA);
>>>>> + // Add the branch to the exit block (around the unrolling loop)
>>>>> + B.CreateCondBr(BrLoopExit, EpilogPreHeader, Exit);
>>>>> InsertPt->eraseFromParent();
>>>>> }
>>>>>
>>>>> /// Create a clone of the blocks in a loop and connect them together.
>>>>> -/// If UnrollProlog is true, loop structure will not be cloned, otherwise a
>>>>> new
>>>>> -/// loop will be created including all cloned blocks, and the iterator of
>>>>> it
>>>>> -/// switches to count NewIter down to 0.
>>>>> +/// If CreateRemainderLoop is false, loop structure will not be cloned,
>>>>> +/// otherwise a new loop will be created including all cloned blocks, and
>>>>> the
>>>>> +/// iterator of it switches to count NewIter down to 0.
>>>>> +/// The cloned blocks should be inserted between InsertTop and InsertBot.
>>>>> +/// If loop structure is cloned InsertTop should be new preheader,
>>>>> InsertBot
>>>>> +/// new loop exit.
>>>>> ///
>>>>> -static void CloneLoopBlocks(Loop *L, Value *NewIter, const bool
>>>>> UnrollProlog,
>>>>> +static void CloneLoopBlocks(Loop *L, Value *NewIter,
>>>>> + const bool CreateRemainderLoop,
>>>>> + const bool UseEpilogRemainder,
>>>>> BasicBlock *InsertTop, BasicBlock *InsertBot,
>>>>> + BasicBlock *Preheader,
>>>>> std::vector<BasicBlock *> &NewBlocks,
>>>>> LoopBlocksDFS &LoopBlocks, ValueToValueMapTy
>>>>> &VMap,
>>>>> LoopInfo *LI) {
>>>>> - BasicBlock *Preheader = L->getLoopPreheader();
>>>>> + StringRef suffix = UseEpilogRemainder ? "epil" : "prol";
>>>>> BasicBlock *Header = L->getHeader();
>>>>> BasicBlock *Latch = L->getLoopLatch();
>>>>> Function *F = Header->getParent();
>>>>> @@ -152,7 +281,7 @@ static void CloneLoopBlocks(Loop *L, Val
>>>>> LoopBlocksDFS::RPOIterator BlockEnd = LoopBlocks.endRPO();
>>>>> Loop *NewLoop = nullptr;
>>>>> Loop *ParentLoop = L->getParentLoop();
>>>>> - if (!UnrollProlog) {
>>>>> + if (CreateRemainderLoop) {
>>>>> NewLoop = new Loop();
>>>>> if (ParentLoop)
>>>>> ParentLoop->addChildLoop(NewLoop);
>>>>> @@ -163,7 +292,7 @@ static void CloneLoopBlocks(Loop *L, Val
>>>>> // For each block in the original loop, create a new copy,
>>>>> // and update the value map with the newly created values.
>>>>> for (LoopBlocksDFS::RPOIterator BB = BlockBegin; BB != BlockEnd; ++BB) {
>>>>> - BasicBlock *NewBB = CloneBasicBlock(*BB, VMap, ".prol", F);
>>>>> + BasicBlock *NewBB = CloneBasicBlock(*BB, VMap, "." + suffix, F);
>>>>> NewBlocks.push_back(NewBB);
>>>>>
>>>>> if (NewLoop)
>>>>> @@ -179,16 +308,17 @@ static void CloneLoopBlocks(Loop *L, Val
>>>>> }
>>>>>
>>>>> if (Latch == *BB) {
>>>>> - // For the last block, if UnrollProlog is true, create a direct jump
>>>>> to
>>>>> - // InsertBot. If not, create a loop back to cloned head.
>>>>> + // For the last block, if CreateRemainderLoop is false, create a
>>>>> direct
>>>>> + // jump to InsertBot. If not, create a loop back to cloned head.
>>>>> VMap.erase((*BB)->getTerminator());
>>>>> BasicBlock *FirstLoopBB = cast<BasicBlock>(VMap[Header]);
>>>>> BranchInst *LatchBR = cast<BranchInst>(NewBB->getTerminator());
>>>>> IRBuilder<> Builder(LatchBR);
>>>>> - if (UnrollProlog) {
>>>>> + if (!CreateRemainderLoop) {
>>>>> Builder.CreateBr(InsertBot);
>>>>> } else {
>>>>> - PHINode *NewIdx = PHINode::Create(NewIter->getType(), 2,
>>>>> "prol.iter",
>>>>> + PHINode *NewIdx = PHINode::Create(NewIter->getType(), 2,
>>>>> + suffix + ".iter",
>>>>> FirstLoopBB->getFirstNonPHI());
>>>>> Value *IdxSub =
>>>>> Builder.CreateSub(NewIdx, ConstantInt::get(NewIdx->getType(), 1),
>>>>> @@ -207,9 +337,15 @@ static void CloneLoopBlocks(Loop *L, Val
>>>>> // cloned loop.
>>>>> for (BasicBlock::iterator I = Header->begin(); isa<PHINode>(I); ++I) {
>>>>> PHINode *NewPHI = cast<PHINode>(VMap[&*I]);
>>>>> - if (UnrollProlog) {
>>>>> - VMap[&*I] = NewPHI->getIncomingValueForBlock(Preheader);
>>>>> - cast<BasicBlock>(VMap[Header])->getInstList().erase(NewPHI);
>>>>> + if (!CreateRemainderLoop) {
>>>>> + if (UseEpilogRemainder) {
>>>>> + unsigned idx = NewPHI->getBasicBlockIndex(Preheader);
>>>>> + NewPHI->setIncomingBlock(idx, InsertTop);
>>>>> + NewPHI->removeIncomingValue(Latch, false);
>>>>> + } else {
>>>>> + VMap[&*I] = NewPHI->getIncomingValueForBlock(Preheader);
>>>>> + cast<BasicBlock>(VMap[Header])->getInstList().erase(NewPHI);
>>>>> + }
>>>>> } else {
>>>>> unsigned idx = NewPHI->getBasicBlockIndex(Preheader);
>>>>> NewPHI->setIncomingBlock(idx, InsertTop);
>>>>> @@ -254,7 +390,7 @@ static void CloneLoopBlocks(Loop *L, Val
>>>>> }
>>>>> }
>>>>>
>>>>> -/// Insert code in the prolog code when unrolling a loop with a
>>>>> +/// Insert code in the prolog/epilog code when unrolling a loop with a
>>>>> /// run-time trip-count.
>>>>> ///
>>>>> /// This method assumes that the loop unroll factor is total number
>>>>> @@ -266,6 +402,7 @@ static void CloneLoopBlocks(Loop *L, Val
>>>>> /// instruction in SimplifyCFG.cpp. Then, the backend decides how code for
>>>>> /// the switch instruction is generated.
>>>>> ///
>>>>> +/// ***Prolog case***
>>>>> /// extraiters = tripcount % loopfactor
>>>>> /// if (extraiters == 0) jump Loop:
>>>>> /// else jump Prol
>>>>> @@ -277,17 +414,35 @@ static void CloneLoopBlocks(Loop *L, Val
>>>>> /// ...
>>>>> /// End:
>>>>> ///
>>>>> -bool llvm::UnrollRuntimeLoopProlog(Loop *L, unsigned Count,
>>>>> - bool AllowExpensiveTripCount, LoopInfo
>>>>> *LI,
>>>>> - ScalarEvolution *SE, DominatorTree *DT,
>>>>> - bool PreserveLCSSA) {
>>>>> - // For now, only unroll loops that contain a single exit.
>>>>> +/// ***Epilog case***
>>>>> +/// extraiters = tripcount % loopfactor
>>>>> +/// if (extraiters == tripcount) jump LoopExit:
>>>>> +/// unroll_iters = tripcount - extraiters
>>>>> +/// Loop: LoopBody; (executes unroll_iter times);
>>>>> +/// unroll_iter -= 1
>>>>> +/// if (unroll_iter != 0) jump Loop:
>>>>> +/// LoopExit:
>>>>> +/// if (extraiters == 0) jump EpilExit:
>>>>> +/// Epil: LoopBody; (executes extraiters times)
>>>>> +/// extraiters -= 1 // Omitted if unroll factor is
>>>>> 2.
>>>>> +/// if (extraiters != 0) jump Epil: // Omitted if unroll factor is
>>>>> 2.
>>>>> +/// EpilExit:
>>>>> +
>>>>> +bool llvm::UnrollRuntimeLoopRemainder(Loop *L, unsigned Count,
>>>>> + bool AllowExpensiveTripCount,
>>>>> + bool UseEpilogRemainder,
>>>>> + LoopInfo *LI, ScalarEvolution *SE,
>>>>> + DominatorTree *DT, bool
>>>>> PreserveLCSSA) {
>>>>> + // for now, only unroll loops that contain a single exit
>>>>> if (!L->getExitingBlock())
>>>>> return false;
>>>>>
>>>>> // Make sure the loop is in canonical form, and there is a single
>>>>> // exit block only.
>>>>> - if (!L->isLoopSimplifyForm() || !L->getUniqueExitBlock())
>>>>> + if (!L->isLoopSimplifyForm())
>>>>> + return false;
>>>>> + BasicBlock *Exit = L->getUniqueExitBlock(); // successor out of loop
>>>>> + if (!Exit)
>>>>> return false;
>>>>>
>>>>> // Use Scalar Evolution to compute the trip count. This allows more loops to
>>>>> @@ -311,8 +466,8 @@ bool llvm::UnrollRuntimeLoopProlog(Loop
>>>>> return false;
>>>>>
>>>>> BasicBlock *Header = L->getHeader();
>>>>> - BasicBlock *PH = L->getLoopPreheader();
>>>>> - BranchInst *PreHeaderBR = cast<BranchInst>(PH->getTerminator());
>>>>> + BasicBlock *PreHeader = L->getLoopPreheader();
>>>>> + BranchInst *PreHeaderBR = cast<BranchInst>(PreHeader->getTerminator());
>>>>> const DataLayout &DL = Header->getModule()->getDataLayout();
>>>>> SCEVExpander Expander(*SE, DL, "loop-unroll");
>>>>> if (!AllowExpensiveTripCount &&
>>>>> @@ -330,26 +485,75 @@ bool llvm::UnrollRuntimeLoopProlog(Loop
>>>>> SE->forgetLoop(ParentLoop);
>>>>>
>>>>> BasicBlock *Latch = L->getLoopLatch();
>>>>> - // It helps to split the original preheader twice, one for the end of the
>>>>> - // prolog code and one for a new loop preheader.
>>>>> - BasicBlock *PEnd = SplitEdge(PH, Header, DT, LI);
>>>>> - BasicBlock *NewPH = SplitBlock(PEnd, PEnd->getTerminator(), DT, LI);
>>>>> - PreHeaderBR = cast<BranchInst>(PH->getTerminator());
>>>>>
>>>>> + // Loop structure is the following:
>>>>> + //
>>>>> + // PreHeader
>>>>> + // Header
>>>>> + // ...
>>>>> + // Latch
>>>>> + // Exit
>>>>> +
>>>>> + BasicBlock *NewPreHeader;
>>>>> + BasicBlock *NewExit = nullptr;
>>>>> + BasicBlock *PrologExit = nullptr;
>>>>> + BasicBlock *EpilogPreHeader = nullptr;
>>>>> + BasicBlock *PrologPreHeader = nullptr;
>>>>> +
>>>>> + if (UseEpilogRemainder) {
>>>>> + // If epilog remainder
>>>>> + // Split PreHeader to insert a branch around loop for unrolling.
>>>>> + NewPreHeader = SplitBlock(PreHeader, PreHeader->getTerminator(), DT,
>>>>> LI);
>>>>> + NewPreHeader->setName(PreHeader->getName() + ".new");
>>>>> + // Split Exit to create phi nodes from branch above.
>>>>> + SmallVector<BasicBlock*, 4> Preds(predecessors(Exit));
>>>>> + NewExit = SplitBlockPredecessors(Exit, Preds, ".unr-lcssa",
>>>>> + DT, LI, PreserveLCSSA);
>>>>> + // Split NewExit to insert epilog remainder loop.
>>>>> + EpilogPreHeader = SplitBlock(NewExit, NewExit->getTerminator(), DT,
>>>>> LI);
>>>>> + EpilogPreHeader->setName(Header->getName() + ".epil.preheader");
>>>>> + } else {
>>>>> + // If prolog remainder
>>>>> + // Split the original preheader twice to insert prolog remainder loop
>>>>> + PrologPreHeader = SplitEdge(PreHeader, Header, DT, LI);
>>>>> + PrologPreHeader->setName(Header->getName() + ".prol.preheader");
>>>>> + PrologExit = SplitBlock(PrologPreHeader,
>>>>> PrologPreHeader->getTerminator(),
>>>>> + DT, LI);
>>>>> + PrologExit->setName(Header->getName() + ".prol.loopexit");
>>>>> + // Split PrologExit to get NewPreHeader.
>>>>> + NewPreHeader = SplitBlock(PrologExit, PrologExit->getTerminator(), DT,
>>>>> LI);
>>>>> + NewPreHeader->setName(PreHeader->getName() + ".new");
>>>>> + }
>>>>> + // Loop structure should be the following:
>>>>> + // Epilog Prolog
>>>>> + //
>>>>> + // PreHeader PreHeader
>>>>> + // *NewPreHeader *PrologPreHeader
>>>>> + // Header *PrologExit
>>>>> + // ... *NewPreHeader
>>>>> + // Latch Header
>>>>> + // *NewExit ...
>>>>> + // *EpilogPreHeader Latch
>>>>> + // Exit Exit
>>>>> +
>>>>> + // Calculate conditions for branch around loop for unrolling
>>>>> + // in epilog case and around prolog remainder loop in prolog case.
>>>>> // Compute the number of extra iterations required, which is:
>>>>> - // extra iterations = run-time trip count % (loop unroll factor + 1)
>>>>> + // extra iterations = run-time trip count % loop unroll factor
>>>>> + PreHeaderBR = cast<BranchInst>(PreHeader->getTerminator());
>>>>> Value *TripCount = Expander.expandCodeFor(TripCountSC,
>>>>> TripCountSC->getType(),
>>>>> PreHeaderBR);
>>>>> Value *BECount = Expander.expandCodeFor(BECountSC, BECountSC->getType(),
>>>>> PreHeaderBR);
>>>>> -
>>>>> IRBuilder<> B(PreHeaderBR);
>>>>> Value *ModVal;
>>>>> // Calculate ModVal = (BECount + 1) % Count.
>>>>> // Note that TripCount is BECount + 1.
>>>>> if (isPowerOf2_32(Count)) {
>>>>> + // When Count is power of 2 we don't BECount for epilog case, however
>>>>> we'll
>>>>> + // need it for a branch around unrolling loop for prolog case.
>>>>> ModVal = B.CreateAnd(TripCount, Count - 1, "xtraiter");
>>>>> - // 1. There are no iterations to be run in the prologue loop.
>>>>> + // 1. There are no iterations to be run in the prolog/epilog loop.
>>>>> // OR
>>>>> // 2. The addition computing TripCount overflowed.
>>>>> //
>>>>> @@ -371,18 +575,18 @@ bool llvm::UnrollRuntimeLoopProlog(Loop
>>>>> ConstantInt::get(BECount->getType(), Count),
>>>>> "xtraiter");
>>>>> }
>>>>> - Value *BranchVal = B.CreateIsNotNull(ModVal, "lcmp.mod");
>>>>> -
>>>>> - // Branch to either the extra iterations or the cloned/unrolled loop.
>>>>> - // We will fix up the true branch label when adding loop body copies.
>>>>> - B.CreateCondBr(BranchVal, PEnd, PEnd);
>>>>> - assert(PreHeaderBR->isUnconditional() &&
>>>>> - PreHeaderBR->getSuccessor(0) == PEnd &&
>>>>> - "CFG edges in Preheader are not correct");
>>>>> + Value *CmpOperand =
>>>>> + UseEpilogRemainder ? TripCount :
>>>>> + ConstantInt::get(TripCount->getType(), 0);
>>>>> + Value *BranchVal = B.CreateICmpNE(ModVal, CmpOperand, "lcmp.mod");
>>>>> + BasicBlock *FirstLoop = UseEpilogRemainder ? NewPreHeader :
>>>>> PrologPreHeader;
>>>>> + BasicBlock *SecondLoop = UseEpilogRemainder ? NewExit : PrologExit;
>>>>> + // Branch to either remainder (extra iterations) loop or unrolling loop.
>>>>> + B.CreateCondBr(BranchVal, FirstLoop, SecondLoop);
>>>>> PreHeaderBR->eraseFromParent();
>>>>> Function *F = Header->getParent();
>>>>> // Get an ordered list of blocks in the loop to help with the ordering of
>>>>> the
>>>>> - // cloned blocks in the prolog code.
>>>>> + // cloned blocks in the prolog/epilog code
>>>>> LoopBlocksDFS LoopBlocks(L);
>>>>> LoopBlocks.perform(LI);
>>>>>
>>>>> @@ -394,17 +598,38 @@ bool llvm::UnrollRuntimeLoopProlog(Loop
>>>>> std::vector<BasicBlock *> NewBlocks;
>>>>> ValueToValueMapTy VMap;
>>>>>
>>>>> - bool UnrollPrologue = Count == 2;
>>>>> + // For unroll factor 2 remainder loop will have 1 iterations.
>>>>> + // Do not create 1 iteration loop.
>>>>> + bool CreateRemainderLoop = (Count != 2);
>>>>>
>>>>> // Clone all the basic blocks in the loop. If Count is 2, we don't clone
>>>>> // the loop, otherwise we create a cloned loop to execute the extra
>>>>> // iterations. This function adds the appropriate CFG connections.
>>>>> - CloneLoopBlocks(L, ModVal, UnrollPrologue, PH, PEnd, NewBlocks,
>>>>> LoopBlocks,
>>>>> - VMap, LI);
>>>>> + BasicBlock *InsertBot = UseEpilogRemainder ? Exit : PrologExit;
>>>>> + BasicBlock *InsertTop = UseEpilogRemainder ? EpilogPreHeader :
>>>>> PrologPreHeader;
>>>>> + CloneLoopBlocks(L, ModVal, CreateRemainderLoop, UseEpilogRemainder,
>>>>> InsertTop,
>>>>> + InsertBot, NewPreHeader, NewBlocks, LoopBlocks, VMap,
>>>>> LI);
>>>>> +
>>>>> + // Insert the cloned blocks into the function.
>>>>> + F->getBasicBlockList().splice(InsertBot->getIterator(),
>>>>> + F->getBasicBlockList(),
>>>>> + NewBlocks[0]->getIterator(),
>>>>> + F->end());
>>>>>
>>>>> - // Insert the cloned blocks into the function just before the original
>>>>> loop.
>>>>> - F->getBasicBlockList().splice(PEnd->getIterator(),
>>>>> F->getBasicBlockList(),
>>>>> - NewBlocks[0]->getIterator(), F->end());
>>>>> + // Loop structure should be the following:
>>>>> + // Epilog Prolog
>>>>> + //
>>>>> + // PreHeader PreHeader
>>>>> + // NewPreHeader PrologPreHeader
>>>>> + // Header PrologHeader
>>>>> + // ... ...
>>>>> + // Latch PrologLatch
>>>>> + // NewExit PrologExit
>>>>> + // EpilogPreHeader NewPreHeader
>>>>> + // EpilogHeader Header
>>>>> + // ... ...
>>>>> + // EpilogLatch Latch
>>>>> + // Exit Exit
>>>>>
>>>>> // Rewrite the cloned instruction operands to use the values created when
>>>>> the
>>>>> // clone is created.
>>>>> @@ -415,11 +640,38 @@ bool llvm::UnrollRuntimeLoopProlog(Loop
>>>>> }
>>>>> }
>>>>>
>>>>> - // Connect the prolog code to the original loop and update the
>>>>> - // PHI functions.
>>>>> - BasicBlock *LastLoopBB = cast<BasicBlock>(VMap[Latch]);
>>>>> - ConnectProlog(L, BECount, Count, LastLoopBB, PEnd, PH, NewPH, VMap, DT,
>>>>> LI,
>>>>> - PreserveLCSSA);
>>>>> + if (UseEpilogRemainder) {
>>>>> + // Connect the epilog code to the original loop and update the
>>>>> + // PHI functions.
>>>>> + ConnectEpilog(L, ModVal, NewExit, Exit, PreHeader,
>>>>> + EpilogPreHeader, NewPreHeader, VMap, DT, LI,
>>>>> + PreserveLCSSA);
>>>>> +
>>>>> + // Update counter in loop for unrolling.
>>>>> + // I should be multiply of Count.
>>>>> + IRBuilder<> B2(NewPreHeader->getTerminator());
>>>>> + Value *TestVal = B2.CreateSub(TripCount, ModVal, "unroll_iter");
>>>>> + BranchInst *LatchBR = cast<BranchInst>(Latch->getTerminator());
>>>>> + B2.SetInsertPoint(LatchBR);
>>>>> + PHINode *NewIdx = PHINode::Create(TestVal->getType(), 2, "niter",
>>>>> + Header->getFirstNonPHI());
>>>>> + Value *IdxSub =
>>>>> + B2.CreateSub(NewIdx, ConstantInt::get(NewIdx->getType(), 1),
>>>>> + NewIdx->getName() + ".nsub");
>>>>> + Value *IdxCmp;
>>>>> + if (LatchBR->getSuccessor(0) == Header)
>>>>> + IdxCmp = B2.CreateIsNotNull(IdxSub, NewIdx->getName() + ".ncmp");
>>>>> + else
>>>>> + IdxCmp = B2.CreateIsNull(IdxSub, NewIdx->getName() + ".ncmp");
>>>>> + NewIdx->addIncoming(TestVal, NewPreHeader);
>>>>> + NewIdx->addIncoming(IdxSub, Latch);
>>>>> + LatchBR->setCondition(IdxCmp);
>>>>> + } else {
>>>>> + // Connect the prolog code to the original loop and update the
>>>>> + // PHI functions.
>>>>> + ConnectProlog(L, BECount, Count, PrologExit, PreHeader, NewPreHeader,
>>>>> + VMap, DT, LI, PreserveLCSSA);
>>>>> + }
>>>>> NumRuntimeUnrolled++;
>>>>> return true;
>>>>> }
>>>>>
>>>>> Modified: llvm/trunk/test/Transforms/LoopUnroll/AArch64/runtime-loop.ll
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/LoopUnroll/AArch64/runtime-loop.ll?rev=265388&r1=265387&r2=265388&view=diff
>>>>> ==============================================================================
>>>>> --- llvm/trunk/test/Transforms/LoopUnroll/AArch64/runtime-loop.ll (original)
>>>>> +++ llvm/trunk/test/Transforms/LoopUnroll/AArch64/runtime-loop.ll Tue Apr 5
>>>>> 07:19:35 2016
>>>>> @@ -1,13 +1,21 @@
>>>>> -; RUN: opt < %s -S -loop-unroll -mtriple aarch64 -mcpu=cortex-a57 |
>>>>> FileCheck %s
>>>>> +; RUN: opt < %s -S -loop-unroll -mtriple aarch64 -mcpu=cortex-a57 |
>>>>> FileCheck %s -check-prefix=EPILOG
>>>>> +; RUN: opt < %s -S -loop-unroll -mtriple aarch64 -mcpu=cortex-a57
>>>>> -unroll-runtime-epilog=false | FileCheck %s -check-prefix=PROLOG
>>>>>
>>>>> ; Tests for unrolling loops with run-time trip counts
>>>>>
>>>>> -; CHECK: %xtraiter = and i32 %n
>>>>> -; CHECK: %lcmp.mod = icmp ne i32 %xtraiter, 0
>>>>> -; CHECK: br i1 %lcmp.mod, label %for.body.prol, label
>>>>> %for.body.preheader.split
>>>>> +; EPILOG: %xtraiter = and i32 %n
>>>>> +; EPILOG: %lcmp.mod = icmp ne i32 %xtraiter, %n
>>>>> +; EPILOG: br i1 %lcmp.mod, label %for.body.preheader.new, label
>>>>> %for.end.loopexit.unr-lcssa
>>>>>
>>>>> -; CHECK: for.body.prol:
>>>>> -; CHECK: for.body:
>>>>> +; PROLOG: %xtraiter = and i32 %n
>>>>> +; PROLOG: %lcmp.mod = icmp ne i32 %xtraiter, 0
>>>>> +; PROLOG: br i1 %lcmp.mod, label %for.body.prol.preheader, label
>>>>> %for.body.prol.loopexit
>>>>> +
>>>>> +; EPILOG: for.body:
>>>>> +; EPILOG: for.body.epil:
>>>>> +
>>>>> +; PROLOG: for.body.prol:
>>>>> +; PROLOG: for.body:
>>>>>
>>>>> define i32 @test(i32* nocapture %a, i32 %n) nounwind uwtable readonly {
>>>>> entry:
>>>>>
>>>>> Modified: llvm/trunk/test/Transforms/LoopUnroll/PowerPC/a2-unrolling.ll
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/LoopUnroll/PowerPC/a2-unrolling.ll?rev=265388&r1=265387&r2=265388&view=diff
>>>>> ==============================================================================
>>>>> --- llvm/trunk/test/Transforms/LoopUnroll/PowerPC/a2-unrolling.ll (original)
>>>>> +++ llvm/trunk/test/Transforms/LoopUnroll/PowerPC/a2-unrolling.ll Tue Apr 5
>>>>> 07:19:35 2016
>>>>> @@ -1,4 +1,5 @@
>>>>> -; RUN: opt < %s -S -mtriple=powerpc64-unknown-linux-gnu -mcpu=a2
>>>>> -loop-unroll | FileCheck %s
>>>>> +; RUN: opt < %s -S -mtriple=powerpc64-unknown-linux-gnu -mcpu=a2
>>>>> -loop-unroll | FileCheck %s -check-prefix=EPILOG
>>>>> +; RUN: opt < %s -S -mtriple=powerpc64-unknown-linux-gnu -mcpu=a2
>>>>> -loop-unroll -unroll-runtime-epilog=false | FileCheck %s
>>>>> -check-prefix=PROLOG
>>>>> define void @unroll_opt_for_size() nounwind optsize {
>>>>> entry:
>>>>> br label %loop
>>>>> @@ -13,11 +14,17 @@ exit:
>>>>> ret void
>>>>> }
>>>>>
>>>>> -; CHECK-LABEL: @unroll_opt_for_size
>>>>> -; CHECK: add
>>>>> -; CHECK-NEXT: add
>>>>> -; CHECK-NEXT: add
>>>>> -; CHECK: icmp
>>>>> +; EPILOG-LABEL: @unroll_opt_for_size
>>>>> +; EPILOG: add
>>>>> +; EPILOG-NEXT: add
>>>>> +; EPILOG-NEXT: add
>>>>> +; EPILOG: icmp
>>>>> +
>>>>> +; PROLOG-LABEL: @unroll_opt_for_size
>>>>> +; PROLOG: add
>>>>> +; PROLOG-NEXT: add
>>>>> +; PROLOG-NEXT: add
>>>>> +; PROLOG: icmp
>>>>>
>>>>> define i32 @test(i32* nocapture %a, i32 %n) nounwind uwtable readonly {
>>>>> entry:
>>>>> @@ -40,8 +47,13 @@ for.end:
>>>>> ret i32 %sum.0.lcssa
>>>>> }
>>>>>
>>>>> -; CHECK-LABEL: @test
>>>>> -; CHECK: for.body.prol{{.*}}:
>>>>> -; CHECK: for.body:
>>>>> -; CHECK: br i1 %exitcond.7, label %for.end.loopexit{{.*}}, label %for.body
>>>>> +; EPILOG-LABEL: @test
>>>>> +; EPILOG: for.body:
>>>>> +; EPILOG: br i1 %niter.ncmp.7, label %for.end.loopexit{{.*}}, label
>>>>> %for.body
>>>>> +; EPILOG: for.body.epil{{.*}}:
>>>>> +
>>>>> +; PROLOG-LABEL: @test
>>>>> +; PROLOG: for.body.prol{{.*}}:
>>>>> +; PROLOG: for.body:
>>>>> +; PROLOG: br i1 %exitcond.7, label %for.end.loopexit{{.*}}, label %for.body
>>>>>
>>>>>
>>>>> Modified: llvm/trunk/test/Transforms/LoopUnroll/X86/mmx.ll
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/LoopUnroll/X86/mmx.ll?rev=265388&r1=265387&r2=265388&view=diff
>>>>> ==============================================================================
>>>>> --- llvm/trunk/test/Transforms/LoopUnroll/X86/mmx.ll (original)
>>>>> +++ llvm/trunk/test/Transforms/LoopUnroll/X86/mmx.ll Tue Apr 5 07:19:35
>>>>> 2016
>>>>> @@ -14,9 +14,9 @@ for.body:
>>>>>
>>>>> exit: ; preds = %for.body
>>>>> %ret = phi x86_mmx [ undef, %for.body ]
>>>>> - ; CHECK: %[[ret_unr:.*]] = phi x86_mmx [ undef,
>>>>> - ; CHECK: %[[ret_ph:.*]] = phi x86_mmx [ undef,
>>>>> - ; CHECK: %[[ret:.*]] = phi x86_mmx [ %[[ret_unr]], {{.*}} ], [
>>>>> %[[ret_ph]]
>>>>> + ; CHECK: %[[ret_ph:.*]] = phi x86_mmx [ undef, %entry
>>>>> + ; CHECK: %[[ret_ph1:.*]] = phi x86_mmx [ undef,
>>>>> + ; CHECK: %[[ret:.*]] = phi x86_mmx [ %[[ret_ph]], {{.*}} ], [
>>>>> %[[ret_ph1]],
>>>>> ; CHECK: ret x86_mmx %[[ret]]
>>>>> ret x86_mmx %ret
>>>>> }
>>>>>
>>>>> Modified:
>>>>> llvm/trunk/test/Transforms/LoopUnroll/high-cost-trip-count-computation.ll
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/LoopUnroll/high-cost-trip-count-computation.ll?rev=265388&r1=265387&r2=265388&view=diff
>>>>> ==============================================================================
>>>>> ---
>>>>> llvm/trunk/test/Transforms/LoopUnroll/high-cost-trip-count-computation.ll
>>>>> (original)
>>>>> +++
>>>>> llvm/trunk/test/Transforms/LoopUnroll/high-cost-trip-count-computation.ll
>>>>> Tue Apr 5 07:19:35 2016
>>>>> @@ -34,7 +34,7 @@ define i32 @test2(i64* %loc, i64 %conv7)
>>>>> ; CHECK: udiv
>>>>> ; CHECK: udiv
>>>>> ; CHECK-NOT: udiv
>>>>> -; CHECK-LABEL: for.body.prol
>>>>> +; CHECK-LABEL: for.body
>>>>> entry:
>>>>> %rem0 = load i64, i64* %loc, align 8
>>>>> %ExpensiveComputation = udiv i64 %rem0, 42 ; <<< Extra computations are
>>>>> added to the trip-count expression
>>>>>
>>>>> Modified: llvm/trunk/test/Transforms/LoopUnroll/runtime-loop.ll
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/LoopUnroll/runtime-loop.ll?rev=265388&r1=265387&r2=265388&view=diff
>>>>> ==============================================================================
>>>>> --- llvm/trunk/test/Transforms/LoopUnroll/runtime-loop.ll (original)
>>>>> +++ llvm/trunk/test/Transforms/LoopUnroll/runtime-loop.ll Tue Apr 5
>>>>> 07:19:35 2016
>>>>> @@ -1,18 +1,30 @@
>>>>> -; RUN: opt < %s -S -loop-unroll -unroll-runtime=true | FileCheck %s
>>>>> +; RUN: opt < %s -S -loop-unroll -unroll-runtime=true | FileCheck %s
>>>>> -check-prefix=EPILOG
>>>>> +; RUN: opt < %s -S -loop-unroll -unroll-runtime=true
>>>>> -unroll-runtime-epilog=false | FileCheck %s -check-prefix=PROLOG
>>>>>
>>>>> target datalayout =
>>>>> "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
>>>>>
>>>>> ; Tests for unrolling loops with run-time trip counts
>>>>>
>>>>> -; CHECK: %xtraiter = and i32 %n
>>>>> -; CHECK: %lcmp.mod = icmp ne i32 %xtraiter, 0
>>>>> -; CHECK: br i1 %lcmp.mod, label %for.body.prol, label
>>>>> %for.body.preheader.split
>>>>> -
>>>>> -; CHECK: for.body.prol:
>>>>> -; CHECK: %indvars.iv.prol = phi i64 [ %indvars.iv.next.prol, %for.body.prol
>>>>> ], [ 0, %for.body.preheader ]
>>>>> -; CHECK: %prol.iter.sub = sub i32 %prol.iter, 1
>>>>> -; CHECK: %prol.iter.cmp = icmp ne i32 %prol.iter.sub, 0
>>>>> -; CHECK: br i1 %prol.iter.cmp, label %for.body.prol, label
>>>>> %for.body.preheader.split, !llvm.loop !0
>>>>> +; EPILOG: %xtraiter = and i32 %n
>>>>> +; EPILOG: %lcmp.mod = icmp ne i32 %xtraiter, %n
>>>>> +; EPILOG: br i1 %lcmp.mod, label %for.body.preheader.new, label
>>>>> %for.end.loopexit.unr-lcssa
>>>>> +
>>>>> +; PROLOG: %xtraiter = and i32 %n
>>>>> +; PROLOG: %lcmp.mod = icmp ne i32 %xtraiter, 0
>>>>> +; PROLOG: br i1 %lcmp.mod, label %for.body.prol.preheader, label
>>>>> %for.body.prol.loopexit
>>>>> +
>>>>> +; EPILOG: for.body.epil:
>>>>> +; EPILOG: %indvars.iv.epil = phi i64 [ %indvars.iv.next.epil,
>>>>> %for.body.epil ], [ %indvars.iv.unr, %for.body.epil.preheader ]
>>>>> +; EPILOG: %epil.iter.sub = sub i32 %epil.iter, 1
>>>>> +; EPILOG: %epil.iter.cmp = icmp ne i32 %epil.iter.sub, 0
>>>>> +; EPILOG: br i1 %epil.iter.cmp, label %for.body.epil, label
>>>>> %for.end.loopexit.epilog-lcssa, !llvm.loop !0
>>>>> +
>>>>> +; PROLOG: for.body.prol:
>>>>> +; PROLOG: %indvars.iv.prol = phi i64 [ %indvars.iv.next.prol,
>>>>> %for.body.prol ], [ 0, %for.body.prol.preheader ]
>>>>> +; PROLOG: %prol.iter.sub = sub i32 %prol.iter, 1
>>>>> +; PROLOG: %prol.iter.cmp = icmp ne i32 %prol.iter.sub, 0
>>>>> +; PROLOG: br i1 %prol.iter.cmp, label %for.body.prol, label
>>>>> %for.body.prol.loopexit, !llvm.loop !0
>>>>> +
>>>>>
>>>>> define i32 @test(i32* nocapture %a, i32 %n) nounwind uwtable readonly {
>>>>> entry:
>>>>> @@ -39,8 +51,11 @@ for.end:
>>>>> ; Still try to completely unroll loops with compile-time trip counts
>>>>> ; even if the -unroll-runtime is specified
>>>>>
>>>>> -; CHECK: for.body:
>>>>> -; CHECK-NOT: for.body.prol:
>>>>> +; EPILOG: for.body:
>>>>> +; EPILOG-NOT: for.body.epil:
>>>>> +
>>>>> +; PROLOG: for.body:
>>>>> +; PROLOG-NOT: for.body.prol:
>>>>>
>>>>> define i32 @test1(i32* nocapture %a) nounwind uwtable readonly {
>>>>> entry:
>>>>> @@ -64,7 +79,8 @@ for.end:
>>>>> ; This is test 2007-05-09-UnknownTripCount.ll which can be unrolled now
>>>>> ; if the -unroll-runtime option is turned on
>>>>>
>>>>> -; CHECK: bb72.2:
>>>>> +; EPILOG: bb72.2:
>>>>> +; PROLOG: bb72.2:
>>>>>
>>>>> define void @foo(i32 %trips) {
>>>>> entry:
>>>>> @@ -86,8 +102,11 @@ cond_true138:
>>>>>
>>>>> ; Test run-time unrolling for a loop that counts down by -2.
>>>>>
>>>>> -; CHECK: for.body.prol:
>>>>> -; CHECK: br i1 %prol.iter.cmp, label %for.body.prol, label
>>>>> %for.body.preheader.split
>>>>> +; EPILOG: for.body.epil:
>>>>> +; EPILOG: br i1 %epil.iter.cmp, label %for.body.epil, label
>>>>> %for.cond.for.end_crit_edge.epilog-lcssa
>>>>> +
>>>>> +; PROLOG: for.body.prol:
>>>>> +; PROLOG: br i1 %prol.iter.cmp, label %for.body.prol, label
>>>>> %for.body.prol.loopexit
>>>>>
>>>>> define zeroext i16 @down(i16* nocapture %p, i32 %len) nounwind uwtable
>>>>> readonly {
>>>>> entry:
>>>>> @@ -116,8 +135,11 @@ for.end:
>>>>> }
>>>>>
>>>>> ; Test run-time unrolling disable metadata.
>>>>> -; CHECK: for.body:
>>>>> -; CHECK-NOT: for.body.prol:
>>>>> +; EPILOG: for.body:
>>>>> +; EPILOG-NOT: for.body.epil:
>>>>> +
>>>>> +; PROLOG: for.body:
>>>>> +; PROLOG-NOT: for.body.prol:
>>>>>
>>>>> define zeroext i16 @test2(i16* nocapture %p, i32 %len) nounwind uwtable
>>>>> readonly {
>>>>> entry:
>>>>> @@ -148,6 +170,8 @@ for.end:
>>>>> !0 = distinct !{!0, !1}
>>>>> !1 = !{!"llvm.loop.unroll.runtime.disable"}
>>>>>
>>>>> -; CHECK: !0 = distinct !{!0, !1}
>>>>> -; CHECK: !1 = !{!"llvm.loop.unroll.disable"}
>>>>> +; EPILOG: !0 = distinct !{!0, !1}
>>>>> +; EPILOG: !1 = !{!"llvm.loop.unroll.disable"}
>>>>>
>>>>> +; PROLOG: !0 = distinct !{!0, !1}
>>>>> +; PROLOG: !1 = !{!"llvm.loop.unroll.disable"}
>>>>>
>>>>> Modified: llvm/trunk/test/Transforms/LoopUnroll/runtime-loop1.ll
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/LoopUnroll/runtime-loop1.ll?rev=265388&r1=265387&r2=265388&view=diff
>>>>> ==============================================================================
>>>>> --- llvm/trunk/test/Transforms/LoopUnroll/runtime-loop1.ll (original)
>>>>> +++ llvm/trunk/test/Transforms/LoopUnroll/runtime-loop1.ll Tue Apr 5
>>>>> 07:19:35 2016
>>>>> @@ -1,19 +1,35 @@
>>>>> -; RUN: opt < %s -S -loop-unroll -unroll-runtime -unroll-count=2 | FileCheck
>>>>> %s
>>>>> +; RUN: opt < %s -S -loop-unroll -unroll-runtime -unroll-count=2 | FileCheck
>>>>> %s -check-prefix=EPILOG
>>>>> +; RUN: opt < %s -S -loop-unroll -unroll-runtime -unroll-count=2
>>>>> -unroll-runtime-epilog=false | FileCheck %s -check-prefix=PROLOG
>>>>>
>>>>> ; This tests that setting the unroll count works
>>>>>
>>>>> -; CHECK: for.body.preheader:
>>>>> -; CHECK: br {{.*}} label %for.body.prol, label %for.body.preheader.split,
>>>>> !dbg [[PH_LOC:![0-9]+]]
>>>>> -; CHECK: for.body.prol:
>>>>> -; CHECK: br label %for.body.preheader.split, !dbg [[BODY_LOC:![0-9]+]]
>>>>> -; CHECK: for.body.preheader.split:
>>>>> -; CHECK: br {{.*}} label %for.end.loopexit, label
>>>>> %for.body.preheader.split.split, !dbg [[PH_LOC]]
>>>>> -; CHECK: for.body:
>>>>> -; CHECK: br i1 %exitcond.1, label %for.end.loopexit.unr-lcssa, label
>>>>> %for.body, !dbg [[BODY_LOC]]
>>>>> -; CHECK-NOT: br i1 %exitcond.4, label %for.end.loopexit{{.*}}, label
>>>>> %for.body
>>>>>
>>>>> -; CHECK-DAG: [[PH_LOC]] = !DILocation(line: 101, column: 1, scope: !{{.*}})
>>>>> -; CHECK-DAG: [[BODY_LOC]] = !DILocation(line: 102, column: 1, scope:
>>>>> !{{.*}})
>>>>> +; EPILOG: for.body.preheader:
>>>>> +; EPILOG: br i1 %lcmp.mod, label %for.body.preheader.new, label
>>>>> %for.end.loopexit.unr-lcssa, !dbg [[PH_LOC:![0-9]+]]
>>>>> +; EPILOG: for.body:
>>>>> +; EPILOG: br i1 %niter.ncmp.1, label
>>>>> %for.end.loopexit.unr-lcssa.loopexit, label %for.body, !dbg
>>>>> [[BODY_LOC:![0-9]+]]
>>>>> +; EPILOG-NOT: br i1 %niter.ncmp.2, label %for.end.loopexit{{.*}}, label
>>>>> %for.body
>>>>> +; EPILOG: for.body.epil.preheader:
>>>>> +; EPILOG: br label %for.body.epil, !dbg [[EXIT_LOC:![0-9]+]]
>>>>> +; EPILOG: for.body.epil:
>>>>> +; EPILOG: br label %for.end.loopexit.epilog-lcssa, !dbg
>>>>> [[BODY_LOC:![0-9]+]]
>>>>> +
>>>>> +; EPILOG-DAG: [[PH_LOC]] = !DILocation(line: 101, column: 1, scope:
>>>>> !{{.*}})
>>>>> +; EPILOG-DAG: [[BODY_LOC]] = !DILocation(line: 102, column: 1, scope:
>>>>> !{{.*}})
>>>>> +; EPILOG-DAG: [[EXIT_LOC]] = !DILocation(line: 103, column: 1, scope:
>>>>> !{{.*}})
>>>>> +
>>>>> +; PROLOG: for.body.preheader:
>>>>> +; PROLOG: br {{.*}} label %for.body.prol.preheader, label
>>>>> %for.body.prol.loopexit, !dbg [[PH_LOC:![0-9]+]]
>>>>> +; PROLOG: for.body.prol:
>>>>> +; PROLOG: br label %for.body.prol.loopexit, !dbg [[BODY_LOC:![0-9]+]]
>>>>> +; PROLOG: for.body.prol.loopexit:
>>>>> +; PROLOG: br {{.*}} label %for.end.loopexit, label
>>>>> %for.body.preheader.new, !dbg [[PH_LOC]]
>>>>> +; PROLOG: for.body:
>>>>> +; PROLOG: br i1 %exitcond.1, label %for.end.loopexit.unr-lcssa, label
>>>>> %for.body, !dbg [[BODY_LOC]]
>>>>> +; PROLOG-NOT: br i1 %exitcond.4, label %for.end.loopexit{{.*}}, label
>>>>> %for.body
>>>>> +
>>>>> +; PROLOG-DAG: [[PH_LOC]] = !DILocation(line: 101, column: 1, scope:
>>>>> !{{.*}})
>>>>> +; PROLOG-DAG: [[BODY_LOC]] = !DILocation(line: 102, column: 1, scope:
>>>>> !{{.*}})
>>>>>
>>>>> define i32 @test(i32* nocapture %a, i32 %n) nounwind uwtable readonly !dbg
>>>>> !6 {
>>>>> entry:
>>>>>
>>>>> Modified: llvm/trunk/test/Transforms/LoopUnroll/runtime-loop2.ll
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/LoopUnroll/runtime-loop2.ll?rev=265388&r1=265387&r2=265388&view=diff
>>>>> ==============================================================================
>>>>> --- llvm/trunk/test/Transforms/LoopUnroll/runtime-loop2.ll (original)
>>>>> +++ llvm/trunk/test/Transforms/LoopUnroll/runtime-loop2.ll Tue Apr 5
>>>>> 07:19:35 2016
>>>>> @@ -1,12 +1,18 @@
>>>>> -; RUN: opt < %s -S -loop-unroll -unroll-threshold=25 -unroll-runtime
>>>>> -unroll-count=8 | FileCheck %s
>>>>> +; RUN: opt < %s -S -loop-unroll -unroll-threshold=25 -unroll-runtime
>>>>> -unroll-count=8 | FileCheck %s -check-prefix=EPILOG
>>>>> +; RUN: opt < %s -S -loop-unroll -unroll-threshold=25 -unroll-runtime
>>>>> -unroll-runtime-epilog=false | FileCheck %s -check-prefix=PROLOG
>>>>>
>>>>> ; Choose a smaller, power-of-two, unroll count if the loop is too large.
>>>>> ; This test makes sure we're not unrolling 'odd' counts
>>>>>
>>>>> -; CHECK: for.body.prol:
>>>>> -; CHECK: for.body:
>>>>> -; CHECK: br i1 %exitcond.3, label %for.end.loopexit{{.*}}, label %for.body
>>>>> -; CHECK-NOT: br i1 %exitcond.4, label %for.end.loopexit{{.*}}, label
>>>>> %for.body
>>>>> +; EPILOG: for.body:
>>>>> +; EPILOG: br i1 %niter.ncmp.3, label
>>>>> %for.end.loopexit.unr-lcssa.loopexit{{.*}}, label %for.body
>>>>> +; EPILOG-NOT: br i1 %niter.ncmp.4, label
>>>>> %for.end.loopexit.unr-lcssa.loopexit{{.*}}, label %for.body
>>>>> +; EPILOG: for.body.epil:
>>>>> +
>>>>> +; PROLOG: for.body.prol:
>>>>> +; PROLOG: for.body:
>>>>> +; PROLOG: br i1 %exitcond.3, label %for.end.loopexit{{.*}}, label %for.body
>>>>> +; PROLOG-NOT: br i1 %exitcond.4, label %for.end.loopexit{{.*}}, label
>>>>> %for.body
>>>>>
>>>>> define i32 @test(i32* nocapture %a, i32 %n) nounwind uwtable readonly {
>>>>> entry:
>>>>>
>>>>> Modified: llvm/trunk/test/Transforms/LoopUnroll/runtime-loop4.ll
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/LoopUnroll/runtime-loop4.ll?rev=265388&r1=265387&r2=265388&view=diff
>>>>> ==============================================================================
>>>>> --- llvm/trunk/test/Transforms/LoopUnroll/runtime-loop4.ll (original)
>>>>> +++ llvm/trunk/test/Transforms/LoopUnroll/runtime-loop4.ll Tue Apr 5
>>>>> 07:19:35 2016
>>>>> @@ -1,13 +1,21 @@
>>>>> -; RUN: opt < %s -S -O2 -unroll-runtime=true | FileCheck %s
>>>>> +; RUN: opt < %s -S -O2 -unroll-runtime=true | FileCheck %s
>>>>> -check-prefix=EPILOG
>>>>> +; RUN: opt < %s -S -O2 -unroll-runtime=true -unroll-runtime-epilog=false |
>>>>> FileCheck %s -check-prefix=PROLOG
>>>>>
>>>>> ; Check runtime unrolling prologue can be promoted by LICM pass.
>>>>>
>>>>> -; CHECK: entry:
>>>>> -; CHECK: %xtraiter
>>>>> -; CHECK: %lcmp.mod
>>>>> -; CHECK: loop1:
>>>>> -; CHECK: br i1 %lcmp.mod
>>>>> -; CHECK: loop2.prol:
>>>>> +; EPILOG: entry:
>>>>> +; EPILOG: %xtraiter
>>>>> +; EPILOG: %lcmp.mod
>>>>> +; EPILOG: loop1:
>>>>> +; EPILOG: br i1 %lcmp.mod
>>>>> +; EPILOG: loop2.epil:
>>>>> +
>>>>> +; PROLOG: entry:
>>>>> +; PROLOG: %xtraiter
>>>>> +; PROLOG: %lcmp.mod
>>>>> +; PROLOG: loop1:
>>>>> +; PROLOG: br i1 %lcmp.mod
>>>>> +; PROLOG: loop2.prol:
>>>>>
>>>>> define void @unroll(i32 %iter, i32* %addr1, i32* %addr2) nounwind {
>>>>> entry:
>>>>>
>>>>> Modified: llvm/trunk/test/Transforms/LoopUnroll/runtime-loop5.ll
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/LoopUnroll/runtime-loop5.ll?rev=265388&r1=265387&r2=265388&view=diff
>>>>> ==============================================================================
>>>>> --- llvm/trunk/test/Transforms/LoopUnroll/runtime-loop5.ll (original)
>>>>> +++ llvm/trunk/test/Transforms/LoopUnroll/runtime-loop5.ll Tue Apr 5
>>>>> 07:19:35 2016
>>>>> @@ -11,9 +11,6 @@ entry:
>>>>> %cmp1 = icmp eq i3 %n, 0
>>>>> br i1 %cmp1, label %for.end, label %for.body
>>>>>
>>>>> -; UNROLL-16-NOT: for.body.prol:
>>>>> -; UNROLL-4: for.body.prol:
>>>>> -
>>>>> for.body: ; preds = %for.body,
>>>>> %entry
>>>>> ; UNROLL-16-LABEL: for.body:
>>>>> ; UNROLL-4-LABEL: for.body:
>>>>> @@ -39,6 +36,10 @@ for.body:
>>>>>
>>>>> ; UNROLL-16-LABEL: for.end
>>>>> ; UNROLL-4-LABEL: for.end
>>>>> +
>>>>> +; UNROLL-16-NOT: for.body.epil:
>>>>> +; UNROLL-4: for.body.epil:
>>>>> +
>>>>> for.end: ; preds = %for.body,
>>>>> %entry
>>>>> %sum.0.lcssa = phi i3 [ 0, %entry ], [ %add, %for.body ]
>>>>> ret i3 %sum.0.lcssa
>>>>>
>>>>> Modified: llvm/trunk/test/Transforms/LoopUnroll/tripcount-overflow.ll
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/LoopUnroll/tripcount-overflow.ll?rev=265388&r1=265387&r2=265388&view=diff
>>>>> ==============================================================================
>>>>> --- llvm/trunk/test/Transforms/LoopUnroll/tripcount-overflow.ll (original)
>>>>> +++ llvm/trunk/test/Transforms/LoopUnroll/tripcount-overflow.ll Tue Apr 5
>>>>> 07:19:35 2016
>>>>> @@ -13,13 +13,13 @@ target datalayout = "e-m:o-i64:64-f80:12
>>>>> ; CHECK: entry:
>>>>> ; CHECK-NEXT: %0 = add i32 %N, 1
>>>>> ; CHECK-NEXT: %xtraiter = and i32 %0, 1
>>>>> -; CHECK-NEXT: %lcmp.mod = icmp ne i32 %xtraiter, 0
>>>>> -; CHECK-NEXT: br i1 %lcmp.mod, label %while.body.prol, label %entry.split
>>>>> +; CHECK-NEXT: %lcmp.mod = icmp ne i32 %xtraiter, %0
>>>>> +; CHECK-NEXT: br i1 %lcmp.mod, label %entry.new, label %while.end.unr-lcssa
>>>>>
>>>>> -; CHECK: while.body.prol:
>>>>> -; CHECK: br label %entry.split
>>>>> +; CHECK: while.body.epil:
>>>>> +; CHECK: br label %while.end.epilog-lcssa
>>>>>
>>>>> -; CHECK: entry.split:
>>>>> +; CHECK: while.end.epilog-lcssa:
>>>>>
>>>>> ; Function Attrs: nounwind readnone ssp uwtable
>>>>> define i32 @foo(i32 %N) {
>>>>>
>>>>> Modified: llvm/trunk/test/Transforms/LoopUnroll/unroll-cleanup.ll
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/LoopUnroll/unroll-cleanup.ll?rev=265388&r1=265387&r2=265388&view=diff
>>>>> ==============================================================================
>>>>> --- llvm/trunk/test/Transforms/LoopUnroll/unroll-cleanup.ll (original)
>>>>> +++ llvm/trunk/test/Transforms/LoopUnroll/unroll-cleanup.ll Tue Apr 5
>>>>> 07:19:35 2016
>>>>> @@ -4,14 +4,14 @@
>>>>> ; RUN: opt < %s -O2 -S | FileCheck %s
>>>>>
>>>>> ; After loop unroll:
>>>>> -; %dec18 = add nsw i32 %dec18.in, -1
>>>>> +; %niter.nsub = add nsw i32 %niter, -1
>>>>> ; ...
>>>>> -; %dec18.1 = add nsw i32 %dec18, -1
>>>>> +; %niter.nsub.1 = add nsw i32 %niter.nsub, -1
>>>>> ; should be merged to:
>>>>> -; %dec18.1 = add nsw i32 %dec18.in, -2
>>>>> +; %dec18.1 = add nsw i32 %niter, -2
>>>>> ;
>>>>> ; CHECK-LABEL: @_Z3fn1v(
>>>>> -; CHECK: %dec18.1 = add nsw i32 %dec18.in, -2
>>>>> +; CHECK: %niter.nsub.1 = add i32 %niter, -2
>>>>>
>>>>> ; ModuleID = '<stdin>'
>>>>> target triple = "x86_64-unknown-linux-gnu"
>>>>>
>>>>> Modified: llvm/trunk/test/Transforms/LoopUnroll/unroll-pragmas.ll
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/LoopUnroll/unroll-pragmas.ll?rev=265388&r1=265387&r2=265388&view=diff
>>>>> ==============================================================================
>>>>> --- llvm/trunk/test/Transforms/LoopUnroll/unroll-pragmas.ll (original)
>>>>> +++ llvm/trunk/test/Transforms/LoopUnroll/unroll-pragmas.ll Tue Apr 5
>>>>> 07:19:35 2016
>>>>> @@ -171,10 +171,6 @@ for.end:
>>>>> ; should be duplicated (original and 4x unrolled).
>>>>> ;
>>>>> ; CHECK-LABEL: @runtime_loop_with_count4(
>>>>> -; CHECK: for.body.prol:
>>>>> -; CHECK: store
>>>>> -; CHECK-NOT: store
>>>>> -; CHECK: br i1
>>>>> ; CHECK: for.body
>>>>> ; CHECK: store
>>>>> ; CHECK: store
>>>>> @@ -182,6 +178,10 @@ for.end:
>>>>> ; CHECK: store
>>>>> ; CHECK-NOT: store
>>>>> ; CHECK: br i1
>>>>> +; CHECK: for.body.epil:
>>>>> +; CHECK: store
>>>>> +; CHECK-NOT: store
>>>>> +; CHECK: br i1
>>>>> define void @runtime_loop_with_count4(i32* nocapture %a, i32 %b) {
>>>>> entry:
>>>>> %cmp3 = icmp sgt i32 %b, 0
>>>>> @@ -287,10 +287,6 @@ for.end:
>>>>> ; (original and 8x).
>>>>> ;
>>>>> ; CHECK-LABEL: @runtime_loop_with_enable(
>>>>> -; CHECK: for.body.prol:
>>>>> -; CHECK: store
>>>>> -; CHECK-NOT: store
>>>>> -; CHECK: br i1
>>>>> ; CHECK: for.body:
>>>>> ; CHECK: store i32
>>>>> ; CHECK: store i32
>>>>> @@ -302,6 +298,10 @@ for.end:
>>>>> ; CHECK: store i32
>>>>> ; CHECK-NOT: store i32
>>>>> ; CHECK: br i1
>>>>> +; CHECK: for.body.epil:
>>>>> +; CHECK: store
>>>>> +; CHECK-NOT: store
>>>>> +; CHECK: br i1
>>>>> define void @runtime_loop_with_enable(i32* nocapture %a, i32 %b) {
>>>>> entry:
>>>>> %cmp3 = icmp sgt i32 %b, 0
>>>>> @@ -328,16 +328,16 @@ for.end:
>>>>> ; should be duplicated (original and 3x unrolled).
>>>>> ;
>>>>> ; CHECK-LABEL: @runtime_loop_with_count3(
>>>>> -; CHECK: for.body.prol:
>>>>> -; CHECK: store
>>>>> -; CHECK-NOT: store
>>>>> -; CHECK: br i1
>>>>> ; CHECK: for.body
>>>>> ; CHECK: store
>>>>> ; CHECK: store
>>>>> ; CHECK: store
>>>>> ; CHECK-NOT: store
>>>>> ; CHECK: br i1
>>>>> +; CHECK: for.body.epil:
>>>>> +; CHECK: store
>>>>> +; CHECK-NOT: store
>>>>> +; CHECK: br i1
>>>>> define void @runtime_loop_with_count3(i32* nocapture %a, i32 %b) {
>>>>> entry:
>>>>> %cmp3 = icmp sgt i32 %b, 0
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> llvm-commits mailing list
>>>>> llvm-commits at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> llvm-commits mailing list
>>>>> llvm-commits at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>>>>>
>>>>>
>>>
>>
More information about the llvm-commits
mailing list