<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Jul 25, 2016, at 5:15 PM, Gerolf Hoflehner <<a href="mailto:ghoflehner@apple.com" class="">ghoflehner@apple.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class=""><br class="Apple-interchange-newline">On Jul 25, 2016, at 3:53 PM, Michael Zolotukhin via llvm-commits <<a href="mailto:llvm-commits@lists.llvm.org" class="">llvm-commits@lists.llvm.org</a>> wrote:<br class=""><br class=""><br class=""><blockquote type="cite" class="">On Jul 24, 2016, at 3:10 AM, Evgeny Stupachenko <<a href="mailto:evstupac@gmail.com" class="">evstupac@gmail.com</a>> wrote:<br class=""><br class="">Hi Michael,<br class=""></blockquote>Hi Evgeny,<br class=""><blockquote type="cite" class=""><br class="">Ok. I'm on vacation right now. With limit access to testing. I'll back<br class="">in the middle of August, run the testing and revert to prologue.<br class="">Should we get one more vote to revert this to prologue or I can do<br class="">this right when I'm back from from vacation?<br class=""></blockquote>I can switch it to the previous value (prologue) for you, and then you fix the test/LSR and switch it back (to epilogue). Does it sound good?<br class=""></blockquote><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">+1</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">I think this is the right approach dealing with big regressions.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""></div></blockquote><div>Hi Evgeny,</div><div><br class=""></div>I switched the flag to 'false' in r277524 and adjusted the tests accordingly. Please feel free to reenable it back once the regression is addressed.</div><div><br class=""></div><div>Thanks,</div><div>Michael<br class=""><blockquote type="cite" class=""><div class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" class=""><br class="">Could you please point me to someone who is responsible for the<br class="">performance tests you are running? I'll address the issue.<br class="">Sorry for staying so long without addressing the issue. I've missed<br class="">the importance of the test and its performance.<br class=""></blockquote>It’s not about that particular test, it’s about 90% regression.<br class=""><br class=""><blockquote type="cite" class="">The regression is not compiler dependent its a "bad luck" caused by<br class="">test structure. I can prove this to the person who is responsible for<br class="">the test.<br class=""></blockquote>If the test is flaky, then I’m completely fine with fixing the test. But otherwise the “bad luck” argument doesn’t convince me - what if a user found a similar regression from a previous compiler version on his code?<br class=""><br class=""><blockquote type="cite" class="">My LSR patch is fixing only static addresses in the loop. The<br class="">regression stays unchanged. The root cause of the regression related<br class="">to code alignment.<br class=""></blockquote><br class="">Thanks,<br class="">Michael<br class=""><br class="">PS: Sorry to disrupt your vacation with this!<br class=""><blockquote type="cite" class=""><br class="">Thanks,<br class="">Evgeny<br class=""><br class=""><br class=""><br class="">On Thu, Jul 21, 2016 at 11:57 AM, Mikhail Zolotukhin<br class=""><<a href="mailto:mzolotukhin@apple.com" class="">mzolotukhin@apple.com</a>> wrote:<br class=""><blockquote type="cite" class=""><br class=""><blockquote type="cite" class="">On Jul 21, 2016, at 11:36 AM, Evgeny Stupachenko <<a href="mailto:evstupac@gmail.com" class="">evstupac@gmail.com</a>> wrote:<br class=""><br class="">Hi Michael,<br class=""></blockquote>Hi Evgeny,<br class=""><blockquote type="cite" class=""><br class="">Turning back to default will decrease LSR positive effect.<br class=""></blockquote>When the LSR changes are committed, we certainly can switch it back.<br class=""><blockquote type="cite" class="">As for the test itself. It does not represent a real case.<br class=""></blockquote>I won't argue that, though the "real case" is a very vague term. What matters is that this test is currently in the testsuite.<br class=""><blockquote type="cite" class="">Global arrays with big amount of data are not ok.<br class="">If we switch to malloc instead, the regression disappears.<br class=""></blockquote>The problem here is that we have a big regression with this change, and it's not addressed so far. It might be a test issue or an LSR issue, but whatever it is, we need to address it, and if it takes too long, turn it back off until it's fixed.<br class=""><br class="">Thanks,<br class="">Michael<br class=""><blockquote type="cite" class="">I'd better modify the test or exclude it.<br class=""><br class="">Thanks,<br class="">Evgeny<br class=""><br class="">On Wed, Jul 20, 2016 at 1:06 PM, Michael Zolotukhin<br class=""><<a href="mailto:mzolotukhin@apple.com" class="">mzolotukhin@apple.com</a>> wrote:<br class=""><blockquote type="cite" class="">Hi Evgeny,<br class=""><br class="">Thanks for the detailed update! After a second thought - can we please switch the default to the previous value while you’re working on the patch? Generally, it’s bad to have such a big regression even though your original patch just exposed an issue in another place.<br class=""><br class="">Thanks,<br class="">Michael<br class=""><blockquote type="cite" class="">On Jul 11, 2016, at 9:08 PM, Evgeny Stupachenko <<a href="mailto:evstupac@gmail.com" class="">evstupac@gmail.com</a>> wrote:<br class=""><br class="">Hi Michael,<br class=""><br class="">The patch is under internal review. It changes a lot. I plan to send<br class="">it here this month or in August.<br class="">The patch fixes addressing, but not the regression.<br class="">We looked into the regression itself on simulator. It appears that the<br class="">big issue is branch misprediction.<br class=""><br class="">Regarding addresses.<br class="">When we have prolog (with unroll on 2):<br class=""><br class="">for_body<br class="">i = phi [i.next, i0]; //i0 is 0 or 1 depending on the prolog execution<br class="">x = *(array + i);<br class="">...<br class="">cmp i, N<br class="">br for_body<br class=""><br class="">LSR is trying to simplify ind vars to one.<br class="">lsr.i = phi [lsr.i.next, 0];<br class="">or<br class="">lsr.i = phi [lsr.i.next, -N];<br class="">That way array access will have base: “array + i0” or “array + i0 - N”<br class="">which is not static even if array is static.<br class=""><br class="">x = *((array + i0) + lsr.i);<br class=""><br class="">So it will require a register inside loop. Right now number of<br class="">registers has the highest priority in LSR when it chooses a solution.<br class=""><br class="">When we have epilog, the same access is without “i0”<br class=""><br class="">x = *(array + lsr.i);<br class=""><br class="">If array is static we do not need an additional register.<br class="">In the test where regression occurred the array is static and N is constant.<br class=""><br class="">However if we have enough registers we can use register instead of<br class="">constant address. The patch should address this.<br class=""><br class="">Thanks,<br class="">Evgeny<br class=""><br class="">On Mon, Jul 11, 2016 at 6:04 PM, Michael Zolotukhin<br class=""><<a href="mailto:mzolotukhin@apple.com" class="">mzolotukhin@apple.com</a>> wrote:<br class=""><blockquote type="cite" class=""><br class=""><blockquote type="cite" class="">On Apr 22, 2016, at 11:35 AM, Michael Zolotukhin <<a href="mailto:mzolotukhin@apple.com" class="">mzolotukhin@apple.com</a>> wrote:<br class=""><br class=""><br class=""><blockquote type="cite" class="">On Apr 20, 2016, at 4:43 PM, Evgeny Stupachenko <<a href="mailto:evstupac@gmail.com" class="">evstupac@gmail.com</a>> wrote:<br class=""><br class="">Hi,<br class=""><br class="">Yes if the test is that important. However I believe that the<br class="">regression caused randomly, but not by epilog unrolling.<br class="">On my Corei7 "I was able to reproduce it only in 32 bit mode, 64 bit<br class="">mode epilog is ~30% faster.".<br class="">The analysis showed no critical changes in the hottest loop.<br class=""><br class="">As for LSR issue.<br class="">Sorry for missing your request for posting PR. I'm preparing LSR patch<br class="">fixing the issue. And issue itself is related to already filed PR23384<br class="">(on an inefficient x86 LSR transformation).<br class=""></blockquote></blockquote>Hi Evgeny,<br class=""><br class="">Is there any progress with the LSR patch? Or is there something blocking you?<br class=""><br class="">Thanks,<br class="">Michael<br class=""><blockquote type="cite" class="">Hi Evgeny,<br class=""><br class="">It’s fine with me to keep it on, I just want to make sure we address the regression. So, if you’re working on the patch, there are no concerns from my side.<br class=""><br class="">Thanks,<br class="">Michael<br class=""><br class=""><blockquote type="cite" class=""><br class="">Thanks,<br class="">Evgeny<br class=""><br class="">On Wed, Apr 20, 2016 at 11:41 AM, Michael Zolotukhin<br class=""><<a href="mailto:mzolotukhin@apple.com" class="">mzolotukhin@apple.com</a>> wrote:<br class=""><blockquote type="cite" class="">Hi,<br class=""><br class="">Could we change the default value to the original one until this 90%<br class="">regression is addressed?<br class=""><br class="">Thanks,<br class="">Michael<br class=""><br class="">On Apr 15, 2016, at 12:37 PM, Mikhail Zolotukhin <<a href="mailto:mzolotukhin@apple.com" class="">mzolotukhin@apple.com</a>><br class="">wrote:<br class=""><br class=""><br class="">On Apr 8, 2016, at 5:39 PM, Evgeny Stupachenko <<a href="mailto:evstupac@gmail.com" class="">evstupac@gmail.com</a>> wrote:<br class=""><br class="">Investigation showed that the regression is most likely LSR related,<br class="">but not loop unroll.<br class="">I was able to reproduce it only in 32 bit mode, 64 bit mode epilog is<br class="">~30% faster.<br class=""><br class="">Going back to the changes in the test. Epilog variant has 1 move more<br class="">in the hottest loop (which is redundant and should be deleted) and use<br class="">constant address in moves inside the loop. However manual deleting of<br class="">the move does not give any significant performance gain.<br class="">The different behavior of LSR comes from additional inductive variable<br class="">in unrolled loop. For Epilog case we add "k" (supposed to be<br class="">eliminated after LSR):<br class=""><br class="">k = n - n % (unroll factor));<br class="">...<br class="">access a[i];<br class="">i++;<br class="">k--;<br class="">if (!k) break;<br class=""><br class="">Moving code of the loop modifying "palign" before loop showed up to 2<br class="">times perf difference. So most likely epilog unroll itself does not<br class="">influence on performance here.<br class="">However, the fact that LSR leave constant address inside the loop<br class="">moves (making them much longer), should be addressed to LSR. I'll<br class="">submit a bug report on this.<br class=""><br class="">Hi Evgeny,<br class=""><br class="">Could you please post the PR number here as well?<br class=""><br class="">Michael<br class=""><br class=""><br class=""><br class="">On Fri, Apr 8, 2016 at 2:09 PM, Evgeny Stupachenko <<a href="mailto:evstupac@gmail.com" class="">evstupac@gmail.com</a>><br class="">wrote:<br class=""><br class="">Hi Michael,<br class=""><br class="">Yes I do have the data.<br class="">There were ~10% improvements on some EEMBC tests; spec2000 performance<br class="">was almost flat (within 2%).<br class=""><br class="">Let me look into BubbleSort case and come back with analysis and<br class="">possible improvement.<br class=""><br class="">Thanks,<br class="">Evgeny<br class=""><br class=""><br class="">On Fri, Apr 8, 2016 at 1:23 PM, Michael Zolotukhin via llvm-commits<br class=""><<a href="mailto:llvm-commits@lists.llvm.org" class="">llvm-commits@lists.llvm.org</a>> wrote:<br class=""><br class="">Hi Evgeny,<br class=""><br class="">We’ve found several performance regressions on LLVM testsuite caused by this<br class="">patch. Do you have a data from your performance experiments to back-up the<br class="">decision to make epilogues the default strategy?<br class=""><br class="">One of the biggest regressions we see is 89% on<br class="">SingleSource/Benchmarks/Stanford/Bubblesort (on x86), while the biggest gain<br class="">is only 30%.<br class=""><br class="">Thanks,<br class="">Michael<br class=""><br class=""><br class="">On Apr 5, 2016, at 5:19 AM, David L Kreitzer via llvm-commits<br class=""><<a href="mailto:llvm-commits@lists.llvm.org" class="">llvm-commits@lists.llvm.org</a>> wrote:<br class=""><br class="">Author: dlkreitz<br class="">Date: Tue Apr 5 07:19:35 2016<br class="">New Revision: 265388<br class=""><br class="">URL: <a href="http://llvm.org/viewvc/llvm-project?rev=265388&view=rev" class="">http://llvm.org/viewvc/llvm-project?rev=265388&view=rev</a><br class="">Log:<br class="">Adds the ability to use an epilog remainder loop during loop unrolling and<br class="">makes<br class="">this the default behavior.<br class=""><br class="">Patch by Evgeny Stupachenko (<a href="mailto:evstupac@gmail.com" class="">evstupac@gmail.com</a>).<br class=""><br class="">Differential Revision: <a href="http://reviews.llvm.org/D18158" class="">http://reviews.llvm.org/D18158</a><br class=""><br class="">Modified:<br class="">llvm/trunk/include/llvm/Transforms/Utils/UnrollLoop.h<br class="">llvm/trunk/lib/Transforms/Utils/LoopUnroll.cpp<br class="">llvm/trunk/lib/Transforms/Utils/LoopUnrollRuntime.cpp<br class="">llvm/trunk/test/Transforms/LoopUnroll/AArch64/runtime-loop.ll<br class="">llvm/trunk/test/Transforms/LoopUnroll/PowerPC/a2-unrolling.ll<br class="">llvm/trunk/test/Transforms/LoopUnroll/X86/mmx.ll<br class="">llvm/trunk/test/Transforms/LoopUnroll/high-cost-trip-count-computation.ll<br class="">llvm/trunk/test/Transforms/LoopUnroll/runtime-loop.ll<br class="">llvm/trunk/test/Transforms/LoopUnroll/runtime-loop1.ll<br class="">llvm/trunk/test/Transforms/LoopUnroll/runtime-loop2.ll<br class="">llvm/trunk/test/Transforms/LoopUnroll/runtime-loop4.ll<br class="">llvm/trunk/test/Transforms/LoopUnroll/runtime-loop5.ll<br class="">llvm/trunk/test/Transforms/LoopUnroll/tripcount-overflow.ll<br class="">llvm/trunk/test/Transforms/LoopUnroll/unroll-cleanup.ll<br class="">llvm/trunk/test/Transforms/LoopUnroll/unroll-pragmas.ll<br class=""><br class="">Modified: llvm/trunk/include/llvm/Transforms/Utils/UnrollLoop.h<br class="">URL:<br class=""><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Transforms/Utils/UnrollLoop.h?rev=265388&r1=265387&r2=265388&view=diff" class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Transforms/Utils/UnrollLoop.h?rev=265388&r1=265387&r2=265388&view=diff</a><br class="">==============================================================================<br class="">--- llvm/trunk/include/llvm/Transforms/Utils/UnrollLoop.h (original)<br class="">+++ llvm/trunk/include/llvm/Transforms/Utils/UnrollLoop.h Tue Apr 5<br class="">07:19:35 2016<br class="">@@ -34,10 +34,11 @@ bool UnrollLoop(Loop *L, unsigned Count,<br class=""> LoopInfo *LI, ScalarEvolution *SE, DominatorTree *DT,<br class=""> AssumptionCache *AC, bool PreserveLCSSA);<br class=""><br class="">-bool UnrollRuntimeLoopProlog(Loop *L, unsigned Count,<br class="">- bool AllowExpensiveTripCount, LoopInfo *LI,<br class="">- ScalarEvolution *SE, DominatorTree *DT,<br class="">- bool PreserveLCSSA);<br class="">+bool UnrollRuntimeLoopRemainder(Loop *L, unsigned Count,<br class="">+ bool AllowExpensiveTripCount,<br class="">+ bool UseEpilogRemainder, LoopInfo *LI,<br class="">+ ScalarEvolution *SE, DominatorTree *DT,<br class="">+ bool PreserveLCSSA);<br class=""><br class="">MDNode *GetUnrollMetadata(MDNode *LoopID, StringRef Name);<br class="">}<br class=""><br class="">Modified: llvm/trunk/lib/Transforms/Utils/LoopUnroll.cpp<br class="">URL:<br class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Utils/LoopUnroll.cpp?rev=265388&r1=265387&r2=265388&view=diff<br class="">==============================================================================<br class="">--- llvm/trunk/lib/Transforms/Utils/LoopUnroll.cpp (original)<br class="">+++ llvm/trunk/lib/Transforms/Utils/LoopUnroll.cpp Tue Apr 5 07:19:35 2016<br class="">@@ -44,6 +44,11 @@ using namespace llvm;<br class="">STATISTIC(NumCompletelyUnrolled, "Number of loops completely unrolled");<br class="">STATISTIC(NumUnrolled, "Number of loops unrolled (completely or<br class="">otherwise)");<br class=""><br class="">+static cl::opt<bool><br class="">+UnrollRuntimeEpilog("unroll-runtime-epilog", cl::init(true), cl::Hidden,<br class="">+ cl::desc("Allow runtime unrolled loops to be unrolled "<br class="">+ "with epilog instead of prolog."));<br class="">+<br class="">/// Convert the instruction operands from referencing the current values<br class="">into<br class="">/// those specified by VMap.<br class="">static inline void remapInstruction(Instruction *I,<br class="">@@ -288,12 +293,13 @@ bool llvm::UnrollLoop(Loop *L, unsigned<br class=""> "convergent "<br class=""> "operation.");<br class="">});<br class="">- // Don't output the runtime loop prolog if Count is a multiple of<br class="">- // TripMultiple. Such a prolog is never needed, and is unsafe if the<br class="">loop<br class="">+ // Don't output the runtime loop remainder if Count is a multiple of<br class="">+ // TripMultiple. Such a remainder is never needed, and is unsafe if the<br class="">loop<br class="">// contains a convergent instruction.<br class="">if (RuntimeTripCount && TripMultiple % Count != 0 &&<br class="">- !UnrollRuntimeLoopProlog(L, Count, AllowExpensiveTripCount, LI, SE,<br class="">DT,<br class="">- PreserveLCSSA))<br class="">+ !UnrollRuntimeLoopRemainder(L, Count, AllowExpensiveTripCount,<br class="">+ UnrollRuntimeEpilog, LI, SE, DT,<br class="">+ PreserveLCSSA))<br class="">return false;<br class=""><br class="">// Notify ScalarEvolution that the loop will be substantially changed,<br class=""><br class="">Modified: llvm/trunk/lib/Transforms/Utils/LoopUnrollRuntime.cpp<br class="">URL:<br class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Utils/LoopUnrollRuntime.cpp?rev=265388&r1=265387&r2=265388&view=diff<br class="">==============================================================================<br class="">--- llvm/trunk/lib/Transforms/Utils/LoopUnrollRuntime.cpp (original)<br class="">+++ llvm/trunk/lib/Transforms/Utils/LoopUnrollRuntime.cpp Tue Apr 5<br class="">07:19:35 2016<br class="">@@ -16,8 +16,8 @@<br class="">// case, we need to generate code to execute these 'left over' iterations.<br class="">//<br class="">// The current strategy generates an if-then-else sequence prior to the<br class="">-// unrolled loop to execute the 'left over' iterations. Other strategies<br class="">-// include generate a loop before or after the unrolled loop.<br class="">+// unrolled loop to execute the 'left over' iterations before or after the<br class="">+// unrolled loop.<br class="">//<br class="">//===----------------------------------------------------------------------===//<br class=""><br class="">@@ -60,33 +60,35 @@ STATISTIC(NumRuntimeUnrolled,<br class="">/// than the unroll factor.<br class="">///<br class="">static void ConnectProlog(Loop *L, Value *BECount, unsigned Count,<br class="">- BasicBlock *LastPrologBB, BasicBlock *PrologEnd,<br class="">- BasicBlock *OrigPH, BasicBlock *NewPH,<br class="">- ValueToValueMapTy &VMap, DominatorTree *DT,<br class="">- LoopInfo *LI, bool PreserveLCSSA) {<br class="">+ BasicBlock *PrologExit, BasicBlock *PreHeader,<br class="">+ BasicBlock *NewPreHeader, ValueToValueMapTy<br class="">&VMap,<br class="">+ DominatorTree *DT, LoopInfo *LI, bool<br class="">PreserveLCSSA) {<br class="">BasicBlock *Latch = L->getLoopLatch();<br class="">assert(Latch && "Loop must have a latch");<br class="">+ BasicBlock *PrologLatch = cast<BasicBlock>(VMap[Latch]);<br class=""><br class="">// Create a PHI node for each outgoing value from the original loop<br class="">// (which means it is an outgoing value from the prolog code too).<br class="">// The new PHI node is inserted in the prolog end basic block.<br class="">- // The new PHI name is added as an operand of a PHI node in either<br class="">+ // The new PHI node value is added as an operand of a PHI node in either<br class="">// the loop header or the loop exit block.<br class="">- for (succ_iterator SBI = succ_begin(Latch), SBE = succ_end(Latch);<br class="">- SBI != SBE; ++SBI) {<br class="">- for (BasicBlock::iterator BBI = (*SBI)->begin();<br class="">- PHINode *PN = dyn_cast<PHINode>(BBI); ++BBI) {<br class="">-<br class="">+ for (BasicBlock *Succ : successors(Latch)) {<br class="">+ for (Instruction &BBI : *Succ) {<br class="">+ PHINode *PN = dyn_cast<PHINode>(&BBI);<br class="">+ // Exit when we passed all PHI nodes.<br class="">+ if (!PN)<br class="">+ break;<br class="">// Add a new PHI node to the prolog end block and add the<br class="">// appropriate incoming values.<br class="">- PHINode *NewPN = PHINode::Create(PN->getType(), 2,<br class="">PN->getName()+".unr",<br class="">- PrologEnd->getTerminator());<br class="">+ PHINode *NewPN = PHINode::Create(PN->getType(), 2, PN->getName() +<br class="">".unr",<br class="">+ PrologExit->getFirstNonPHI());<br class="">// Adding a value to the new PHI node from the original loop preheader.<br class="">// This is the value that skips all the prolog code.<br class="">if (L->contains(PN)) {<br class="">- NewPN->addIncoming(PN->getIncomingValueForBlock(NewPH), OrigPH);<br class="">+ NewPN->addIncoming(PN->getIncomingValueForBlock(NewPreHeader),<br class="">+ PreHeader);<br class="">} else {<br class="">- NewPN->addIncoming(UndefValue::get(PN->getType()), OrigPH);<br class="">+ NewPN->addIncoming(UndefValue::get(PN->getType()), PreHeader);<br class="">}<br class=""><br class="">Value *V = PN->getIncomingValueForBlock(Latch);<br class="">@@ -97,22 +99,22 @@ static void ConnectProlog(Loop *L, Value<br class="">}<br class="">// Adding a value to the new PHI node from the last prolog block<br class="">// that was created.<br class="">- NewPN->addIncoming(V, LastPrologBB);<br class="">+ NewPN->addIncoming(V, PrologLatch);<br class=""><br class="">// Update the existing PHI node operand with the value from the<br class="">// new PHI node. How this is done depends on if the existing<br class="">// PHI node is in the original loop block, or the exit block.<br class="">if (L->contains(PN)) {<br class="">- PN->setIncomingValue(PN->getBasicBlockIndex(NewPH), NewPN);<br class="">+ PN->setIncomingValue(PN->getBasicBlockIndex(NewPreHeader), NewPN);<br class="">} else {<br class="">- PN->addIncoming(NewPN, PrologEnd);<br class="">+ PN->addIncoming(NewPN, PrologExit);<br class="">}<br class="">}<br class="">}<br class=""><br class="">// Create a branch around the original loop, which is taken if there are no<br class="">// iterations remaining to be executed after running the prologue.<br class="">- Instruction *InsertPt = PrologEnd->getTerminator();<br class="">+ Instruction *InsertPt = PrologExit->getTerminator();<br class="">IRBuilder<> B(InsertPt);<br class=""><br class="">assert(Count != 0 && "nonsensical Count!");<br class="">@@ -126,25 +128,152 @@ static void ConnectProlog(Loop *L, Value<br class="">BasicBlock *Exit = L->getUniqueExitBlock();<br class="">assert(Exit && "Loop must have a single exit block only");<br class="">// Split the exit to maintain loop canonicalization guarantees<br class="">- SmallVector<BasicBlock*, 4> Preds(pred_begin(Exit), pred_end(Exit));<br class="">+ SmallVector<BasicBlock*, 4> Preds(predecessors(Exit));<br class="">SplitBlockPredecessors(Exit, Preds, ".unr-lcssa", DT, LI,<br class=""> PreserveLCSSA);<br class="">// Add the branch to the exit block (around the unrolled loop)<br class="">- B.CreateCondBr(BrLoopExit, Exit, NewPH);<br class="">+ B.CreateCondBr(BrLoopExit, Exit, NewPreHeader);<br class="">+ InsertPt->eraseFromParent();<br class="">+}<br class="">+<br class="">+/// Connect the unrolling epilog code to the original loop.<br class="">+/// The unrolling epilog code contains code to execute the<br class="">+/// 'extra' iterations if the run-time trip count modulo the<br class="">+/// unroll count is non-zero.<br class="">+///<br class="">+/// This function performs the following:<br class="">+/// - Update PHI nodes at the unrolling loop exit and epilog loop exit<br class="">+/// - Create PHI nodes at the unrolling loop exit to combine<br class="">+/// values that exit the unrolling loop code and jump around it.<br class="">+/// - Update PHI operands in the epilog loop by the new PHI nodes<br class="">+/// - Branch around the epilog loop if extra iters (ModVal) is zero.<br class="">+///<br class="">+static void ConnectEpilog(Loop *L, Value *ModVal, BasicBlock *NewExit,<br class="">+ BasicBlock *Exit, BasicBlock *PreHeader,<br class="">+ BasicBlock *EpilogPreHeader, BasicBlock<br class="">*NewPreHeader,<br class="">+ ValueToValueMapTy &VMap, DominatorTree *DT,<br class="">+ LoopInfo *LI, bool PreserveLCSSA) {<br class="">+ BasicBlock *Latch = L->getLoopLatch();<br class="">+ assert(Latch && "Loop must have a latch");<br class="">+ BasicBlock *EpilogLatch = cast<BasicBlock>(VMap[Latch]);<br class="">+<br class="">+ // Loop structure should be the following:<br class="">+ //<br class="">+ // PreHeader<br class="">+ // NewPreHeader<br class="">+ // Header<br class="">+ // ...<br class="">+ // Latch<br class="">+ // NewExit (PN)<br class="">+ // EpilogPreHeader<br class="">+ // EpilogHeader<br class="">+ // ...<br class="">+ // EpilogLatch<br class="">+ // Exit (EpilogPN)<br class="">+<br class="">+ // Update PHI nodes at NewExit and Exit.<br class="">+ for (Instruction &BBI : *NewExit) {<br class="">+ PHINode *PN = dyn_cast<PHINode>(&BBI);<br class="">+ // Exit when we passed all PHI nodes.<br class="">+ if (!PN)<br class="">+ break;<br class="">+ // PN should be used in another PHI located in Exit block as<br class="">+ // Exit was split by SplitBlockPredecessors into Exit and NewExit<br class="">+ // Basicaly it should look like:<br class="">+ // NewExit:<br class="">+ // PN = PHI [I, Latch]<br class="">+ // ...<br class="">+ // Exit:<br class="">+ // EpilogPN = PHI [PN, EpilogPreHeader]<br class="">+ //<br class="">+ // There is EpilogPreHeader incoming block instead of NewExit as<br class="">+ // NewExit was spilt 1 more time to get EpilogPreHeader.<br class="">+ assert(PN->hasOneUse() && "The phi should have 1 use");<br class="">+ PHINode *EpilogPN = cast<PHINode> (PN->use_begin()->getUser());<br class="">+ assert(EpilogPN->getParent() == Exit && "EpilogPN should be in Exit<br class="">block");<br class="">+<br class="">+ // Add incoming PreHeader from branch around the Loop<br class="">+ PN->addIncoming(UndefValue::get(PN->getType()), PreHeader);<br class="">+<br class="">+ Value *V = PN->getIncomingValueForBlock(Latch);<br class="">+ Instruction *I = dyn_cast<Instruction>(V);<br class="">+ if (I && L->contains(I))<br class="">+ // If value comes from an instruction in the loop add VMap value.<br class="">+ V = VMap[I];<br class="">+ // For the instruction out of the loop, constant or undefined value<br class="">+ // insert value itself.<br class="">+ EpilogPN->addIncoming(V, EpilogLatch);<br class="">+<br class="">+ assert(EpilogPN->getBasicBlockIndex(EpilogPreHeader) >= 0 &&<br class="">+ "EpilogPN should have EpilogPreHeader incoming block");<br class="">+ // Change EpilogPreHeader incoming block to NewExit.<br class="">+<br class="">EpilogPN->setIncomingBlock(EpilogPN->getBasicBlockIndex(EpilogPreHeader),<br class="">+ NewExit);<br class="">+ // Now PHIs should look like:<br class="">+ // NewExit:<br class="">+ // PN = PHI [I, Latch], [undef, PreHeader]<br class="">+ // ...<br class="">+ // Exit:<br class="">+ // EpilogPN = PHI [PN, NewExit], [VMap[I], EpilogLatch]<br class="">+ }<br class="">+<br class="">+ // Create PHI nodes at NewExit (from the unrolling loop Latch and<br class="">PreHeader).<br class="">+ // Update corresponding PHI nodes in epilog loop.<br class="">+ for (BasicBlock *Succ : successors(Latch)) {<br class="">+ // Skip this as we already updated phis in exit blocks.<br class="">+ if (!L->contains(Succ))<br class="">+ continue;<br class="">+ for (Instruction &BBI : *Succ) {<br class="">+ PHINode *PN = dyn_cast<PHINode>(&BBI);<br class="">+ // Exit when we passed all PHI nodes.<br class="">+ if (!PN)<br class="">+ break;<br class="">+ // Add new PHI nodes to the loop exit block and update epilog<br class="">+ // PHIs with the new PHI values.<br class="">+ PHINode *NewPN = PHINode::Create(PN->getType(), 2, PN->getName() +<br class="">".unr",<br class="">+ NewExit->getFirstNonPHI());<br class="">+ // Adding a value to the new PHI node from the unrolling loop<br class="">preheader.<br class="">+ NewPN->addIncoming(PN->getIncomingValueForBlock(NewPreHeader),<br class="">PreHeader);<br class="">+ // Adding a value to the new PHI node from the unrolling loop latch.<br class="">+ NewPN->addIncoming(PN->getIncomingValueForBlock(Latch), Latch);<br class="">+<br class="">+ // Update the existing PHI node operand with the value from the new<br class="">PHI<br class="">+ // node. Corresponding instruction in epilog loop should be PHI.<br class="">+ PHINode *VPN = cast<PHINode>(VMap[&BBI]);<br class="">+ VPN->setIncomingValue(VPN->getBasicBlockIndex(EpilogPreHeader),<br class="">NewPN);<br class="">+ }<br class="">+ }<br class="">+<br class="">+ Instruction *InsertPt = NewExit->getTerminator();<br class="">+ IRBuilder<> B(InsertPt);<br class="">+ Value *BrLoopExit = B.CreateIsNotNull(ModVal);<br class="">+ assert(Exit && "Loop must have a single exit block only");<br class="">+ // Split the exit to maintain loop canonicalization guarantees<br class="">+ SmallVector<BasicBlock*, 4> Preds(predecessors(Exit));<br class="">+ SplitBlockPredecessors(Exit, Preds, ".epilog-lcssa", DT, LI,<br class="">+ PreserveLCSSA);<br class="">+ // Add the branch to the exit block (around the unrolling loop)<br class="">+ B.CreateCondBr(BrLoopExit, EpilogPreHeader, Exit);<br class="">InsertPt->eraseFromParent();<br class="">}<br class=""><br class="">/// Create a clone of the blocks in a loop and connect them together.<br class="">-/// If UnrollProlog is true, loop structure will not be cloned, otherwise a<br class="">new<br class="">-/// loop will be created including all cloned blocks, and the iterator of<br class="">it<br class="">-/// switches to count NewIter down to 0.<br class="">+/// If CreateRemainderLoop is false, loop structure will not be cloned,<br class="">+/// otherwise a new loop will be created including all cloned blocks, and<br class="">the<br class="">+/// iterator of it switches to count NewIter down to 0.<br class="">+/// The cloned blocks should be inserted between InsertTop and InsertBot.<br class="">+/// If loop structure is cloned InsertTop should be new preheader,<br class="">InsertBot<br class="">+/// new loop exit.<br class="">///<br class="">-static void CloneLoopBlocks(Loop *L, Value *NewIter, const bool<br class="">UnrollProlog,<br class="">+static void CloneLoopBlocks(Loop *L, Value *NewIter,<br class="">+ const bool CreateRemainderLoop,<br class="">+ const bool UseEpilogRemainder,<br class=""> BasicBlock *InsertTop, BasicBlock *InsertBot,<br class="">+ BasicBlock *Preheader,<br class=""> std::vector<BasicBlock *> &NewBlocks,<br class=""> LoopBlocksDFS &LoopBlocks, ValueToValueMapTy<br class="">&VMap,<br class=""> LoopInfo *LI) {<br class="">- BasicBlock *Preheader = L->getLoopPreheader();<br class="">+ StringRef suffix = UseEpilogRemainder ? "epil" : "prol";<br class="">BasicBlock *Header = L->getHeader();<br class="">BasicBlock *Latch = L->getLoopLatch();<br class="">Function *F = Header->getParent();<br class="">@@ -152,7 +281,7 @@ static void CloneLoopBlocks(Loop *L, Val<br class="">LoopBlocksDFS::RPOIterator BlockEnd = LoopBlocks.endRPO();<br class="">Loop *NewLoop = nullptr;<br class="">Loop *ParentLoop = L->getParentLoop();<br class="">- if (!UnrollProlog) {<br class="">+ if (CreateRemainderLoop) {<br class="">NewLoop = new Loop();<br class="">if (ParentLoop)<br class="">ParentLoop->addChildLoop(NewLoop);<br class="">@@ -163,7 +292,7 @@ static void CloneLoopBlocks(Loop *L, Val<br class="">// For each block in the original loop, create a new copy,<br class="">// and update the value map with the newly created values.<br class="">for (LoopBlocksDFS::RPOIterator BB = BlockBegin; BB != BlockEnd; ++BB) {<br class="">- BasicBlock *NewBB = CloneBasicBlock(*BB, VMap, ".prol", F);<br class="">+ BasicBlock *NewBB = CloneBasicBlock(*BB, VMap, "." + suffix, F);<br class="">NewBlocks.push_back(NewBB);<br class=""><br class="">if (NewLoop)<br class="">@@ -179,16 +308,17 @@ static void CloneLoopBlocks(Loop *L, Val<br class="">}<br class=""><br class="">if (Latch == *BB) {<br class="">- // For the last block, if UnrollProlog is true, create a direct jump<br class="">to<br class="">- // InsertBot. If not, create a loop back to cloned head.<br class="">+ // For the last block, if CreateRemainderLoop is false, create a<br class="">direct<br class="">+ // jump to InsertBot. If not, create a loop back to cloned head.<br class="">VMap.erase((*BB)->getTerminator());<br class="">BasicBlock *FirstLoopBB = cast<BasicBlock>(VMap[Header]);<br class="">BranchInst *LatchBR = cast<BranchInst>(NewBB->getTerminator());<br class="">IRBuilder<> Builder(LatchBR);<br class="">- if (UnrollProlog) {<br class="">+ if (!CreateRemainderLoop) {<br class="">Builder.CreateBr(InsertBot);<br class="">} else {<br class="">- PHINode *NewIdx = PHINode::Create(NewIter->getType(), 2,<br class="">"prol.iter",<br class="">+ PHINode *NewIdx = PHINode::Create(NewIter->getType(), 2,<br class="">+ suffix + ".iter",<br class=""> FirstLoopBB->getFirstNonPHI());<br class="">Value *IdxSub =<br class=""> Builder.CreateSub(NewIdx, ConstantInt::get(NewIdx->getType(), 1),<br class="">@@ -207,9 +337,15 @@ static void CloneLoopBlocks(Loop *L, Val<br class="">// cloned loop.<br class="">for (BasicBlock::iterator I = Header->begin(); isa<PHINode>(I); ++I) {<br class="">PHINode *NewPHI = cast<PHINode>(VMap[&*I]);<br class="">- if (UnrollProlog) {<br class="">- VMap[&*I] = NewPHI->getIncomingValueForBlock(Preheader);<br class="">- cast<BasicBlock>(VMap[Header])->getInstList().erase(NewPHI);<br class="">+ if (!CreateRemainderLoop) {<br class="">+ if (UseEpilogRemainder) {<br class="">+ unsigned idx = NewPHI->getBasicBlockIndex(Preheader);<br class="">+ NewPHI->setIncomingBlock(idx, InsertTop);<br class="">+ NewPHI->removeIncomingValue(Latch, false);<br class="">+ } else {<br class="">+ VMap[&*I] = NewPHI->getIncomingValueForBlock(Preheader);<br class="">+ cast<BasicBlock>(VMap[Header])->getInstList().erase(NewPHI);<br class="">+ }<br class="">} else {<br class="">unsigned idx = NewPHI->getBasicBlockIndex(Preheader);<br class="">NewPHI->setIncomingBlock(idx, InsertTop);<br class="">@@ -254,7 +390,7 @@ static void CloneLoopBlocks(Loop *L, Val<br class="">}<br class="">}<br class=""><br class="">-/// Insert code in the prolog code when unrolling a loop with a<br class="">+/// Insert code in the prolog/epilog code when unrolling a loop with a<br class="">/// run-time trip-count.<br class="">///<br class="">/// This method assumes that the loop unroll factor is total number<br class="">@@ -266,6 +402,7 @@ static void CloneLoopBlocks(Loop *L, Val<br class="">/// instruction in SimplifyCFG.cpp. Then, the backend decides how code for<br class="">/// the switch instruction is generated.<br class="">///<br class="">+/// ***Prolog case***<br class="">/// extraiters = tripcount % loopfactor<br class="">/// if (extraiters == 0) jump Loop:<br class="">/// else jump Prol<br class="">@@ -277,17 +414,35 @@ static void CloneLoopBlocks(Loop *L, Val<br class="">/// ...<br class="">/// End:<br class="">///<br class="">-bool llvm::UnrollRuntimeLoopProlog(Loop *L, unsigned Count,<br class="">- bool AllowExpensiveTripCount, LoopInfo<br class="">*LI,<br class="">- ScalarEvolution *SE, DominatorTree *DT,<br class="">- bool PreserveLCSSA) {<br class="">- // For now, only unroll loops that contain a single exit.<br class="">+/// ***Epilog case***<br class="">+/// extraiters = tripcount % loopfactor<br class="">+/// if (extraiters == tripcount) jump LoopExit:<br class="">+/// unroll_iters = tripcount - extraiters<br class="">+/// Loop: LoopBody; (executes unroll_iter times);<br class="">+/// unroll_iter -= 1<br class="">+/// if (unroll_iter != 0) jump Loop:<br class="">+/// LoopExit:<br class="">+/// if (extraiters == 0) jump EpilExit:<br class="">+/// Epil: LoopBody; (executes extraiters times)<br class="">+/// extraiters -= 1 // Omitted if unroll factor is<br class="">2.<br class="">+/// if (extraiters != 0) jump Epil: // Omitted if unroll factor is<br class="">2.<br class="">+/// EpilExit:<br class="">+<br class="">+bool llvm::UnrollRuntimeLoopRemainder(Loop *L, unsigned Count,<br class="">+ bool AllowExpensiveTripCount,<br class="">+ bool UseEpilogRemainder,<br class="">+ LoopInfo *LI, ScalarEvolution *SE,<br class="">+ DominatorTree *DT, bool<br class="">PreserveLCSSA) {<br class="">+ // for now, only unroll loops that contain a single exit<br class="">if (!L->getExitingBlock())<br class="">return false;<br class=""><br class="">// Make sure the loop is in canonical form, and there is a single<br class="">// exit block only.<br class="">- if (!L->isLoopSimplifyForm() || !L->getUniqueExitBlock())<br class="">+ if (!L->isLoopSimplifyForm())<br class="">+ return false;<br class="">+ BasicBlock *Exit = L->getUniqueExitBlock(); // successor out of loop<br class="">+ if (!Exit)<br class="">return false;<br class=""><br class="">// Use Scalar Evolution to compute the trip count. This allows more loops to<br class="">@@ -311,8 +466,8 @@ bool llvm::UnrollRuntimeLoopProlog(Loop<br class="">return false;<br class=""><br class="">BasicBlock *Header = L->getHeader();<br class="">- BasicBlock *PH = L->getLoopPreheader();<br class="">- BranchInst *PreHeaderBR = cast<BranchInst>(PH->getTerminator());<br class="">+ BasicBlock *PreHeader = L->getLoopPreheader();<br class="">+ BranchInst *PreHeaderBR = cast<BranchInst>(PreHeader->getTerminator());<br class="">const DataLayout &DL = Header->getModule()->getDataLayout();<br class="">SCEVExpander Expander(*SE, DL, "loop-unroll");<br class="">if (!AllowExpensiveTripCount &&<br class="">@@ -330,26 +485,75 @@ bool llvm::UnrollRuntimeLoopProlog(Loop<br class="">SE->forgetLoop(ParentLoop);<br class=""><br class="">BasicBlock *Latch = L->getLoopLatch();<br class="">- // It helps to split the original preheader twice, one for the end of the<br class="">- // prolog code and one for a new loop preheader.<br class="">- BasicBlock *PEnd = SplitEdge(PH, Header, DT, LI);<br class="">- BasicBlock *NewPH = SplitBlock(PEnd, PEnd->getTerminator(), DT, LI);<br class="">- PreHeaderBR = cast<BranchInst>(PH->getTerminator());<br class=""><br class="">+ // Loop structure is the following:<br class="">+ //<br class="">+ // PreHeader<br class="">+ // Header<br class="">+ // ...<br class="">+ // Latch<br class="">+ // Exit<br class="">+<br class="">+ BasicBlock *NewPreHeader;<br class="">+ BasicBlock *NewExit = nullptr;<br class="">+ BasicBlock *PrologExit = nullptr;<br class="">+ BasicBlock *EpilogPreHeader = nullptr;<br class="">+ BasicBlock *PrologPreHeader = nullptr;<br class="">+<br class="">+ if (UseEpilogRemainder) {<br class="">+ // If epilog remainder<br class="">+ // Split PreHeader to insert a branch around loop for unrolling.<br class="">+ NewPreHeader = SplitBlock(PreHeader, PreHeader->getTerminator(), DT,<br class="">LI);<br class="">+ NewPreHeader->setName(PreHeader->getName() + ".new");<br class="">+ // Split Exit to create phi nodes from branch above.<br class="">+ SmallVector<BasicBlock*, 4> Preds(predecessors(Exit));<br class="">+ NewExit = SplitBlockPredecessors(Exit, Preds, ".unr-lcssa",<br class="">+ DT, LI, PreserveLCSSA);<br class="">+ // Split NewExit to insert epilog remainder loop.<br class="">+ EpilogPreHeader = SplitBlock(NewExit, NewExit->getTerminator(), DT,<br class="">LI);<br class="">+ EpilogPreHeader->setName(Header->getName() + ".epil.preheader");<br class="">+ } else {<br class="">+ // If prolog remainder<br class="">+ // Split the original preheader twice to insert prolog remainder loop<br class="">+ PrologPreHeader = SplitEdge(PreHeader, Header, DT, LI);<br class="">+ PrologPreHeader->setName(Header->getName() + ".prol.preheader");<br class="">+ PrologExit = SplitBlock(PrologPreHeader,<br class="">PrologPreHeader->getTerminator(),<br class="">+ DT, LI);<br class="">+ PrologExit->setName(Header->getName() + ".prol.loopexit");<br class="">+ // Split PrologExit to get NewPreHeader.<br class="">+ NewPreHeader = SplitBlock(PrologExit, PrologExit->getTerminator(), DT,<br class="">LI);<br class="">+ NewPreHeader->setName(PreHeader->getName() + ".new");<br class="">+ }<br class="">+ // Loop structure should be the following:<br class="">+ // Epilog Prolog<br class="">+ //<br class="">+ // PreHeader PreHeader<br class="">+ // *NewPreHeader *PrologPreHeader<br class="">+ // Header *PrologExit<br class="">+ // ... *NewPreHeader<br class="">+ // Latch Header<br class="">+ // *NewExit ...<br class="">+ // *EpilogPreHeader Latch<br class="">+ // Exit Exit<br class="">+<br class="">+ // Calculate conditions for branch around loop for unrolling<br class="">+ // in epilog case and around prolog remainder loop in prolog case.<br class="">// Compute the number of extra iterations required, which is:<br class="">- // extra iterations = run-time trip count % (loop unroll factor + 1)<br class="">+ // extra iterations = run-time trip count % loop unroll factor<br class="">+ PreHeaderBR = cast<BranchInst>(PreHeader->getTerminator());<br class="">Value *TripCount = Expander.expandCodeFor(TripCountSC,<br class="">TripCountSC->getType(),<br class=""> PreHeaderBR);<br class="">Value *BECount = Expander.expandCodeFor(BECountSC, BECountSC->getType(),<br class=""> PreHeaderBR);<br class="">-<br class="">IRBuilder<> B(PreHeaderBR);<br class="">Value *ModVal;<br class="">// Calculate ModVal = (BECount + 1) % Count.<br class="">// Note that TripCount is BECount + 1.<br class="">if (isPowerOf2_32(Count)) {<br class="">+ // When Count is power of 2 we don't BECount for epilog case, however<br class="">we'll<br class="">+ // need it for a branch around unrolling loop for prolog case.<br class="">ModVal = B.CreateAnd(TripCount, Count - 1, "xtraiter");<br class="">- // 1. There are no iterations to be run in the prologue loop.<br class="">+ // 1. There are no iterations to be run in the prolog/epilog loop.<br class="">// OR<br class="">// 2. The addition computing TripCount overflowed.<br class="">//<br class="">@@ -371,18 +575,18 @@ bool llvm::UnrollRuntimeLoopProlog(Loop<br class=""> ConstantInt::get(BECount->getType(), Count),<br class=""> "xtraiter");<br class="">}<br class="">- Value *BranchVal = B.CreateIsNotNull(ModVal, "lcmp.mod");<br class="">-<br class="">- // Branch to either the extra iterations or the cloned/unrolled loop.<br class="">- // We will fix up the true branch label when adding loop body copies.<br class="">- B.CreateCondBr(BranchVal, PEnd, PEnd);<br class="">- assert(PreHeaderBR->isUnconditional() &&<br class="">- PreHeaderBR->getSuccessor(0) == PEnd &&<br class="">- "CFG edges in Preheader are not correct");<br class="">+ Value *CmpOperand =<br class="">+ UseEpilogRemainder ? TripCount :<br class="">+ ConstantInt::get(TripCount->getType(), 0);<br class="">+ Value *BranchVal = B.CreateICmpNE(ModVal, CmpOperand, "lcmp.mod");<br class="">+ BasicBlock *FirstLoop = UseEpilogRemainder ? NewPreHeader :<br class="">PrologPreHeader;<br class="">+ BasicBlock *SecondLoop = UseEpilogRemainder ? NewExit : PrologExit;<br class="">+ // Branch to either remainder (extra iterations) loop or unrolling loop.<br class="">+ B.CreateCondBr(BranchVal, FirstLoop, SecondLoop);<br class="">PreHeaderBR->eraseFromParent();<br class="">Function *F = Header->getParent();<br class="">// Get an ordered list of blocks in the loop to help with the ordering of<br class="">the<br class="">- // cloned blocks in the prolog code.<br class="">+ // cloned blocks in the prolog/epilog code<br class="">LoopBlocksDFS LoopBlocks(L);<br class="">LoopBlocks.perform(LI);<br class=""><br class="">@@ -394,17 +598,38 @@ bool llvm::UnrollRuntimeLoopProlog(Loop<br class="">std::vector<BasicBlock *> NewBlocks;<br class="">ValueToValueMapTy VMap;<br class=""><br class="">- bool UnrollPrologue = Count == 2;<br class="">+ // For unroll factor 2 remainder loop will have 1 iterations.<br class="">+ // Do not create 1 iteration loop.<br class="">+ bool CreateRemainderLoop = (Count != 2);<br class=""><br class="">// Clone all the basic blocks in the loop. If Count is 2, we don't clone<br class="">// the loop, otherwise we create a cloned loop to execute the extra<br class="">// iterations. This function adds the appropriate CFG connections.<br class="">- CloneLoopBlocks(L, ModVal, UnrollPrologue, PH, PEnd, NewBlocks,<br class="">LoopBlocks,<br class="">- VMap, LI);<br class="">+ BasicBlock *InsertBot = UseEpilogRemainder ? Exit : PrologExit;<br class="">+ BasicBlock *InsertTop = UseEpilogRemainder ? EpilogPreHeader :<br class="">PrologPreHeader;<br class="">+ CloneLoopBlocks(L, ModVal, CreateRemainderLoop, UseEpilogRemainder,<br class="">InsertTop,<br class="">+ InsertBot, NewPreHeader, NewBlocks, LoopBlocks, VMap,<br class="">LI);<br class="">+<br class="">+ // Insert the cloned blocks into the function.<br class="">+ F->getBasicBlockList().splice(InsertBot->getIterator(),<br class="">+ F->getBasicBlockList(),<br class="">+ NewBlocks[0]->getIterator(),<br class="">+ F->end());<br class=""><br class="">- // Insert the cloned blocks into the function just before the original<br class="">loop.<br class="">- F->getBasicBlockList().splice(PEnd->getIterator(),<br class="">F->getBasicBlockList(),<br class="">- NewBlocks[0]->getIterator(), F->end());<br class="">+ // Loop structure should be the following:<br class="">+ // Epilog Prolog<br class="">+ //<br class="">+ // PreHeader PreHeader<br class="">+ // NewPreHeader PrologPreHeader<br class="">+ // Header PrologHeader<br class="">+ // ... ...<br class="">+ // Latch PrologLatch<br class="">+ // NewExit PrologExit<br class="">+ // EpilogPreHeader NewPreHeader<br class="">+ // EpilogHeader Header<br class="">+ // ... ...<br class="">+ // EpilogLatch Latch<br class="">+ // Exit Exit<br class=""><br class="">// Rewrite the cloned instruction operands to use the values created when<br class="">the<br class="">// clone is created.<br class="">@@ -415,11 +640,38 @@ bool llvm::UnrollRuntimeLoopProlog(Loop<br class="">}<br class="">}<br class=""><br class="">- // Connect the prolog code to the original loop and update the<br class="">- // PHI functions.<br class="">- BasicBlock *LastLoopBB = cast<BasicBlock>(VMap[Latch]);<br class="">- ConnectProlog(L, BECount, Count, LastLoopBB, PEnd, PH, NewPH, VMap, DT,<br class="">LI,<br class="">- PreserveLCSSA);<br class="">+ if (UseEpilogRemainder) {<br class="">+ // Connect the epilog code to the original loop and update the<br class="">+ // PHI functions.<br class="">+ ConnectEpilog(L, ModVal, NewExit, Exit, PreHeader,<br class="">+ EpilogPreHeader, NewPreHeader, VMap, DT, LI,<br class="">+ PreserveLCSSA);<br class="">+<br class="">+ // Update counter in loop for unrolling.<br class="">+ // I should be multiply of Count.<br class="">+ IRBuilder<> B2(NewPreHeader->getTerminator());<br class="">+ Value *TestVal = B2.CreateSub(TripCount, ModVal, "unroll_iter");<br class="">+ BranchInst *LatchBR = cast<BranchInst>(Latch->getTerminator());<br class="">+ B2.SetInsertPoint(LatchBR);<br class="">+ PHINode *NewIdx = PHINode::Create(TestVal->getType(), 2, "niter",<br class="">+ Header->getFirstNonPHI());<br class="">+ Value *IdxSub =<br class="">+ B2.CreateSub(NewIdx, ConstantInt::get(NewIdx->getType(), 1),<br class="">+ NewIdx->getName() + ".nsub");<br class="">+ Value *IdxCmp;<br class="">+ if (LatchBR->getSuccessor(0) == Header)<br class="">+ IdxCmp = B2.CreateIsNotNull(IdxSub, NewIdx->getName() + ".ncmp");<br class="">+ else<br class="">+ IdxCmp = B2.CreateIsNull(IdxSub, NewIdx->getName() + ".ncmp");<br class="">+ NewIdx->addIncoming(TestVal, NewPreHeader);<br class="">+ NewIdx->addIncoming(IdxSub, Latch);<br class="">+ LatchBR->setCondition(IdxCmp);<br class="">+ } else {<br class="">+ // Connect the prolog code to the original loop and update the<br class="">+ // PHI functions.<br class="">+ ConnectProlog(L, BECount, Count, PrologExit, PreHeader, NewPreHeader,<br class="">+ VMap, DT, LI, PreserveLCSSA);<br class="">+ }<br class="">NumRuntimeUnrolled++;<br class="">return true;<br class="">}<br class=""><br class="">Modified: llvm/trunk/test/Transforms/LoopUnroll/AArch64/runtime-loop.ll<br class="">URL:<br class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/LoopUnroll/AArch64/runtime-loop.ll?rev=265388&r1=265387&r2=265388&view=diff<br class="">==============================================================================<br class="">--- llvm/trunk/test/Transforms/LoopUnroll/AArch64/runtime-loop.ll (original)<br class="">+++ llvm/trunk/test/Transforms/LoopUnroll/AArch64/runtime-loop.ll Tue Apr 5<br class="">07:19:35 2016<br class="">@@ -1,13 +1,21 @@<br class="">-; RUN: opt < %s -S -loop-unroll -mtriple aarch64 -mcpu=cortex-a57 |<br class="">FileCheck %s<br class="">+; RUN: opt < %s -S -loop-unroll -mtriple aarch64 -mcpu=cortex-a57 |<br class="">FileCheck %s -check-prefix=EPILOG<br class="">+; RUN: opt < %s -S -loop-unroll -mtriple aarch64 -mcpu=cortex-a57<br class="">-unroll-runtime-epilog=false | FileCheck %s -check-prefix=PROLOG<br class=""><br class="">; Tests for unrolling loops with run-time trip counts<br class=""><br class="">-; CHECK: %xtraiter = and i32 %n<br class="">-; CHECK: %lcmp.mod = icmp ne i32 %xtraiter, 0<br class="">-; CHECK: br i1 %lcmp.mod, label %for.body.prol, label<br class="">%for.body.preheader.split<br class="">+; EPILOG: %xtraiter = and i32 %n<br class="">+; EPILOG: %lcmp.mod = icmp ne i32 %xtraiter, %n<br class="">+; EPILOG: br i1 %lcmp.mod, label %for.body.preheader.new, label<br class="">%for.end.loopexit.unr-lcssa<br class=""><br class="">-; CHECK: for.body.prol:<br class="">-; CHECK: for.body:<br class="">+; PROLOG: %xtraiter = and i32 %n<br class="">+; PROLOG: %lcmp.mod = icmp ne i32 %xtraiter, 0<br class="">+; PROLOG: br i1 %lcmp.mod, label %for.body.prol.preheader, label<br class="">%for.body.prol.loopexit<br class="">+<br class="">+; EPILOG: for.body:<br class="">+; EPILOG: for.body.epil:<br class="">+<br class="">+; PROLOG: for.body.prol:<br class="">+; PROLOG: for.body:<br class=""><br class="">define i32 @test(i32* nocapture %a, i32 %n) nounwind uwtable readonly {<br class="">entry:<br class=""><br class="">Modified: llvm/trunk/test/Transforms/LoopUnroll/PowerPC/a2-unrolling.ll<br class="">URL:<br class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/LoopUnroll/PowerPC/a2-unrolling.ll?rev=265388&r1=265387&r2=265388&view=diff<br class="">==============================================================================<br class="">--- llvm/trunk/test/Transforms/LoopUnroll/PowerPC/a2-unrolling.ll (original)<br class="">+++ llvm/trunk/test/Transforms/LoopUnroll/PowerPC/a2-unrolling.ll Tue Apr 5<br class="">07:19:35 2016<br class="">@@ -1,4 +1,5 @@<br class="">-; RUN: opt < %s -S -mtriple=powerpc64-unknown-linux-gnu -mcpu=a2<br class="">-loop-unroll | FileCheck %s<br class="">+; RUN: opt < %s -S -mtriple=powerpc64-unknown-linux-gnu -mcpu=a2<br class="">-loop-unroll | FileCheck %s -check-prefix=EPILOG<br class="">+; RUN: opt < %s -S -mtriple=powerpc64-unknown-linux-gnu -mcpu=a2<br class="">-loop-unroll -unroll-runtime-epilog=false | FileCheck %s<br class="">-check-prefix=PROLOG<br class="">define void @unroll_opt_for_size() nounwind optsize {<br class="">entry:<br class="">br label %loop<br class="">@@ -13,11 +14,17 @@ exit:<br class="">ret void<br class="">}<br class=""><br class="">-; CHECK-LABEL: @unroll_opt_for_size<br class="">-; CHECK: add<br class="">-; CHECK-NEXT: add<br class="">-; CHECK-NEXT: add<br class="">-; CHECK: icmp<br class="">+; EPILOG-LABEL: @unroll_opt_for_size<br class="">+; EPILOG: add<br class="">+; EPILOG-NEXT: add<br class="">+; EPILOG-NEXT: add<br class="">+; EPILOG: icmp<br class="">+<br class="">+; PROLOG-LABEL: @unroll_opt_for_size<br class="">+; PROLOG: add<br class="">+; PROLOG-NEXT: add<br class="">+; PROLOG-NEXT: add<br class="">+; PROLOG: icmp<br class=""><br class="">define i32 @test(i32* nocapture %a, i32 %n) nounwind uwtable readonly {<br class="">entry:<br class="">@@ -40,8 +47,13 @@ for.end:<br class="">ret i32 %sum.0.lcssa<br class="">}<br class=""><br class="">-; CHECK-LABEL: @test<br class="">-; CHECK: for.body.prol{{.*}}:<br class="">-; CHECK: for.body:<br class="">-; CHECK: br i1 %exitcond.7, label %for.end.loopexit{{.*}}, label %for.body<br class="">+; EPILOG-LABEL: @test<br class="">+; EPILOG: for.body:<br class="">+; EPILOG: br i1 %niter.ncmp.7, label %for.end.loopexit{{.*}}, label<br class="">%for.body<br class="">+; EPILOG: for.body.epil{{.*}}:<br class="">+<br class="">+; PROLOG-LABEL: @test<br class="">+; PROLOG: for.body.prol{{.*}}:<br class="">+; PROLOG: for.body:<br class="">+; PROLOG: br i1 %exitcond.7, label %for.end.loopexit{{.*}}, label %for.body<br class=""><br class=""><br class="">Modified: llvm/trunk/test/Transforms/LoopUnroll/X86/mmx.ll<br class="">URL:<br class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/LoopUnroll/X86/mmx.ll?rev=265388&r1=265387&r2=265388&view=diff<br class="">==============================================================================<br class="">--- llvm/trunk/test/Transforms/LoopUnroll/X86/mmx.ll (original)<br class="">+++ llvm/trunk/test/Transforms/LoopUnroll/X86/mmx.ll Tue Apr 5 07:19:35<br class="">2016<br class="">@@ -14,9 +14,9 @@ for.body:<br class=""><br class="">exit: ; preds = %for.body<br class="">%ret = phi x86_mmx [ undef, %for.body ]<br class="">- ; CHECK: %[[ret_unr:.*]] = phi x86_mmx [ undef,<br class="">- ; CHECK: %[[ret_ph:.*]] = phi x86_mmx [ undef,<br class="">- ; CHECK: %[[ret:.*]] = phi x86_mmx [ %[[ret_unr]], {{.*}} ], [<br class="">%[[ret_ph]]<br class="">+ ; CHECK: %[[ret_ph:.*]] = phi x86_mmx [ undef, %entry<br class="">+ ; CHECK: %[[ret_ph1:.*]] = phi x86_mmx [ undef,<br class="">+ ; CHECK: %[[ret:.*]] = phi x86_mmx [ %[[ret_ph]], {{.*}} ], [<br class="">%[[ret_ph1]],<br class="">; CHECK: ret x86_mmx %[[ret]]<br class="">ret x86_mmx %ret<br class="">}<br class=""><br class="">Modified:<br class="">llvm/trunk/test/Transforms/LoopUnroll/high-cost-trip-count-computation.ll<br class="">URL:<br class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/LoopUnroll/high-cost-trip-count-computation.ll?rev=265388&r1=265387&r2=265388&view=diff<br class="">==============================================================================<br class="">---<br class="">llvm/trunk/test/Transforms/LoopUnroll/high-cost-trip-count-computation.ll<br class="">(original)<br class="">+++<br class="">llvm/trunk/test/Transforms/LoopUnroll/high-cost-trip-count-computation.ll<br class="">Tue Apr 5 07:19:35 2016<br class="">@@ -34,7 +34,7 @@ define i32 @test2(i64* %loc, i64 %conv7)<br class="">; CHECK: udiv<br class="">; CHECK: udiv<br class="">; CHECK-NOT: udiv<br class="">-; CHECK-LABEL: for.body.prol<br class="">+; CHECK-LABEL: for.body<br class="">entry:<br class="">%rem0 = load i64, i64* %loc, align 8<br class="">%ExpensiveComputation = udiv i64 %rem0, 42 ; <<< Extra computations are<br class="">added to the trip-count expression<br class=""><br class="">Modified: llvm/trunk/test/Transforms/LoopUnroll/runtime-loop.ll<br class="">URL:<br class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/LoopUnroll/runtime-loop.ll?rev=265388&r1=265387&r2=265388&view=diff<br class="">==============================================================================<br class="">--- llvm/trunk/test/Transforms/LoopUnroll/runtime-loop.ll (original)<br class="">+++ llvm/trunk/test/Transforms/LoopUnroll/runtime-loop.ll Tue Apr 5<br class="">07:19:35 2016<br class="">@@ -1,18 +1,30 @@<br class="">-; RUN: opt < %s -S -loop-unroll -unroll-runtime=true | FileCheck %s<br class="">+; RUN: opt < %s -S -loop-unroll -unroll-runtime=true | FileCheck %s<br class="">-check-prefix=EPILOG<br class="">+; RUN: opt < %s -S -loop-unroll -unroll-runtime=true<br class="">-unroll-runtime-epilog=false | FileCheck %s -check-prefix=PROLOG<br class=""><br class="">target datalayout =<br class="">"e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"<br class=""><br class="">; Tests for unrolling loops with run-time trip counts<br class=""><br class="">-; CHECK: %xtraiter = and i32 %n<br class="">-; CHECK: %lcmp.mod = icmp ne i32 %xtraiter, 0<br class="">-; CHECK: br i1 %lcmp.mod, label %for.body.prol, label<br class="">%for.body.preheader.split<br class="">-<br class="">-; CHECK: for.body.prol:<br class="">-; CHECK: %indvars.iv.prol = phi i64 [ %indvars.iv.next.prol, %for.body.prol<br class="">], [ 0, %for.body.preheader ]<br class="">-; CHECK: %prol.iter.sub = sub i32 %prol.iter, 1<br class="">-; CHECK: %prol.iter.cmp = icmp ne i32 %prol.iter.sub, 0<br class="">-; CHECK: br i1 %prol.iter.cmp, label %for.body.prol, label<br class="">%for.body.preheader.split, !llvm.loop !0<br class="">+; EPILOG: %xtraiter = and i32 %n<br class="">+; EPILOG: %lcmp.mod = icmp ne i32 %xtraiter, %n<br class="">+; EPILOG: br i1 %lcmp.mod, label %for.body.preheader.new, label<br class="">%for.end.loopexit.unr-lcssa<br class="">+<br class="">+; PROLOG: %xtraiter = and i32 %n<br class="">+; PROLOG: %lcmp.mod = icmp ne i32 %xtraiter, 0<br class="">+; PROLOG: br i1 %lcmp.mod, label %for.body.prol.preheader, label<br class="">%for.body.prol.loopexit<br class="">+<br class="">+; EPILOG: for.body.epil:<br class="">+; EPILOG: %indvars.iv.epil = phi i64 [ %indvars.iv.next.epil,<br class="">%for.body.epil ], [ %indvars.iv.unr, %for.body.epil.preheader ]<br class="">+; EPILOG: %epil.iter.sub = sub i32 %epil.iter, 1<br class="">+; EPILOG: %epil.iter.cmp = icmp ne i32 %epil.iter.sub, 0<br class="">+; EPILOG: br i1 %epil.iter.cmp, label %for.body.epil, label<br class="">%for.end.loopexit.epilog-lcssa, !llvm.loop !0<br class="">+<br class="">+; PROLOG: for.body.prol:<br class="">+; PROLOG: %indvars.iv.prol = phi i64 [ %indvars.iv.next.prol,<br class="">%for.body.prol ], [ 0, %for.body.prol.preheader ]<br class="">+; PROLOG: %prol.iter.sub = sub i32 %prol.iter, 1<br class="">+; PROLOG: %prol.iter.cmp = icmp ne i32 %prol.iter.sub, 0<br class="">+; PROLOG: br i1 %prol.iter.cmp, label %for.body.prol, label<br class="">%for.body.prol.loopexit, !llvm.loop !0<br class="">+<br class=""><br class="">define i32 @test(i32* nocapture %a, i32 %n) nounwind uwtable readonly {<br class="">entry:<br class="">@@ -39,8 +51,11 @@ for.end:<br class="">; Still try to completely unroll loops with compile-time trip counts<br class="">; even if the -unroll-runtime is specified<br class=""><br class="">-; CHECK: for.body:<br class="">-; CHECK-NOT: for.body.prol:<br class="">+; EPILOG: for.body:<br class="">+; EPILOG-NOT: for.body.epil:<br class="">+<br class="">+; PROLOG: for.body:<br class="">+; PROLOG-NOT: for.body.prol:<br class=""><br class="">define i32 @test1(i32* nocapture %a) nounwind uwtable readonly {<br class="">entry:<br class="">@@ -64,7 +79,8 @@ for.end:<br class="">; This is test 2007-05-09-UnknownTripCount.ll which can be unrolled now<br class="">; if the -unroll-runtime option is turned on<br class=""><br class="">-; CHECK: bb72.2:<br class="">+; EPILOG: bb72.2:<br class="">+; PROLOG: bb72.2:<br class=""><br class="">define void @foo(i32 %trips) {<br class="">entry:<br class="">@@ -86,8 +102,11 @@ cond_true138:<br class=""><br class="">; Test run-time unrolling for a loop that counts down by -2.<br class=""><br class="">-; CHECK: for.body.prol:<br class="">-; CHECK: br i1 %prol.iter.cmp, label %for.body.prol, label<br class="">%for.body.preheader.split<br class="">+; EPILOG: for.body.epil:<br class="">+; EPILOG: br i1 %epil.iter.cmp, label %for.body.epil, label<br class="">%for.cond.for.end_crit_edge.epilog-lcssa<br class="">+<br class="">+; PROLOG: for.body.prol:<br class="">+; PROLOG: br i1 %prol.iter.cmp, label %for.body.prol, label<br class="">%for.body.prol.loopexit<br class=""><br class="">define zeroext i16 @down(i16* nocapture %p, i32 %len) nounwind uwtable<br class="">readonly {<br class="">entry:<br class="">@@ -116,8 +135,11 @@ for.end:<br class="">}<br class=""><br class="">; Test run-time unrolling disable metadata.<br class="">-; CHECK: for.body:<br class="">-; CHECK-NOT: for.body.prol:<br class="">+; EPILOG: for.body:<br class="">+; EPILOG-NOT: for.body.epil:<br class="">+<br class="">+; PROLOG: for.body:<br class="">+; PROLOG-NOT: for.body.prol:<br class=""><br class="">define zeroext i16 @test2(i16* nocapture %p, i32 %len) nounwind uwtable<br class="">readonly {<br class="">entry:<br class="">@@ -148,6 +170,8 @@ for.end:<br class="">!0 = distinct !{!0, !1}<br class="">!1 = !{!"llvm.loop.unroll.runtime.disable"}<br class=""><br class="">-; CHECK: !0 = distinct !{!0, !1}<br class="">-; CHECK: !1 = !{!"llvm.loop.unroll.disable"}<br class="">+; EPILOG: !0 = distinct !{!0, !1}<br class="">+; EPILOG: !1 = !{!"llvm.loop.unroll.disable"}<br class=""><br class="">+; PROLOG: !0 = distinct !{!0, !1}<br class="">+; PROLOG: !1 = !{!"llvm.loop.unroll.disable"}<br class=""><br class="">Modified: llvm/trunk/test/Transforms/LoopUnroll/runtime-loop1.ll<br class="">URL:<br class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/LoopUnroll/runtime-loop1.ll?rev=265388&r1=265387&r2=265388&view=diff<br class="">==============================================================================<br class="">--- llvm/trunk/test/Transforms/LoopUnroll/runtime-loop1.ll (original)<br class="">+++ llvm/trunk/test/Transforms/LoopUnroll/runtime-loop1.ll Tue Apr 5<br class="">07:19:35 2016<br class="">@@ -1,19 +1,35 @@<br class="">-; RUN: opt < %s -S -loop-unroll -unroll-runtime -unroll-count=2 | FileCheck<br class="">%s<br class="">+; RUN: opt < %s -S -loop-unroll -unroll-runtime -unroll-count=2 | FileCheck<br class="">%s -check-prefix=EPILOG<br class="">+; RUN: opt < %s -S -loop-unroll -unroll-runtime -unroll-count=2<br class="">-unroll-runtime-epilog=false | FileCheck %s -check-prefix=PROLOG<br class=""><br class="">; This tests that setting the unroll count works<br class=""><br class="">-; CHECK: for.body.preheader:<br class="">-; CHECK: br {{.*}} label %for.body.prol, label %for.body.preheader.split,<br class="">!dbg [[PH_LOC:![0-9]+]]<br class="">-; CHECK: for.body.prol:<br class="">-; CHECK: br label %for.body.preheader.split, !dbg [[BODY_LOC:![0-9]+]]<br class="">-; CHECK: for.body.preheader.split:<br class="">-; CHECK: br {{.*}} label %for.end.loopexit, label<br class="">%for.body.preheader.split.split, !dbg [[PH_LOC]]<br class="">-; CHECK: for.body:<br class="">-; CHECK: br i1 %exitcond.1, label %for.end.loopexit.unr-lcssa, label<br class="">%for.body, !dbg [[BODY_LOC]]<br class="">-; CHECK-NOT: br i1 %exitcond.4, label %for.end.loopexit{{.*}}, label<br class="">%for.body<br class=""><br class="">-; CHECK-DAG: [[PH_LOC]] = !DILocation(line: 101, column: 1, scope: !{{.*}})<br class="">-; CHECK-DAG: [[BODY_LOC]] = !DILocation(line: 102, column: 1, scope:<br class="">!{{.*}})<br class="">+; EPILOG: for.body.preheader:<br class="">+; EPILOG: br i1 %lcmp.mod, label %for.body.preheader.new, label<br class="">%for.end.loopexit.unr-lcssa, !dbg [[PH_LOC:![0-9]+]]<br class="">+; EPILOG: for.body:<br class="">+; EPILOG: br i1 %niter.ncmp.1, label<br class="">%for.end.loopexit.unr-lcssa.loopexit, label %for.body, !dbg<br class="">[[BODY_LOC:![0-9]+]]<br class="">+; EPILOG-NOT: br i1 %niter.ncmp.2, label %for.end.loopexit{{.*}}, label<br class="">%for.body<br class="">+; EPILOG: for.body.epil.preheader:<br class="">+; EPILOG: br label %for.body.epil, !dbg [[EXIT_LOC:![0-9]+]]<br class="">+; EPILOG: for.body.epil:<br class="">+; EPILOG: br label %for.end.loopexit.epilog-lcssa, !dbg<br class="">[[BODY_LOC:![0-9]+]]<br class="">+<br class="">+; EPILOG-DAG: [[PH_LOC]] = !DILocation(line: 101, column: 1, scope:<br class="">!{{.*}})<br class="">+; EPILOG-DAG: [[BODY_LOC]] = !DILocation(line: 102, column: 1, scope:<br class="">!{{.*}})<br class="">+; EPILOG-DAG: [[EXIT_LOC]] = !DILocation(line: 103, column: 1, scope:<br class="">!{{.*}})<br class="">+<br class="">+; PROLOG: for.body.preheader:<br class="">+; PROLOG: br {{.*}} label %for.body.prol.preheader, label<br class="">%for.body.prol.loopexit, !dbg [[PH_LOC:![0-9]+]]<br class="">+; PROLOG: for.body.prol:<br class="">+; PROLOG: br label %for.body.prol.loopexit, !dbg [[BODY_LOC:![0-9]+]]<br class="">+; PROLOG: for.body.prol.loopexit:<br class="">+; PROLOG: br {{.*}} label %for.end.loopexit, label<br class="">%for.body.preheader.new, !dbg [[PH_LOC]]<br class="">+; PROLOG: for.body:<br class="">+; PROLOG: br i1 %exitcond.1, label %for.end.loopexit.unr-lcssa, label<br class="">%for.body, !dbg [[BODY_LOC]]<br class="">+; PROLOG-NOT: br i1 %exitcond.4, label %for.end.loopexit{{.*}}, label<br class="">%for.body<br class="">+<br class="">+; PROLOG-DAG: [[PH_LOC]] = !DILocation(line: 101, column: 1, scope:<br class="">!{{.*}})<br class="">+; PROLOG-DAG: [[BODY_LOC]] = !DILocation(line: 102, column: 1, scope:<br class="">!{{.*}})<br class=""><br class="">define i32 @test(i32* nocapture %a, i32 %n) nounwind uwtable readonly !dbg<br class="">!6 {<br class="">entry:<br class=""><br class="">Modified: llvm/trunk/test/Transforms/LoopUnroll/runtime-loop2.ll<br class="">URL:<br class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/LoopUnroll/runtime-loop2.ll?rev=265388&r1=265387&r2=265388&view=diff<br class="">==============================================================================<br class="">--- llvm/trunk/test/Transforms/LoopUnroll/runtime-loop2.ll (original)<br class="">+++ llvm/trunk/test/Transforms/LoopUnroll/runtime-loop2.ll Tue Apr 5<br class="">07:19:35 2016<br class="">@@ -1,12 +1,18 @@<br class="">-; RUN: opt < %s -S -loop-unroll -unroll-threshold=25 -unroll-runtime<br class="">-unroll-count=8 | FileCheck %s<br class="">+; RUN: opt < %s -S -loop-unroll -unroll-threshold=25 -unroll-runtime<br class="">-unroll-count=8 | FileCheck %s -check-prefix=EPILOG<br class="">+; RUN: opt < %s -S -loop-unroll -unroll-threshold=25 -unroll-runtime<br class="">-unroll-runtime-epilog=false | FileCheck %s -check-prefix=PROLOG<br class=""><br class="">; Choose a smaller, power-of-two, unroll count if the loop is too large.<br class="">; This test makes sure we're not unrolling 'odd' counts<br class=""><br class="">-; CHECK: for.body.prol:<br class="">-; CHECK: for.body:<br class="">-; CHECK: br i1 %exitcond.3, label %for.end.loopexit{{.*}}, label %for.body<br class="">-; CHECK-NOT: br i1 %exitcond.4, label %for.end.loopexit{{.*}}, label<br class="">%for.body<br class="">+; EPILOG: for.body:<br class="">+; EPILOG: br i1 %niter.ncmp.3, label<br class="">%for.end.loopexit.unr-lcssa.loopexit{{.*}}, label %for.body<br class="">+; EPILOG-NOT: br i1 %niter.ncmp.4, label<br class="">%for.end.loopexit.unr-lcssa.loopexit{{.*}}, label %for.body<br class="">+; EPILOG: for.body.epil:<br class="">+<br class="">+; PROLOG: for.body.prol:<br class="">+; PROLOG: for.body:<br class="">+; PROLOG: br i1 %exitcond.3, label %for.end.loopexit{{.*}}, label %for.body<br class="">+; PROLOG-NOT: br i1 %exitcond.4, label %for.end.loopexit{{.*}}, label<br class="">%for.body<br class=""><br class="">define i32 @test(i32* nocapture %a, i32 %n) nounwind uwtable readonly {<br class="">entry:<br class=""><br class="">Modified: llvm/trunk/test/Transforms/LoopUnroll/runtime-loop4.ll<br class="">URL:<br class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/LoopUnroll/runtime-loop4.ll?rev=265388&r1=265387&r2=265388&view=diff<br class="">==============================================================================<br class="">--- llvm/trunk/test/Transforms/LoopUnroll/runtime-loop4.ll (original)<br class="">+++ llvm/trunk/test/Transforms/LoopUnroll/runtime-loop4.ll Tue Apr 5<br class="">07:19:35 2016<br class="">@@ -1,13 +1,21 @@<br class="">-; RUN: opt < %s -S -O2 -unroll-runtime=true | FileCheck %s<br class="">+; RUN: opt < %s -S -O2 -unroll-runtime=true | FileCheck %s<br class="">-check-prefix=EPILOG<br class="">+; RUN: opt < %s -S -O2 -unroll-runtime=true -unroll-runtime-epilog=false |<br class="">FileCheck %s -check-prefix=PROLOG<br class=""><br class="">; Check runtime unrolling prologue can be promoted by LICM pass.<br class=""><br class="">-; CHECK: entry:<br class="">-; CHECK: %xtraiter<br class="">-; CHECK: %lcmp.mod<br class="">-; CHECK: loop1:<br class="">-; CHECK: br i1 %lcmp.mod<br class="">-; CHECK: loop2.prol:<br class="">+; EPILOG: entry:<br class="">+; EPILOG: %xtraiter<br class="">+; EPILOG: %lcmp.mod<br class="">+; EPILOG: loop1:<br class="">+; EPILOG: br i1 %lcmp.mod<br class="">+; EPILOG: loop2.epil:<br class="">+<br class="">+; PROLOG: entry:<br class="">+; PROLOG: %xtraiter<br class="">+; PROLOG: %lcmp.mod<br class="">+; PROLOG: loop1:<br class="">+; PROLOG: br i1 %lcmp.mod<br class="">+; PROLOG: loop2.prol:<br class=""><br class="">define void @unroll(i32 %iter, i32* %addr1, i32* %addr2) nounwind {<br class="">entry:<br class=""><br class="">Modified: llvm/trunk/test/Transforms/LoopUnroll/runtime-loop5.ll<br class="">URL:<br class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/LoopUnroll/runtime-loop5.ll?rev=265388&r1=265387&r2=265388&view=diff<br class="">==============================================================================<br class="">--- llvm/trunk/test/Transforms/LoopUnroll/runtime-loop5.ll (original)<br class="">+++ llvm/trunk/test/Transforms/LoopUnroll/runtime-loop5.ll Tue Apr 5<br class="">07:19:35 2016<br class="">@@ -11,9 +11,6 @@ entry:<br class="">%cmp1 = icmp eq i3 %n, 0<br class="">br i1 %cmp1, label %for.end, label %for.body<br class=""><br class="">-; UNROLL-16-NOT: for.body.prol:<br class="">-; UNROLL-4: for.body.prol:<br class="">-<br class="">for.body: ; preds = %for.body,<br class="">%entry<br class="">; UNROLL-16-LABEL: for.body:<br class="">; UNROLL-4-LABEL: for.body:<br class="">@@ -39,6 +36,10 @@ for.body:<br class=""><br class="">; UNROLL-16-LABEL: for.end<br class="">; UNROLL-4-LABEL: for.end<br class="">+<br class="">+; UNROLL-16-NOT: for.body.epil:<br class="">+; UNROLL-4: for.body.epil:<br class="">+<br class="">for.end: ; preds = %for.body,<br class="">%entry<br class="">%sum.0.lcssa = phi i3 [ 0, %entry ], [ %add, %for.body ]<br class="">ret i3 %sum.0.lcssa<br class=""><br class="">Modified: llvm/trunk/test/Transforms/LoopUnroll/tripcount-overflow.ll<br class="">URL:<br class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/LoopUnroll/tripcount-overflow.ll?rev=265388&r1=265387&r2=265388&view=diff<br class="">==============================================================================<br class="">--- llvm/trunk/test/Transforms/LoopUnroll/tripcount-overflow.ll (original)<br class="">+++ llvm/trunk/test/Transforms/LoopUnroll/tripcount-overflow.ll Tue Apr 5<br class="">07:19:35 2016<br class="">@@ -13,13 +13,13 @@ target datalayout = "e-m:o-i64:64-f80:12<br class="">; CHECK: entry:<br class="">; CHECK-NEXT: %0 = add i32 %N, 1<br class="">; CHECK-NEXT: %xtraiter = and i32 %0, 1<br class="">-; CHECK-NEXT: %lcmp.mod = icmp ne i32 %xtraiter, 0<br class="">-; CHECK-NEXT: br i1 %lcmp.mod, label %while.body.prol, label %entry.split<br class="">+; CHECK-NEXT: %lcmp.mod = icmp ne i32 %xtraiter, %0<br class="">+; CHECK-NEXT: br i1 %lcmp.mod, label %entry.new, label %while.end.unr-lcssa<br class=""><br class="">-; CHECK: while.body.prol:<br class="">-; CHECK: br label %entry.split<br class="">+; CHECK: while.body.epil:<br class="">+; CHECK: br label %while.end.epilog-lcssa<br class=""><br class="">-; CHECK: entry.split:<br class="">+; CHECK: while.end.epilog-lcssa:<br class=""><br class="">; Function Attrs: nounwind readnone ssp uwtable<br class="">define i32 @foo(i32 %N) {<br class=""><br class="">Modified: llvm/trunk/test/Transforms/LoopUnroll/unroll-cleanup.ll<br class="">URL:<br class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/LoopUnroll/unroll-cleanup.ll?rev=265388&r1=265387&r2=265388&view=diff<br class="">==============================================================================<br class="">--- llvm/trunk/test/Transforms/LoopUnroll/unroll-cleanup.ll (original)<br class="">+++ llvm/trunk/test/Transforms/LoopUnroll/unroll-cleanup.ll Tue Apr 5<br class="">07:19:35 2016<br class="">@@ -4,14 +4,14 @@<br class="">; RUN: opt < %s -O2 -S | FileCheck %s<br class=""><br class="">; After loop unroll:<br class="">-; %dec18 = add nsw i32 %dec18.in, -1<br class="">+; %niter.nsub = add nsw i32 %niter, -1<br class="">; ...<br class="">-; %dec18.1 = add nsw i32 %dec18, -1<br class="">+; %niter.nsub.1 = add nsw i32 %niter.nsub, -1<br class="">; should be merged to:<br class="">-; %dec18.1 = add nsw i32 %dec18.in, -2<br class="">+; %dec18.1 = add nsw i32 %niter, -2<br class="">;<br class="">; CHECK-LABEL: @_Z3fn1v(<br class="">-; CHECK: %dec18.1 = add nsw i32 %dec18.in, -2<br class="">+; CHECK: %niter.nsub.1 = add i32 %niter, -2<br class=""><br class="">; ModuleID = '<stdin>'<br class="">target triple = "x86_64-unknown-linux-gnu"<br class=""><br class="">Modified: llvm/trunk/test/Transforms/LoopUnroll/unroll-pragmas.ll<br class="">URL:<br class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/LoopUnroll/unroll-pragmas.ll?rev=265388&r1=265387&r2=265388&view=diff<br class="">==============================================================================<br class="">--- llvm/trunk/test/Transforms/LoopUnroll/unroll-pragmas.ll (original)<br class="">+++ llvm/trunk/test/Transforms/LoopUnroll/unroll-pragmas.ll Tue Apr 5<br class="">07:19:35 2016<br class="">@@ -171,10 +171,6 @@ for.end:<br class="">; should be duplicated (original and 4x unrolled).<br class="">;<br class="">; CHECK-LABEL: @runtime_loop_with_count4(<br class="">-; CHECK: for.body.prol:<br class="">-; CHECK: store<br class="">-; CHECK-NOT: store<br class="">-; CHECK: br i1<br class="">; CHECK: for.body<br class="">; CHECK: store<br class="">; CHECK: store<br class="">@@ -182,6 +178,10 @@ for.end:<br class="">; CHECK: store<br class="">; CHECK-NOT: store<br class="">; CHECK: br i1<br class="">+; CHECK: for.body.epil:<br class="">+; CHECK: store<br class="">+; CHECK-NOT: store<br class="">+; CHECK: br i1<br class="">define void @runtime_loop_with_count4(i32* nocapture %a, i32 %b) {<br class="">entry:<br class="">%cmp3 = icmp sgt i32 %b, 0<br class="">@@ -287,10 +287,6 @@ for.end:<br class="">; (original and 8x).<br class="">;<br class="">; CHECK-LABEL: @runtime_loop_with_enable(<br class="">-; CHECK: for.body.prol:<br class="">-; CHECK: store<br class="">-; CHECK-NOT: store<br class="">-; CHECK: br i1<br class="">; CHECK: for.body:<br class="">; CHECK: store i32<br class="">; CHECK: store i32<br class="">@@ -302,6 +298,10 @@ for.end:<br class="">; CHECK: store i32<br class="">; CHECK-NOT: store i32<br class="">; CHECK: br i1<br class="">+; CHECK: for.body.epil:<br class="">+; CHECK: store<br class="">+; CHECK-NOT: store<br class="">+; CHECK: br i1<br class="">define void @runtime_loop_with_enable(i32* nocapture %a, i32 %b) {<br class="">entry:<br class="">%cmp3 = icmp sgt i32 %b, 0<br class="">@@ -328,16 +328,16 @@ for.end:<br class="">; should be duplicated (original and 3x unrolled).<br class="">;<br class="">; CHECK-LABEL: @runtime_loop_with_count3(<br class="">-; CHECK: for.body.prol:<br class="">-; CHECK: store<br class="">-; CHECK-NOT: store<br class="">-; CHECK: br i1<br class="">; CHECK: for.body<br class="">; CHECK: store<br class="">; CHECK: store<br class="">; CHECK: store<br class="">; CHECK-NOT: store<br class="">; CHECK: br i1<br class="">+; CHECK: for.body.epil:<br class="">+; CHECK: store<br class="">+; CHECK-NOT: store<br class="">+; CHECK: br i1<br class="">define void @runtime_loop_with_count3(i32* nocapture %a, i32 %b) {<br class="">entry:<br class="">%cmp3 = icmp sgt i32 %b, 0<br class=""><br class=""><br class="">_______________________________________________<br class="">llvm-commits mailing list<br class="">llvm-commits@lists.llvm.org<br class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits<br class=""><br class=""><br class="">_______________________________________________<br class="">llvm-commits mailing list<br class="">llvm-commits@lists.llvm.org<br class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits<br class=""><br class=""><br class=""></blockquote></blockquote><br class=""></blockquote><br class=""></blockquote></blockquote><br class=""></blockquote></blockquote><br class=""></blockquote></blockquote><br class="">_______________________________________________<br class="">llvm-commits mailing list<br class=""><a href="mailto:llvm-commits@lists.llvm.org" class="">llvm-commits@lists.llvm.org</a><br class=""><a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits</a></blockquote></div></blockquote></div><br class=""></body></html>