<div dir="ltr"><div dir="ltr">This showed up in the google internal testing on PPC, a segmentation fault was triggered, and then I looked for whether there are also failures in open source on one of the existing buildbots. I didn't check whether the buildbot is one of those that are notifying or not and whether it has the exact same problem I saw internally, sorry for that. But based on the testing I did (I ran a binary search over LLVM changes), I can say with certainty that your change introduced this segmentation fault on PPC. And after the revert, it was again working fine. I will ask people more knowledgeable than me how I can create a reproducer for you. I was the one responsible for integrating the LLVM changes into google, and it was a bad week and I could barely keep up with the rate of breakages and/or things needing adjustments.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Feb 5, 2021 at 6:05 PM Philip Reames <<a href="mailto:listmail@philipreames.com">listmail@philipreames.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Adrian,<br>
<br>
I'm going to need you to provide a bit more information here. The test <br>
failure in stage1 was fixed at the time you reverted this patch. The <br>
remaining failure in the bot is very unclear. What is a execution_time <br>
failure? From the log output, the "failing" run finished in 0.5 seconds, <br>
whereas the previous "succeeding" run finished in 11 seconds. Without <br>
further context, I'd say that's no failure.<br>
<br>
I'll also note that I did not receive email from this bot. I received <br>
notice from the various other bots and fixed the ARM test issue, but <br>
unless I missed it in with the others, this bot is not notifying.<br>
<br>
In general, I'm a fan of fast reverts, but I have to admit, this one <br>
appears borderline at the moment.<br>
<br>
Philip<br>
<br>
On 2/5/21 3:53 AM, Adrian Kuegel via llvm-commits wrote:<br>
> Author: Adrian Kuegel<br>
> Date: 2021-02-05T12:51:03+01:00<br>
> New Revision: 7fe41ac3dff2d44c3d2c31b28554fbe4a86eaa6c<br>
><br>
> URL: <a href="https://github.com/llvm/llvm-project/commit/7fe41ac3dff2d44c3d2c31b28554fbe4a86eaa6c" rel="noreferrer" target="_blank">https://github.com/llvm/llvm-project/commit/7fe41ac3dff2d44c3d2c31b28554fbe4a86eaa6c</a><br>
> DIFF: <a href="https://github.com/llvm/llvm-project/commit/7fe41ac3dff2d44c3d2c31b28554fbe4a86eaa6c.diff" rel="noreferrer" target="_blank">https://github.com/llvm/llvm-project/commit/7fe41ac3dff2d44c3d2c31b28554fbe4a86eaa6c.diff</a><br>
><br>
> LOG: Revert "[LV] Unconditionally branch from middle to scalar preheader if the scalar loop must execute"<br>
><br>
> This reverts commit 3e5ce49e5371ce4feadbf97dd5c2b652d9db3d1d.<br>
><br>
> Tests started failing on PPC, for example:<br>
> <a href="http://lab.llvm.org:8011/#/builders/105/builds/5569" rel="noreferrer" target="_blank">http://lab.llvm.org:8011/#/builders/105/builds/5569</a><br>
><br>
> Added:<br>
> <br>
><br>
> Modified:<br>
> llvm/lib/Transforms/Utils/LoopVersioning.cpp<br>
> llvm/lib/Transforms/Vectorize/LoopVectorize.cpp<br>
> llvm/test/Transforms/LoopVectorize/first-order-recurrence-complex.ll<br>
> llvm/test/Transforms/LoopVectorize/interleaved-accesses.ll<br>
> llvm/test/Transforms/LoopVectorize/loop-form.ll<br>
><br>
> Removed:<br>
> <br>
><br>
><br>
> ################################################################################<br>
> diff --git a/llvm/lib/Transforms/Utils/LoopVersioning.cpp b/llvm/lib/Transforms/Utils/LoopVersioning.cpp<br>
> index 8a89158788cf..de4fb446fdf2 100644<br>
> --- a/llvm/lib/Transforms/Utils/LoopVersioning.cpp<br>
> +++ b/llvm/lib/Transforms/Utils/LoopVersioning.cpp<br>
> @@ -44,11 +44,11 @@ LoopVersioning::LoopVersioning(const LoopAccessInfo &LAI,<br>
> AliasChecks(Checks.begin(), Checks.end()),<br>
> Preds(LAI.getPSE().getUnionPredicate()), LAI(LAI), LI(LI), DT(DT),<br>
> SE(SE) {<br>
> + assert(L->getUniqueExitBlock() && "No single exit block");<br>
> }<br>
> <br>
> void LoopVersioning::versionLoop(<br>
> const SmallVectorImpl<Instruction *> &DefsUsedOutside) {<br>
> - assert(VersionedLoop->getUniqueExitBlock() && "No single exit block");<br>
> assert(VersionedLoop->isLoopSimplifyForm() &&<br>
> "Loop is not in loop-simplify form");<br>
> <br>
><br>
> diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp<br>
> index 3277842edbfe..6bce0caeb36f 100644<br>
> --- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp<br>
> +++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp<br>
> @@ -852,7 +852,7 @@ class InnerLoopVectorizer {<br>
> /// Middle Block between the vector and the scalar.<br>
> BasicBlock *LoopMiddleBlock;<br>
> <br>
> - /// The unique ExitBlock of the scalar loop if one exists. Note that<br>
> + /// The (unique) ExitBlock of the scalar loop. Note that<br>
> /// there can be multiple exiting edges reaching this block.<br>
> BasicBlock *LoopExitBlock;<br>
> <br>
> @@ -3147,13 +3147,9 @@ void InnerLoopVectorizer::emitMinimumIterationCountCheck(Loop *L,<br>
> DT->getNode(Bypass)->getIDom()) &&<br>
> "TC check is expected to dominate Bypass");<br>
> <br>
> - // Update dominator for Bypass & LoopExit (if needed).<br>
> + // Update dominator for Bypass & LoopExit.<br>
> DT->changeImmediateDominator(Bypass, TCCheckBlock);<br>
> - if (!Cost->requiresScalarEpilogue())<br>
> - // If there is an epilogue which must run, there's no edge from the<br>
> - // middle block to exit blocks and thus no need to update the immediate<br>
> - // dominator of the exit blocks.<br>
> - DT->changeImmediateDominator(LoopExitBlock, TCCheckBlock);<br>
> + DT->changeImmediateDominator(LoopExitBlock, TCCheckBlock);<br>
> <br>
> ReplaceInstWithInst(<br>
> TCCheckBlock->getTerminator(),<br>
> @@ -3192,11 +3188,7 @@ void InnerLoopVectorizer::emitSCEVChecks(Loop *L, BasicBlock *Bypass) {<br>
> // Update dominator only if this is first RT check.<br>
> if (LoopBypassBlocks.empty()) {<br>
> DT->changeImmediateDominator(Bypass, SCEVCheckBlock);<br>
> - if (!Cost->requiresScalarEpilogue())<br>
> - // If there is an epilogue which must run, there's no edge from the<br>
> - // middle block to exit blocks and thus no need to update the immediate<br>
> - // dominator of the exit blocks.<br>
> - DT->changeImmediateDominator(LoopExitBlock, SCEVCheckBlock);<br>
> + DT->changeImmediateDominator(LoopExitBlock, SCEVCheckBlock);<br>
> }<br>
> <br>
> ReplaceInstWithInst(<br>
> @@ -3252,11 +3244,7 @@ void InnerLoopVectorizer::emitMemRuntimeChecks(Loop *L, BasicBlock *Bypass) {<br>
> // Update dominator only if this is first RT check.<br>
> if (LoopBypassBlocks.empty()) {<br>
> DT->changeImmediateDominator(Bypass, MemCheckBlock);<br>
> - if (!Cost->requiresScalarEpilogue())<br>
> - // If there is an epilogue which must run, there's no edge from the<br>
> - // middle block to exit blocks and thus no need to update the immediate<br>
> - // dominator of the exit blocks.<br>
> - DT->changeImmediateDominator(LoopExitBlock, MemCheckBlock);<br>
> + DT->changeImmediateDominator(LoopExitBlock, MemCheckBlock);<br>
> }<br>
> <br>
> Instruction *FirstCheckInst;<br>
> @@ -3381,10 +3369,9 @@ Value *InnerLoopVectorizer::emitTransformedIndex(<br>
> Loop *InnerLoopVectorizer::createVectorLoopSkeleton(StringRef Prefix) {<br>
> LoopScalarBody = OrigLoop->getHeader();<br>
> LoopVectorPreHeader = OrigLoop->getLoopPreheader();<br>
> + LoopExitBlock = OrigLoop->getUniqueExitBlock();<br>
> + assert(LoopExitBlock && "Must have an exit block");<br>
> assert(LoopVectorPreHeader && "Invalid loop structure");<br>
> - LoopExitBlock = OrigLoop->getUniqueExitBlock(); // may be nullptr<br>
> - assert((LoopExitBlock || Cost->requiresScalarEpilogue()) &&<br>
> - "multiple exit loop without required epilogue?");<br>
> <br>
> LoopMiddleBlock =<br>
> SplitBlock(LoopVectorPreHeader, LoopVectorPreHeader->getTerminator(), DT,<br>
> @@ -3393,20 +3380,12 @@ Loop *InnerLoopVectorizer::createVectorLoopSkeleton(StringRef Prefix) {<br>
> SplitBlock(LoopMiddleBlock, LoopMiddleBlock->getTerminator(), DT, LI,<br>
> nullptr, Twine(Prefix) + "<a href="http://scalar.ph" rel="noreferrer" target="_blank">scalar.ph</a>");<br>
> <br>
> + // Set up branch from middle block to the exit and scalar preheader blocks.<br>
> + // completeLoopSkeleton will update the condition to use an iteration check,<br>
> + // if required to decide whether to execute the remainder.<br>
> + BranchInst *BrInst =<br>
> + BranchInst::Create(LoopExitBlock, LoopScalarPreHeader, Builder.getTrue());<br>
> auto *ScalarLatchTerm = OrigLoop->getLoopLatch()->getTerminator();<br>
> -<br>
> - // Set up the middle block terminator. Two cases:<br>
> - // 1) If we know that we must execute the scalar epilogue, emit an<br>
> - // unconditional branch.<br>
> - // 2) Otherwise, we must have a single unique exit block (due to how we<br>
> - // implement the multiple exit case). In this case, set up a conditonal<br>
> - // branch from the middle block to the loop scalar preheader, and the<br>
> - // exit block. completeLoopSkeleton will update the condition to use an<br>
> - // iteration check, if required to decide whether to execute the remainder.<br>
> - BranchInst *BrInst = Cost->requiresScalarEpilogue() ?<br>
> - BranchInst::Create(LoopScalarPreHeader) :<br>
> - BranchInst::Create(LoopExitBlock, LoopScalarPreHeader,<br>
> - Builder.getTrue());<br>
> BrInst->setDebugLoc(ScalarLatchTerm->getDebugLoc());<br>
> ReplaceInstWithInst(LoopMiddleBlock->getTerminator(), BrInst);<br>
> <br>
> @@ -3418,11 +3397,7 @@ Loop *InnerLoopVectorizer::createVectorLoopSkeleton(StringRef Prefix) {<br>
> nullptr, nullptr, Twine(Prefix) + "vector.body");<br>
> <br>
> // Update dominator for loop exit.<br>
> - if (!Cost->requiresScalarEpilogue())<br>
> - // If there is an epilogue which must run, there's no edge from the<br>
> - // middle block to exit blocks and thus no need to update the immediate<br>
> - // dominator of the exit blocks.<br>
> - DT->changeImmediateDominator(LoopExitBlock, LoopMiddleBlock);<br>
> + DT->changeImmediateDominator(LoopExitBlock, LoopMiddleBlock);<br>
> <br>
> // Create and register the new vector loop.<br>
> Loop *Lp = LI->AllocateLoop();<br>
> @@ -3519,14 +3494,10 @@ BasicBlock *InnerLoopVectorizer::completeLoopSkeleton(Loop *L,<br>
> auto *ScalarLatchTerm = OrigLoop->getLoopLatch()->getTerminator();<br>
> <br>
> // Add a check in the middle block to see if we have completed<br>
> - // all of the iterations in the first vector loop. Three cases:<br>
> - // 1) If we require a scalar epilogue, there is no conditional branch as<br>
> - // we unconditionally branch to the scalar preheader. Do nothing.<br>
> - // 2) If (N - N%VF) == N, then we *don't* need to run the remainder.<br>
> - // Thus if tail is to be folded, we know we don't need to run the<br>
> - // remainder and we can use the previous value for the condition (true).<br>
> - // 3) Otherwise, construct a runtime check.<br>
> - if (!Cost->requiresScalarEpilogue() && !Cost->foldTailByMasking()) {<br>
> + // all of the iterations in the first vector loop.<br>
> + // If (N - N%VF) == N, then we *don't* need to run the remainder.<br>
> + // If tail is to be folded, we know we don't need to run the remainder.<br>
> + if (!Cost->foldTailByMasking()) {<br>
> Instruction *CmpN = CmpInst::Create(Instruction::ICmp, CmpInst::ICMP_EQ,<br>
> Count, VectorTripCount, "cmp.n",<br>
> LoopMiddleBlock->getTerminator());<br>
> @@ -3590,17 +3561,17 @@ BasicBlock *InnerLoopVectorizer::createVectorizedLoopSkeleton() {<br>
> | [ ]_| <-- vector loop.<br>
> | |<br>
> | v<br>
> - \ -[ ] <--- middle-block.<br>
> - \/ |<br>
> - /\ v<br>
> - | ->[ ] <--- new preheader.<br>
> + | -[ ] <--- middle-block.<br>
> + | / |<br>
> + | / v<br>
> + -|- >[ ] <--- new preheader.<br>
> | |<br>
> - (opt) v <-- edge from middle to exit iff epilogue is not required.<br>
> + | v<br>
> | [ ] \<br>
> - | [ ]_| <-- old scalar loop to handle remainder (scalar epilogue).<br>
> + | [ ]_| <-- old scalar loop to handle remainder.<br>
> \ |<br>
> \ v<br>
> - >[ ] <-- exit block(s).<br>
> + >[ ] <-- exit block.<br>
> ...<br>
> */<br>
> <br>
> @@ -4021,18 +3992,13 @@ void InnerLoopVectorizer::fixVectorizedLoop() {<br>
> // Forget the original basic block.<br>
> PSE.getSE()->forgetLoop(OrigLoop);<br>
> <br>
> - // If we inserted an edge from the middle block to the unique exit block,<br>
> - // update uses outside the loop (phis) to account for the newly inserted<br>
> - // edge.<br>
> - if (!Cost->requiresScalarEpilogue()) {<br>
> - // Fix-up external users of the induction variables.<br>
> - for (auto &Entry : Legal->getInductionVars())<br>
> - fixupIVUsers(Entry.first, Entry.second,<br>
> - getOrCreateVectorTripCount(LI->getLoopFor(LoopVectorBody)),<br>
> - IVEndValues[Entry.first], LoopMiddleBlock);<br>
> + // Fix-up external users of the induction variables.<br>
> + for (auto &Entry : Legal->getInductionVars())<br>
> + fixupIVUsers(Entry.first, Entry.second,<br>
> + getOrCreateVectorTripCount(LI->getLoopFor(LoopVectorBody)),<br>
> + IVEndValues[Entry.first], LoopMiddleBlock);<br>
> <br>
> - fixLCSSAPHIs();<br>
> - }<br>
> + fixLCSSAPHIs();<br>
> for (Instruction *PI : PredicatedInstructions)<br>
> sinkScalarOperands(&*PI);<br>
> <br>
> @@ -4250,13 +4216,12 @@ void InnerLoopVectorizer::fixFirstOrderRecurrence(PHINode *Phi) {<br>
> // recurrence in the exit block, and then add an edge for the middle block.<br>
> // Note that LCSSA does not imply single entry when the original scalar loop<br>
> // had multiple exiting edges (as we always run the last iteration in the<br>
> - // scalar epilogue); in that case, there is no edge from middle to exit and<br>
> - // and thus no phis which needed updated.<br>
> - if (!Cost->requiresScalarEpilogue())<br>
> - for (PHINode &LCSSAPhi : LoopExitBlock->phis())<br>
> - if (any_of(LCSSAPhi.incoming_values(),<br>
> - [Phi](Value *V) { return V == Phi; }))<br>
> - LCSSAPhi.addIncoming(ExtractForPhiUsedOutsideLoop, LoopMiddleBlock);<br>
> + // scalar epilogue); in that case, the exiting path through middle will be<br>
> + // dynamically dead and the value picked for the phi doesn't matter.<br>
> + for (PHINode &LCSSAPhi : LoopExitBlock->phis())<br>
> + if (any_of(LCSSAPhi.incoming_values(),<br>
> + [Phi](Value *V) { return V == Phi; }))<br>
> + LCSSAPhi.addIncoming(ExtractForPhiUsedOutsideLoop, LoopMiddleBlock);<br>
> }<br>
> <br>
> void InnerLoopVectorizer::fixReduction(PHINode *Phi) {<br>
> @@ -4421,11 +4386,10 @@ void InnerLoopVectorizer::fixReduction(PHINode *Phi) {<br>
> // We know that the loop is in LCSSA form. We need to update the PHI nodes<br>
> // in the exit blocks. See comment on analogous loop in<br>
> // fixFirstOrderRecurrence for a more complete explaination of the logic.<br>
> - if (!Cost->requiresScalarEpilogue())<br>
> - for (PHINode &LCSSAPhi : LoopExitBlock->phis())<br>
> - if (any_of(LCSSAPhi.incoming_values(),<br>
> - [LoopExitInst](Value *V) { return V == LoopExitInst; }))<br>
> - LCSSAPhi.addIncoming(ReducedPartRdx, LoopMiddleBlock);<br>
> + for (PHINode &LCSSAPhi : LoopExitBlock->phis())<br>
> + if (any_of(LCSSAPhi.incoming_values(),<br>
> + [LoopExitInst](Value *V) { return V == LoopExitInst; }))<br>
> + LCSSAPhi.addIncoming(ReducedPartRdx, LoopMiddleBlock);<br>
> <br>
> // Fix the scalar loop reduction variable with the incoming reduction sum<br>
> // from the vector body and from the backedge value.<br>
> @@ -8074,11 +8038,7 @@ BasicBlock *EpilogueVectorizerMainLoop::emitMinimumIterationCountCheck(<br>
> <br>
> // Update dominator for Bypass & LoopExit.<br>
> DT->changeImmediateDominator(Bypass, TCCheckBlock);<br>
> - if (!Cost->requiresScalarEpilogue())<br>
> - // For loops with multiple exits, there's no edge from the middle block<br>
> - // to exit blocks (as the epilogue must run) and thus no need to update<br>
> - // the immediate dominator of the exit blocks.<br>
> - DT->changeImmediateDominator(LoopExitBlock, TCCheckBlock);<br>
> + DT->changeImmediateDominator(LoopExitBlock, TCCheckBlock);<br>
> <br>
> LoopBypassBlocks.push_back(TCCheckBlock);<br>
> <br>
> @@ -8142,12 +8102,7 @@ EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton() {<br>
> <br>
> DT->changeImmediateDominator(LoopScalarPreHeader,<br>
> EPI.EpilogueIterationCountCheck);<br>
> - if (!Cost->requiresScalarEpilogue())<br>
> - // If there is an epilogue which must run, there's no edge from the<br>
> - // middle block to exit blocks and thus no need to update the immediate<br>
> - // dominator of the exit blocks.<br>
> - DT->changeImmediateDominator(LoopExitBlock,<br>
> - EPI.EpilogueIterationCountCheck);<br>
> + DT->changeImmediateDominator(LoopExitBlock, EPI.EpilogueIterationCountCheck);<br>
> <br>
> // Keep track of bypass blocks, as they feed start values to the induction<br>
> // phis in the scalar loop preheader.<br>
><br>
> diff --git a/llvm/test/Transforms/LoopVectorize/first-order-recurrence-complex.ll b/llvm/test/Transforms/LoopVectorize/first-order-recurrence-complex.ll<br>
> index ec280bf5d5e4..7d4a3c5c9935 100644<br>
> --- a/llvm/test/Transforms/LoopVectorize/first-order-recurrence-complex.ll<br>
> +++ b/llvm/test/Transforms/LoopVectorize/first-order-recurrence-complex.ll<br>
> @@ -471,9 +471,10 @@ define i16 @multiple_exit(i16* %p, i32 %n) {<br>
> ; CHECK-NEXT: [[TMP15:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]<br>
> ; CHECK-NEXT: br i1 [[TMP15]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], [[LOOP6:!llvm.loop !.*]]<br>
> ; CHECK: middle.block:<br>
> +; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i32 [[TMP2]], [[N_VEC]]<br>
> ; CHECK-NEXT: [[VECTOR_RECUR_EXTRACT:%.*]] = extractelement <4 x i16> [[WIDE_LOAD]], i32 3<br>
> ; CHECK-NEXT: [[VECTOR_RECUR_EXTRACT_FOR_PHI:%.*]] = extractelement <4 x i16> [[WIDE_LOAD]], i32 2<br>
> -; CHECK-NEXT: br label [[SCALAR_PH]]<br>
> +; CHECK-NEXT: br i1 [[CMP_N]], label [[IF_END:%.*]], label [[SCALAR_PH]]<br>
> ; CHECK: <a href="http://scalar.ph" rel="noreferrer" target="_blank">scalar.ph</a>:<br>
> ; CHECK-NEXT: [[SCALAR_RECUR_INIT:%.*]] = phi i16 [ 0, [[ENTRY:%.*]] ], [ [[VECTOR_RECUR_EXTRACT]], [[MIDDLE_BLOCK]] ]<br>
> ; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i32 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY]] ]<br>
> @@ -485,14 +486,14 @@ define i16 @multiple_exit(i16* %p, i32 %n) {<br>
> ; CHECK-NEXT: [[B:%.*]] = getelementptr inbounds i16, i16* [[P]], i64 [[IPROM]]<br>
> ; CHECK-NEXT: [[REC_NEXT]] = load i16, i16* [[B]], align 2<br>
> ; CHECK-NEXT: [[CMP:%.*]] = icmp slt i32 [[I]], [[N]]<br>
> -; CHECK-NEXT: br i1 [[CMP]], label [[FOR_BODY]], label [[IF_END:%.*]]<br>
> +; CHECK-NEXT: br i1 [[CMP]], label [[FOR_BODY]], label [[IF_END]]<br>
> ; CHECK: for.body:<br>
> ; CHECK-NEXT: store i16 [[SCALAR_RECUR]], i16* [[B]], align 4<br>
> ; CHECK-NEXT: [[INC]] = add nsw i32 [[I]], 1<br>
> ; CHECK-NEXT: [[CMP2:%.*]] = icmp slt i32 [[I]], 2096<br>
> ; CHECK-NEXT: br i1 [[CMP2]], label [[FOR_COND]], label [[IF_END]], [[LOOP7:!llvm.loop !.*]]<br>
> ; CHECK: if.end:<br>
> -; CHECK-NEXT: [[REC_LCSSA:%.*]] = phi i16 [ [[SCALAR_RECUR]], [[FOR_BODY]] ], [ [[SCALAR_RECUR]], [[FOR_COND]] ]<br>
> +; CHECK-NEXT: [[REC_LCSSA:%.*]] = phi i16 [ [[SCALAR_RECUR]], [[FOR_BODY]] ], [ [[SCALAR_RECUR]], [[FOR_COND]] ], [ [[VECTOR_RECUR_EXTRACT_FOR_PHI]], [[MIDDLE_BLOCK]] ]<br>
> ; CHECK-NEXT: ret i16 [[REC_LCSSA]]<br>
> ;<br>
> entry:<br>
> @@ -557,9 +558,10 @@ define i16 @multiple_exit2(i16* %p, i32 %n) {<br>
> ; CHECK-NEXT: [[TMP15:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]<br>
> ; CHECK-NEXT: br i1 [[TMP15]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], [[LOOP8:!llvm.loop !.*]]<br>
> ; CHECK: middle.block:<br>
> +; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i32 [[TMP2]], [[N_VEC]]<br>
> ; CHECK-NEXT: [[VECTOR_RECUR_EXTRACT:%.*]] = extractelement <4 x i16> [[WIDE_LOAD]], i32 3<br>
> ; CHECK-NEXT: [[VECTOR_RECUR_EXTRACT_FOR_PHI:%.*]] = extractelement <4 x i16> [[WIDE_LOAD]], i32 2<br>
> -; CHECK-NEXT: br label [[SCALAR_PH]]<br>
> +; CHECK-NEXT: br i1 [[CMP_N]], label [[IF_END:%.*]], label [[SCALAR_PH]]<br>
> ; CHECK: <a href="http://scalar.ph" rel="noreferrer" target="_blank">scalar.ph</a>:<br>
> ; CHECK-NEXT: [[SCALAR_RECUR_INIT:%.*]] = phi i16 [ 0, [[ENTRY:%.*]] ], [ [[VECTOR_RECUR_EXTRACT]], [[MIDDLE_BLOCK]] ]<br>
> ; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i32 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY]] ]<br>
> @@ -571,14 +573,14 @@ define i16 @multiple_exit2(i16* %p, i32 %n) {<br>
> ; CHECK-NEXT: [[B:%.*]] = getelementptr inbounds i16, i16* [[P]], i64 [[IPROM]]<br>
> ; CHECK-NEXT: [[REC_NEXT]] = load i16, i16* [[B]], align 2<br>
> ; CHECK-NEXT: [[CMP:%.*]] = icmp slt i32 [[I]], [[N]]<br>
> -; CHECK-NEXT: br i1 [[CMP]], label [[FOR_BODY]], label [[IF_END:%.*]]<br>
> +; CHECK-NEXT: br i1 [[CMP]], label [[FOR_BODY]], label [[IF_END]]<br>
> ; CHECK: for.body:<br>
> ; CHECK-NEXT: store i16 [[SCALAR_RECUR]], i16* [[B]], align 4<br>
> ; CHECK-NEXT: [[INC]] = add nsw i32 [[I]], 1<br>
> ; CHECK-NEXT: [[CMP2:%.*]] = icmp slt i32 [[I]], 2096<br>
> ; CHECK-NEXT: br i1 [[CMP2]], label [[FOR_COND]], label [[IF_END]], [[LOOP9:!llvm.loop !.*]]<br>
> ; CHECK: if.end:<br>
> -; CHECK-NEXT: [[REC_LCSSA:%.*]] = phi i16 [ [[SCALAR_RECUR]], [[FOR_COND]] ], [ 10, [[FOR_BODY]] ]<br>
> +; CHECK-NEXT: [[REC_LCSSA:%.*]] = phi i16 [ [[SCALAR_RECUR]], [[FOR_COND]] ], [ 10, [[FOR_BODY]] ], [ [[VECTOR_RECUR_EXTRACT_FOR_PHI]], [[MIDDLE_BLOCK]] ]<br>
> ; CHECK-NEXT: ret i16 [[REC_LCSSA]]<br>
> ;<br>
> entry:<br>
><br>
> diff --git a/llvm/test/Transforms/LoopVectorize/interleaved-accesses.ll b/llvm/test/Transforms/LoopVectorize/interleaved-accesses.ll<br>
> index f0ba677348ab..0d4bdf0ecac3 100644<br>
> --- a/llvm/test/Transforms/LoopVectorize/interleaved-accesses.ll<br>
> +++ b/llvm/test/Transforms/LoopVectorize/interleaved-accesses.ll<br>
> @@ -447,7 +447,7 @@ define void @even_load_static_tc(i32* noalias nocapture readonly %A, i32* noalia<br>
> ; CHECK-NEXT: [[TMP6:%.*]] = icmp eq i64 [[INDEX_NEXT]], 508<br>
> ; CHECK-NEXT: br i1 [[TMP6]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], [[LOOP12:!llvm.loop !.*]]<br>
> ; CHECK: middle.block:<br>
> -; CHECK-NEXT: br label [[SCALAR_PH]]<br>
> +; CHECK-NEXT: br i1 false, label [[FOR_COND_CLEANUP:%.*]], label [[SCALAR_PH]]<br>
> ; CHECK: <a href="http://scalar.ph" rel="noreferrer" target="_blank">scalar.ph</a>:<br>
> ; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ 1016, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]<br>
> ; CHECK-NEXT: br label [[FOR_BODY:%.*]]<br>
> @@ -463,7 +463,7 @@ define void @even_load_static_tc(i32* noalias nocapture readonly %A, i32* noalia<br>
> ; CHECK-NEXT: store i32 [[MUL]], i32* [[ARRAYIDX2]], align 4<br>
> ; CHECK-NEXT: [[INDVARS_IV_NEXT]] = add nuw nsw i64 [[INDVARS_IV]], 2<br>
> ; CHECK-NEXT: [[CMP:%.*]] = icmp ult i64 [[INDVARS_IV]], 1022<br>
> -; CHECK-NEXT: br i1 [[CMP]], label [[FOR_BODY]], label [[FOR_COND_CLEANUP:%.*]], [[LOOP13:!llvm.loop !.*]]<br>
> +; CHECK-NEXT: br i1 [[CMP]], label [[FOR_BODY]], label [[FOR_COND_CLEANUP]], [[LOOP13:!llvm.loop !.*]]<br>
> ;<br>
> entry:<br>
> br label %for.body<br>
> @@ -528,7 +528,7 @@ define void @even_load_dynamic_tc(i32* noalias nocapture readonly %A, i32* noali<br>
> ; CHECK-NEXT: [[TMP12:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]<br>
> ; CHECK-NEXT: br i1 [[TMP12]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], [[LOOP14:!llvm.loop !.*]]<br>
> ; CHECK: middle.block:<br>
> -; CHECK-NEXT: br label [[SCALAR_PH]]<br>
> +; CHECK-NEXT: br i1 false, label [[FOR_COND_CLEANUP:%.*]], label [[SCALAR_PH]]<br>
> ; CHECK: <a href="http://scalar.ph" rel="noreferrer" target="_blank">scalar.ph</a>:<br>
> ; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ [[IND_END]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]<br>
> ; CHECK-NEXT: br label [[FOR_BODY:%.*]]<br>
> @@ -544,7 +544,7 @@ define void @even_load_dynamic_tc(i32* noalias nocapture readonly %A, i32* noali<br>
> ; CHECK-NEXT: store i32 [[MUL]], i32* [[ARRAYIDX2]], align 4<br>
> ; CHECK-NEXT: [[INDVARS_IV_NEXT]] = add nuw nsw i64 [[INDVARS_IV]], 2<br>
> ; CHECK-NEXT: [[CMP:%.*]] = icmp ult i64 [[INDVARS_IV_NEXT]], [[N]]<br>
> -; CHECK-NEXT: br i1 [[CMP]], label [[FOR_BODY]], label [[FOR_COND_CLEANUP:%.*]], [[LOOP15:!llvm.loop !.*]]<br>
> +; CHECK-NEXT: br i1 [[CMP]], label [[FOR_BODY]], label [[FOR_COND_CLEANUP]], [[LOOP15:!llvm.loop !.*]]<br>
> ;<br>
> entry:<br>
> br label %for.body<br>
> @@ -973,7 +973,7 @@ define void @PR27626_0(%pair.i32 *%p, i32 %z, i64 %n) {<br>
> ; CHECK-NEXT: [[TMP19:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]<br>
> ; CHECK-NEXT: br i1 [[TMP19]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], [[LOOP24:!llvm.loop !.*]]<br>
> ; CHECK: middle.block:<br>
> -; CHECK-NEXT: br label [[SCALAR_PH]]<br>
> +; CHECK-NEXT: br i1 false, label [[FOR_END:%.*]], label [[SCALAR_PH]]<br>
> ; CHECK: <a href="http://scalar.ph" rel="noreferrer" target="_blank">scalar.ph</a>:<br>
> ; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]<br>
> ; CHECK-NEXT: br label [[FOR_BODY:%.*]]<br>
> @@ -985,7 +985,7 @@ define void @PR27626_0(%pair.i32 *%p, i32 %z, i64 %n) {<br>
> ; CHECK-NEXT: store i32 [[Z]], i32* [[P_I_Y]], align 4<br>
> ; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 1<br>
> ; CHECK-NEXT: [[COND:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]<br>
> -; CHECK-NEXT: br i1 [[COND]], label [[FOR_BODY]], label [[FOR_END:%.*]], [[LOOP25:!llvm.loop !.*]]<br>
> +; CHECK-NEXT: br i1 [[COND]], label [[FOR_BODY]], label [[FOR_END]], [[LOOP25:!llvm.loop !.*]]<br>
> ; CHECK: for.end:<br>
> ; CHECK-NEXT: ret void<br>
> ;<br>
> @@ -1066,7 +1066,7 @@ define i32 @PR27626_1(%pair.i32 *%p, i64 %n) {<br>
> ; CHECK-NEXT: [[RDX_SHUF3:%.*]] = shufflevector <4 x i32> [[BIN_RDX]], <4 x i32> poison, <4 x i32> <i32 1, i32 undef, i32 undef, i32 undef><br>
> ; CHECK-NEXT: [[BIN_RDX4:%.*]] = add <4 x i32> [[BIN_RDX]], [[RDX_SHUF3]]<br>
> ; CHECK-NEXT: [[TMP19:%.*]] = extractelement <4 x i32> [[BIN_RDX4]], i32 0<br>
> -; CHECK-NEXT: br label [[SCALAR_PH]]<br>
> +; CHECK-NEXT: br i1 false, label [[FOR_END:%.*]], label [[SCALAR_PH]]<br>
> ; CHECK: <a href="http://scalar.ph" rel="noreferrer" target="_blank">scalar.ph</a>:<br>
> ; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]<br>
> ; CHECK-NEXT: [[BC_MERGE_RDX:%.*]] = phi i32 [ [[TMP19]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY]] ]<br>
> @@ -1081,9 +1081,10 @@ define i32 @PR27626_1(%pair.i32 *%p, i64 %n) {<br>
> ; CHECK-NEXT: [[TMP21]] = add nsw i32 [[TMP20]], [[S]]<br>
> ; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 1<br>
> ; CHECK-NEXT: [[COND:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]<br>
> -; CHECK-NEXT: br i1 [[COND]], label [[FOR_BODY]], label [[FOR_END:%.*]], [[LOOP27:!llvm.loop !.*]]<br>
> +; CHECK-NEXT: br i1 [[COND]], label [[FOR_BODY]], label [[FOR_END]], [[LOOP27:!llvm.loop !.*]]<br>
> ; CHECK: for.end:<br>
> -; CHECK-NEXT: ret i32 [[TMP21]]<br>
> +; CHECK-NEXT: [[TMP22:%.*]] = phi i32 [ [[TMP21]], [[FOR_BODY]] ], [ [[TMP19]], [[MIDDLE_BLOCK]] ]<br>
> +; CHECK-NEXT: ret i32 [[TMP22]]<br>
> ;<br>
> entry:<br>
> br label %for.body<br>
> @@ -1162,7 +1163,7 @@ define void @PR27626_2(%pair.i32 *%p, i64 %n, i32 %z) {<br>
> ; CHECK-NEXT: [[TMP20:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]<br>
> ; CHECK-NEXT: br i1 [[TMP20]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], [[LOOP28:!llvm.loop !.*]]<br>
> ; CHECK: middle.block:<br>
> -; CHECK-NEXT: br label [[SCALAR_PH]]<br>
> +; CHECK-NEXT: br i1 false, label [[FOR_END:%.*]], label [[SCALAR_PH]]<br>
> ; CHECK: <a href="http://scalar.ph" rel="noreferrer" target="_blank">scalar.ph</a>:<br>
> ; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]<br>
> ; CHECK-NEXT: br label [[FOR_BODY:%.*]]<br>
> @@ -1176,7 +1177,7 @@ define void @PR27626_2(%pair.i32 *%p, i64 %n, i32 %z) {<br>
> ; CHECK-NEXT: store i32 [[TMP21]], i32* [[P_I_Y]], align 4<br>
> ; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 1<br>
> ; CHECK-NEXT: [[COND:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]<br>
> -; CHECK-NEXT: br i1 [[COND]], label [[FOR_BODY]], label [[FOR_END:%.*]], [[LOOP29:!llvm.loop !.*]]<br>
> +; CHECK-NEXT: br i1 [[COND]], label [[FOR_BODY]], label [[FOR_END]], [[LOOP29:!llvm.loop !.*]]<br>
> ; CHECK: for.end:<br>
> ; CHECK-NEXT: ret void<br>
> ;<br>
> @@ -1263,7 +1264,7 @@ define i32 @PR27626_3(%pair.i32 *%p, i64 %n, i32 %z) {<br>
> ; CHECK-NEXT: [[RDX_SHUF3:%.*]] = shufflevector <4 x i32> [[BIN_RDX]], <4 x i32> poison, <4 x i32> <i32 1, i32 undef, i32 undef, i32 undef><br>
> ; CHECK-NEXT: [[BIN_RDX4:%.*]] = add <4 x i32> [[BIN_RDX]], [[RDX_SHUF3]]<br>
> ; CHECK-NEXT: [[TMP22:%.*]] = extractelement <4 x i32> [[BIN_RDX4]], i32 0<br>
> -; CHECK-NEXT: br label [[SCALAR_PH]]<br>
> +; CHECK-NEXT: br i1 false, label [[FOR_END:%.*]], label [[SCALAR_PH]]<br>
> ; CHECK: <a href="http://scalar.ph" rel="noreferrer" target="_blank">scalar.ph</a>:<br>
> ; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]<br>
> ; CHECK-NEXT: [[BC_MERGE_RDX:%.*]] = phi i32 [ [[TMP22]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY]] ]<br>
> @@ -1281,9 +1282,10 @@ define i32 @PR27626_3(%pair.i32 *%p, i64 %n, i32 %z) {<br>
> ; CHECK-NEXT: [[TMP25]] = add nsw i32 [[TMP24]], [[S]]<br>
> ; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 1<br>
> ; CHECK-NEXT: [[COND:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]<br>
> -; CHECK-NEXT: br i1 [[COND]], label [[FOR_BODY]], label [[FOR_END:%.*]], [[LOOP31:!llvm.loop !.*]]<br>
> +; CHECK-NEXT: br i1 [[COND]], label [[FOR_BODY]], label [[FOR_END]], [[LOOP31:!llvm.loop !.*]]<br>
> ; CHECK: for.end:<br>
> -; CHECK-NEXT: ret i32 [[TMP25]]<br>
> +; CHECK-NEXT: [[TMP26:%.*]] = phi i32 [ [[TMP25]], [[FOR_BODY]] ], [ [[TMP22]], [[MIDDLE_BLOCK]] ]<br>
> +; CHECK-NEXT: ret i32 [[TMP26]]<br>
> ;<br>
> entry:<br>
> br label %for.body<br>
><br>
> diff --git a/llvm/test/Transforms/LoopVectorize/loop-form.ll b/llvm/test/Transforms/LoopVectorize/loop-form.ll<br>
> index f32002fae2b6..91780789088b 100644<br>
> --- a/llvm/test/Transforms/LoopVectorize/loop-form.ll<br>
> +++ b/llvm/test/Transforms/LoopVectorize/loop-form.ll<br>
> @@ -146,14 +146,15 @@ define void @early_exit(i16* %p, i32 %n) {<br>
> ; CHECK-NEXT: [[TMP10:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]<br>
> ; CHECK-NEXT: br i1 [[TMP10]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], [[LOOP4:!llvm.loop !.*]]<br>
> ; CHECK: middle.block:<br>
> -; CHECK-NEXT: br label [[SCALAR_PH]]<br>
> +; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i32 [[TMP1]], [[N_VEC]]<br>
> +; CHECK-NEXT: br i1 [[CMP_N]], label [[IF_END:%.*]], label [[SCALAR_PH]]<br>
> ; CHECK: <a href="http://scalar.ph" rel="noreferrer" target="_blank">scalar.ph</a>:<br>
> ; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i32 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]<br>
> ; CHECK-NEXT: br label [[FOR_COND:%.*]]<br>
> ; CHECK: for.cond:<br>
> ; CHECK-NEXT: [[I:%.*]] = phi i32 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[INC:%.*]], [[FOR_BODY:%.*]] ]<br>
> ; CHECK-NEXT: [[CMP:%.*]] = icmp slt i32 [[I]], [[N]]<br>
> -; CHECK-NEXT: br i1 [[CMP]], label [[FOR_BODY]], label [[IF_END:%.*]]<br>
> +; CHECK-NEXT: br i1 [[CMP]], label [[FOR_BODY]], label [[IF_END]]<br>
> ; CHECK: for.body:<br>
> ; CHECK-NEXT: [[IPROM:%.*]] = sext i32 [[I]] to i64<br>
> ; CHECK-NEXT: [[B:%.*]] = getelementptr inbounds i16, i16* [[P]], i64 [[IPROM]]<br>
> @@ -285,14 +286,15 @@ define void @multiple_unique_exit(i16* %p, i32 %n) {<br>
> ; CHECK-NEXT: [[TMP11:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]<br>
> ; CHECK-NEXT: br i1 [[TMP11]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], [[LOOP6:!llvm.loop !.*]]<br>
> ; CHECK: middle.block:<br>
> -; CHECK-NEXT: br label [[SCALAR_PH]]<br>
> +; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i32 [[TMP2]], [[N_VEC]]<br>
> +; CHECK-NEXT: br i1 [[CMP_N]], label [[IF_END:%.*]], label [[SCALAR_PH]]<br>
> ; CHECK: <a href="http://scalar.ph" rel="noreferrer" target="_blank">scalar.ph</a>:<br>
> ; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i32 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]<br>
> ; CHECK-NEXT: br label [[FOR_COND:%.*]]<br>
> ; CHECK: for.cond:<br>
> ; CHECK-NEXT: [[I:%.*]] = phi i32 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[INC:%.*]], [[FOR_BODY:%.*]] ]<br>
> ; CHECK-NEXT: [[CMP:%.*]] = icmp slt i32 [[I]], [[N]]<br>
> -; CHECK-NEXT: br i1 [[CMP]], label [[FOR_BODY]], label [[IF_END:%.*]]<br>
> +; CHECK-NEXT: br i1 [[CMP]], label [[FOR_BODY]], label [[IF_END]]<br>
> ; CHECK: for.body:<br>
> ; CHECK-NEXT: [[IPROM:%.*]] = sext i32 [[I]] to i64<br>
> ; CHECK-NEXT: [[B:%.*]] = getelementptr inbounds i16, i16* [[P]], i64 [[IPROM]]<br>
> @@ -372,14 +374,17 @@ define i32 @multiple_unique_exit2(i16* %p, i32 %n) {<br>
> ; CHECK-NEXT: [[TMP11:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]<br>
> ; CHECK-NEXT: br i1 [[TMP11]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], [[LOOP8:!llvm.loop !.*]]<br>
> ; CHECK: middle.block:<br>
> -; CHECK-NEXT: br label [[SCALAR_PH]]<br>
> +; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i32 [[TMP2]], [[N_VEC]]<br>
> +; CHECK-NEXT: [[IND_ESCAPE:%.*]] = sub i32 [[N_VEC]], 1<br>
> +; CHECK-NEXT: [[IND_ESCAPE1:%.*]] = sub i32 [[N_VEC]], 1<br>
> +; CHECK-NEXT: br i1 [[CMP_N]], label [[IF_END:%.*]], label [[SCALAR_PH]]<br>
> ; CHECK: <a href="http://scalar.ph" rel="noreferrer" target="_blank">scalar.ph</a>:<br>
> ; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i32 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]<br>
> ; CHECK-NEXT: br label [[FOR_COND:%.*]]<br>
> ; CHECK: for.cond:<br>
> ; CHECK-NEXT: [[I:%.*]] = phi i32 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[INC:%.*]], [[FOR_BODY:%.*]] ]<br>
> ; CHECK-NEXT: [[CMP:%.*]] = icmp slt i32 [[I]], [[N]]<br>
> -; CHECK-NEXT: br i1 [[CMP]], label [[FOR_BODY]], label [[IF_END:%.*]]<br>
> +; CHECK-NEXT: br i1 [[CMP]], label [[FOR_BODY]], label [[IF_END]]<br>
> ; CHECK: for.body:<br>
> ; CHECK-NEXT: [[IPROM:%.*]] = sext i32 [[I]] to i64<br>
> ; CHECK-NEXT: [[B:%.*]] = getelementptr inbounds i16, i16* [[P]], i64 [[IPROM]]<br>
> @@ -388,7 +393,7 @@ define i32 @multiple_unique_exit2(i16* %p, i32 %n) {<br>
> ; CHECK-NEXT: [[CMP2:%.*]] = icmp slt i32 [[I]], 2096<br>
> ; CHECK-NEXT: br i1 [[CMP2]], label [[FOR_COND]], label [[IF_END]], [[LOOP9:!llvm.loop !.*]]<br>
> ; CHECK: if.end:<br>
> -; CHECK-NEXT: [[I_LCSSA:%.*]] = phi i32 [ [[I]], [[FOR_BODY]] ], [ [[I]], [[FOR_COND]] ]<br>
> +; CHECK-NEXT: [[I_LCSSA:%.*]] = phi i32 [ [[I]], [[FOR_BODY]] ], [ [[I]], [[FOR_COND]] ], [ [[IND_ESCAPE1]], [[MIDDLE_BLOCK]] ]<br>
> ; CHECK-NEXT: ret i32 [[I_LCSSA]]<br>
> ;<br>
> ; TAILFOLD-LABEL: @multiple_unique_exit2(<br>
> @@ -461,14 +466,15 @@ define i32 @multiple_unique_exit3(i16* %p, i32 %n) {<br>
> ; CHECK-NEXT: [[TMP11:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]<br>
> ; CHECK-NEXT: br i1 [[TMP11]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], [[LOOP10:!llvm.loop !.*]]<br>
> ; CHECK: middle.block:<br>
> -; CHECK-NEXT: br label [[SCALAR_PH]]<br>
> +; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i32 [[TMP2]], [[N_VEC]]<br>
> +; CHECK-NEXT: br i1 [[CMP_N]], label [[IF_END:%.*]], label [[SCALAR_PH]]<br>
> ; CHECK: <a href="http://scalar.ph" rel="noreferrer" target="_blank">scalar.ph</a>:<br>
> ; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i32 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]<br>
> ; CHECK-NEXT: br label [[FOR_COND:%.*]]<br>
> ; CHECK: for.cond:<br>
> ; CHECK-NEXT: [[I:%.*]] = phi i32 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[INC:%.*]], [[FOR_BODY:%.*]] ]<br>
> ; CHECK-NEXT: [[CMP:%.*]] = icmp slt i32 [[I]], [[N]]<br>
> -; CHECK-NEXT: br i1 [[CMP]], label [[FOR_BODY]], label [[IF_END:%.*]]<br>
> +; CHECK-NEXT: br i1 [[CMP]], label [[FOR_BODY]], label [[IF_END]]<br>
> ; CHECK: for.body:<br>
> ; CHECK-NEXT: [[IPROM:%.*]] = sext i32 [[I]] to i64<br>
> ; CHECK-NEXT: [[B:%.*]] = getelementptr inbounds i16, i16* [[P]], i64 [[IPROM]]<br>
> @@ -477,7 +483,7 @@ define i32 @multiple_unique_exit3(i16* %p, i32 %n) {<br>
> ; CHECK-NEXT: [[CMP2:%.*]] = icmp slt i32 [[I]], 2096<br>
> ; CHECK-NEXT: br i1 [[CMP2]], label [[FOR_COND]], label [[IF_END]], [[LOOP11:!llvm.loop !.*]]<br>
> ; CHECK: if.end:<br>
> -; CHECK-NEXT: [[EXIT:%.*]] = phi i32 [ 0, [[FOR_COND]] ], [ 1, [[FOR_BODY]] ]<br>
> +; CHECK-NEXT: [[EXIT:%.*]] = phi i32 [ 0, [[FOR_COND]] ], [ 1, [[FOR_BODY]] ], [ 0, [[MIDDLE_BLOCK]] ]<br>
> ; CHECK-NEXT: ret i32 [[EXIT]]<br>
> ;<br>
> ; TAILFOLD-LABEL: @multiple_unique_exit3(<br>
> @@ -994,7 +1000,8 @@ define void @scalar_predication(float* %addr) {<br>
> ; CHECK-NEXT: [[TMP10:%.*]] = icmp eq i64 [[INDEX_NEXT]], 200<br>
> ; CHECK-NEXT: br i1 [[TMP10]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], [[LOOP12:!llvm.loop !.*]]<br>
> ; CHECK: middle.block:<br>
> -; CHECK-NEXT: br label [[SCALAR_PH]]<br>
> +; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i64 201, 200<br>
> +; CHECK-NEXT: br i1 [[CMP_N]], label [[EXIT:%.*]], label [[SCALAR_PH]]<br>
> ; CHECK: <a href="http://scalar.ph" rel="noreferrer" target="_blank">scalar.ph</a>:<br>
> ; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ 200, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]<br>
> ; CHECK-NEXT: br label [[LOOP_HEADER:%.*]]<br>
> @@ -1002,7 +1009,7 @@ define void @scalar_predication(float* %addr) {<br>
> ; CHECK-NEXT: [[IV:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[IV_NEXT:%.*]], [[LOOP_LATCH:%.*]] ]<br>
> ; CHECK-NEXT: [[GEP:%.*]] = getelementptr float, float* [[ADDR]], i64 [[IV]]<br>
> ; CHECK-NEXT: [[EXITCOND_NOT:%.*]] = icmp eq i64 [[IV]], 200<br>
> -; CHECK-NEXT: br i1 [[EXITCOND_NOT]], label [[EXIT:%.*]], label [[LOOP_BODY:%.*]]<br>
> +; CHECK-NEXT: br i1 [[EXITCOND_NOT]], label [[EXIT]], label [[LOOP_BODY:%.*]]<br>
> ; CHECK: loop.body:<br>
> ; CHECK-NEXT: [[TMP11:%.*]] = load float, float* [[GEP]], align 4<br>
> ; CHECK-NEXT: [[PRED:%.*]] = fcmp oeq float [[TMP11]], 0.000000e+00<br>
> @@ -1088,7 +1095,8 @@ define i32 @me_reduction(i32* %addr) {<br>
> ; CHECK-NEXT: [[RDX_SHUF:%.*]] = shufflevector <2 x i32> [[TMP5]], <2 x i32> poison, <2 x i32> <i32 1, i32 undef><br>
> ; CHECK-NEXT: [[BIN_RDX:%.*]] = add <2 x i32> [[TMP5]], [[RDX_SHUF]]<br>
> ; CHECK-NEXT: [[TMP7:%.*]] = extractelement <2 x i32> [[BIN_RDX]], i32 0<br>
> -; CHECK-NEXT: br label [[SCALAR_PH]]<br>
> +; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i64 201, 200<br>
> +; CHECK-NEXT: br i1 [[CMP_N]], label [[EXIT:%.*]], label [[SCALAR_PH]]<br>
> ; CHECK: <a href="http://scalar.ph" rel="noreferrer" target="_blank">scalar.ph</a>:<br>
> ; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ 200, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]<br>
> ; CHECK-NEXT: [[BC_MERGE_RDX:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[TMP7]], [[MIDDLE_BLOCK]] ]<br>
> @@ -1098,7 +1106,7 @@ define i32 @me_reduction(i32* %addr) {<br>
> ; CHECK-NEXT: [[ACCUM:%.*]] = phi i32 [ [[BC_MERGE_RDX]], [[SCALAR_PH]] ], [ [[ACCUM_NEXT:%.*]], [[LOOP_LATCH]] ]<br>
> ; CHECK-NEXT: [[GEP:%.*]] = getelementptr i32, i32* [[ADDR]], i64 [[IV]]<br>
> ; CHECK-NEXT: [[EXITCOND_NOT:%.*]] = icmp eq i64 [[IV]], 200<br>
> -; CHECK-NEXT: br i1 [[EXITCOND_NOT]], label [[EXIT:%.*]], label [[LOOP_LATCH]]<br>
> +; CHECK-NEXT: br i1 [[EXITCOND_NOT]], label [[EXIT]], label [[LOOP_LATCH]]<br>
> ; CHECK: loop.latch:<br>
> ; CHECK-NEXT: [[TMP8:%.*]] = load i32, i32* [[GEP]], align 4<br>
> ; CHECK-NEXT: [[ACCUM_NEXT]] = add i32 [[ACCUM]], [[TMP8]]<br>
> @@ -1106,7 +1114,7 @@ define i32 @me_reduction(i32* %addr) {<br>
> ; CHECK-NEXT: [[EXITCOND2_NOT:%.*]] = icmp eq i64 [[IV]], 400<br>
> ; CHECK-NEXT: br i1 [[EXITCOND2_NOT]], label [[EXIT]], label [[LOOP_HEADER]], [[LOOP15:!llvm.loop !.*]]<br>
> ; CHECK: exit:<br>
> -; CHECK-NEXT: [[LCSSA:%.*]] = phi i32 [ 0, [[LOOP_HEADER]] ], [ [[ACCUM_NEXT]], [[LOOP_LATCH]] ]<br>
> +; CHECK-NEXT: [[LCSSA:%.*]] = phi i32 [ 0, [[LOOP_HEADER]] ], [ [[ACCUM_NEXT]], [[LOOP_LATCH]] ], [ [[TMP7]], [[MIDDLE_BLOCK]] ]<br>
> ; CHECK-NEXT: ret i32 [[LCSSA]]<br>
> ;<br>
> ; TAILFOLD-LABEL: @me_reduction(<br>
><br>
><br>
> <br>
> _______________________________________________<br>
> llvm-commits mailing list<br>
> <a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a><br>
> <a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><span><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><img src="https://lh4.googleusercontent.com/tI3sB3YqaUCwaqwKhQNZHIw38fP0nxtBm4sEYjL-7r65mm0qsyQR9Gs9808J0p8qR8Dq3toLlvf4Amm9BKqyOIsTTkEqP4YX8K4pI2hZCVcaTxSvCEVIq8FZwIxhnAud7bg=s1600" width="100px;" height="58px;" style="color: rgb(102, 102, 102); font-family: Arial; font-size: 13.3333px; white-space: pre-wrap; line-height: 1.38; border: none; background-color: transparent;"><br></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13.3333px;font-family:Arial;color:rgb(102,102,102);vertical-align:baseline;white-space:pre-wrap;background-color:transparent">Google Germany GmbH</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="color:rgba(0,0,0,0.75);font-family:sans-serif;font-size:13.2px;line-height:normal">Erika</span><span style="color:rgba(0,0,0,0.75);font-family:sans-serif;font-size:13.2px;line-height:normal">-</span><span style="color:rgba(0,0,0,0.75);font-family:sans-serif;font-size:13.2px;line-height:normal">Mann</span><span style="color:rgba(0,0,0,0.75);font-family:sans-serif;font-size:13.2px;line-height:normal">-</span><span style="color:rgba(0,0,0,0.75);font-family:sans-serif;font-size:13.2px;line-height:normal">Straße</span><span style="color:rgba(0,0,0,0.75);font-family:sans-serif;font-size:13.2px;line-height:normal"> 33</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="color:rgba(0,0,0,0.75);font-family:sans-serif;font-size:13.2px;line-height:normal">80636 München</span><br></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="background-color:transparent;color:rgb(183,183,183);font-family:Arial;font-size:10.6667px;white-space:pre-wrap">Geschäftsführer: Paul Manicle, Halimah DeLaine Prado</span><br></p></span><span><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10.6667px;font-family:Arial;color:rgb(183,183,183);vertical-align:baseline;white-space:pre-wrap;background-color:transparent">Registergericht und -nummer: Hamburg, HRB 86891</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10.6667px;font-family:Arial;color:rgb(183,183,183);vertical-align:baseline;white-space:pre-wrap;background-color:transparent">Sitz der Gesellschaft: Hamburg</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10.6667px;font-family:Arial;color:rgb(183,183,183);vertical-align:baseline;white-space:pre-wrap;background-color:transparent">Diese E-Mail ist vertraulich. Wenn Sie nicht der richtige Adressat sind, leiten Sie diese bitte nicht weiter, informieren Sie den Absender und löschen Sie die E-Mail und alle Anhänge. Vielen Dank.</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10.6667px;font-family:Arial;color:rgb(183,183,183);vertical-align:baseline;white-space:pre-wrap;background-color:transparent"> </span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10.6667px;font-family:Arial;color:rgb(183,183,183);vertical-align:baseline;white-space:pre-wrap;background-color:transparent">This e-mail is confidential. If you are not the right addressee please do not forward it, please inform the sender, and please erase this e-mail including any attachments. Thanks.<br></span></p></span></div></div></div></div></div></div></div></div>