<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <p>I tried another cycle to see if this had been resolved, but am
      still seeing build bot failures, nearly exclusively on PPC.  (And
      one arm self host bot.)  <br>
    </p>
    <p>@PPC Bot Owner - I need help reducing a test case for the failure
      seen here:
      <a class="moz-txt-link-freetext" href="https://lab.llvm.org/buildbot#builders/100/builds/5762">https://lab.llvm.org/buildbot#builders/100/builds/5762</a>.  I am
      stuck, and unable to make progress for nearly 3 months now.  I
      would greatly appreciate help.  <br>
    </p>
    <p>Philip<br>
    </p>
    <div class="moz-cite-prefix">On 5/5/21 11:59 PM, Adrian Kuegel
      wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CAHhV=OX3NbAExy4YsQWLH0fC-zXiG_VpToAtB72NhkUwwYekRQ@mail.gmail.com">
      <meta http-equiv="content-type" content="text/html; charset=UTF-8">
      <div dir="ltr">
        <div dir="ltr">Sounds good to me, and thanks for sending the
          heads up :)</div>
        <br>
        <div class="gmail_quote">
          <div dir="ltr" class="gmail_attr">On Wed, May 5, 2021 at 10:01
            PM Philip Reames <<a
              href="mailto:listmail@philipreames.com"
              moz-do-not-send="true">listmail@philipreames.com</a>>
            wrote:<br>
          </div>
          <blockquote class="gmail_quote" style="margin:0px 0px 0px
            0.8ex;border-left:1px solid
            rgb(204,204,204);padding-left:1ex">FYI, I'm going to try
            recommitting this without changes in a day or so.<br>
            <br>
            I never heard back from a PPC bot owner, and I don't have
            enough <br>
            information to really debug anything from the builtbot log. 
            I did run <br>
            across a latent issue which this patch may very well have
            exposed at <br>
            much higher frequency; the previous patch in this series
            (which is much <br>
            more restrictive) does appear to have increased frequency. 
            That was <br>
            worked around in 80e80250.  My educated guess is that same
            issue <br>
            triggered the miscompile seen on the ppc bot, but that is
            more of a <br>
            guess than I'd really prefer.<br>
            <br>
            I'm going to submit this during off hours, and watch the
            bots fairly <br>
            closely after submit.  Hopefully this either cycles clean or
            I get a <br>
            better clue as to what the root issue is.<br>
            <br>
            Philip<br>
            <br>
            On 2/8/21 9:09 PM, Philip Reames via llvm-commits wrote:<br>
            > Ben,<br>
            ><br>
            > Thanks for the clarification.  The log does not make
            the fact this is <br>
            > an execution failure obvious.<br>
            ><br>
            > No, I don't have access to a PPC machine.<br>
            ><br>
            > I am going to need some assistance from the bot owner
            on this. At a <br>
            > minimum, IR for the test in question (before
            optimization, but on <br>
            > target platform) seems like a reasonable ask.<br>
            ><br>
            > I strongly suspect this change is simply exposing
            another latent <br>
            > issue.  Or at least, I've reviewed the change and don't
            see anything <br>
            > likely to cause runtime crashes w/o also tripping
            compiler asserts.<br>
            ><br>
            > Philip<br>
            ><br>
            ><br>
            > On 2/8/21 5:21 AM, Benjamin Kramer wrote:<br>
            >> `execution_time` failures mean that the bot
            succeeded building a test<br>
            >> but it failed when running it. I'm relatively
            certain that this is the<br>
            >> same issue Adrian is seeing -- binaries segfaulting
            early on PPC.<br>
            >><br>
            >> The bot log output isn't helpful at all for
            investigating why this is<br>
            >> happening. Do you happen to have access to a PPC
            machine?<br>
            >><br>
            >> On Fri, Feb 5, 2021 at 6:05 PM Philip Reames via
            llvm-commits<br>
            >> <<a href="mailto:llvm-commits@lists.llvm.org"
              target="_blank" moz-do-not-send="true">llvm-commits@lists.llvm.org</a>>
            wrote:<br>
            >>> Adrian,<br>
            >>><br>
            >>> I'm going to need you to provide a bit more
            information here. The test<br>
            >>> failure in stage1 was fixed at the time you
            reverted this patch.  The<br>
            >>> remaining failure in the bot is very unclear. 
            What is a execution_time<br>
            >>> failure? From the log output, the "failing" run
            finished in 0.5 <br>
            >>> seconds,<br>
            >>> whereas the previous "succeeding" run finished
            in 11 seconds. Without<br>
            >>> further context, I'd say that's no failure.<br>
            >>><br>
            >>> I'll also note that I did not receive email
            from this bot.  I received<br>
            >>> notice from the various other bots and fixed
            the ARM test issue, but<br>
            >>> unless I missed it in with the others, this bot
            is not notifying.<br>
            >>><br>
            >>> In general, I'm a fan of fast reverts, but I
            have to admit, this one<br>
            >>> appears borderline at the moment.<br>
            >>><br>
            >>> Philip<br>
            >>><br>
            >>> On 2/5/21 3:53 AM, Adrian Kuegel via
            llvm-commits wrote:<br>
            >>>> Author: Adrian Kuegel<br>
            >>>> Date: 2021-02-05T12:51:03+01:00<br>
            >>>> New Revision:
            7fe41ac3dff2d44c3d2c31b28554fbe4a86eaa6c<br>
            >>>><br>
            >>>> URL: <br>
            >>>> <a
href="https://github.com/llvm/llvm-project/commit/7fe41ac3dff2d44c3d2c31b28554fbe4a86eaa6c"
              rel="noreferrer" target="_blank" moz-do-not-send="true">https://github.com/llvm/llvm-project/commit/7fe41ac3dff2d44c3d2c31b28554fbe4a86eaa6c</a><br>
            >>>> DIFF: <br>
            >>>> <a
href="https://github.com/llvm/llvm-project/commit/7fe41ac3dff2d44c3d2c31b28554fbe4a86eaa6c.diff"
              rel="noreferrer" target="_blank" moz-do-not-send="true">https://github.com/llvm/llvm-project/commit/7fe41ac3dff2d44c3d2c31b28554fbe4a86eaa6c.diff</a><br>
            >>>><br>
            >>>> LOG: Revert "[LV] Unconditionally branch
            from middle to scalar <br>
            >>>> preheader if the scalar loop must execute"<br>
            >>>><br>
            >>>> This reverts commit
            3e5ce49e5371ce4feadbf97dd5c2b652d9db3d1d.<br>
            >>>><br>
            >>>> Tests started failing on PPC, for example:<br>
            >>>> <a
              href="http://lab.llvm.org:8011/#/builders/105/builds/5569"
              rel="noreferrer" target="_blank" moz-do-not-send="true">http://lab.llvm.org:8011/#/builders/105/builds/5569</a><br>
            >>>><br>
            >>>> Added:<br>
            >>>><br>
            >>>><br>
            >>>> Modified:<br>
            >>>>      
            llvm/lib/Transforms/Utils/LoopVersioning.cpp<br>
            >>>>      
            llvm/lib/Transforms/Vectorize/LoopVectorize.cpp<br>
            >>>>
            llvm/test/Transforms/LoopVectorize/first-order-recurrence-complex.ll<br>
            >>>>
            llvm/test/Transforms/LoopVectorize/interleaved-accesses.ll<br>
            >>>>      
            llvm/test/Transforms/LoopVectorize/loop-form.ll<br>
            >>>><br>
            >>>> Removed:<br>
            >>>><br>
            >>>><br>
            >>>><br>
            >>>>
################################################################################
            <br>
            >>>><br>
            >>>> diff  --git
            a/llvm/lib/Transforms/Utils/LoopVersioning.cpp <br>
            >>>>
            b/llvm/lib/Transforms/Utils/LoopVersioning.cpp<br>
            >>>> index 8a89158788cf..de4fb446fdf2 100644<br>
            >>>> ---
            a/llvm/lib/Transforms/Utils/LoopVersioning.cpp<br>
            >>>> +++
            b/llvm/lib/Transforms/Utils/LoopVersioning.cpp<br>
            >>>> @@ -44,11 +44,11 @@
            LoopVersioning::LoopVersioning(const <br>
            >>>> LoopAccessInfo &LAI,<br>
            >>>>          AliasChecks(Checks.begin(),
            Checks.end()),<br>
            >>>>         
            Preds(LAI.getPSE().getUnionPredicate()), LAI(LAI), LI(LI), <br>
            >>>> DT(DT),<br>
            >>>>          SE(SE) {<br>
            >>>> +  assert(L->getUniqueExitBlock()
            && "No single exit block");<br>
            >>>>    }<br>
            >>>><br>
            >>>>    void LoopVersioning::versionLoop(<br>
            >>>>        const SmallVectorImpl<Instruction
            *> &DefsUsedOutside) {<br>
            >>>> - 
            assert(VersionedLoop->getUniqueExitBlock() && "No
            single exit <br>
            >>>> block");<br>
            >>>>     
            assert(VersionedLoop->isLoopSimplifyForm() &&<br>
            >>>>             "Loop is not in loop-simplify
            form");<br>
            >>>><br>
            >>>><br>
            >>>> diff  --git
            a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp <br>
            >>>>
            b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp<br>
            >>>> index 3277842edbfe..6bce0caeb36f 100644<br>
            >>>> ---
            a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp<br>
            >>>> +++
            b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp<br>
            >>>> @@ -852,7 +852,7 @@ class
            InnerLoopVectorizer {<br>
            >>>>      /// Middle Block between the vector
            and the scalar.<br>
            >>>>      BasicBlock *LoopMiddleBlock;<br>
            >>>><br>
            >>>> -  /// The unique ExitBlock of the scalar
            loop if one exists.  Note <br>
            >>>> that<br>
            >>>> +  /// The (unique) ExitBlock of the scalar
            loop.  Note that<br>
            >>>>      /// there can be multiple exiting
            edges reaching this block.<br>
            >>>>      BasicBlock *LoopExitBlock;<br>
            >>>><br>
            >>>> @@ -3147,13 +3147,9 @@ void <br>
            >>>>
            InnerLoopVectorizer::emitMinimumIterationCountCheck(Loop *L,<br>
            >>>> DT->getNode(Bypass)->getIDom())
            &&<br>
            >>>>             "TC check is expected to
            dominate Bypass");<br>
            >>>><br>
            >>>> -  // Update dominator for Bypass &
            LoopExit (if needed).<br>
            >>>> +  // Update dominator for Bypass &
            LoopExit.<br>
            >>>>     
            DT->changeImmediateDominator(Bypass, TCCheckBlock);<br>
            >>>> -  if (!Cost->requiresScalarEpilogue())<br>
            >>>> -    // If there is an epilogue which must
            run, there's no edge <br>
            >>>> from the<br>
            >>>> -    // middle block to exit blocks  and
            thus no need to update the <br>
            >>>> immediate<br>
            >>>> -    // dominator of the exit blocks.<br>
            >>>> -   
            DT->changeImmediateDominator(LoopExitBlock,
            TCCheckBlock);<br>
            >>>> + 
            DT->changeImmediateDominator(LoopExitBlock,
            TCCheckBlock);<br>
            >>>><br>
            >>>>      ReplaceInstWithInst(<br>
            >>>>          TCCheckBlock->getTerminator(),<br>
            >>>> @@ -3192,11 +3188,7 @@ void <br>
            >>>> InnerLoopVectorizer::emitSCEVChecks(Loop
            *L, BasicBlock *Bypass) {<br>
            >>>>      // Update dominator only if this is
            first RT check.<br>
            >>>>      if (LoopBypassBlocks.empty()) {<br>
            >>>>       
            DT->changeImmediateDominator(Bypass, SCEVCheckBlock);<br>
            >>>> -    if
            (!Cost->requiresScalarEpilogue())<br>
            >>>> -      // If there is an epilogue which
            must run, there's no edge <br>
            >>>> from the<br>
            >>>> -      // middle block to exit blocks  and
            thus no need to update <br>
            >>>> the immediate<br>
            >>>> -      // dominator of the exit blocks.<br>
            >>>> -     
            DT->changeImmediateDominator(LoopExitBlock,
            SCEVCheckBlock);<br>
            >>>> +   
            DT->changeImmediateDominator(LoopExitBlock,
            SCEVCheckBlock);<br>
            >>>>      }<br>
            >>>><br>
            >>>>      ReplaceInstWithInst(<br>
            >>>> @@ -3252,11 +3244,7 @@ void <br>
            >>>>
            InnerLoopVectorizer::emitMemRuntimeChecks(Loop *L,
            BasicBlock <br>
            >>>> *Bypass) {<br>
            >>>>      // Update dominator only if this is
            first RT check.<br>
            >>>>      if (LoopBypassBlocks.empty()) {<br>
            >>>>       
            DT->changeImmediateDominator(Bypass, MemCheckBlock);<br>
            >>>> -    if
            (!Cost->requiresScalarEpilogue())<br>
            >>>> -      // If there is an epilogue which
            must run, there's no edge <br>
            >>>> from the<br>
            >>>> -      // middle block to exit blocks  and
            thus no need to update <br>
            >>>> the immediate<br>
            >>>> -      // dominator of the exit blocks.<br>
            >>>> -     
            DT->changeImmediateDominator(LoopExitBlock,
            MemCheckBlock);<br>
            >>>> +   
            DT->changeImmediateDominator(LoopExitBlock,
            MemCheckBlock);<br>
            >>>>      }<br>
            >>>><br>
            >>>>      Instruction *FirstCheckInst;<br>
            >>>> @@ -3381,10 +3369,9 @@ Value <br>
            >>>> *InnerLoopVectorizer::emitTransformedIndex(<br>
            >>>>    Loop
            *InnerLoopVectorizer::createVectorLoopSkeleton(StringRef <br>
            >>>> Prefix) {<br>
            >>>>      LoopScalarBody =
            OrigLoop->getHeader();<br>
            >>>>      LoopVectorPreHeader =
            OrigLoop->getLoopPreheader();<br>
            >>>> +  LoopExitBlock =
            OrigLoop->getUniqueExitBlock();<br>
            >>>> +  assert(LoopExitBlock && "Must
            have an exit block");<br>
            >>>>      assert(LoopVectorPreHeader &&
            "Invalid loop structure");<br>
            >>>> -  LoopExitBlock =
            OrigLoop->getUniqueExitBlock(); // may be nullptr<br>
            >>>> -  assert((LoopExitBlock ||
            Cost->requiresScalarEpilogue()) &&<br>
            >>>> -         "multiple exit loop without
            required epilogue?");<br>
            >>>><br>
            >>>>      LoopMiddleBlock =<br>
            >>>>          SplitBlock(LoopVectorPreHeader, <br>
            >>>> LoopVectorPreHeader->getTerminator(),
            DT,<br>
            >>>> @@ -3393,20 +3380,12 @@ Loop <br>
            >>>>
            *InnerLoopVectorizer::createVectorLoopSkeleton(StringRef
            Prefix) {<br>
            >>>>          SplitBlock(LoopMiddleBlock, <br>
            >>>> LoopMiddleBlock->getTerminator(), DT,
            LI,<br>
            >>>>                     nullptr, Twine(Prefix)
            + "<a href="http://scalar.ph" rel="noreferrer"
              target="_blank" moz-do-not-send="true">scalar.ph</a>");<br>
            >>>><br>
            >>>> +  // Set up branch from middle block to
            the exit and scalar <br>
            >>>> preheader blocks.<br>
            >>>> +  // completeLoopSkeleton will update the
            condition to use an <br>
            >>>> iteration check,<br>
            >>>> +  // if required to decide whether to
            execute the remainder.<br>
            >>>> +  BranchInst *BrInst =<br>
            >>>> +      BranchInst::Create(LoopExitBlock,
            LoopScalarPreHeader, <br>
            >>>> Builder.getTrue());<br>
            >>>>      auto *ScalarLatchTerm = <br>
            >>>>
            OrigLoop->getLoopLatch()->getTerminator();<br>
            >>>> -<br>
            >>>> -  // Set up the middle block terminator. 
            Two cases:<br>
            >>>> -  // 1) If we know that we must execute
            the scalar epilogue, emit an<br>
            >>>> -  //    unconditional branch.<br>
            >>>> -  // 2) Otherwise, we must have a single
            unique exit block (due to <br>
            >>>> how we<br>
            >>>> -  //    implement the multiple exit
            case).  In this case, set up a <br>
            >>>> conditonal<br>
            >>>> -  //    branch from the middle block to
            the loop scalar preheader, <br>
            >>>> and the<br>
            >>>> -  //    exit block.  completeLoopSkeleton
            will update the <br>
            >>>> condition to use an<br>
            >>>> -  //    iteration check, if required to
            decide whether to execute <br>
            >>>> the remainder.<br>
            >>>> -  BranchInst *BrInst =
            Cost->requiresScalarEpilogue() ?<br>
            >>>> -   
            BranchInst::Create(LoopScalarPreHeader) :<br>
            >>>> -    BranchInst::Create(LoopExitBlock,
            LoopScalarPreHeader,<br>
            >>>> -                       Builder.getTrue());<br>
            >>>>
            BrInst->setDebugLoc(ScalarLatchTerm->getDebugLoc());<br>
            >>>>
            ReplaceInstWithInst(LoopMiddleBlock->getTerminator(),
            BrInst);<br>
            >>>><br>
            >>>> @@ -3418,11 +3397,7 @@ Loop <br>
            >>>>
            *InnerLoopVectorizer::createVectorLoopSkeleton(StringRef
            Prefix) {<br>
            >>>>                     nullptr, nullptr,
            Twine(Prefix) + "vector.body");<br>
            >>>><br>
            >>>>      // Update dominator for loop exit.<br>
            >>>> -  if (!Cost->requiresScalarEpilogue())<br>
            >>>> -    // If there is an epilogue which must
            run, there's no edge <br>
            >>>> from the<br>
            >>>> -    // middle block to exit blocks  and
            thus no need to update the <br>
            >>>> immediate<br>
            >>>> -    // dominator of the exit blocks.<br>
            >>>> -   
            DT->changeImmediateDominator(LoopExitBlock,
            LoopMiddleBlock);<br>
            >>>> + 
            DT->changeImmediateDominator(LoopExitBlock,
            LoopMiddleBlock);<br>
            >>>><br>
            >>>>      // Create and register the new vector
            loop.<br>
            >>>>      Loop *Lp = LI->AllocateLoop();<br>
            >>>> @@ -3519,14 +3494,10 @@ BasicBlock <br>
            >>>>
            *InnerLoopVectorizer::completeLoopSkeleton(Loop *L,<br>
            >>>>      auto *ScalarLatchTerm = <br>
            >>>>
            OrigLoop->getLoopLatch()->getTerminator();<br>
            >>>><br>
            >>>>      // Add a check in the middle block to
            see if we have completed<br>
            >>>> -  // all of the iterations in the first
            vector loop.  Three cases:<br>
            >>>> -  // 1) If we require a scalar epilogue,
            there is no conditional <br>
            >>>> branch as<br>
            >>>> -  //    we unconditionally branch to the
            scalar preheader. Do <br>
            >>>> nothing.<br>
            >>>> -  // 2) If (N - N%VF) == N, then we
            *don't* need to run the <br>
            >>>> remainder.<br>
            >>>> -  //    Thus if tail is to be folded, we
            know we don't need to run <br>
            >>>> the<br>
            >>>> -  //    remainder and we can use the
            previous value for the <br>
            >>>> condition (true).<br>
            >>>> -  // 3) Otherwise, construct a runtime
            check.<br>
            >>>> -  if (!Cost->requiresScalarEpilogue()
            && <br>
            >>>> !Cost->foldTailByMasking()) {<br>
            >>>> +  // all of the iterations in the first
            vector loop.<br>
            >>>> +  // If (N - N%VF) == N, then we *don't*
            need to run the remainder.<br>
            >>>> +  // If tail is to be folded, we know we
            don't need to run the <br>
            >>>> remainder.<br>
            >>>> +  if (!Cost->foldTailByMasking()) {<br>
            >>>>        Instruction *CmpN =
            CmpInst::Create(Instruction::ICmp, <br>
            >>>> CmpInst::ICMP_EQ,<br>
            >>>>                                           
            Count, VectorTripCount, <br>
            >>>> "cmp.n",<br>
            >>>> LoopMiddleBlock->getTerminator());<br>
            >>>> @@ -3590,17 +3561,17 @@ BasicBlock <br>
            >>>>
            *InnerLoopVectorizer::createVectorizedLoopSkeleton() {<br>
            >>>>      |    [  ]_|   <-- vector loop.<br>
            >>>>      |     |<br>
            >>>>      |     v<br>
            >>>> -  \   -[ ]   <--- middle-block.<br>
            >>>> -   \/   |<br>
            >>>> -   /\   v<br>
            >>>> -   | ->[ ]     <--- new preheader.<br>
            >>>> +  |   -[ ]   <--- middle-block.<br>
            >>>> +  |  /  |<br>
            >>>> +  | /   v<br>
            >>>> +  -|- >[ ]     <--- new preheader.<br>
            >>>>       |    |<br>
            >>>> - (opt)  v      <-- edge from middle to
            exit iff epilogue is not <br>
            >>>> required.<br>
            >>>> +   |    v<br>
            >>>>       |   [ ] \<br>
            >>>> -   |   [ ]_|   <-- old scalar loop to
            handle remainder (scalar <br>
            >>>> epilogue).<br>
            >>>> +   |   [ ]_|   <-- old scalar loop to
            handle remainder.<br>
            >>>>        \   |<br>
            >>>>         \  v<br>
            >>>> -      >[ ]     <-- exit block(s).<br>
            >>>> +      >[ ]     <-- exit block.<br>
            >>>>       ...<br>
            >>>>       */<br>
            >>>><br>
            >>>> @@ -4021,18 +3992,13 @@ void <br>
            >>>> InnerLoopVectorizer::fixVectorizedLoop() {<br>
            >>>>      // Forget the original basic block.<br>
            >>>>      PSE.getSE()->forgetLoop(OrigLoop);<br>
            >>>><br>
            >>>> -  // If we inserted an edge from the
            middle block to the unique <br>
            >>>> exit block,<br>
            >>>> -  // update uses outside the loop (phis)
            to account for the newly <br>
            >>>> inserted<br>
            >>>> -  // edge.<br>
            >>>> -  if (!Cost->requiresScalarEpilogue())
            {<br>
            >>>> -    // Fix-up external users of the
            induction variables.<br>
            >>>> -    for (auto &Entry :
            Legal->getInductionVars())<br>
            >>>> -      fixupIVUsers(Entry.first,
            Entry.second,<br>
            >>>> -
            getOrCreateVectorTripCount(LI->getLoopFor(LoopVectorBody)),<br>
            >>>> -                  
            IVEndValues[Entry.first], LoopMiddleBlock);<br>
            >>>> +  // Fix-up external users of the
            induction variables.<br>
            >>>> +  for (auto &Entry :
            Legal->getInductionVars())<br>
            >>>> +    fixupIVUsers(Entry.first,
            Entry.second,<br>
            >>>> +
            getOrCreateVectorTripCount(LI->getLoopFor(LoopVectorBody)),<br>
            >>>> +                 IVEndValues[Entry.first],
            LoopMiddleBlock);<br>
            >>>><br>
            >>>> -    fixLCSSAPHIs();<br>
            >>>> -  }<br>
            >>>> +  fixLCSSAPHIs();<br>
            >>>>      for (Instruction *PI :
            PredicatedInstructions)<br>
            >>>>        sinkScalarOperands(&*PI);<br>
            >>>><br>
            >>>> @@ -4250,13 +4216,12 @@ void <br>
            >>>>
            InnerLoopVectorizer::fixFirstOrderRecurrence(PHINode *Phi) {<br>
            >>>>      // recurrence in the exit block, and
            then add an edge for the <br>
            >>>> middle block.<br>
            >>>>      // Note that LCSSA does not imply
            single entry when the <br>
            >>>> original scalar loop<br>
            >>>>      // had multiple exiting edges (as we
            always run the last <br>
            >>>> iteration in the<br>
            >>>> -  // scalar epilogue); in that case, there
            is no edge from middle <br>
            >>>> to exit and<br>
            >>>> -  // and thus no phis which needed
            updated.<br>
            >>>> -  if (!Cost->requiresScalarEpilogue())<br>
            >>>> -    for (PHINode &LCSSAPhi :
            LoopExitBlock->phis())<br>
            >>>> -      if
            (any_of(LCSSAPhi.incoming_values(),<br>
            >>>> -                 [Phi](Value *V) { return
            V == Phi; }))<br>
            >>>> -       
            LCSSAPhi.addIncoming(ExtractForPhiUsedOutsideLoop, <br>
            >>>> LoopMiddleBlock);<br>
            >>>> +  // scalar epilogue); in that case, the
            exiting path through <br>
            >>>> middle will be<br>
            >>>> +  // dynamically dead and the value picked
            for the phi doesn't <br>
            >>>> matter.<br>
            >>>> +  for (PHINode &LCSSAPhi :
            LoopExitBlock->phis())<br>
            >>>> +    if (any_of(LCSSAPhi.incoming_values(),<br>
            >>>> +               [Phi](Value *V) { return V
            == Phi; }))<br>
            >>>> +     
            LCSSAPhi.addIncoming(ExtractForPhiUsedOutsideLoop, <br>
            >>>> LoopMiddleBlock);<br>
            >>>>    }<br>
            >>>><br>
            >>>>    void
            InnerLoopVectorizer::fixReduction(PHINode *Phi) {<br>
            >>>> @@ -4421,11 +4386,10 @@ void <br>
            >>>> InnerLoopVectorizer::fixReduction(PHINode
            *Phi) {<br>
            >>>>      // We know that the loop is in LCSSA
            form. We need to update <br>
            >>>> the PHI nodes<br>
            >>>>      // in the exit blocks.  See comment on
            analogous loop in<br>
            >>>>      // fixFirstOrderRecurrence for a more
            complete explaination of <br>
            >>>> the logic.<br>
            >>>> -  if (!Cost->requiresScalarEpilogue())<br>
            >>>> -    for (PHINode &LCSSAPhi :
            LoopExitBlock->phis())<br>
            >>>> -      if
            (any_of(LCSSAPhi.incoming_values(),<br>
            >>>> -                 [LoopExitInst](Value *V)
            { return V == <br>
            >>>> LoopExitInst; }))<br>
            >>>> -       
            LCSSAPhi.addIncoming(ReducedPartRdx, LoopMiddleBlock);<br>
            >>>> +  for (PHINode &LCSSAPhi :
            LoopExitBlock->phis())<br>
            >>>> +    if (any_of(LCSSAPhi.incoming_values(),<br>
            >>>> +               [LoopExitInst](Value *V) {
            return V == <br>
            >>>> LoopExitInst; }))<br>
            >>>> +      LCSSAPhi.addIncoming(ReducedPartRdx,
            LoopMiddleBlock);<br>
            >>>><br>
            >>>>      // Fix the scalar loop reduction
            variable with the incoming <br>
            >>>> reduction sum<br>
            >>>>      // from the vector body and from the
            backedge value.<br>
            >>>> @@ -8074,11 +8038,7 @@ BasicBlock <br>
            >>>>
            *EpilogueVectorizerMainLoop::emitMinimumIterationCountCheck(<br>
            >>>><br>
            >>>>        // Update dominator for Bypass &
            LoopExit.<br>
            >>>>       
            DT->changeImmediateDominator(Bypass, TCCheckBlock);<br>
            >>>> -    if
            (!Cost->requiresScalarEpilogue())<br>
            >>>> -      // For loops with multiple exits,
            there's no edge from the <br>
            >>>> middle block<br>
            >>>> -      // to exit blocks (as the epilogue
            must run) and thus no <br>
            >>>> need to update<br>
            >>>> -      // the immediate dominator of the
            exit blocks.<br>
            >>>> -     
            DT->changeImmediateDominator(LoopExitBlock,
            TCCheckBlock);<br>
            >>>> +   
            DT->changeImmediateDominator(LoopExitBlock,
            TCCheckBlock);<br>
            >>>><br>
            >>>>       
            LoopBypassBlocks.push_back(TCCheckBlock);<br>
            >>>><br>
            >>>> @@ -8142,12 +8102,7 @@ <br>
            >>>>
            EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton()
            <br>
            >>>> {<br>
            >>>><br>
            >>>>     
            DT->changeImmediateDominator(LoopScalarPreHeader,<br>
            >>>> EPI.EpilogueIterationCountCheck);<br>
            >>>> -  if (!Cost->requiresScalarEpilogue())<br>
            >>>> -    // If there is an epilogue which must
            run, there's no edge <br>
            >>>> from the<br>
            >>>> -    // middle block to exit blocks  and
            thus no need to update the <br>
            >>>> immediate<br>
            >>>> -    // dominator of the exit blocks.<br>
            >>>> -   
            DT->changeImmediateDominator(LoopExitBlock,<br>
            >>>> - EPI.EpilogueIterationCountCheck);<br>
            >>>> + 
            DT->changeImmediateDominator(LoopExitBlock, <br>
            >>>> EPI.EpilogueIterationCountCheck);<br>
            >>>><br>
            >>>>      // Keep track of bypass blocks, as
            they feed start values to <br>
            >>>> the induction<br>
            >>>>      // phis in the scalar loop preheader.<br>
            >>>><br>
            >>>> diff  --git <br>
            >>>>
            a/llvm/test/Transforms/LoopVectorize/first-order-recurrence-complex.ll
            <br>
            >>>>
            b/llvm/test/Transforms/LoopVectorize/first-order-recurrence-complex.ll<br>
            >>>> index ec280bf5d5e4..7d4a3c5c9935 100644<br>
            >>>> --- <br>
            >>>>
            a/llvm/test/Transforms/LoopVectorize/first-order-recurrence-complex.ll<br>
            >>>> +++ <br>
            >>>>
            b/llvm/test/Transforms/LoopVectorize/first-order-recurrence-complex.ll<br>
            >>>> @@ -471,9 +471,10 @@ define i16
            @multiple_exit(i16* %p, i32 %n) {<br>
            >>>>    ; CHECK-NEXT:    [[TMP15:%.*]] = icmp eq
            i32 [[INDEX_NEXT]], <br>
            >>>> [[N_VEC]]<br>
            >>>>    ; CHECK-NEXT:    br i1 [[TMP15]], label
            [[MIDDLE_BLOCK:%.*]], <br>
            >>>> label [[VECTOR_BODY]], [[LOOP6:!llvm.loop
            !.*]]<br>
            >>>>    ; CHECK:       middle.block:<br>
            >>>> +; CHECK-NEXT:    [[CMP_N:%.*]] = icmp eq
            i32 [[TMP2]], [[N_VEC]]<br>
            >>>>    ; CHECK-NEXT:   
            [[VECTOR_RECUR_EXTRACT:%.*]] = extractelement <br>
            >>>> <4 x i16> [[WIDE_LOAD]], i32 3<br>
            >>>>    ; CHECK-NEXT:   
            [[VECTOR_RECUR_EXTRACT_FOR_PHI:%.*]] = <br>
            >>>> extractelement <4 x i16>
            [[WIDE_LOAD]], i32 2<br>
            >>>> -; CHECK-NEXT:    br label [[SCALAR_PH]]<br>
            >>>> +; CHECK-NEXT:    br i1 [[CMP_N]], label
            [[IF_END:%.*]], label <br>
            >>>> [[SCALAR_PH]]<br>
            >>>>    ; CHECK:       <a
              href="http://scalar.ph" rel="noreferrer" target="_blank"
              moz-do-not-send="true">scalar.ph</a>:<br>
            >>>>    ; CHECK-NEXT:   
            [[SCALAR_RECUR_INIT:%.*]] = phi i16 [ 0, <br>
            >>>> [[ENTRY:%.*]] ], [
            [[VECTOR_RECUR_EXTRACT]], [[MIDDLE_BLOCK]] ]<br>
            >>>>    ; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] =
            phi i32 [ [[N_VEC]], <br>
            >>>> [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY]] ]<br>
            >>>> @@ -485,14 +486,14 @@ define i16
            @multiple_exit(i16* %p, i32 %n) {<br>
            >>>>    ; CHECK-NEXT:    [[B:%.*]] =
            getelementptr inbounds i16, i16* <br>
            >>>> [[P]], i64 [[IPROM]]<br>
            >>>>    ; CHECK-NEXT:    [[REC_NEXT]] = load
            i16, i16* [[B]], align 2<br>
            >>>>    ; CHECK-NEXT:    [[CMP:%.*]] = icmp slt
            i32 [[I]], [[N]]<br>
            >>>> -; CHECK-NEXT:    br i1 [[CMP]], label
            [[FOR_BODY]], label <br>
            >>>> [[IF_END:%.*]]<br>
            >>>> +; CHECK-NEXT:    br i1 [[CMP]], label
            [[FOR_BODY]], label [[IF_END]]<br>
            >>>>    ; CHECK:       for.body:<br>
            >>>>    ; CHECK-NEXT:    store i16
            [[SCALAR_RECUR]], i16* [[B]], align 4<br>
            >>>>    ; CHECK-NEXT:    [[INC]] = add nsw i32
            [[I]], 1<br>
            >>>>    ; CHECK-NEXT:    [[CMP2:%.*]] = icmp slt
            i32 [[I]], 2096<br>
            >>>>    ; CHECK-NEXT:    br i1 [[CMP2]], label
            [[FOR_COND]], label <br>
            >>>> [[IF_END]], [[LOOP7:!llvm.loop !.*]]<br>
            >>>>    ; CHECK:       if.end:<br>
            >>>> -; CHECK-NEXT:    [[REC_LCSSA:%.*]] = phi
            i16 [ [[SCALAR_RECUR]], <br>
            >>>> [[FOR_BODY]] ], [ [[SCALAR_RECUR]],
            [[FOR_COND]] ]<br>
            >>>> +; CHECK-NEXT:    [[REC_LCSSA:%.*]] = phi
            i16 [ [[SCALAR_RECUR]], <br>
            >>>> [[FOR_BODY]] ], [ [[SCALAR_RECUR]],
            [[FOR_COND]] ], [ <br>
            >>>> [[VECTOR_RECUR_EXTRACT_FOR_PHI]],
            [[MIDDLE_BLOCK]] ]<br>
            >>>>    ; CHECK-NEXT:    ret i16 [[REC_LCSSA]]<br>
            >>>>    ;<br>
            >>>>    entry:<br>
            >>>> @@ -557,9 +558,10 @@ define i16
            @multiple_exit2(i16* %p, i32 %n) {<br>
            >>>>    ; CHECK-NEXT:    [[TMP15:%.*]] = icmp eq
            i32 [[INDEX_NEXT]], <br>
            >>>> [[N_VEC]]<br>
            >>>>    ; CHECK-NEXT:    br i1 [[TMP15]], label
            [[MIDDLE_BLOCK:%.*]], <br>
            >>>> label [[VECTOR_BODY]], [[LOOP8:!llvm.loop
            !.*]]<br>
            >>>>    ; CHECK:       middle.block:<br>
            >>>> +; CHECK-NEXT:    [[CMP_N:%.*]] = icmp eq
            i32 [[TMP2]], [[N_VEC]]<br>
            >>>>    ; CHECK-NEXT:   
            [[VECTOR_RECUR_EXTRACT:%.*]] = extractelement <br>
            >>>> <4 x i16> [[WIDE_LOAD]], i32 3<br>
            >>>>    ; CHECK-NEXT:   
            [[VECTOR_RECUR_EXTRACT_FOR_PHI:%.*]] = <br>
            >>>> extractelement <4 x i16>
            [[WIDE_LOAD]], i32 2<br>
            >>>> -; CHECK-NEXT:    br label [[SCALAR_PH]]<br>
            >>>> +; CHECK-NEXT:    br i1 [[CMP_N]], label
            [[IF_END:%.*]], label <br>
            >>>> [[SCALAR_PH]]<br>
            >>>>    ; CHECK:       <a
              href="http://scalar.ph" rel="noreferrer" target="_blank"
              moz-do-not-send="true">scalar.ph</a>:<br>
            >>>>    ; CHECK-NEXT:   
            [[SCALAR_RECUR_INIT:%.*]] = phi i16 [ 0, <br>
            >>>> [[ENTRY:%.*]] ], [
            [[VECTOR_RECUR_EXTRACT]], [[MIDDLE_BLOCK]] ]<br>
            >>>>    ; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] =
            phi i32 [ [[N_VEC]], <br>
            >>>> [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY]] ]<br>
            >>>> @@ -571,14 +573,14 @@ define i16
            @multiple_exit2(i16* %p, i32 %n) {<br>
            >>>>    ; CHECK-NEXT:    [[B:%.*]] =
            getelementptr inbounds i16, i16* <br>
            >>>> [[P]], i64 [[IPROM]]<br>
            >>>>    ; CHECK-NEXT:    [[REC_NEXT]] = load
            i16, i16* [[B]], align 2<br>
            >>>>    ; CHECK-NEXT:    [[CMP:%.*]] = icmp slt
            i32 [[I]], [[N]]<br>
            >>>> -; CHECK-NEXT:    br i1 [[CMP]], label
            [[FOR_BODY]], label <br>
            >>>> [[IF_END:%.*]]<br>
            >>>> +; CHECK-NEXT:    br i1 [[CMP]], label
            [[FOR_BODY]], label [[IF_END]]<br>
            >>>>    ; CHECK:       for.body:<br>
            >>>>    ; CHECK-NEXT:    store i16
            [[SCALAR_RECUR]], i16* [[B]], align 4<br>
            >>>>    ; CHECK-NEXT:    [[INC]] = add nsw i32
            [[I]], 1<br>
            >>>>    ; CHECK-NEXT:    [[CMP2:%.*]] = icmp slt
            i32 [[I]], 2096<br>
            >>>>    ; CHECK-NEXT:    br i1 [[CMP2]], label
            [[FOR_COND]], label <br>
            >>>> [[IF_END]], [[LOOP9:!llvm.loop !.*]]<br>
            >>>>    ; CHECK:       if.end:<br>
            >>>> -; CHECK-NEXT:    [[REC_LCSSA:%.*]] = phi
            i16 [ [[SCALAR_RECUR]], <br>
            >>>> [[FOR_COND]] ], [ 10, [[FOR_BODY]] ]<br>
            >>>> +; CHECK-NEXT:    [[REC_LCSSA:%.*]] = phi
            i16 [ [[SCALAR_RECUR]], <br>
            >>>> [[FOR_COND]] ], [ 10, [[FOR_BODY]] ], [ <br>
            >>>> [[VECTOR_RECUR_EXTRACT_FOR_PHI]],
            [[MIDDLE_BLOCK]] ]<br>
            >>>>    ; CHECK-NEXT:    ret i16 [[REC_LCSSA]]<br>
            >>>>    ;<br>
            >>>>    entry:<br>
            >>>><br>
            >>>> diff  --git <br>
            >>>>
            a/llvm/test/Transforms/LoopVectorize/interleaved-accesses.ll
            <br>
            >>>>
            b/llvm/test/Transforms/LoopVectorize/interleaved-accesses.ll<br>
            >>>> index f0ba677348ab..0d4bdf0ecac3 100644<br>
            >>>> ---
            a/llvm/test/Transforms/LoopVectorize/interleaved-accesses.ll<br>
            >>>> +++
            b/llvm/test/Transforms/LoopVectorize/interleaved-accesses.ll<br>
            >>>> @@ -447,7 +447,7 @@ define void
            @even_load_static_tc(i32* noalias <br>
            >>>> nocapture readonly %A, i32* noalia<br>
            >>>>    ; CHECK-NEXT:    [[TMP6:%.*]] = icmp eq
            i64 [[INDEX_NEXT]], 508<br>
            >>>>    ; CHECK-NEXT:    br i1 [[TMP6]], label
            [[MIDDLE_BLOCK:%.*]], <br>
            >>>> label [[VECTOR_BODY]], [[LOOP12:!llvm.loop
            !.*]]<br>
            >>>>    ; CHECK:       middle.block:<br>
            >>>> -; CHECK-NEXT:    br label [[SCALAR_PH]]<br>
            >>>> +; CHECK-NEXT:    br i1 false, label
            [[FOR_COND_CLEANUP:%.*]], <br>
            >>>> label [[SCALAR_PH]]<br>
            >>>>    ; CHECK:       <a
              href="http://scalar.ph" rel="noreferrer" target="_blank"
              moz-do-not-send="true">scalar.ph</a>:<br>
            >>>>    ; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] =
            phi i64 [ 1016, <br>
            >>>> [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]<br>
            >>>>    ; CHECK-NEXT:    br label
            [[FOR_BODY:%.*]]<br>
            >>>> @@ -463,7 +463,7 @@ define void
            @even_load_static_tc(i32* noalias <br>
            >>>> nocapture readonly %A, i32* noalia<br>
            >>>>    ; CHECK-NEXT:    store i32 [[MUL]], i32*
            [[ARRAYIDX2]], align 4<br>
            >>>>    ; CHECK-NEXT:    [[INDVARS_IV_NEXT]] =
            add nuw nsw i64 <br>
            >>>> [[INDVARS_IV]], 2<br>
            >>>>    ; CHECK-NEXT:    [[CMP:%.*]] = icmp ult
            i64 [[INDVARS_IV]], 1022<br>
            >>>> -; CHECK-NEXT:    br i1 [[CMP]], label
            [[FOR_BODY]], label <br>
            >>>> [[FOR_COND_CLEANUP:%.*]],
            [[LOOP13:!llvm.loop !.*]]<br>
            >>>> +; CHECK-NEXT:    br i1 [[CMP]], label
            [[FOR_BODY]], label <br>
            >>>> [[FOR_COND_CLEANUP]], [[LOOP13:!llvm.loop
            !.*]]<br>
            >>>>    ;<br>
            >>>>    entry:<br>
            >>>>      br label %for.body<br>
            >>>> @@ -528,7 +528,7 @@ define void
            @even_load_dynamic_tc(i32* noalias <br>
            >>>> nocapture readonly %A, i32* noali<br>
            >>>>    ; CHECK-NEXT:    [[TMP12:%.*]] = icmp eq
            i64 [[INDEX_NEXT]], <br>
            >>>> [[N_VEC]]<br>
            >>>>    ; CHECK-NEXT:    br i1 [[TMP12]], label
            [[MIDDLE_BLOCK:%.*]], <br>
            >>>> label [[VECTOR_BODY]], [[LOOP14:!llvm.loop
            !.*]]<br>
            >>>>    ; CHECK:       middle.block:<br>
            >>>> -; CHECK-NEXT:    br label [[SCALAR_PH]]<br>
            >>>> +; CHECK-NEXT:    br i1 false, label
            [[FOR_COND_CLEANUP:%.*]], <br>
            >>>> label [[SCALAR_PH]]<br>
            >>>>    ; CHECK:       <a
              href="http://scalar.ph" rel="noreferrer" target="_blank"
              moz-do-not-send="true">scalar.ph</a>:<br>
            >>>>    ; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] =
            phi i64 [ [[IND_END]], <br>
            >>>> [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]<br>
            >>>>    ; CHECK-NEXT:    br label
            [[FOR_BODY:%.*]]<br>
            >>>> @@ -544,7 +544,7 @@ define void
            @even_load_dynamic_tc(i32* noalias <br>
            >>>> nocapture readonly %A, i32* noali<br>
            >>>>    ; CHECK-NEXT:    store i32 [[MUL]], i32*
            [[ARRAYIDX2]], align 4<br>
            >>>>    ; CHECK-NEXT:    [[INDVARS_IV_NEXT]] =
            add nuw nsw i64 <br>
            >>>> [[INDVARS_IV]], 2<br>
            >>>>    ; CHECK-NEXT:    [[CMP:%.*]] = icmp ult
            i64 [[INDVARS_IV_NEXT]], <br>
            >>>> [[N]]<br>
            >>>> -; CHECK-NEXT:    br i1 [[CMP]], label
            [[FOR_BODY]], label <br>
            >>>> [[FOR_COND_CLEANUP:%.*]],
            [[LOOP15:!llvm.loop !.*]]<br>
            >>>> +; CHECK-NEXT:    br i1 [[CMP]], label
            [[FOR_BODY]], label <br>
            >>>> [[FOR_COND_CLEANUP]], [[LOOP15:!llvm.loop
            !.*]]<br>
            >>>>    ;<br>
            >>>>    entry:<br>
            >>>>      br label %for.body<br>
            >>>> @@ -973,7 +973,7 @@ define void
            @PR27626_0(%pair.i32 *%p, i32 %z, <br>
            >>>> i64 %n) {<br>
            >>>>    ; CHECK-NEXT:    [[TMP19:%.*]] = icmp eq
            i64 [[INDEX_NEXT]], <br>
            >>>> [[N_VEC]]<br>
            >>>>    ; CHECK-NEXT:    br i1 [[TMP19]], label
            [[MIDDLE_BLOCK:%.*]], <br>
            >>>> label [[VECTOR_BODY]], [[LOOP24:!llvm.loop
            !.*]]<br>
            >>>>    ; CHECK:       middle.block:<br>
            >>>> -; CHECK-NEXT:    br label [[SCALAR_PH]]<br>
            >>>> +; CHECK-NEXT:    br i1 false, label
            [[FOR_END:%.*]], label <br>
            >>>> [[SCALAR_PH]]<br>
            >>>>    ; CHECK:       <a
              href="http://scalar.ph" rel="noreferrer" target="_blank"
              moz-do-not-send="true">scalar.ph</a>:<br>
            >>>>    ; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] =
            phi i64 [ [[N_VEC]], <br>
            >>>> [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]<br>
            >>>>    ; CHECK-NEXT:    br label
            [[FOR_BODY:%.*]]<br>
            >>>> @@ -985,7 +985,7 @@ define void
            @PR27626_0(%pair.i32 *%p, i32 %z, <br>
            >>>> i64 %n) {<br>
            >>>>    ; CHECK-NEXT:    store i32 [[Z]], i32*
            [[P_I_Y]], align 4<br>
            >>>>    ; CHECK-NEXT:    [[I_NEXT]] = add nuw
            nsw i64 [[I]], 1<br>
            >>>>    ; CHECK-NEXT:    [[COND:%.*]] = icmp slt
            i64 [[I_NEXT]], [[N]]<br>
            >>>> -; CHECK-NEXT:    br i1 [[COND]], label
            [[FOR_BODY]], label <br>
            >>>> [[FOR_END:%.*]], [[LOOP25:!llvm.loop !.*]]<br>
            >>>> +; CHECK-NEXT:    br i1 [[COND]], label
            [[FOR_BODY]], label <br>
            >>>> [[FOR_END]], [[LOOP25:!llvm.loop !.*]]<br>
            >>>>    ; CHECK:       for.end:<br>
            >>>>    ; CHECK-NEXT:    ret void<br>
            >>>>    ;<br>
            >>>> @@ -1066,7 +1066,7 @@ define i32
            @PR27626_1(%pair.i32 *%p, i64 %n) {<br>
            >>>>    ; CHECK-NEXT:    [[RDX_SHUF3:%.*]] =
            shufflevector <4 x i32> <br>
            >>>> [[BIN_RDX]], <4 x i32> poison, <4
            x i32> <i32 1, i32 undef, i32 <br>
            >>>> undef, i32 undef><br>
            >>>>    ; CHECK-NEXT:    [[BIN_RDX4:%.*]] = add
            <4 x i32> [[BIN_RDX]], <br>
            >>>> [[RDX_SHUF3]]<br>
            >>>>    ; CHECK-NEXT:    [[TMP19:%.*]] =
            extractelement <4 x i32> <br>
            >>>> [[BIN_RDX4]], i32 0<br>
            >>>> -; CHECK-NEXT:    br label [[SCALAR_PH]]<br>
            >>>> +; CHECK-NEXT:    br i1 false, label
            [[FOR_END:%.*]], label <br>
            >>>> [[SCALAR_PH]]<br>
            >>>>    ; CHECK:       <a
              href="http://scalar.ph" rel="noreferrer" target="_blank"
              moz-do-not-send="true">scalar.ph</a>:<br>
            >>>>    ; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] =
            phi i64 [ [[N_VEC]], <br>
            >>>> [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]<br>
            >>>>    ; CHECK-NEXT:    [[BC_MERGE_RDX:%.*]] =
            phi i32 [ [[TMP19]], <br>
            >>>> [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY]] ]<br>
            >>>> @@ -1081,9 +1081,10 @@ define i32
            @PR27626_1(%pair.i32 *%p, i64 %n) {<br>
            >>>>    ; CHECK-NEXT:    [[TMP21]] = add nsw i32
            [[TMP20]], [[S]]<br>
            >>>>    ; CHECK-NEXT:    [[I_NEXT]] = add nuw
            nsw i64 [[I]], 1<br>
            >>>>    ; CHECK-NEXT:    [[COND:%.*]] = icmp slt
            i64 [[I_NEXT]], [[N]]<br>
            >>>> -; CHECK-NEXT:    br i1 [[COND]], label
            [[FOR_BODY]], label <br>
            >>>> [[FOR_END:%.*]], [[LOOP27:!llvm.loop !.*]]<br>
            >>>> +; CHECK-NEXT:    br i1 [[COND]], label
            [[FOR_BODY]], label <br>
            >>>> [[FOR_END]], [[LOOP27:!llvm.loop !.*]]<br>
            >>>>    ; CHECK:       for.end:<br>
            >>>> -; CHECK-NEXT:    ret i32 [[TMP21]]<br>
            >>>> +; CHECK-NEXT:    [[TMP22:%.*]] = phi i32 [
            [[TMP21]], [[FOR_BODY]] <br>
            >>>> ], [ [[TMP19]], [[MIDDLE_BLOCK]] ]<br>
            >>>> +; CHECK-NEXT:    ret i32 [[TMP22]]<br>
            >>>>    ;<br>
            >>>>    entry:<br>
            >>>>      br label %for.body<br>
            >>>> @@ -1162,7 +1163,7 @@ define void
            @PR27626_2(%pair.i32 *%p, i64 %n, <br>
            >>>> i32 %z) {<br>
            >>>>    ; CHECK-NEXT:    [[TMP20:%.*]] = icmp eq
            i64 [[INDEX_NEXT]], <br>
            >>>> [[N_VEC]]<br>
            >>>>    ; CHECK-NEXT:    br i1 [[TMP20]], label
            [[MIDDLE_BLOCK:%.*]], <br>
            >>>> label [[VECTOR_BODY]], [[LOOP28:!llvm.loop
            !.*]]<br>
            >>>>    ; CHECK:       middle.block:<br>
            >>>> -; CHECK-NEXT:    br label [[SCALAR_PH]]<br>
            >>>> +; CHECK-NEXT:    br i1 false, label
            [[FOR_END:%.*]], label <br>
            >>>> [[SCALAR_PH]]<br>
            >>>>    ; CHECK:       <a
              href="http://scalar.ph" rel="noreferrer" target="_blank"
              moz-do-not-send="true">scalar.ph</a>:<br>
            >>>>    ; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] =
            phi i64 [ [[N_VEC]], <br>
            >>>> [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]<br>
            >>>>    ; CHECK-NEXT:    br label
            [[FOR_BODY:%.*]]<br>
            >>>> @@ -1176,7 +1177,7 @@ define void
            @PR27626_2(%pair.i32 *%p, i64 %n, <br>
            >>>> i32 %z) {<br>
            >>>>    ; CHECK-NEXT:    store i32 [[TMP21]],
            i32* [[P_I_Y]], align 4<br>
            >>>>    ; CHECK-NEXT:    [[I_NEXT]] = add nuw
            nsw i64 [[I]], 1<br>
            >>>>    ; CHECK-NEXT:    [[COND:%.*]] = icmp slt
            i64 [[I_NEXT]], [[N]]<br>
            >>>> -; CHECK-NEXT:    br i1 [[COND]], label
            [[FOR_BODY]], label <br>
            >>>> [[FOR_END:%.*]], [[LOOP29:!llvm.loop !.*]]<br>
            >>>> +; CHECK-NEXT:    br i1 [[COND]], label
            [[FOR_BODY]], label <br>
            >>>> [[FOR_END]], [[LOOP29:!llvm.loop !.*]]<br>
            >>>>    ; CHECK:       for.end:<br>
            >>>>    ; CHECK-NEXT:    ret void<br>
            >>>>    ;<br>
            >>>> @@ -1263,7 +1264,7 @@ define i32
            @PR27626_3(%pair.i32 *%p, i64 %n, <br>
            >>>> i32 %z) {<br>
            >>>>    ; CHECK-NEXT:    [[RDX_SHUF3:%.*]] =
            shufflevector <4 x i32> <br>
            >>>> [[BIN_RDX]], <4 x i32> poison, <4
            x i32> <i32 1, i32 undef, i32 <br>
            >>>> undef, i32 undef><br>
            >>>>    ; CHECK-NEXT:    [[BIN_RDX4:%.*]] = add
            <4 x i32> [[BIN_RDX]], <br>
            >>>> [[RDX_SHUF3]]<br>
            >>>>    ; CHECK-NEXT:    [[TMP22:%.*]] =
            extractelement <4 x i32> <br>
            >>>> [[BIN_RDX4]], i32 0<br>
            >>>> -; CHECK-NEXT:    br label [[SCALAR_PH]]<br>
            >>>> +; CHECK-NEXT:    br i1 false, label
            [[FOR_END:%.*]], label <br>
            >>>> [[SCALAR_PH]]<br>
            >>>>    ; CHECK:       <a
              href="http://scalar.ph" rel="noreferrer" target="_blank"
              moz-do-not-send="true">scalar.ph</a>:<br>
            >>>>    ; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] =
            phi i64 [ [[N_VEC]], <br>
            >>>> [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]<br>
            >>>>    ; CHECK-NEXT:    [[BC_MERGE_RDX:%.*]] =
            phi i32 [ [[TMP22]], <br>
            >>>> [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY]] ]<br>
            >>>> @@ -1281,9 +1282,10 @@ define i32
            @PR27626_3(%pair.i32 *%p, i64 %n, <br>
            >>>> i32 %z) {<br>
            >>>>    ; CHECK-NEXT:    [[TMP25]] = add nsw i32
            [[TMP24]], [[S]]<br>
            >>>>    ; CHECK-NEXT:    [[I_NEXT]] = add nuw
            nsw i64 [[I]], 1<br>
            >>>>    ; CHECK-NEXT:    [[COND:%.*]] = icmp slt
            i64 [[I_NEXT]], [[N]]<br>
            >>>> -; CHECK-NEXT:    br i1 [[COND]], label
            [[FOR_BODY]], label <br>
            >>>> [[FOR_END:%.*]], [[LOOP31:!llvm.loop !.*]]<br>
            >>>> +; CHECK-NEXT:    br i1 [[COND]], label
            [[FOR_BODY]], label <br>
            >>>> [[FOR_END]], [[LOOP31:!llvm.loop !.*]]<br>
            >>>>    ; CHECK:       for.end:<br>
            >>>> -; CHECK-NEXT:    ret i32 [[TMP25]]<br>
            >>>> +; CHECK-NEXT:    [[TMP26:%.*]] = phi i32 [
            [[TMP25]], [[FOR_BODY]] <br>
            >>>> ], [ [[TMP22]], [[MIDDLE_BLOCK]] ]<br>
            >>>> +; CHECK-NEXT:    ret i32 [[TMP26]]<br>
            >>>>    ;<br>
            >>>>    entry:<br>
            >>>>      br label %for.body<br>
            >>>><br>
            >>>> diff  --git
            a/llvm/test/Transforms/LoopVectorize/loop-form.ll <br>
            >>>>
            b/llvm/test/Transforms/LoopVectorize/loop-form.ll<br>
            >>>> index f32002fae2b6..91780789088b 100644<br>
            >>>> ---
            a/llvm/test/Transforms/LoopVectorize/loop-form.ll<br>
            >>>> +++
            b/llvm/test/Transforms/LoopVectorize/loop-form.ll<br>
            >>>> @@ -146,14 +146,15 @@ define void
            @early_exit(i16* %p, i32 %n) {<br>
            >>>>    ; CHECK-NEXT:    [[TMP10:%.*]] = icmp eq
            i32 [[INDEX_NEXT]], <br>
            >>>> [[N_VEC]]<br>
            >>>>    ; CHECK-NEXT:    br i1 [[TMP10]], label
            [[MIDDLE_BLOCK:%.*]], <br>
            >>>> label [[VECTOR_BODY]], [[LOOP4:!llvm.loop
            !.*]]<br>
            >>>>    ; CHECK:       middle.block:<br>
            >>>> -; CHECK-NEXT:    br label [[SCALAR_PH]]<br>
            >>>> +; CHECK-NEXT:    [[CMP_N:%.*]] = icmp eq
            i32 [[TMP1]], [[N_VEC]]<br>
            >>>> +; CHECK-NEXT:    br i1 [[CMP_N]], label
            [[IF_END:%.*]], label <br>
            >>>> [[SCALAR_PH]]<br>
            >>>>    ; CHECK:       <a
              href="http://scalar.ph" rel="noreferrer" target="_blank"
              moz-do-not-send="true">scalar.ph</a>:<br>
            >>>>    ; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] =
            phi i32 [ [[N_VEC]], <br>
            >>>> [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]<br>
            >>>>    ; CHECK-NEXT:    br label
            [[FOR_COND:%.*]]<br>
            >>>>    ; CHECK:       for.cond:<br>
            >>>>    ; CHECK-NEXT:    [[I:%.*]] = phi i32 [
            [[BC_RESUME_VAL]], <br>
            >>>> [[SCALAR_PH]] ], [ [[INC:%.*]],
            [[FOR_BODY:%.*]] ]<br>
            >>>>    ; CHECK-NEXT:    [[CMP:%.*]] = icmp slt
            i32 [[I]], [[N]]<br>
            >>>> -; CHECK-NEXT:    br i1 [[CMP]], label
            [[FOR_BODY]], label <br>
            >>>> [[IF_END:%.*]]<br>
            >>>> +; CHECK-NEXT:    br i1 [[CMP]], label
            [[FOR_BODY]], label [[IF_END]]<br>
            >>>>    ; CHECK:       for.body:<br>
            >>>>    ; CHECK-NEXT:    [[IPROM:%.*]] = sext
            i32 [[I]] to i64<br>
            >>>>    ; CHECK-NEXT:    [[B:%.*]] =
            getelementptr inbounds i16, i16* <br>
            >>>> [[P]], i64 [[IPROM]]<br>
            >>>> @@ -285,14 +286,15 @@ define void
            @multiple_unique_exit(i16* %p, <br>
            >>>> i32 %n) {<br>
            >>>>    ; CHECK-NEXT:    [[TMP11:%.*]] = icmp eq
            i32 [[INDEX_NEXT]], <br>
            >>>> [[N_VEC]]<br>
            >>>>    ; CHECK-NEXT:    br i1 [[TMP11]], label
            [[MIDDLE_BLOCK:%.*]], <br>
            >>>> label [[VECTOR_BODY]], [[LOOP6:!llvm.loop
            !.*]]<br>
            >>>>    ; CHECK:       middle.block:<br>
            >>>> -; CHECK-NEXT:    br label [[SCALAR_PH]]<br>
            >>>> +; CHECK-NEXT:    [[CMP_N:%.*]] = icmp eq
            i32 [[TMP2]], [[N_VEC]]<br>
            >>>> +; CHECK-NEXT:    br i1 [[CMP_N]], label
            [[IF_END:%.*]], label <br>
            >>>> [[SCALAR_PH]]<br>
            >>>>    ; CHECK:       <a
              href="http://scalar.ph" rel="noreferrer" target="_blank"
              moz-do-not-send="true">scalar.ph</a>:<br>
            >>>>    ; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] =
            phi i32 [ [[N_VEC]], <br>
            >>>> [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]<br>
            >>>>    ; CHECK-NEXT:    br label
            [[FOR_COND:%.*]]<br>
            >>>>    ; CHECK:       for.cond:<br>
            >>>>    ; CHECK-NEXT:    [[I:%.*]] = phi i32 [
            [[BC_RESUME_VAL]], <br>
            >>>> [[SCALAR_PH]] ], [ [[INC:%.*]],
            [[FOR_BODY:%.*]] ]<br>
            >>>>    ; CHECK-NEXT:    [[CMP:%.*]] = icmp slt
            i32 [[I]], [[N]]<br>
            >>>> -; CHECK-NEXT:    br i1 [[CMP]], label
            [[FOR_BODY]], label <br>
            >>>> [[IF_END:%.*]]<br>
            >>>> +; CHECK-NEXT:    br i1 [[CMP]], label
            [[FOR_BODY]], label [[IF_END]]<br>
            >>>>    ; CHECK:       for.body:<br>
            >>>>    ; CHECK-NEXT:    [[IPROM:%.*]] = sext
            i32 [[I]] to i64<br>
            >>>>    ; CHECK-NEXT:    [[B:%.*]] =
            getelementptr inbounds i16, i16* <br>
            >>>> [[P]], i64 [[IPROM]]<br>
            >>>> @@ -372,14 +374,17 @@ define i32
            @multiple_unique_exit2(i16* %p, <br>
            >>>> i32 %n) {<br>
            >>>>    ; CHECK-NEXT:    [[TMP11:%.*]] = icmp eq
            i32 [[INDEX_NEXT]], <br>
            >>>> [[N_VEC]]<br>
            >>>>    ; CHECK-NEXT:    br i1 [[TMP11]], label
            [[MIDDLE_BLOCK:%.*]], <br>
            >>>> label [[VECTOR_BODY]], [[LOOP8:!llvm.loop
            !.*]]<br>
            >>>>    ; CHECK:       middle.block:<br>
            >>>> -; CHECK-NEXT:    br label [[SCALAR_PH]]<br>
            >>>> +; CHECK-NEXT:    [[CMP_N:%.*]] = icmp eq
            i32 [[TMP2]], [[N_VEC]]<br>
            >>>> +; CHECK-NEXT:    [[IND_ESCAPE:%.*]] = sub
            i32 [[N_VEC]], 1<br>
            >>>> +; CHECK-NEXT:    [[IND_ESCAPE1:%.*]] = sub
            i32 [[N_VEC]], 1<br>
            >>>> +; CHECK-NEXT:    br i1 [[CMP_N]], label
            [[IF_END:%.*]], label <br>
            >>>> [[SCALAR_PH]]<br>
            >>>>    ; CHECK:       <a
              href="http://scalar.ph" rel="noreferrer" target="_blank"
              moz-do-not-send="true">scalar.ph</a>:<br>
            >>>>    ; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] =
            phi i32 [ [[N_VEC]], <br>
            >>>> [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]<br>
            >>>>    ; CHECK-NEXT:    br label
            [[FOR_COND:%.*]]<br>
            >>>>    ; CHECK:       for.cond:<br>
            >>>>    ; CHECK-NEXT:    [[I:%.*]] = phi i32 [
            [[BC_RESUME_VAL]], <br>
            >>>> [[SCALAR_PH]] ], [ [[INC:%.*]],
            [[FOR_BODY:%.*]] ]<br>
            >>>>    ; CHECK-NEXT:    [[CMP:%.*]] = icmp slt
            i32 [[I]], [[N]]<br>
            >>>> -; CHECK-NEXT:    br i1 [[CMP]], label
            [[FOR_BODY]], label <br>
            >>>> [[IF_END:%.*]]<br>
            >>>> +; CHECK-NEXT:    br i1 [[CMP]], label
            [[FOR_BODY]], label [[IF_END]]<br>
            >>>>    ; CHECK:       for.body:<br>
            >>>>    ; CHECK-NEXT:    [[IPROM:%.*]] = sext
            i32 [[I]] to i64<br>
            >>>>    ; CHECK-NEXT:    [[B:%.*]] =
            getelementptr inbounds i16, i16* <br>
            >>>> [[P]], i64 [[IPROM]]<br>
            >>>> @@ -388,7 +393,7 @@ define i32
            @multiple_unique_exit2(i16* %p, i32 <br>
            >>>> %n) {<br>
            >>>>    ; CHECK-NEXT:    [[CMP2:%.*]] = icmp slt
            i32 [[I]], 2096<br>
            >>>>    ; CHECK-NEXT:    br i1 [[CMP2]], label
            [[FOR_COND]], label <br>
            >>>> [[IF_END]], [[LOOP9:!llvm.loop !.*]]<br>
            >>>>    ; CHECK:       if.end:<br>
            >>>> -; CHECK-NEXT:    [[I_LCSSA:%.*]] = phi i32
            [ [[I]], [[FOR_BODY]] <br>
            >>>> ], [ [[I]], [[FOR_COND]] ]<br>
            >>>> +; CHECK-NEXT:    [[I_LCSSA:%.*]] = phi i32
            [ [[I]], [[FOR_BODY]] <br>
            >>>> ], [ [[I]], [[FOR_COND]] ], [
            [[IND_ESCAPE1]], [[MIDDLE_BLOCK]] ]<br>
            >>>>    ; CHECK-NEXT:    ret i32 [[I_LCSSA]]<br>
            >>>>    ;<br>
            >>>>    ; TAILFOLD-LABEL:
            @multiple_unique_exit2(<br>
            >>>> @@ -461,14 +466,15 @@ define i32
            @multiple_unique_exit3(i16* %p, <br>
            >>>> i32 %n) {<br>
            >>>>    ; CHECK-NEXT:    [[TMP11:%.*]] = icmp eq
            i32 [[INDEX_NEXT]], <br>
            >>>> [[N_VEC]]<br>
            >>>>    ; CHECK-NEXT:    br i1 [[TMP11]], label
            [[MIDDLE_BLOCK:%.*]], <br>
            >>>> label [[VECTOR_BODY]], [[LOOP10:!llvm.loop
            !.*]]<br>
            >>>>    ; CHECK:       middle.block:<br>
            >>>> -; CHECK-NEXT:    br label [[SCALAR_PH]]<br>
            >>>> +; CHECK-NEXT:    [[CMP_N:%.*]] = icmp eq
            i32 [[TMP2]], [[N_VEC]]<br>
            >>>> +; CHECK-NEXT:    br i1 [[CMP_N]], label
            [[IF_END:%.*]], label <br>
            >>>> [[SCALAR_PH]]<br>
            >>>>    ; CHECK:       <a
              href="http://scalar.ph" rel="noreferrer" target="_blank"
              moz-do-not-send="true">scalar.ph</a>:<br>
            >>>>    ; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] =
            phi i32 [ [[N_VEC]], <br>
            >>>> [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]<br>
            >>>>    ; CHECK-NEXT:    br label
            [[FOR_COND:%.*]]<br>
            >>>>    ; CHECK:       for.cond:<br>
            >>>>    ; CHECK-NEXT:    [[I:%.*]] = phi i32 [
            [[BC_RESUME_VAL]], <br>
            >>>> [[SCALAR_PH]] ], [ [[INC:%.*]],
            [[FOR_BODY:%.*]] ]<br>
            >>>>    ; CHECK-NEXT:    [[CMP:%.*]] = icmp slt
            i32 [[I]], [[N]]<br>
            >>>> -; CHECK-NEXT:    br i1 [[CMP]], label
            [[FOR_BODY]], label <br>
            >>>> [[IF_END:%.*]]<br>
            >>>> +; CHECK-NEXT:    br i1 [[CMP]], label
            [[FOR_BODY]], label [[IF_END]]<br>
            >>>>    ; CHECK:       for.body:<br>
            >>>>    ; CHECK-NEXT:    [[IPROM:%.*]] = sext
            i32 [[I]] to i64<br>
            >>>>    ; CHECK-NEXT:    [[B:%.*]] =
            getelementptr inbounds i16, i16* <br>
            >>>> [[P]], i64 [[IPROM]]<br>
            >>>> @@ -477,7 +483,7 @@ define i32
            @multiple_unique_exit3(i16* %p, i32 <br>
            >>>> %n) {<br>
            >>>>    ; CHECK-NEXT:    [[CMP2:%.*]] = icmp slt
            i32 [[I]], 2096<br>
            >>>>    ; CHECK-NEXT:    br i1 [[CMP2]], label
            [[FOR_COND]], label <br>
            >>>> [[IF_END]], [[LOOP11:!llvm.loop !.*]]<br>
            >>>>    ; CHECK:       if.end:<br>
            >>>> -; CHECK-NEXT:    [[EXIT:%.*]] = phi i32 [
            0, [[FOR_COND]] ], [ 1, <br>
            >>>> [[FOR_BODY]] ]<br>
            >>>> +; CHECK-NEXT:    [[EXIT:%.*]] = phi i32 [
            0, [[FOR_COND]] ], [ 1, <br>
            >>>> [[FOR_BODY]] ], [ 0, [[MIDDLE_BLOCK]] ]<br>
            >>>>    ; CHECK-NEXT:    ret i32 [[EXIT]]<br>
            >>>>    ;<br>
            >>>>    ; TAILFOLD-LABEL:
            @multiple_unique_exit3(<br>
            >>>> @@ -994,7 +1000,8 @@ define void
            @scalar_predication(float* %addr) {<br>
            >>>>    ; CHECK-NEXT:    [[TMP10:%.*]] = icmp eq
            i64 [[INDEX_NEXT]], 200<br>
            >>>>    ; CHECK-NEXT:    br i1 [[TMP10]], label
            [[MIDDLE_BLOCK:%.*]], <br>
            >>>> label [[VECTOR_BODY]], [[LOOP12:!llvm.loop
            !.*]]<br>
            >>>>    ; CHECK:       middle.block:<br>
            >>>> -; CHECK-NEXT:    br label [[SCALAR_PH]]<br>
            >>>> +; CHECK-NEXT:    [[CMP_N:%.*]] = icmp eq
            i64 201, 200<br>
            >>>> +; CHECK-NEXT:    br i1 [[CMP_N]], label
            [[EXIT:%.*]], label <br>
            >>>> [[SCALAR_PH]]<br>
            >>>>    ; CHECK:       <a
              href="http://scalar.ph" rel="noreferrer" target="_blank"
              moz-do-not-send="true">scalar.ph</a>:<br>
            >>>>    ; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] =
            phi i64 [ 200, <br>
            >>>> [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]<br>
            >>>>    ; CHECK-NEXT:    br label
            [[LOOP_HEADER:%.*]]<br>
            >>>> @@ -1002,7 +1009,7 @@ define void
            @scalar_predication(float* %addr) {<br>
            >>>>    ; CHECK-NEXT:    [[IV:%.*]] = phi i64 [
            [[BC_RESUME_VAL]], <br>
            >>>> [[SCALAR_PH]] ], [ [[IV_NEXT:%.*]],
            [[LOOP_LATCH:%.*]] ]<br>
            >>>>    ; CHECK-NEXT:    [[GEP:%.*]] =
            getelementptr float, float* <br>
            >>>> [[ADDR]], i64 [[IV]]<br>
            >>>>    ; CHECK-NEXT:    [[EXITCOND_NOT:%.*]] =
            icmp eq i64 [[IV]], 200<br>
            >>>> -; CHECK-NEXT:    br i1 [[EXITCOND_NOT]],
            label [[EXIT:%.*]], label <br>
            >>>> [[LOOP_BODY:%.*]]<br>
            >>>> +; CHECK-NEXT:    br i1 [[EXITCOND_NOT]],
            label [[EXIT]], label <br>
            >>>> [[LOOP_BODY:%.*]]<br>
            >>>>    ; CHECK:       loop.body:<br>
            >>>>    ; CHECK-NEXT:    [[TMP11:%.*]] = load
            float, float* [[GEP]], <br>
            >>>> align 4<br>
            >>>>    ; CHECK-NEXT:    [[PRED:%.*]] = fcmp oeq
            float [[TMP11]], <br>
            >>>> 0.000000e+00<br>
            >>>> @@ -1088,7 +1095,8 @@ define i32
            @me_reduction(i32* %addr) {<br>
            >>>>    ; CHECK-NEXT:    [[RDX_SHUF:%.*]] =
            shufflevector <2 x i32> <br>
            >>>> [[TMP5]], <2 x i32> poison, <2 x
            i32> <i32 1, i32 undef><br>
            >>>>    ; CHECK-NEXT:    [[BIN_RDX:%.*]] = add
            <2 x i32> [[TMP5]], <br>
            >>>> [[RDX_SHUF]]<br>
            >>>>    ; CHECK-NEXT:    [[TMP7:%.*]] =
            extractelement <2 x i32> <br>
            >>>> [[BIN_RDX]], i32 0<br>
            >>>> -; CHECK-NEXT:    br label [[SCALAR_PH]]<br>
            >>>> +; CHECK-NEXT:    [[CMP_N:%.*]] = icmp eq
            i64 201, 200<br>
            >>>> +; CHECK-NEXT:    br i1 [[CMP_N]], label
            [[EXIT:%.*]], label <br>
            >>>> [[SCALAR_PH]]<br>
            >>>>    ; CHECK:       <a
              href="http://scalar.ph" rel="noreferrer" target="_blank"
              moz-do-not-send="true">scalar.ph</a>:<br>
            >>>>    ; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] =
            phi i64 [ 200, <br>
            >>>> [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]<br>
            >>>>    ; CHECK-NEXT:    [[BC_MERGE_RDX:%.*]] =
            phi i32 [ 0, [[ENTRY]] <br>
            >>>> ], [ [[TMP7]], [[MIDDLE_BLOCK]] ]<br>
            >>>> @@ -1098,7 +1106,7 @@ define i32
            @me_reduction(i32* %addr) {<br>
            >>>>    ; CHECK-NEXT:    [[ACCUM:%.*]] = phi i32
            [ [[BC_MERGE_RDX]], <br>
            >>>> [[SCALAR_PH]] ], [ [[ACCUM_NEXT:%.*]],
            [[LOOP_LATCH]] ]<br>
            >>>>    ; CHECK-NEXT:    [[GEP:%.*]] =
            getelementptr i32, i32* [[ADDR]], <br>
            >>>> i64 [[IV]]<br>
            >>>>    ; CHECK-NEXT:    [[EXITCOND_NOT:%.*]] =
            icmp eq i64 [[IV]], 200<br>
            >>>> -; CHECK-NEXT:    br i1 [[EXITCOND_NOT]],
            label [[EXIT:%.*]], label <br>
            >>>> [[LOOP_LATCH]]<br>
            >>>> +; CHECK-NEXT:    br i1 [[EXITCOND_NOT]],
            label [[EXIT]], label <br>
            >>>> [[LOOP_LATCH]]<br>
            >>>>    ; CHECK:       loop.latch:<br>
            >>>>    ; CHECK-NEXT:    [[TMP8:%.*]] = load
            i32, i32* [[GEP]], align 4<br>
            >>>>    ; CHECK-NEXT:    [[ACCUM_NEXT]] = add
            i32 [[ACCUM]], [[TMP8]]<br>
            >>>> @@ -1106,7 +1114,7 @@ define i32
            @me_reduction(i32* %addr) {<br>
            >>>>    ; CHECK-NEXT:    [[EXITCOND2_NOT:%.*]] =
            icmp eq i64 [[IV]], 400<br>
            >>>>    ; CHECK-NEXT:    br i1
            [[EXITCOND2_NOT]], label [[EXIT]], label <br>
            >>>> [[LOOP_HEADER]], [[LOOP15:!llvm.loop !.*]]<br>
            >>>>    ; CHECK:       exit:<br>
            >>>> -; CHECK-NEXT:    [[LCSSA:%.*]] = phi i32 [
            0, [[LOOP_HEADER]] ], [ <br>
            >>>> [[ACCUM_NEXT]], [[LOOP_LATCH]] ]<br>
            >>>> +; CHECK-NEXT:    [[LCSSA:%.*]] = phi i32 [
            0, [[LOOP_HEADER]] ], [ <br>
            >>>> [[ACCUM_NEXT]], [[LOOP_LATCH]] ], [
            [[TMP7]], [[MIDDLE_BLOCK]] ]<br>
            >>>>    ; CHECK-NEXT:    ret i32 [[LCSSA]]<br>
            >>>>    ;<br>
            >>>>    ; TAILFOLD-LABEL: @me_reduction(<br>
            >>>><br>
            >>>><br>
            >>>><br>
            >>>>
            _______________________________________________<br>
            >>>> llvm-commits mailing list<br>
            >>>> <a
              href="mailto:llvm-commits@lists.llvm.org" target="_blank"
              moz-do-not-send="true">llvm-commits@lists.llvm.org</a><br>
            >>>> <a
              href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits"
              rel="noreferrer" target="_blank" moz-do-not-send="true">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits</a><br>
            >>> _______________________________________________<br>
            >>> llvm-commits mailing list<br>
            >>> <a href="mailto:llvm-commits@lists.llvm.org"
              target="_blank" moz-do-not-send="true">llvm-commits@lists.llvm.org</a><br>
            >>> <a
              href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits"
              rel="noreferrer" target="_blank" moz-do-not-send="true">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits</a><br>
            > _______________________________________________<br>
            > llvm-commits mailing list<br>
            > <a href="mailto:llvm-commits@lists.llvm.org"
              target="_blank" moz-do-not-send="true">llvm-commits@lists.llvm.org</a><br>
            > <a
              href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits"
              rel="noreferrer" target="_blank" moz-do-not-send="true">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits</a></blockquote>
        </div>
      </div>
    </blockquote>
  </body>
</html>