<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>If you have the before and after IR, that's really all I probably
need. If you share those, I can take the investigation from
there.</p>
<p>Philip<br>
</p>
<div class="moz-cite-prefix">On 5/25/21 2:12 PM, Nemanja Ivanovic
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:OFC99B9E23.4F6BE4A3-ON002586E0.00745021-002586E0.00748300@notes.na.collabserv.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div class="socmaildefaultfont" dir="ltr"
style="font-family:Arial, Helvetica, sans-serif;font-size:10pt">
<div dir="ltr">Hi Philip,</div>
<div dir="ltr">I am sorry about the late reply - this was a 4
day weekend here in Toronto. We kind of started with looking
at the difference in the post-LV IR with and without your
patch for the SingleSource test case. Stefan is currently
looking at it and we're hoping to figure out what's going on
soon. Would it help you if we shared the two IR files with you
to get your opinion on what might be happening as well?</div>
<div dir="ltr"> </div>
<div dir="ltr">Nemanja Ivanovic<br>
LLVM PPC Backend Development<br>
IBM Toronto Lab<br>
Email: <a class="moz-txt-link-abbreviated" href="mailto:nemanjai@ca.ibm.com">nemanjai@ca.ibm.com</a><br>
Phone: 905-413-3388</div>
<div dir="ltr"> </div>
<div dir="ltr"> </div>
<blockquote data-history-content-modified="1" dir="ltr"
style="border-left:solid #aaaaaa 2px; margin-left:5px;
padding-left:5px; direction:ltr; margin-right:0px">-----
Original message -----<br>
From: Philip Reames <a class="moz-txt-link-rfc2396E" href="mailto:listmail@philipreames.com"><listmail@philipreames.com></a><br>
To: Nemanja Ivanovic <a class="moz-txt-link-rfc2396E" href="mailto:nemanjai@ca.ibm.com"><nemanjai@ca.ibm.com></a><br>
Cc: <a class="moz-txt-link-abbreviated" href="mailto:akuegel@google.com">akuegel@google.com</a>, <a class="moz-txt-link-abbreviated" href="mailto:benny.kra@gmail.com">benny.kra@gmail.com</a>,
<a class="moz-txt-link-abbreviated" href="mailto:llvm-commits@lists.llvm.org">llvm-commits@lists.llvm.org</a>, <a class="moz-txt-link-abbreviated" href="mailto:llvmlistbot@llvm.org">llvmlistbot@llvm.org</a>, LLVM on
Power <a class="moz-txt-link-rfc2396E" href="mailto:powerllvm@ca.ibm.com"><powerllvm@ca.ibm.com></a><br>
Subject: [EXTERNAL] Re: [llvm] 7fe41ac - Revert "[LV]
Unconditionally branch from middle to scalar preheader if the
scalar loop must execute"<br>
Date: Wed, May 19, 2021 2:15 PM<br>
<br>
<!--Notes ACF
<meta http-equiv="Content-Type" content="text/html; charset=utf8" >-->
<p>Looking at the various failures, I see both LE and BE bots
failing. The BE bot shows a miscompile in one of the test
suite benchmarks. The LE bots appear to be showing a crash
while using a stage1 build clang to build stage2 clang. </p>
<p>LE example: <a
href="https://lab.llvm.org/buildbot#builders/19/builds/4236"
moz-do-not-send="true" target="_blank">https://lab.llvm.org/buildbot#builders/19/builds/4236</a><br>
BE example: <a
href="https://lab.llvm.org/buildbot#builders/100/builds/5762"
target="_blank" moz-do-not-send="true">https://lab.llvm.org/buildbot#builders/100/builds/5762</a></p>
<p>So both are showing miscompiles, just with different
symptoms. Frankly, the BE looks easier to debug (much code
making it into the miscompiled binary.)</p>
<p>Oddly, pretty much only PPC bots are failing. The one
exception is a stage2 failure on AArch64 (<a
href="https://lab.llvm.org/buildbot/#/builders/111/builds/2027"
target="_blank" moz-do-not-send="true">https://lab.llvm.org/buildbot/#/builders/111/builds/2027</a>)
which looks similar to the LE failure above. </p>
<p>Given this appears to be target specific, I am *guessing*
there's some vectorizer hook which is causing a different
codepath to be executed. Before we start trying to get me
access to hardware, do you have any guesses on what that
hook might be? If you can give me a good hint on where to
look, I suspect I can probably find the issue that way. </p>
<p>Philip</p>
<div>On 5/18/21 3:07 AM, Nemanja Ivanovic wrote:</div>
<blockquote
cite="mid:OF443062AC.F5998286-ON002586D9.00373B50-002586D9.0037A584@notes.na.collabserv.com"
type="cite"><!--Notes ACF
<meta http-equiv="content-type" content="text/html; charset=utf8" >-->
<div dir="ltr" style="font-family:Arial, Helvetica,
sans-serif;font-size:10pt">
<div dir="ltr">Hi Philip,</div>
<div dir="ltr">I am not sure what happened with your first
attempt to contact us and how we missed it. We would be
more than happy to help you debug this issue. Do you
know if this only affects the big endian bot or if it
also fails on little endian bots? In the latter case, we
can certainly provide access to a little endian machine
hosted at OSU/OSL. In the former case, we don't have a
machine available and we'll have to do the debugging and
report to you (which might take a bit longer).</div>
<div dir="ltr"> </div>
<div dir="ltr">Nemanja Ivanovic<br>
LLVM PPC Backend Development<br>
IBM Toronto Lab<br>
Email: <a href="mailto:nemanjai@ca.ibm.com"
target="_blank" moz-do-not-send="true">nemanjai@ca.ibm.com</a><br>
Phone: 905-413-3388</div>
<div dir="ltr"> </div>
<div dir="ltr"> </div>
<blockquote data-history-content-modified="1"
data-history-expanded="1" dir="ltr"
style="border-left:solid #aaaaaa 2px; margin-left:5px;
padding-left:5px; direction:ltr; margin-right:0px">-----
Original message -----<br>
From: Philip Reames <a
href="mailto:listmail@philipreames.com"
target="_blank" moz-do-not-send="true"><listmail@philipreames.com></a><br>
To: Adrian Kuegel <a href="mailto:akuegel@google.com"
target="_blank" moz-do-not-send="true"><akuegel@google.com></a><br>
Cc: Benjamin Kramer <a
href="mailto:benny.kra@gmail.com" target="_blank"
moz-do-not-send="true"><benny.kra@gmail.com></a>,
Adrian Kuegel <a href="mailto:llvmlistbot@llvm.org"
target="_blank" moz-do-not-send="true"><llvmlistbot@llvm.org></a>,
llvm-commits <a
href="mailto:llvm-commits@lists.llvm.org"
target="_blank" moz-do-not-send="true"><llvm-commits@lists.llvm.org></a>,
<a href="mailto:powerllvm@ca.ibm.com" target="_blank"
moz-do-not-send="true">powerllvm@ca.ibm.com</a><br>
Subject: [EXTERNAL] Re: [llvm] 7fe41ac - Revert "[LV]
Unconditionally branch from middle to scalar preheader
if the scalar loop must execute"<br>
Date: Mon, May 17, 2021 11:59 PM<br>
<br>
<!-- BaNnErBlUrFlE-HeAdEr-start -->
<!-- BaNnErBlUrFlE-HeAdEr-end -->
<!--Notes ACF
<meta http-equiv="Content-Type" content="text/html; charset=utf8" >-->
<!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN --> <span
style="display:none!important;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;max-width:0px;opacity:0;overflow:hidden;">
I tried another cycle to see if this had been
resolved, but am still seeing build bot failures,
nearly exclusively on PPC. (And one arm self host
bot.) @PPC Bot Owner - I need help reducing a test
case for the failure seen here: </span>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN --> <span
style="display:none!important;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;max-width:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</span>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 3px 0px 16px 0px;" ><tr><td>
<table class="pfptBannerTableMSO" border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%;border-radius:4px;border-top:4px solid #90a4ae;background-color:#d0d8dc;" ><tr><td valign="top" >
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px" >
<tr><td><span class="pfptTitleMSO" style="color:#000000 !important;font-family: 'Arial', sans-serif;font-weight:bold !important;font-size:14px !important;" >
This Message Is From an External Sender
</span></td></tr>
<tr><td><span class="pfptSubtitleMSO" style="color:#000000 !important;font-weight:normal !important;font-family: 'Arial', sans-serif; font-size:12px !important;" >
This message came from outside your organization.
</span></td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<!--[if !((ie)|(mso))]-->
<div style="margin:16px 0px 16px 0px; padding:8px 16px
8px 16px; border-radius: 4px; min-width:
200px;background-color: #d0d8dc; border-top: 4px solid
#90a4ae;">
<div style="display: inline-block; margin: 0px 0px 1px
0px; max-width: 600px;">
<div style="color:#000000 !important;font-family:
'Arial', sans-serif !important;font-weight:bold
!important;font-size:14px
!important;line-height:18px;display:block;">This
Message Is From an External Sender</div>
<div style="color:#000000
!important;font-weight:normal
!important;font-family: 'Arial', sans-serif
!important;font-size:12px
!important;line-height:18px;margin-top:2px;display:block">This
message came from outside your organization.</div>
</div>
</div>
<!--[endif]--> <span
style="display:none!important;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;max-width:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</span><br>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<p>I tried another cycle to see if this had been
resolved, but am still seeing build bot failures,
nearly exclusively on PPC. (And one arm self host
bot.) </p>
<p>@PPC Bot Owner - I need help reducing a test case for
the failure seen here: <a
href="https://lab.llvm.org/buildbot#builders/100/builds/5762"
moz-do-not-send="true" target="_blank">https://lab.llvm.org/buildbot#builders/100/builds/5762</a>.
I am stuck, and unable to make progress for nearly 3
months now. I would greatly appreciate help. </p>
<p>Philip</p>
<div>On 5/5/21 11:59 PM, Adrian Kuegel wrote:</div>
<blockquote
cite="mid:CAHhV=OX3NbAExy4YsQWLH0fC-zXiG_VpToAtB72NhkUwwYekRQ@mail.gmail.com"
type="cite"><!--Notes ACF
<meta http-equiv="content-type" content="text/html; charset=utf8" >-->
<div dir="ltr">
<div dir="ltr">Sounds good to me, and thanks for
sending the heads up :)</div>
<div>
<div dir="ltr">On Wed, May 5, 2021 at 10:01 PM
Philip Reames <<a
href="mailto:listmail@philipreames.com"
moz-do-not-send="true" target="_blank">listmail@philipreames.com</a>>
wrote:</div>
<blockquote style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">FYI, I'm
going to try recommitting this without changes
in a day or so.<br>
<br>
I never heard back from a PPC bot owner, and I
don't have enough<br>
information to really debug anything from the
builtbot log. I did run<br>
across a latent issue which this patch may very
well have exposed at<br>
much higher frequency; the previous patch in
this series (which is much<br>
more restrictive) does appear to have increased
frequency. That was<br>
worked around in 80e80250. My educated guess is
that same issue<br>
triggered the miscompile seen on the ppc bot,
but that is more of a<br>
guess than I'd really prefer.<br>
<br>
I'm going to submit this during off hours, and
watch the bots fairly<br>
closely after submit. Hopefully this either
cycles clean or I get a<br>
better clue as to what the root issue is.<br>
<br>
Philip<br>
<br>
On 2/8/21 9:09 PM, Philip Reames via
llvm-commits wrote:<br>
> Ben,<br>
><br>
> Thanks for the clarification. The log does
not make the fact this is<br>
> an execution failure obvious.<br>
><br>
> No, I don't have access to a PPC machine.<br>
><br>
> I am going to need some assistance from the
bot owner on this. At a<br>
> minimum, IR for the test in question
(before optimization, but on<br>
> target platform) seems like a reasonable
ask.<br>
><br>
> I strongly suspect this change is simply
exposing another latent<br>
> issue. Or at least, I've reviewed the
change and don't see anything<br>
> likely to cause runtime crashes w/o also
tripping compiler asserts.<br>
><br>
> Philip<br>
><br>
><br>
> On 2/8/21 5:21 AM, Benjamin Kramer wrote:<br>
>> `execution_time` failures mean that the
bot succeeded building a test<br>
>> but it failed when running it. I'm
relatively certain that this is the<br>
>> same issue Adrian is seeing -- binaries
segfaulting early on PPC.<br>
>><br>
>> The bot log output isn't helpful at all
for investigating why this is<br>
>> happening. Do you happen to have access
to a PPC machine?<br>
>><br>
>> On Fri, Feb 5, 2021 at 6:05 PM Philip
Reames via llvm-commits<br>
>> <<a
href="mailto:llvm-commits@lists.llvm.org"
moz-do-not-send="true" target="_blank">llvm-commits@lists.llvm.org</a>>
wrote:<br>
>>> Adrian,<br>
>>><br>
>>> I'm going to need you to provide a
bit more information here. The test<br>
>>> failure in stage1 was fixed at the
time you reverted this patch. The<br>
>>> remaining failure in the bot is
very unclear. What is a execution_time<br>
>>> failure? From the log output, the
"failing" run finished in 0.5<br>
>>> seconds,<br>
>>> whereas the previous "succeeding"
run finished in 11 seconds. Without<br>
>>> further context, I'd say that's no
failure.<br>
>>><br>
>>> I'll also note that I did not
receive email from this bot. I received<br>
>>> notice from the various other bots
and fixed the ARM test issue, but<br>
>>> unless I missed it in with the
others, this bot is not notifying.<br>
>>><br>
>>> In general, I'm a fan of fast
reverts, but I have to admit, this one<br>
>>> appears borderline at the moment.<br>
>>><br>
>>> Philip<br>
>>><br>
>>> On 2/5/21 3:53 AM, Adrian Kuegel
via llvm-commits wrote:<br>
>>>> Author: Adrian Kuegel<br>
>>>> Date: 2021-02-05T12:51:03+01:00<br>
>>>> New Revision:
7fe41ac3dff2d44c3d2c31b28554fbe4a86eaa6c<br>
>>>><br>
>>>> URL:<br>
>>>> <a
href="https://github.com/llvm/llvm-project/commit/7fe41ac3dff2d44c3d2c31b28554fbe4a86eaa6c"
moz-do-not-send="true" rel="noreferrer"
target="_blank">https://github.com/llvm/llvm-project/commit/7fe41ac3dff2d44c3d2c31b28554fbe4a86eaa6c</a><br>
>>>> DIFF:<br>
>>>> <a
href="https://github.com/llvm/llvm-project/commit/7fe41ac3dff2d44c3d2c31b28554fbe4a86eaa6c.diff"
moz-do-not-send="true" rel="noreferrer"
target="_blank">https://github.com/llvm/llvm-project/commit/7fe41ac3dff2d44c3d2c31b28554fbe4a86eaa6c.diff</a><br>
>>>><br>
>>>> LOG: Revert "[LV]
Unconditionally branch from middle to scalar<br>
>>>> preheader if the scalar loop
must execute"<br>
>>>><br>
>>>> This reverts commit
3e5ce49e5371ce4feadbf97dd5c2b652d9db3d1d.<br>
>>>><br>
>>>> Tests started failing on PPC,
for example:<br>
>>>> <a
href="http://lab.llvm.org:8011/#/builders/105/builds/5569"
moz-do-not-send="true" rel="noreferrer"
target="_blank">http://lab.llvm.org:8011/#/builders/105/builds/5569</a><br>
>>>><br>
>>>> Added:<br>
>>>><br>
>>>><br>
>>>> Modified:<br>
>>>>
llvm/lib/Transforms/Utils/LoopVersioning.cpp<br>
>>>>
llvm/lib/Transforms/Vectorize/LoopVectorize.cpp<br>
>>>>
llvm/test/Transforms/LoopVectorize/first-order-recurrence-complex.ll<br>
>>>>
llvm/test/Transforms/LoopVectorize/interleaved-accesses.ll<br>
>>>>
llvm/test/Transforms/LoopVectorize/loop-form.ll<br>
>>>><br>
>>>> Removed:<br>
>>>><br>
>>>><br>
>>>><br>
>>>>
################################################################################<br>
>>>><br>
>>>> diff --git
a/llvm/lib/Transforms/Utils/LoopVersioning.cpp<br>
>>>>
b/llvm/lib/Transforms/Utils/LoopVersioning.cpp<br>
>>>> index
8a89158788cf..de4fb446fdf2 100644<br>
>>>> ---
a/llvm/lib/Transforms/Utils/LoopVersioning.cpp<br>
>>>> +++
b/llvm/lib/Transforms/Utils/LoopVersioning.cpp<br>
>>>> @@ -44,11 +44,11 @@
LoopVersioning::LoopVersioning(const<br>
>>>> LoopAccessInfo &LAI,<br>
>>>>
AliasChecks(Checks.begin(), Checks.end()),<br>
>>>>
Preds(LAI.getPSE().getUnionPredicate()),
LAI(LAI), LI(LI),<br>
>>>> DT(DT),<br>
>>>> SE(SE) {<br>
>>>> +
assert(L->getUniqueExitBlock() && "No
single exit block");<br>
>>>> }<br>
>>>><br>
>>>> void
LoopVersioning::versionLoop(<br>
>>>> const
SmallVectorImpl<Instruction *>
&DefsUsedOutside) {<br>
>>>> -
assert(VersionedLoop->getUniqueExitBlock()
&& "No single exit<br>
>>>> block");<br>
>>>>
assert(VersionedLoop->isLoopSimplifyForm()
&&<br>
>>>> "Loop is not in
loop-simplify form");<br>
>>>><br>
>>>><br>
>>>> diff --git
a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp<br>
>>>>
b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp<br>
>>>> index
3277842edbfe..6bce0caeb36f 100644<br>
>>>> ---
a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp<br>
>>>> +++
b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp<br>
>>>> @@ -852,7 +852,7 @@ class
InnerLoopVectorizer {<br>
>>>> /// Middle Block between
the vector and the scalar.<br>
>>>> BasicBlock
*LoopMiddleBlock;<br>
>>>><br>
>>>> - /// The unique ExitBlock of
the scalar loop if one exists. Note<br>
>>>> that<br>
>>>> + /// The (unique) ExitBlock
of the scalar loop. Note that<br>
>>>> /// there can be multiple
exiting edges reaching this block.<br>
>>>> BasicBlock *LoopExitBlock;<br>
>>>><br>
>>>> @@ -3147,13 +3147,9 @@ void<br>
>>>>
InnerLoopVectorizer::emitMinimumIterationCountCheck(Loop
*L,<br>
>>>>
DT->getNode(Bypass)->getIDom()) &&<br>
>>>> "TC check is
expected to dominate Bypass");<br>
>>>><br>
>>>> - // Update dominator for
Bypass & LoopExit (if needed).<br>
>>>> + // Update dominator for
Bypass & LoopExit.<br>
>>>>
DT->changeImmediateDominator(Bypass,
TCCheckBlock);<br>
>>>> - if
(!Cost->requiresScalarEpilogue())<br>
>>>> - // If there is an epilogue
which must run, there's no edge<br>
>>>> from the<br>
>>>> - // middle block to exit
blocks and thus no need to update the<br>
>>>> immediate<br>
>>>> - // dominator of the exit
blocks.<br>
>>>> -
DT->changeImmediateDominator(LoopExitBlock,
TCCheckBlock);<br>
>>>> +
DT->changeImmediateDominator(LoopExitBlock,
TCCheckBlock);<br>
>>>><br>
>>>> ReplaceInstWithInst(<br>
>>>>
TCCheckBlock->getTerminator(),<br>
>>>> @@ -3192,11 +3188,7 @@ void<br>
>>>>
InnerLoopVectorizer::emitSCEVChecks(Loop *L,
BasicBlock *Bypass) {<br>
>>>> // Update dominator only
if this is first RT check.<br>
>>>> if
(LoopBypassBlocks.empty()) {<br>
>>>>
DT->changeImmediateDominator(Bypass,
SCEVCheckBlock);<br>
>>>> - if
(!Cost->requiresScalarEpilogue())<br>
>>>> - // If there is an
epilogue which must run, there's no edge<br>
>>>> from the<br>
>>>> - // middle block to exit
blocks and thus no need to update<br>
>>>> the immediate<br>
>>>> - // dominator of the exit
blocks.<br>
>>>> -
DT->changeImmediateDominator(LoopExitBlock,
SCEVCheckBlock);<br>
>>>> +
DT->changeImmediateDominator(LoopExitBlock,
SCEVCheckBlock);<br>
>>>> }<br>
>>>><br>
>>>> ReplaceInstWithInst(<br>
>>>> @@ -3252,11 +3244,7 @@ void<br>
>>>>
InnerLoopVectorizer::emitMemRuntimeChecks(Loop
*L, BasicBlock<br>
>>>> *Bypass) {<br>
>>>> // Update dominator only
if this is first RT check.<br>
>>>> if
(LoopBypassBlocks.empty()) {<br>
>>>>
DT->changeImmediateDominator(Bypass,
MemCheckBlock);<br>
>>>> - if
(!Cost->requiresScalarEpilogue())<br>
>>>> - // If there is an
epilogue which must run, there's no edge<br>
>>>> from the<br>
>>>> - // middle block to exit
blocks and thus no need to update<br>
>>>> the immediate<br>
>>>> - // dominator of the exit
blocks.<br>
>>>> -
DT->changeImmediateDominator(LoopExitBlock,
MemCheckBlock);<br>
>>>> +
DT->changeImmediateDominator(LoopExitBlock,
MemCheckBlock);<br>
>>>> }<br>
>>>><br>
>>>> Instruction
*FirstCheckInst;<br>
>>>> @@ -3381,10 +3369,9 @@ Value<br>
>>>>
*InnerLoopVectorizer::emitTransformedIndex(<br>
>>>> Loop
*InnerLoopVectorizer::createVectorLoopSkeleton(StringRef<br>
>>>> Prefix) {<br>
>>>> LoopScalarBody =
OrigLoop->getHeader();<br>
>>>> LoopVectorPreHeader =
OrigLoop->getLoopPreheader();<br>
>>>> + LoopExitBlock =
OrigLoop->getUniqueExitBlock();<br>
>>>> + assert(LoopExitBlock
&& "Must have an exit block");<br>
>>>> assert(LoopVectorPreHeader
&& "Invalid loop structure");<br>
>>>> - LoopExitBlock =
OrigLoop->getUniqueExitBlock(); // may be
nullptr<br>
>>>> - assert((LoopExitBlock ||
Cost->requiresScalarEpilogue()) &&<br>
>>>> - "multiple exit loop
without required epilogue?");<br>
>>>><br>
>>>> LoopMiddleBlock =<br>
>>>>
SplitBlock(LoopVectorPreHeader,<br>
>>>>
LoopVectorPreHeader->getTerminator(), DT,<br>
>>>> @@ -3393,20 +3380,12 @@ Loop<br>
>>>>
*InnerLoopVectorizer::createVectorLoopSkeleton(StringRef
Prefix) {<br>
>>>>
SplitBlock(LoopMiddleBlock,<br>
>>>>
LoopMiddleBlock->getTerminator(), DT, LI,<br>
>>>> nullptr,
Twine(Prefix) + "<a href="http://scalar.ph"
moz-do-not-send="true" rel="noreferrer"
target="_blank">scalar.ph</a>");<br>
>>>><br>
>>>> + // Set up branch from middle
block to the exit and scalar<br>
>>>> preheader blocks.<br>
>>>> + // completeLoopSkeleton will
update the condition to use an<br>
>>>> iteration check,<br>
>>>> + // if required to decide
whether to execute the remainder.<br>
>>>> + BranchInst *BrInst =<br>
>>>> +
BranchInst::Create(LoopExitBlock,
LoopScalarPreHeader,<br>
>>>> Builder.getTrue());<br>
>>>> auto *ScalarLatchTerm =<br>
>>>>
OrigLoop->getLoopLatch()->getTerminator();<br>
>>>> -<br>
>>>> - // Set up the middle block
terminator. Two cases:<br>
>>>> - // 1) If we know that we
must execute the scalar epilogue, emit an<br>
>>>> - // unconditional branch.<br>
>>>> - // 2) Otherwise, we must
have a single unique exit block (due to<br>
>>>> how we<br>
>>>> - // implement the multiple
exit case). In this case, set up a<br>
>>>> conditonal<br>
>>>> - // branch from the middle
block to the loop scalar preheader,<br>
>>>> and the<br>
>>>> - // exit block.
completeLoopSkeleton will update the<br>
>>>> condition to use an<br>
>>>> - // iteration check, if
required to decide whether to execute<br>
>>>> the remainder.<br>
>>>> - BranchInst *BrInst =
Cost->requiresScalarEpilogue() ?<br>
>>>> -
BranchInst::Create(LoopScalarPreHeader) :<br>
>>>> -
BranchInst::Create(LoopExitBlock,
LoopScalarPreHeader,<br>
>>>> -
Builder.getTrue());<br>
>>>>
BrInst->setDebugLoc(ScalarLatchTerm->getDebugLoc());<br>
>>>>
ReplaceInstWithInst(LoopMiddleBlock->getTerminator(),
BrInst);<br>
>>>><br>
>>>> @@ -3418,11 +3397,7 @@ Loop<br>
>>>>
*InnerLoopVectorizer::createVectorLoopSkeleton(StringRef
Prefix) {<br>
>>>> nullptr,
nullptr, Twine(Prefix) + "vector.body");<br>
>>>><br>
>>>> // Update dominator for
loop exit.<br>
>>>> - if
(!Cost->requiresScalarEpilogue())<br>
>>>> - // If there is an epilogue
which must run, there's no edge<br>
>>>> from the<br>
>>>> - // middle block to exit
blocks and thus no need to update the<br>
>>>> immediate<br>
>>>> - // dominator of the exit
blocks.<br>
>>>> -
DT->changeImmediateDominator(LoopExitBlock,
LoopMiddleBlock);<br>
>>>> +
DT->changeImmediateDominator(LoopExitBlock,
LoopMiddleBlock);<br>
>>>><br>
>>>> // Create and register the
new vector loop.<br>
>>>> Loop *Lp =
LI->AllocateLoop();<br>
>>>> @@ -3519,14 +3494,10 @@
BasicBlock<br>
>>>>
*InnerLoopVectorizer::completeLoopSkeleton(Loop
*L,<br>
>>>> auto *ScalarLatchTerm =<br>
>>>>
OrigLoop->getLoopLatch()->getTerminator();<br>
>>>><br>
>>>> // Add a check in the
middle block to see if we have completed<br>
>>>> - // all of the iterations in
the first vector loop. Three cases:<br>
>>>> - // 1) If we require a scalar
epilogue, there is no conditional<br>
>>>> branch as<br>
>>>> - // we unconditionally
branch to the scalar preheader. Do<br>
>>>> nothing.<br>
>>>> - // 2) If (N - N%VF) == N,
then we *don't* need to run the<br>
>>>> remainder.<br>
>>>> - // Thus if tail is to be
folded, we know we don't need to run<br>
>>>> the<br>
>>>> - // remainder and we can
use the previous value for the<br>
>>>> condition (true).<br>
>>>> - // 3) Otherwise, construct a
runtime check.<br>
>>>> - if
(!Cost->requiresScalarEpilogue() &&<br>
>>>> !Cost->foldTailByMasking())
{<br>
>>>> + // all of the iterations in
the first vector loop.<br>
>>>> + // If (N - N%VF) == N, then
we *don't* need to run the remainder.<br>
>>>> + // If tail is to be folded,
we know we don't need to run the<br>
>>>> remainder.<br>
>>>> + if
(!Cost->foldTailByMasking()) {<br>
>>>> Instruction *CmpN =
CmpInst::Create(Instruction::ICmp,<br>
>>>> CmpInst::ICMP_EQ,<br>
>>>>
Count, VectorTripCount,<br>
>>>> "cmp.n",<br>
>>>>
LoopMiddleBlock->getTerminator());<br>
>>>> @@ -3590,17 +3561,17 @@
BasicBlock<br>
>>>>
*InnerLoopVectorizer::createVectorizedLoopSkeleton()
{<br>
>>>> | [ ]_| <--
vector loop.<br>
>>>> | |<br>
>>>> | v<br>
>>>> - \ -[ ] <---
middle-block.<br>
>>>> - \/ |<br>
>>>> - /\ v<br>
>>>> - | ->[ ] <--- new
preheader.<br>
>>>> + | -[ ] <---
middle-block.<br>
>>>> + | / |<br>
>>>> + | / v<br>
>>>> + -|- >[ ] <--- new
preheader.<br>
>>>> | |<br>
>>>> - (opt) v <-- edge
from middle to exit iff epilogue is not<br>
>>>> required.<br>
>>>> + | v<br>
>>>> | [ ] \<br>
>>>> - | [ ]_| <-- old
scalar loop to handle remainder (scalar<br>
>>>> epilogue).<br>
>>>> + | [ ]_| <-- old
scalar loop to handle remainder.<br>
>>>> \ |<br>
>>>> \ v<br>
>>>> - >[ ] <-- exit
block(s).<br>
>>>> + >[ ] <-- exit
block.<br>
>>>> ...<br>
>>>> */<br>
>>>><br>
>>>> @@ -4021,18 +3992,13 @@ void<br>
>>>>
InnerLoopVectorizer::fixVectorizedLoop() {<br>
>>>> // Forget the original
basic block.<br>
>>>>
PSE.getSE()->forgetLoop(OrigLoop);<br>
>>>><br>
>>>> - // If we inserted an edge
from the middle block to the unique<br>
>>>> exit block,<br>
>>>> - // update uses outside the
loop (phis) to account for the newly<br>
>>>> inserted<br>
>>>> - // edge.<br>
>>>> - if
(!Cost->requiresScalarEpilogue()) {<br>
>>>> - // Fix-up external users
of the induction variables.<br>
>>>> - for (auto &Entry :
Legal->getInductionVars())<br>
>>>> -
fixupIVUsers(Entry.first, Entry.second,<br>
>>>> -
getOrCreateVectorTripCount(LI->getLoopFor(LoopVectorBody)),<br>
>>>> -
IVEndValues[Entry.first], LoopMiddleBlock);<br>
>>>> + // Fix-up external users of
the induction variables.<br>
>>>> + for (auto &Entry :
Legal->getInductionVars())<br>
>>>> + fixupIVUsers(Entry.first,
Entry.second,<br>
>>>> +
getOrCreateVectorTripCount(LI->getLoopFor(LoopVectorBody)),<br>
>>>> +
IVEndValues[Entry.first], LoopMiddleBlock);<br>
>>>><br>
>>>> - fixLCSSAPHIs();<br>
>>>> - }<br>
>>>> + fixLCSSAPHIs();<br>
>>>> for (Instruction *PI :
PredicatedInstructions)<br>
>>>>
sinkScalarOperands(&*PI);<br>
>>>><br>
>>>> @@ -4250,13 +4216,12 @@ void<br>
>>>>
InnerLoopVectorizer::fixFirstOrderRecurrence(PHINode
*Phi) {<br>
>>>> // recurrence in the exit
block, and then add an edge for the<br>
>>>> middle block.<br>
>>>> // Note that LCSSA does
not imply single entry when the<br>
>>>> original scalar loop<br>
>>>> // had multiple exiting
edges (as we always run the last<br>
>>>> iteration in the<br>
>>>> - // scalar epilogue); in that
case, there is no edge from middle<br>
>>>> to exit and<br>
>>>> - // and thus no phis which
needed updated.<br>
>>>> - if
(!Cost->requiresScalarEpilogue())<br>
>>>> - for (PHINode &LCSSAPhi
: LoopExitBlock->phis())<br>
>>>> - if
(any_of(LCSSAPhi.incoming_values(),<br>
>>>> - [Phi](Value
*V) { return V == Phi; }))<br>
>>>> -
LCSSAPhi.addIncoming(ExtractForPhiUsedOutsideLoop,<br>
>>>> LoopMiddleBlock);<br>
>>>> + // scalar epilogue); in that
case, the exiting path through<br>
>>>> middle will be<br>
>>>> + // dynamically dead and the
value picked for the phi doesn't<br>
>>>> matter.<br>
>>>> + for (PHINode &LCSSAPhi :
LoopExitBlock->phis())<br>
>>>> + if
(any_of(LCSSAPhi.incoming_values(),<br>
>>>> + [Phi](Value *V)
{ return V == Phi; }))<br>
>>>> +
LCSSAPhi.addIncoming(ExtractForPhiUsedOutsideLoop,<br>
>>>> LoopMiddleBlock);<br>
>>>> }<br>
>>>><br>
>>>> void
InnerLoopVectorizer::fixReduction(PHINode *Phi)
{<br>
>>>> @@ -4421,11 +4386,10 @@ void<br>
>>>>
InnerLoopVectorizer::fixReduction(PHINode *Phi)
{<br>
>>>> // We know that the loop
is in LCSSA form. We need to update<br>
>>>> the PHI nodes<br>
>>>> // in the exit blocks.
See comment on analogous loop in<br>
>>>> // fixFirstOrderRecurrence
for a more complete explaination of<br>
>>>> the logic.<br>
>>>> - if
(!Cost->requiresScalarEpilogue())<br>
>>>> - for (PHINode &LCSSAPhi
: LoopExitBlock->phis())<br>
>>>> - if
(any_of(LCSSAPhi.incoming_values(),<br>
>>>> -
[LoopExitInst](Value *V) { return V ==<br>
>>>> LoopExitInst; }))<br>
>>>> -
LCSSAPhi.addIncoming(ReducedPartRdx,
LoopMiddleBlock);<br>
>>>> + for (PHINode &LCSSAPhi :
LoopExitBlock->phis())<br>
>>>> + if
(any_of(LCSSAPhi.incoming_values(),<br>
>>>> +
[LoopExitInst](Value *V) { return V ==<br>
>>>> LoopExitInst; }))<br>
>>>> +
LCSSAPhi.addIncoming(ReducedPartRdx,
LoopMiddleBlock);<br>
>>>><br>
>>>> // Fix the scalar loop
reduction variable with the incoming<br>
>>>> reduction sum<br>
>>>> // from the vector body
and from the backedge value.<br>
>>>> @@ -8074,11 +8038,7 @@
BasicBlock<br>
>>>>
*EpilogueVectorizerMainLoop::emitMinimumIterationCountCheck(<br>
>>>><br>
>>>> // Update dominator for
Bypass & LoopExit.<br>
>>>>
DT->changeImmediateDominator(Bypass,
TCCheckBlock);<br>
>>>> - if
(!Cost->requiresScalarEpilogue())<br>
>>>> - // For loops with
multiple exits, there's no edge from the<br>
>>>> middle block<br>
>>>> - // to exit blocks (as
the epilogue must run) and thus no<br>
>>>> need to update<br>
>>>> - // the immediate
dominator of the exit blocks.<br>
>>>> -
DT->changeImmediateDominator(LoopExitBlock,
TCCheckBlock);<br>
>>>> +
DT->changeImmediateDominator(LoopExitBlock,
TCCheckBlock);<br>
>>>><br>
>>>>
LoopBypassBlocks.push_back(TCCheckBlock);<br>
>>>><br>
>>>> @@ -8142,12 +8102,7 @@<br>
>>>>
EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton()<br>
>>>> {<br>
>>>><br>
>>>>
DT->changeImmediateDominator(LoopScalarPreHeader,<br>
>>>>
EPI.EpilogueIterationCountCheck);<br>
>>>> - if
(!Cost->requiresScalarEpilogue())<br>
>>>> - // If there is an epilogue
which must run, there's no edge<br>
>>>> from the<br>
>>>> - // middle block to exit
blocks and thus no need to update the<br>
>>>> immediate<br>
>>>> - // dominator of the exit
blocks.<br>
>>>> -
DT->changeImmediateDominator(LoopExitBlock,<br>
>>>> -
EPI.EpilogueIterationCountCheck);<br>
>>>> +
DT->changeImmediateDominator(LoopExitBlock,<br>
>>>>
EPI.EpilogueIterationCountCheck);<br>
>>>><br>
>>>> // Keep track of bypass
blocks, as they feed start values to<br>
>>>> the induction<br>
>>>> // phis in the scalar loop
preheader.<br>
>>>><br>
>>>> diff --git<br>
>>>>
a/llvm/test/Transforms/LoopVectorize/first-order-recurrence-complex.ll<br>
>>>>
b/llvm/test/Transforms/LoopVectorize/first-order-recurrence-complex.ll<br>
>>>> index
ec280bf5d5e4..7d4a3c5c9935 100644<br>
>>>> ---<br>
>>>>
a/llvm/test/Transforms/LoopVectorize/first-order-recurrence-complex.ll<br>
>>>> +++<br>
>>>>
b/llvm/test/Transforms/LoopVectorize/first-order-recurrence-complex.ll<br>
>>>> @@ -471,9 +471,10 @@ define i16
@multiple_exit(i16* %p, i32 %n) {<br>
>>>> ; CHECK-NEXT:
[[TMP15:%.*]] = icmp eq i32 [[INDEX_NEXT]],<br>
>>>> [[N_VEC]]<br>
>>>> ; CHECK-NEXT: br i1
[[TMP15]], label [[MIDDLE_BLOCK:%.*]],<br>
>>>> label [[VECTOR_BODY]],
[[LOOP6:!llvm.loop !.*]]<br>
>>>> ; CHECK: middle.block:<br>
>>>> +; CHECK-NEXT: [[CMP_N:%.*]]
= icmp eq i32 [[TMP2]], [[N_VEC]]<br>
>>>> ; CHECK-NEXT:
[[VECTOR_RECUR_EXTRACT:%.*]] = extractelement<br>
>>>> <4 x i16> [[WIDE_LOAD]],
i32 3<br>
>>>> ; CHECK-NEXT:
[[VECTOR_RECUR_EXTRACT_FOR_PHI:%.*]] =<br>
>>>> extractelement <4 x i16>
[[WIDE_LOAD]], i32 2<br>
>>>> -; CHECK-NEXT: br label
[[SCALAR_PH]]<br>
>>>> +; CHECK-NEXT: br i1
[[CMP_N]], label [[IF_END:%.*]], label<br>
>>>> [[SCALAR_PH]]<br>
>>>> ; CHECK: <a
href="http://scalar.ph" moz-do-not-send="true"
rel="noreferrer" target="_blank">scalar.ph</a>:<br>
>>>> ; CHECK-NEXT:
[[SCALAR_RECUR_INIT:%.*]] = phi i16 [ 0,<br>
>>>> [[ENTRY:%.*]] ], [
[[VECTOR_RECUR_EXTRACT]], [[MIDDLE_BLOCK]] ]<br>
>>>> ; CHECK-NEXT:
[[BC_RESUME_VAL:%.*]] = phi i32 [ [[N_VEC]],<br>
>>>> [[MIDDLE_BLOCK]] ], [ 0,
[[ENTRY]] ]<br>
>>>> @@ -485,14 +486,14 @@ define
i16 @multiple_exit(i16* %p, i32 %n) {<br>
>>>> ; CHECK-NEXT: [[B:%.*]] =
getelementptr inbounds i16, i16*<br>
>>>> [[P]], i64 [[IPROM]]<br>
>>>> ; CHECK-NEXT:
[[REC_NEXT]] = load i16, i16* [[B]], align 2<br>
>>>> ; CHECK-NEXT: [[CMP:%.*]]
= icmp slt i32 [[I]], [[N]]<br>
>>>> -; CHECK-NEXT: br i1
[[CMP]], label [[FOR_BODY]], label<br>
>>>> [[IF_END:%.*]]<br>
>>>> +; CHECK-NEXT: br i1
[[CMP]], label [[FOR_BODY]], label [[IF_END]]<br>
>>>> ; CHECK: for.body:<br>
>>>> ; CHECK-NEXT: store i16
[[SCALAR_RECUR]], i16* [[B]], align 4<br>
>>>> ; CHECK-NEXT: [[INC]] =
add nsw i32 [[I]], 1<br>
>>>> ; CHECK-NEXT:
[[CMP2:%.*]] = icmp slt i32 [[I]], 2096<br>
>>>> ; CHECK-NEXT: br i1
[[CMP2]], label [[FOR_COND]], label<br>
>>>> [[IF_END]], [[LOOP7:!llvm.loop
!.*]]<br>
>>>> ; CHECK: if.end:<br>
>>>> -; CHECK-NEXT:
[[REC_LCSSA:%.*]] = phi i16 [ [[SCALAR_RECUR]],<br>
>>>> [[FOR_BODY]] ], [
[[SCALAR_RECUR]], [[FOR_COND]] ]<br>
>>>> +; CHECK-NEXT:
[[REC_LCSSA:%.*]] = phi i16 [ [[SCALAR_RECUR]],<br>
>>>> [[FOR_BODY]] ], [
[[SCALAR_RECUR]], [[FOR_COND]] ], [<br>
>>>>
[[VECTOR_RECUR_EXTRACT_FOR_PHI]],
[[MIDDLE_BLOCK]] ]<br>
>>>> ; CHECK-NEXT: ret i16
[[REC_LCSSA]]<br>
>>>> ;<br>
>>>> entry:<br>
>>>> @@ -557,9 +558,10 @@ define i16
@multiple_exit2(i16* %p, i32 %n) {<br>
>>>> ; CHECK-NEXT:
[[TMP15:%.*]] = icmp eq i32 [[INDEX_NEXT]],<br>
>>>> [[N_VEC]]<br>
>>>> ; CHECK-NEXT: br i1
[[TMP15]], label [[MIDDLE_BLOCK:%.*]],<br>
>>>> label [[VECTOR_BODY]],
[[LOOP8:!llvm.loop !.*]]<br>
>>>> ; CHECK: middle.block:<br>
>>>> +; CHECK-NEXT: [[CMP_N:%.*]]
= icmp eq i32 [[TMP2]], [[N_VEC]]<br>
>>>> ; CHECK-NEXT:
[[VECTOR_RECUR_EXTRACT:%.*]] = extractelement<br>
>>>> <4 x i16> [[WIDE_LOAD]],
i32 3<br>
>>>> ; CHECK-NEXT:
[[VECTOR_RECUR_EXTRACT_FOR_PHI:%.*]] =<br>
>>>> extractelement <4 x i16>
[[WIDE_LOAD]], i32 2<br>
>>>> -; CHECK-NEXT: br label
[[SCALAR_PH]]<br>
>>>> +; CHECK-NEXT: br i1
[[CMP_N]], label [[IF_END:%.*]], label<br>
>>>> [[SCALAR_PH]]<br>
>>>> ; CHECK: <a
href="http://scalar.ph" moz-do-not-send="true"
rel="noreferrer" target="_blank">scalar.ph</a>:<br>
>>>> ; CHECK-NEXT:
[[SCALAR_RECUR_INIT:%.*]] = phi i16 [ 0,<br>
>>>> [[ENTRY:%.*]] ], [
[[VECTOR_RECUR_EXTRACT]], [[MIDDLE_BLOCK]] ]<br>
>>>> ; CHECK-NEXT:
[[BC_RESUME_VAL:%.*]] = phi i32 [ [[N_VEC]],<br>
>>>> [[MIDDLE_BLOCK]] ], [ 0,
[[ENTRY]] ]<br>
>>>> @@ -571,14 +573,14 @@ define
i16 @multiple_exit2(i16* %p, i32 %n) {<br>
>>>> ; CHECK-NEXT: [[B:%.*]] =
getelementptr inbounds i16, i16*<br>
>>>> [[P]], i64 [[IPROM]]<br>
>>>> ; CHECK-NEXT:
[[REC_NEXT]] = load i16, i16* [[B]], align 2<br>
>>>> ; CHECK-NEXT: [[CMP:%.*]]
= icmp slt i32 [[I]], [[N]]<br>
>>>> -; CHECK-NEXT: br i1
[[CMP]], label [[FOR_BODY]], label<br>
>>>> [[IF_END:%.*]]<br>
>>>> +; CHECK-NEXT: br i1
[[CMP]], label [[FOR_BODY]], label [[IF_END]]<br>
>>>> ; CHECK: for.body:<br>
>>>> ; CHECK-NEXT: store i16
[[SCALAR_RECUR]], i16* [[B]], align 4<br>
>>>> ; CHECK-NEXT: [[INC]] =
add nsw i32 [[I]], 1<br>
>>>> ; CHECK-NEXT:
[[CMP2:%.*]] = icmp slt i32 [[I]], 2096<br>
>>>> ; CHECK-NEXT: br i1
[[CMP2]], label [[FOR_COND]], label<br>
>>>> [[IF_END]], [[LOOP9:!llvm.loop
!.*]]<br>
>>>> ; CHECK: if.end:<br>
>>>> -; CHECK-NEXT:
[[REC_LCSSA:%.*]] = phi i16 [ [[SCALAR_RECUR]],<br>
>>>> [[FOR_COND]] ], [ 10,
[[FOR_BODY]] ]<br>
>>>> +; CHECK-NEXT:
[[REC_LCSSA:%.*]] = phi i16 [ [[SCALAR_RECUR]],<br>
>>>> [[FOR_COND]] ], [ 10,
[[FOR_BODY]] ], [<br>
>>>>
[[VECTOR_RECUR_EXTRACT_FOR_PHI]],
[[MIDDLE_BLOCK]] ]<br>
>>>> ; CHECK-NEXT: ret i16
[[REC_LCSSA]]<br>
>>>> ;<br>
>>>> entry:<br>
>>>><br>
>>>> diff --git<br>
>>>>
a/llvm/test/Transforms/LoopVectorize/interleaved-accesses.ll<br>
>>>>
b/llvm/test/Transforms/LoopVectorize/interleaved-accesses.ll<br>
>>>> index
f0ba677348ab..0d4bdf0ecac3 100644<br>
>>>> ---
a/llvm/test/Transforms/LoopVectorize/interleaved-accesses.ll<br>
>>>> +++
b/llvm/test/Transforms/LoopVectorize/interleaved-accesses.ll<br>
>>>> @@ -447,7 +447,7 @@ define void
@even_load_static_tc(i32* noalias<br>
>>>> nocapture readonly %A, i32*
noalia<br>
>>>> ; CHECK-NEXT:
[[TMP6:%.*]] = icmp eq i64 [[INDEX_NEXT]], 508<br>
>>>> ; CHECK-NEXT: br i1
[[TMP6]], label [[MIDDLE_BLOCK:%.*]],<br>
>>>> label [[VECTOR_BODY]],
[[LOOP12:!llvm.loop !.*]]<br>
>>>> ; CHECK: middle.block:<br>
>>>> -; CHECK-NEXT: br label
[[SCALAR_PH]]<br>
>>>> +; CHECK-NEXT: br i1 false,
label [[FOR_COND_CLEANUP:%.*]],<br>
>>>> label [[SCALAR_PH]]<br>
>>>> ; CHECK: <a
href="http://scalar.ph" moz-do-not-send="true"
rel="noreferrer" target="_blank">scalar.ph</a>:<br>
>>>> ; CHECK-NEXT:
[[BC_RESUME_VAL:%.*]] = phi i64 [ 1016,<br>
>>>> [[MIDDLE_BLOCK]] ], [ 0,
[[ENTRY:%.*]] ]<br>
>>>> ; CHECK-NEXT: br label
[[FOR_BODY:%.*]]<br>
>>>> @@ -463,7 +463,7 @@ define void
@even_load_static_tc(i32* noalias<br>
>>>> nocapture readonly %A, i32*
noalia<br>
>>>> ; CHECK-NEXT: store i32
[[MUL]], i32* [[ARRAYIDX2]], align 4<br>
>>>> ; CHECK-NEXT:
[[INDVARS_IV_NEXT]] = add nuw nsw i64<br>
>>>> [[INDVARS_IV]], 2<br>
>>>> ; CHECK-NEXT: [[CMP:%.*]]
= icmp ult i64 [[INDVARS_IV]], 1022<br>
>>>> -; CHECK-NEXT: br i1
[[CMP]], label [[FOR_BODY]], label<br>
>>>> [[FOR_COND_CLEANUP:%.*]],
[[LOOP13:!llvm.loop !.*]]<br>
>>>> +; CHECK-NEXT: br i1
[[CMP]], label [[FOR_BODY]], label<br>
>>>> [[FOR_COND_CLEANUP]],
[[LOOP13:!llvm.loop !.*]]<br>
>>>> ;<br>
>>>> entry:<br>
>>>> br label %for.body<br>
>>>> @@ -528,7 +528,7 @@ define void
@even_load_dynamic_tc(i32* noalias<br>
>>>> nocapture readonly %A, i32*
noali<br>
>>>> ; CHECK-NEXT:
[[TMP12:%.*]] = icmp eq i64 [[INDEX_NEXT]],<br>
>>>> [[N_VEC]]<br>
>>>> ; CHECK-NEXT: br i1
[[TMP12]], label [[MIDDLE_BLOCK:%.*]],<br>
>>>> label [[VECTOR_BODY]],
[[LOOP14:!llvm.loop !.*]]<br>
>>>> ; CHECK: middle.block:<br>
>>>> -; CHECK-NEXT: br label
[[SCALAR_PH]]<br>
>>>> +; CHECK-NEXT: br i1 false,
label [[FOR_COND_CLEANUP:%.*]],<br>
>>>> label [[SCALAR_PH]]<br>
>>>> ; CHECK: <a
href="http://scalar.ph" moz-do-not-send="true"
rel="noreferrer" target="_blank">scalar.ph</a>:<br>
>>>> ; CHECK-NEXT:
[[BC_RESUME_VAL:%.*]] = phi i64 [ [[IND_END]],<br>
>>>> [[MIDDLE_BLOCK]] ], [ 0,
[[ENTRY:%.*]] ]<br>
>>>> ; CHECK-NEXT: br label
[[FOR_BODY:%.*]]<br>
>>>> @@ -544,7 +544,7 @@ define void
@even_load_dynamic_tc(i32* noalias<br>
>>>> nocapture readonly %A, i32*
noali<br>
>>>> ; CHECK-NEXT: store i32
[[MUL]], i32* [[ARRAYIDX2]], align 4<br>
>>>> ; CHECK-NEXT:
[[INDVARS_IV_NEXT]] = add nuw nsw i64<br>
>>>> [[INDVARS_IV]], 2<br>
>>>> ; CHECK-NEXT: [[CMP:%.*]]
= icmp ult i64 [[INDVARS_IV_NEXT]],<br>
>>>> [[N]]<br>
>>>> -; CHECK-NEXT: br i1
[[CMP]], label [[FOR_BODY]], label<br>
>>>> [[FOR_COND_CLEANUP:%.*]],
[[LOOP15:!llvm.loop !.*]]<br>
>>>> +; CHECK-NEXT: br i1
[[CMP]], label [[FOR_BODY]], label<br>
>>>> [[FOR_COND_CLEANUP]],
[[LOOP15:!llvm.loop !.*]]<br>
>>>> ;<br>
>>>> entry:<br>
>>>> br label %for.body<br>
>>>> @@ -973,7 +973,7 @@ define void
@PR27626_0(%pair.i32 *%p, i32 %z,<br>
>>>> i64 %n) {<br>
>>>> ; CHECK-NEXT:
[[TMP19:%.*]] = icmp eq i64 [[INDEX_NEXT]],<br>
>>>> [[N_VEC]]<br>
>>>> ; CHECK-NEXT: br i1
[[TMP19]], label [[MIDDLE_BLOCK:%.*]],<br>
>>>> label [[VECTOR_BODY]],
[[LOOP24:!llvm.loop !.*]]<br>
>>>> ; CHECK: middle.block:<br>
>>>> -; CHECK-NEXT: br label
[[SCALAR_PH]]<br>
>>>> +; CHECK-NEXT: br i1 false,
label [[FOR_END:%.*]], label<br>
>>>> [[SCALAR_PH]]<br>
>>>> ; CHECK: <a
href="http://scalar.ph" moz-do-not-send="true"
rel="noreferrer" target="_blank">scalar.ph</a>:<br>
>>>> ; CHECK-NEXT:
[[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]],<br>
>>>> [[MIDDLE_BLOCK]] ], [ 0,
[[ENTRY:%.*]] ]<br>
>>>> ; CHECK-NEXT: br label
[[FOR_BODY:%.*]]<br>
>>>> @@ -985,7 +985,7 @@ define void
@PR27626_0(%pair.i32 *%p, i32 %z,<br>
>>>> i64 %n) {<br>
>>>> ; CHECK-NEXT: store i32
[[Z]], i32* [[P_I_Y]], align 4<br>
>>>> ; CHECK-NEXT: [[I_NEXT]]
= add nuw nsw i64 [[I]], 1<br>
>>>> ; CHECK-NEXT:
[[COND:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]<br>
>>>> -; CHECK-NEXT: br i1
[[COND]], label [[FOR_BODY]], label<br>
>>>> [[FOR_END:%.*]],
[[LOOP25:!llvm.loop !.*]]<br>
>>>> +; CHECK-NEXT: br i1
[[COND]], label [[FOR_BODY]], label<br>
>>>> [[FOR_END]],
[[LOOP25:!llvm.loop !.*]]<br>
>>>> ; CHECK: for.end:<br>
>>>> ; CHECK-NEXT: ret void<br>
>>>> ;<br>
>>>> @@ -1066,7 +1066,7 @@ define
i32 @PR27626_1(%pair.i32 *%p, i64 %n) {<br>
>>>> ; CHECK-NEXT:
[[RDX_SHUF3:%.*]] = shufflevector <4 x
i32><br>
>>>> [[BIN_RDX]], <4 x i32>
poison, <4 x i32> <i32 1, i32 undef,
i32<br>
>>>> undef, i32 undef><br>
>>>> ; CHECK-NEXT:
[[BIN_RDX4:%.*]] = add <4 x i32>
[[BIN_RDX]],<br>
>>>> [[RDX_SHUF3]]<br>
>>>> ; CHECK-NEXT:
[[TMP19:%.*]] = extractelement <4 x i32><br>
>>>> [[BIN_RDX4]], i32 0<br>
>>>> -; CHECK-NEXT: br label
[[SCALAR_PH]]<br>
>>>> +; CHECK-NEXT: br i1 false,
label [[FOR_END:%.*]], label<br>
>>>> [[SCALAR_PH]]<br>
>>>> ; CHECK: <a
href="http://scalar.ph" moz-do-not-send="true"
rel="noreferrer" target="_blank">scalar.ph</a>:<br>
>>>> ; CHECK-NEXT:
[[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]],<br>
>>>> [[MIDDLE_BLOCK]] ], [ 0,
[[ENTRY:%.*]] ]<br>
>>>> ; CHECK-NEXT:
[[BC_MERGE_RDX:%.*]] = phi i32 [ [[TMP19]],<br>
>>>> [[MIDDLE_BLOCK]] ], [ 0,
[[ENTRY]] ]<br>
>>>> @@ -1081,9 +1081,10 @@ define
i32 @PR27626_1(%pair.i32 *%p, i64 %n) {<br>
>>>> ; CHECK-NEXT: [[TMP21]] =
add nsw i32 [[TMP20]], [[S]]<br>
>>>> ; CHECK-NEXT: [[I_NEXT]]
= add nuw nsw i64 [[I]], 1<br>
>>>> ; CHECK-NEXT:
[[COND:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]<br>
>>>> -; CHECK-NEXT: br i1
[[COND]], label [[FOR_BODY]], label<br>
>>>> [[FOR_END:%.*]],
[[LOOP27:!llvm.loop !.*]]<br>
>>>> +; CHECK-NEXT: br i1
[[COND]], label [[FOR_BODY]], label<br>
>>>> [[FOR_END]],
[[LOOP27:!llvm.loop !.*]]<br>
>>>> ; CHECK: for.end:<br>
>>>> -; CHECK-NEXT: ret i32
[[TMP21]]<br>
>>>> +; CHECK-NEXT: [[TMP22:%.*]]
= phi i32 [ [[TMP21]], [[FOR_BODY]]<br>
>>>> ], [ [[TMP19]],
[[MIDDLE_BLOCK]] ]<br>
>>>> +; CHECK-NEXT: ret i32
[[TMP22]]<br>
>>>> ;<br>
>>>> entry:<br>
>>>> br label %for.body<br>
>>>> @@ -1162,7 +1163,7 @@ define
void @PR27626_2(%pair.i32 *%p, i64 %n,<br>
>>>> i32 %z) {<br>
>>>> ; CHECK-NEXT:
[[TMP20:%.*]] = icmp eq i64 [[INDEX_NEXT]],<br>
>>>> [[N_VEC]]<br>
>>>> ; CHECK-NEXT: br i1
[[TMP20]], label [[MIDDLE_BLOCK:%.*]],<br>
>>>> label [[VECTOR_BODY]],
[[LOOP28:!llvm.loop !.*]]<br>
>>>> ; CHECK: middle.block:<br>
>>>> -; CHECK-NEXT: br label
[[SCALAR_PH]]<br>
>>>> +; CHECK-NEXT: br i1 false,
label [[FOR_END:%.*]], label<br>
>>>> [[SCALAR_PH]]<br>
>>>> ; CHECK: <a
href="http://scalar.ph" moz-do-not-send="true"
rel="noreferrer" target="_blank">scalar.ph</a>:<br>
>>>> ; CHECK-NEXT:
[[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]],<br>
>>>> [[MIDDLE_BLOCK]] ], [ 0,
[[ENTRY:%.*]] ]<br>
>>>> ; CHECK-NEXT: br label
[[FOR_BODY:%.*]]<br>
>>>> @@ -1176,7 +1177,7 @@ define
void @PR27626_2(%pair.i32 *%p, i64 %n,<br>
>>>> i32 %z) {<br>
>>>> ; CHECK-NEXT: store i32
[[TMP21]], i32* [[P_I_Y]], align 4<br>
>>>> ; CHECK-NEXT: [[I_NEXT]]
= add nuw nsw i64 [[I]], 1<br>
>>>> ; CHECK-NEXT:
[[COND:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]<br>
>>>> -; CHECK-NEXT: br i1
[[COND]], label [[FOR_BODY]], label<br>
>>>> [[FOR_END:%.*]],
[[LOOP29:!llvm.loop !.*]]<br>
>>>> +; CHECK-NEXT: br i1
[[COND]], label [[FOR_BODY]], label<br>
>>>> [[FOR_END]],
[[LOOP29:!llvm.loop !.*]]<br>
>>>> ; CHECK: for.end:<br>
>>>> ; CHECK-NEXT: ret void<br>
>>>> ;<br>
>>>> @@ -1263,7 +1264,7 @@ define
i32 @PR27626_3(%pair.i32 *%p, i64 %n,<br>
>>>> i32 %z) {<br>
>>>> ; CHECK-NEXT:
[[RDX_SHUF3:%.*]] = shufflevector <4 x
i32><br>
>>>> [[BIN_RDX]], <4 x i32>
poison, <4 x i32> <i32 1, i32 undef,
i32<br>
>>>> undef, i32 undef><br>
>>>> ; CHECK-NEXT:
[[BIN_RDX4:%.*]] = add <4 x i32>
[[BIN_RDX]],<br>
>>>> [[RDX_SHUF3]]<br>
>>>> ; CHECK-NEXT:
[[TMP22:%.*]] = extractelement <4 x i32><br>
>>>> [[BIN_RDX4]], i32 0<br>
>>>> -; CHECK-NEXT: br label
[[SCALAR_PH]]<br>
>>>> +; CHECK-NEXT: br i1 false,
label [[FOR_END:%.*]], label<br>
>>>> [[SCALAR_PH]]<br>
>>>> ; CHECK: <a
href="http://scalar.ph" moz-do-not-send="true"
rel="noreferrer" target="_blank">scalar.ph</a>:<br>
>>>> ; CHECK-NEXT:
[[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]],<br>
>>>> [[MIDDLE_BLOCK]] ], [ 0,
[[ENTRY:%.*]] ]<br>
>>>> ; CHECK-NEXT:
[[BC_MERGE_RDX:%.*]] = phi i32 [ [[TMP22]],<br>
>>>> [[MIDDLE_BLOCK]] ], [ 0,
[[ENTRY]] ]<br>
>>>> @@ -1281,9 +1282,10 @@ define
i32 @PR27626_3(%pair.i32 *%p, i64 %n,<br>
>>>> i32 %z) {<br>
>>>> ; CHECK-NEXT: [[TMP25]] =
add nsw i32 [[TMP24]], [[S]]<br>
>>>> ; CHECK-NEXT: [[I_NEXT]]
= add nuw nsw i64 [[I]], 1<br>
>>>> ; CHECK-NEXT:
[[COND:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]<br>
>>>> -; CHECK-NEXT: br i1
[[COND]], label [[FOR_BODY]], label<br>
>>>> [[FOR_END:%.*]],
[[LOOP31:!llvm.loop !.*]]<br>
>>>> +; CHECK-NEXT: br i1
[[COND]], label [[FOR_BODY]], label<br>
>>>> [[FOR_END]],
[[LOOP31:!llvm.loop !.*]]<br>
>>>> ; CHECK: for.end:<br>
>>>> -; CHECK-NEXT: ret i32
[[TMP25]]<br>
>>>> +; CHECK-NEXT: [[TMP26:%.*]]
= phi i32 [ [[TMP25]], [[FOR_BODY]]<br>
>>>> ], [ [[TMP22]],
[[MIDDLE_BLOCK]] ]<br>
>>>> +; CHECK-NEXT: ret i32
[[TMP26]]<br>
>>>> ;<br>
>>>> entry:<br>
>>>> br label %for.body<br>
>>>><br>
>>>> diff --git
a/llvm/test/Transforms/LoopVectorize/loop-form.ll<br>
>>>>
b/llvm/test/Transforms/LoopVectorize/loop-form.ll<br>
>>>> index
f32002fae2b6..91780789088b 100644<br>
>>>> ---
a/llvm/test/Transforms/LoopVectorize/loop-form.ll<br>
>>>> +++
b/llvm/test/Transforms/LoopVectorize/loop-form.ll<br>
>>>> @@ -146,14 +146,15 @@ define
void @early_exit(i16* %p, i32 %n) {<br>
>>>> ; CHECK-NEXT:
[[TMP10:%.*]] = icmp eq i32 [[INDEX_NEXT]],<br>
>>>> [[N_VEC]]<br>
>>>> ; CHECK-NEXT: br i1
[[TMP10]], label [[MIDDLE_BLOCK:%.*]],<br>
>>>> label [[VECTOR_BODY]],
[[LOOP4:!llvm.loop !.*]]<br>
>>>> ; CHECK: middle.block:<br>
>>>> -; CHECK-NEXT: br label
[[SCALAR_PH]]<br>
>>>> +; CHECK-NEXT: [[CMP_N:%.*]]
= icmp eq i32 [[TMP1]], [[N_VEC]]<br>
>>>> +; CHECK-NEXT: br i1
[[CMP_N]], label [[IF_END:%.*]], label<br>
>>>> [[SCALAR_PH]]<br>
>>>> ; CHECK: <a
href="http://scalar.ph" moz-do-not-send="true"
rel="noreferrer" target="_blank">scalar.ph</a>:<br>
>>>> ; CHECK-NEXT:
[[BC_RESUME_VAL:%.*]] = phi i32 [ [[N_VEC]],<br>
>>>> [[MIDDLE_BLOCK]] ], [ 0,
[[ENTRY:%.*]] ]<br>
>>>> ; CHECK-NEXT: br label
[[FOR_COND:%.*]]<br>
>>>> ; CHECK: for.cond:<br>
>>>> ; CHECK-NEXT: [[I:%.*]] =
phi i32 [ [[BC_RESUME_VAL]],<br>
>>>> [[SCALAR_PH]] ], [ [[INC:%.*]],
[[FOR_BODY:%.*]] ]<br>
>>>> ; CHECK-NEXT: [[CMP:%.*]]
= icmp slt i32 [[I]], [[N]]<br>
>>>> -; CHECK-NEXT: br i1
[[CMP]], label [[FOR_BODY]], label<br>
>>>> [[IF_END:%.*]]<br>
>>>> +; CHECK-NEXT: br i1
[[CMP]], label [[FOR_BODY]], label [[IF_END]]<br>
>>>> ; CHECK: for.body:<br>
>>>> ; CHECK-NEXT:
[[IPROM:%.*]] = sext i32 [[I]] to i64<br>
>>>> ; CHECK-NEXT: [[B:%.*]] =
getelementptr inbounds i16, i16*<br>
>>>> [[P]], i64 [[IPROM]]<br>
>>>> @@ -285,14 +286,15 @@ define
void @multiple_unique_exit(i16* %p,<br>
>>>> i32 %n) {<br>
>>>> ; CHECK-NEXT:
[[TMP11:%.*]] = icmp eq i32 [[INDEX_NEXT]],<br>
>>>> [[N_VEC]]<br>
>>>> ; CHECK-NEXT: br i1
[[TMP11]], label [[MIDDLE_BLOCK:%.*]],<br>
>>>> label [[VECTOR_BODY]],
[[LOOP6:!llvm.loop !.*]]<br>
>>>> ; CHECK: middle.block:<br>
>>>> -; CHECK-NEXT: br label
[[SCALAR_PH]]<br>
>>>> +; CHECK-NEXT: [[CMP_N:%.*]]
= icmp eq i32 [[TMP2]], [[N_VEC]]<br>
>>>> +; CHECK-NEXT: br i1
[[CMP_N]], label [[IF_END:%.*]], label<br>
>>>> [[SCALAR_PH]]<br>
>>>> ; CHECK: <a
href="http://scalar.ph" moz-do-not-send="true"
rel="noreferrer" target="_blank">scalar.ph</a>:<br>
>>>> ; CHECK-NEXT:
[[BC_RESUME_VAL:%.*]] = phi i32 [ [[N_VEC]],<br>
>>>> [[MIDDLE_BLOCK]] ], [ 0,
[[ENTRY:%.*]] ]<br>
>>>> ; CHECK-NEXT: br label
[[FOR_COND:%.*]]<br>
>>>> ; CHECK: for.cond:<br>
>>>> ; CHECK-NEXT: [[I:%.*]] =
phi i32 [ [[BC_RESUME_VAL]],<br>
>>>> [[SCALAR_PH]] ], [ [[INC:%.*]],
[[FOR_BODY:%.*]] ]<br>
>>>> ; CHECK-NEXT: [[CMP:%.*]]
= icmp slt i32 [[I]], [[N]]<br>
>>>> -; CHECK-NEXT: br i1
[[CMP]], label [[FOR_BODY]], label<br>
>>>> [[IF_END:%.*]]<br>
>>>> +; CHECK-NEXT: br i1
[[CMP]], label [[FOR_BODY]], label [[IF_END]]<br>
>>>> ; CHECK: for.body:<br>
>>>> ; CHECK-NEXT:
[[IPROM:%.*]] = sext i32 [[I]] to i64<br>
>>>> ; CHECK-NEXT: [[B:%.*]] =
getelementptr inbounds i16, i16*<br>
>>>> [[P]], i64 [[IPROM]]<br>
>>>> @@ -372,14 +374,17 @@ define
i32 @multiple_unique_exit2(i16* %p,<br>
>>>> i32 %n) {<br>
>>>> ; CHECK-NEXT:
[[TMP11:%.*]] = icmp eq i32 [[INDEX_NEXT]],<br>
>>>> [[N_VEC]]<br>
>>>> ; CHECK-NEXT: br i1
[[TMP11]], label [[MIDDLE_BLOCK:%.*]],<br>
>>>> label [[VECTOR_BODY]],
[[LOOP8:!llvm.loop !.*]]<br>
>>>> ; CHECK: middle.block:<br>
>>>> -; CHECK-NEXT: br label
[[SCALAR_PH]]<br>
>>>> +; CHECK-NEXT: [[CMP_N:%.*]]
= icmp eq i32 [[TMP2]], [[N_VEC]]<br>
>>>> +; CHECK-NEXT:
[[IND_ESCAPE:%.*]] = sub i32 [[N_VEC]], 1<br>
>>>> +; CHECK-NEXT:
[[IND_ESCAPE1:%.*]] = sub i32 [[N_VEC]], 1<br>
>>>> +; CHECK-NEXT: br i1
[[CMP_N]], label [[IF_END:%.*]], label<br>
>>>> [[SCALAR_PH]]<br>
>>>> ; CHECK: <a
href="http://scalar.ph" moz-do-not-send="true"
rel="noreferrer" target="_blank">scalar.ph</a>:<br>
>>>> ; CHECK-NEXT:
[[BC_RESUME_VAL:%.*]] = phi i32 [ [[N_VEC]],<br>
>>>> [[MIDDLE_BLOCK]] ], [ 0,
[[ENTRY:%.*]] ]<br>
>>>> ; CHECK-NEXT: br label
[[FOR_COND:%.*]]<br>
>>>> ; CHECK: for.cond:<br>
>>>> ; CHECK-NEXT: [[I:%.*]] =
phi i32 [ [[BC_RESUME_VAL]],<br>
>>>> [[SCALAR_PH]] ], [ [[INC:%.*]],
[[FOR_BODY:%.*]] ]<br>
>>>> ; CHECK-NEXT: [[CMP:%.*]]
= icmp slt i32 [[I]], [[N]]<br>
>>>> -; CHECK-NEXT: br i1
[[CMP]], label [[FOR_BODY]], label<br>
>>>> [[IF_END:%.*]]<br>
>>>> +; CHECK-NEXT: br i1
[[CMP]], label [[FOR_BODY]], label [[IF_END]]<br>
>>>> ; CHECK: for.body:<br>
>>>> ; CHECK-NEXT:
[[IPROM:%.*]] = sext i32 [[I]] to i64<br>
>>>> ; CHECK-NEXT: [[B:%.*]] =
getelementptr inbounds i16, i16*<br>
>>>> [[P]], i64 [[IPROM]]<br>
>>>> @@ -388,7 +393,7 @@ define i32
@multiple_unique_exit2(i16* %p, i32<br>
>>>> %n) {<br>
>>>> ; CHECK-NEXT:
[[CMP2:%.*]] = icmp slt i32 [[I]], 2096<br>
>>>> ; CHECK-NEXT: br i1
[[CMP2]], label [[FOR_COND]], label<br>
>>>> [[IF_END]], [[LOOP9:!llvm.loop
!.*]]<br>
>>>> ; CHECK: if.end:<br>
>>>> -; CHECK-NEXT:
[[I_LCSSA:%.*]] = phi i32 [ [[I]], [[FOR_BODY]]<br>
>>>> ], [ [[I]], [[FOR_COND]] ]<br>
>>>> +; CHECK-NEXT:
[[I_LCSSA:%.*]] = phi i32 [ [[I]], [[FOR_BODY]]<br>
>>>> ], [ [[I]], [[FOR_COND]] ], [
[[IND_ESCAPE1]], [[MIDDLE_BLOCK]] ]<br>
>>>> ; CHECK-NEXT: ret i32
[[I_LCSSA]]<br>
>>>> ;<br>
>>>> ; TAILFOLD-LABEL:
@multiple_unique_exit2(<br>
>>>> @@ -461,14 +466,15 @@ define
i32 @multiple_unique_exit3(i16* %p,<br>
>>>> i32 %n) {<br>
>>>> ; CHECK-NEXT:
[[TMP11:%.*]] = icmp eq i32 [[INDEX_NEXT]],<br>
>>>> [[N_VEC]]<br>
>>>> ; CHECK-NEXT: br i1
[[TMP11]], label [[MIDDLE_BLOCK:%.*]],<br>
>>>> label [[VECTOR_BODY]],
[[LOOP10:!llvm.loop !.*]]<br>
>>>> ; CHECK: middle.block:<br>
>>>> -; CHECK-NEXT: br label
[[SCALAR_PH]]<br>
>>>> +; CHECK-NEXT: [[CMP_N:%.*]]
= icmp eq i32 [[TMP2]], [[N_VEC]]<br>
>>>> +; CHECK-NEXT: br i1
[[CMP_N]], label [[IF_END:%.*]], label<br>
>>>> [[SCALAR_PH]]<br>
>>>> ; CHECK: <a
href="http://scalar.ph" moz-do-not-send="true"
rel="noreferrer" target="_blank">scalar.ph</a>:<br>
>>>> ; CHECK-NEXT:
[[BC_RESUME_VAL:%.*]] = phi i32 [ [[N_VEC]],<br>
>>>> [[MIDDLE_BLOCK]] ], [ 0,
[[ENTRY:%.*]] ]<br>
>>>> ; CHECK-NEXT: br label
[[FOR_COND:%.*]]<br>
>>>> ; CHECK: for.cond:<br>
>>>> ; CHECK-NEXT: [[I:%.*]] =
phi i32 [ [[BC_RESUME_VAL]],<br>
>>>> [[SCALAR_PH]] ], [ [[INC:%.*]],
[[FOR_BODY:%.*]] ]<br>
>>>> ; CHECK-NEXT: [[CMP:%.*]]
= icmp slt i32 [[I]], [[N]]<br>
>>>> -; CHECK-NEXT: br i1
[[CMP]], label [[FOR_BODY]], label<br>
>>>> [[IF_END:%.*]]<br>
>>>> +; CHECK-NEXT: br i1
[[CMP]], label [[FOR_BODY]], label [[IF_END]]<br>
>>>> ; CHECK: for.body:<br>
>>>> ; CHECK-NEXT:
[[IPROM:%.*]] = sext i32 [[I]] to i64<br>
>>>> ; CHECK-NEXT: [[B:%.*]] =
getelementptr inbounds i16, i16*<br>
>>>> [[P]], i64 [[IPROM]]<br>
>>>> @@ -477,7 +483,7 @@ define i32
@multiple_unique_exit3(i16* %p, i32<br>
>>>> %n) {<br>
>>>> ; CHECK-NEXT:
[[CMP2:%.*]] = icmp slt i32 [[I]], 2096<br>
>>>> ; CHECK-NEXT: br i1
[[CMP2]], label [[FOR_COND]], label<br>
>>>> [[IF_END]], [[LOOP11:!llvm.loop
!.*]]<br>
>>>> ; CHECK: if.end:<br>
>>>> -; CHECK-NEXT: [[EXIT:%.*]]
= phi i32 [ 0, [[FOR_COND]] ], [ 1,<br>
>>>> [[FOR_BODY]] ]<br>
>>>> +; CHECK-NEXT: [[EXIT:%.*]]
= phi i32 [ 0, [[FOR_COND]] ], [ 1,<br>
>>>> [[FOR_BODY]] ], [ 0,
[[MIDDLE_BLOCK]] ]<br>
>>>> ; CHECK-NEXT: ret i32
[[EXIT]]<br>
>>>> ;<br>
>>>> ; TAILFOLD-LABEL:
@multiple_unique_exit3(<br>
>>>> @@ -994,7 +1000,8 @@ define
void @scalar_predication(float* %addr) {<br>
>>>> ; CHECK-NEXT:
[[TMP10:%.*]] = icmp eq i64 [[INDEX_NEXT]], 200<br>
>>>> ; CHECK-NEXT: br i1
[[TMP10]], label [[MIDDLE_BLOCK:%.*]],<br>
>>>> label [[VECTOR_BODY]],
[[LOOP12:!llvm.loop !.*]]<br>
>>>> ; CHECK: middle.block:<br>
>>>> -; CHECK-NEXT: br label
[[SCALAR_PH]]<br>
>>>> +; CHECK-NEXT: [[CMP_N:%.*]]
= icmp eq i64 201, 200<br>
>>>> +; CHECK-NEXT: br i1
[[CMP_N]], label [[EXIT:%.*]], label<br>
>>>> [[SCALAR_PH]]<br>
>>>> ; CHECK: <a
href="http://scalar.ph" moz-do-not-send="true"
rel="noreferrer" target="_blank">scalar.ph</a>:<br>
>>>> ; CHECK-NEXT:
[[BC_RESUME_VAL:%.*]] = phi i64 [ 200,<br>
>>>> [[MIDDLE_BLOCK]] ], [ 0,
[[ENTRY:%.*]] ]<br>
>>>> ; CHECK-NEXT: br label
[[LOOP_HEADER:%.*]]<br>
>>>> @@ -1002,7 +1009,7 @@ define
void @scalar_predication(float* %addr) {<br>
>>>> ; CHECK-NEXT: [[IV:%.*]]
= phi i64 [ [[BC_RESUME_VAL]],<br>
>>>> [[SCALAR_PH]] ], [
[[IV_NEXT:%.*]], [[LOOP_LATCH:%.*]] ]<br>
>>>> ; CHECK-NEXT: [[GEP:%.*]]
= getelementptr float, float*<br>
>>>> [[ADDR]], i64 [[IV]]<br>
>>>> ; CHECK-NEXT:
[[EXITCOND_NOT:%.*]] = icmp eq i64 [[IV]], 200<br>
>>>> -; CHECK-NEXT: br i1
[[EXITCOND_NOT]], label [[EXIT:%.*]], label<br>
>>>> [[LOOP_BODY:%.*]]<br>
>>>> +; CHECK-NEXT: br i1
[[EXITCOND_NOT]], label [[EXIT]], label<br>
>>>> [[LOOP_BODY:%.*]]<br>
>>>> ; CHECK: loop.body:<br>
>>>> ; CHECK-NEXT:
[[TMP11:%.*]] = load float, float* [[GEP]],<br>
>>>> align 4<br>
>>>> ; CHECK-NEXT:
[[PRED:%.*]] = fcmp oeq float [[TMP11]],<br>
>>>> 0.000000e+00<br>
>>>> @@ -1088,7 +1095,8 @@ define
i32 @me_reduction(i32* %addr) {<br>
>>>> ; CHECK-NEXT:
[[RDX_SHUF:%.*]] = shufflevector <2 x i32><br>
>>>> [[TMP5]], <2 x i32>
poison, <2 x i32> <i32 1, i32 undef><br>
>>>> ; CHECK-NEXT:
[[BIN_RDX:%.*]] = add <2 x i32> [[TMP5]],<br>
>>>> [[RDX_SHUF]]<br>
>>>> ; CHECK-NEXT:
[[TMP7:%.*]] = extractelement <2 x i32><br>
>>>> [[BIN_RDX]], i32 0<br>
>>>> -; CHECK-NEXT: br label
[[SCALAR_PH]]<br>
>>>> +; CHECK-NEXT: [[CMP_N:%.*]]
= icmp eq i64 201, 200<br>
>>>> +; CHECK-NEXT: br i1
[[CMP_N]], label [[EXIT:%.*]], label<br>
>>>> [[SCALAR_PH]]<br>
>>>> ; CHECK: <a
href="http://scalar.ph" moz-do-not-send="true"
rel="noreferrer" target="_blank">scalar.ph</a>:<br>
>>>> ; CHECK-NEXT:
[[BC_RESUME_VAL:%.*]] = phi i64 [ 200,<br>
>>>> [[MIDDLE_BLOCK]] ], [ 0,
[[ENTRY:%.*]] ]<br>
>>>> ; CHECK-NEXT:
[[BC_MERGE_RDX:%.*]] = phi i32 [ 0, [[ENTRY]]<br>
>>>> ], [ [[TMP7]], [[MIDDLE_BLOCK]]
]<br>
>>>> @@ -1098,7 +1106,7 @@ define
i32 @me_reduction(i32* %addr) {<br>
>>>> ; CHECK-NEXT:
[[ACCUM:%.*]] = phi i32 [ [[BC_MERGE_RDX]],<br>
>>>> [[SCALAR_PH]] ], [
[[ACCUM_NEXT:%.*]], [[LOOP_LATCH]] ]<br>
>>>> ; CHECK-NEXT: [[GEP:%.*]]
= getelementptr i32, i32* [[ADDR]],<br>
>>>> i64 [[IV]]<br>
>>>> ; CHECK-NEXT:
[[EXITCOND_NOT:%.*]] = icmp eq i64 [[IV]], 200<br>
>>>> -; CHECK-NEXT: br i1
[[EXITCOND_NOT]], label [[EXIT:%.*]], label<br>
>>>> [[LOOP_LATCH]]<br>
>>>> +; CHECK-NEXT: br i1
[[EXITCOND_NOT]], label [[EXIT]], label<br>
>>>> [[LOOP_LATCH]]<br>
>>>> ; CHECK: loop.latch:<br>
>>>> ; CHECK-NEXT:
[[TMP8:%.*]] = load i32, i32* [[GEP]], align 4<br>
>>>> ; CHECK-NEXT:
[[ACCUM_NEXT]] = add i32 [[ACCUM]], [[TMP8]]<br>
>>>> @@ -1106,7 +1114,7 @@ define
i32 @me_reduction(i32* %addr) {<br>
>>>> ; CHECK-NEXT:
[[EXITCOND2_NOT:%.*]] = icmp eq i64 [[IV]], 400<br>
>>>> ; CHECK-NEXT: br i1
[[EXITCOND2_NOT]], label [[EXIT]], label<br>
>>>> [[LOOP_HEADER]],
[[LOOP15:!llvm.loop !.*]]<br>
>>>> ; CHECK: exit:<br>
>>>> -; CHECK-NEXT: [[LCSSA:%.*]]
= phi i32 [ 0, [[LOOP_HEADER]] ], [<br>
>>>> [[ACCUM_NEXT]], [[LOOP_LATCH]]
]<br>
>>>> +; CHECK-NEXT: [[LCSSA:%.*]]
= phi i32 [ 0, [[LOOP_HEADER]] ], [<br>
>>>> [[ACCUM_NEXT]], [[LOOP_LATCH]]
], [ [[TMP7]], [[MIDDLE_BLOCK]] ]<br>
>>>> ; CHECK-NEXT: ret i32
[[LCSSA]]<br>
>>>> ;<br>
>>>> ; TAILFOLD-LABEL:
@me_reduction(<br>
>>>><br>
>>>><br>
>>>><br>
>>>>
_______________________________________________<br>
>>>> llvm-commits mailing list<br>
>>>> <a
href="mailto:llvm-commits@lists.llvm.org"
moz-do-not-send="true" target="_blank">llvm-commits@lists.llvm.org</a><br>
>>>> <a
href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits"
moz-do-not-send="true" rel="noreferrer"
target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits</a><br>
>>>
_______________________________________________<br>
>>> llvm-commits mailing list<br>
>>> <a
href="mailto:llvm-commits@lists.llvm.org"
moz-do-not-send="true" target="_blank">llvm-commits@lists.llvm.org</a><br>
>>> <a
href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits"
moz-do-not-send="true" rel="noreferrer"
target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits</a><br>
>
_______________________________________________<br>
> llvm-commits mailing list<br>
> <a
href="mailto:llvm-commits@lists.llvm.org"
moz-do-not-send="true" target="_blank">llvm-commits@lists.llvm.org</a><br>
> <a
href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits"
moz-do-not-send="true" rel="noreferrer"
target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits</a></blockquote>
</div>
</div>
</blockquote>
</blockquote>
<div dir="ltr"> </div>
</div>
</blockquote>
</blockquote>
<div dir="ltr"> </div>
</div>
<br>
</blockquote>
</body>
</html>