<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.xmsonormal, li.xmsonormal, div.xmsonormal
{mso-style-name:x_msonormal;
margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.xmsolistparagraph, li.xmsolistparagraph, div.xmsolistparagraph
{mso-style-name:x_msolistparagraph;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.xmsonormal0, li.xmsonormal0, div.xmsonormal0
{mso-style-name:x_msonormal0;
margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.xxmsonormal, li.xxmsonormal, div.xxmsonormal
{mso-style-name:x_xmsonormal;
margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.xxmsochpdefault, li.xxmsochpdefault, div.xxmsochpdefault
{mso-style-name:x_xmsochpdefault;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
p.xmsochpdefault, li.xmsochpdefault, div.xmsochpdefault
{mso-style-name:x_msochpdefault;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
span.xmsohyperlink
{mso-style-name:x_msohyperlink;
color:#0563C1;
text-decoration:underline;}
span.xmsohyperlinkfollowed
{mso-style-name:x_msohyperlinkfollowed;
color:#954F72;
text-decoration:underline;}
span.xxemailstyle19
{mso-style-name:x_xemailstyle19;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.xemailstyle23
{mso-style-name:x_emailstyle23;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.xemailstyle24
{mso-style-name:x_emailstyle24;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle31
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:12.0pt">> </span><span style="font-size:12.0pt;color:black">The compare that we are talking is the compare that compares the induction step and the backedge taken count, and this feeds the masked loads/stores.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">The compare of interest is clear, I think. It compares a Vector Induction Variable with a broadcasted loop invariant value, aka the BTC. Obtaining the latter operand is the goal, clearly, but to
do so, the former operand needs to be recognized as a VIV.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt">What if this compare is not generated by LV’s fold-tail-by-masking transformation?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt">For example, if a programmer folds the tail manually, by doing<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt"> for (int i=0; i<1000; i++) {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt"> if (i<998) {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt"> …<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt"> }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt"> }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt">which LV then vectorizes, generating the desired compare, treating it like any other compare, w/o recognizing that it’s the desired one?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt">Or what if a vectorized loop with the desired compare originates from some front-end, such as OpenCL.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt">Presumably, in all such cases this compare is still/equally of interest.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt">If you can elaborate</span><span style="font-size:12.0pt"> on the complicated cases and the difficulty they entail, possibly
</span><span style="font-size:12.0pt">with concrete IR examples, a general robust way of recognizing the desired compare could perhaps be found. How are doubly nested loops or reductions related here?<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>From:</b> Sjoerd Meijer <Sjoerd.Meijer@arm.com> <br>
<b>Sent:</b> Thursday, May 21, 2020 01:17<br>
<b>To:</b> Zaks, Ayal (Mobileye) <ayal.zaks@intel.com>; Eli Friedman <efriedma@quicinc.com><br>
<b>Cc:</b> llvm-dev@lists.llvm.org<br>
<b>Subject:</b> Re: [llvm-dev] LV: predication<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Hi Ayal,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Let me start with commenting on this:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">> A dedicated intrinsic that freezes the compare instruction, for no apparent reason, may potentially cripple subsequent passes from further optimizing the vectorized loop.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">The point is we have a very good reason, which is that it passes on the right information on the backend, enabling opimisations as opposed to crippling them.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">The compare that we are talking is the compare that compares the induction step and the backedge taken count, and this feeds the masked loads/stores. Thus, for example, we are not talking about
the compare controlling the backedge, and it is not affecting loop control. While it is undoubtedly true that there could optimisation that can't handle this particular icmp instruction, it is difficult to imagine for me at this point that being unable to
analyse this icmp would cripple things.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">> Could you elaborate on these more complicated cases and the difficulty they entail?
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">The problem that we are solving is that we need the scalar loop backedge taken count (BTC), or just the iteration count, of the original scalar loop for a given vector loop. Just to be clear, we
do not only need the vector iteration count, but again also the scalar loop Iteration Count (IC). We need this for a certain form of predication. This information, the scalar loop IC is produced by vectoriser, and is materialised in the form of the instructions
that generate the predicates for the masked loads/stores: this icmp with induction step and the scalar IC.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Our current approach works for simple cases, because we pattern match the IR, and look for the scalar IC in these icmps that feed masked loads/stores. To make sure we let's say don't accidentally
pattern match a random icmp, we compare this with SCEV information. Thus, we have to match up a SCEV expression with pattern matched IR. I could give IR examples, but hopefully it's easy to imagine that this pattern matching and matching up with SCEV info
is becoming a bit horrible for doubly nested loops or reductions. This icmp materliased as @llvm.get.active.lanes.mask(%IV, %BTC) avoids all of this, as we can just pick up %BTC in the backend.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">As we are looking for the scalar loop iteration count, not the VIV, I don't think SCEV for vector loops is going to be helpful.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Please let me know if I can elaborate further, or if things are not clear.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Cheers,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Sjoerd.<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="divRplyFwdMsg">
<p class="MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> Zaks, Ayal (Mobileye) <<a href="mailto:ayal.zaks@intel.com">ayal.zaks@intel.com</a>><br>
<b>Sent:</b> 20 May 2020 20:39<br>
<b>To:</b> Sjoerd Meijer <<a href="mailto:Sjoerd.Meijer@arm.com">Sjoerd.Meijer@arm.com</a>>; Eli Friedman <<a href="mailto:efriedma@quicinc.com">efriedma@quicinc.com</a>><br>
<b>Cc:</b> <a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a> <<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a>><br>
<b>Subject:</b> RE: [llvm-dev] LV: predication</span> <o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="xmsonormal">I realize this discussion and <span style="font-size:12.0pt;color:black">
D79100 </span>have progressed, sorry, but could we revisit the “simplest path” of deriving the desired number?<o:p></o:p></p>
<p class="xmsonormal"> <o:p></o:p></p>
<p class="xmsonormal">> This is what we are currently doing and works excellent for simpler cases. For the more complicated cases that we now what to handle as well, the pattern matching just becomes a bit too horrible, and it is fragile too.<o:p></o:p></p>
<p class="xmsonormal"> <o:p></o:p></p>
<p class="xmsonormal">Could you elaborate on these more complicated cases and the difficulty they entail? Presumably a vector compare of a “Vector Induction Variable” with a broadcasted invariant value is sought, to be RAUW’d by a hardware configured mask.
Is it the recognition of VIV’s that’s becoming horrible and fragile? It may be generally useful to have a robust utility and/or analysis that identifies such VIV, effectively extending SCEV to reason about vector values, rather than complicating any backend
pass. Middle-end passes may find this information useful too, operating after LV, or on vector IR produced elsewhere.<o:p></o:p></p>
<p class="xmsonormal"> <o:p></o:p></p>
<p class="xmsonormal">This is somewhat analogous to the argument about relying on a canonical induction variable versus employing SCEV to derive it, <a href="http://lists.llvm.org/pipermail/llvm-dev/2020-April/140572.html">http://lists.llvm.org/pipermail/llvm-dev/2020-April/140572.html</a>.<o:p></o:p></p>
<p class="xmsonormal"> <o:p></o:p></p>
<p class="xmsonormal">A dedicated intrinsic that freezes the compare instruction, for no apparent reason, may potentially cripple subsequent passes from further optimizing the vectorized loop.<o:p></o:p></p>
<p class="xmsonormal"> <o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="xmsonormal"><b>From:</b> llvm-dev <<a href="mailto:llvm-dev-bounces@lists.llvm.org">llvm-dev-bounces@lists.llvm.org</a>>
<b>On Behalf Of </b>Sjoerd Meijer via llvm-dev<br>
<b>Sent:</b> Friday, May 01, 2020 21:54<br>
<b>To:</b> Eli Friedman <<a href="mailto:efriedma@quicinc.com">efriedma@quicinc.com</a>>; llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a>><br>
<b>Subject:</b> Re: [llvm-dev] LV: predication<o:p></o:p></p>
</div>
</div>
<p class="xmsonormal"> <o:p></o:p></p>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black">Hi Eli,</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black">> The problem with your proposal, as written, is that the vectorizer is producing the intrinsic. Because we don’t impose any ordering on optimizations before codegen, every optimization pass
in LLVM would have to be taught to preserve any @llvm.set.loop.elements.i32 whenever it makes any change. This is completely impractical because the intrinsic isn’t related to anything optimizations would normally look for: it’s a random intrinsic in the
middle of nowhere.</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black">I do see that point. But is that also not the beauty of it? It just sits in the preheader, if gets removed, then so be it. And if it not recognised, then also no harm done?</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black">> Probably the simplest path to get this working is to derive the number of elements in the backend (in HardwareLoops, or your tail predication pass). You should be able to figure it from the
masks used in the llvm.masked.load/store instructions in the loop.</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black">This is what we are currently doing and works excellent for simpler cases. For the more complicated cases that we now what to handle as well, the pattern matching just becomes a bit too horrible,
and it is fragile too. All we need is the information that the vectoriser already has, and pass this on somehow.</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black">As I am really keen to simply our backend pass, would there be another way to pass this information on? If emitting an intrinsic is a blocker, could this be done with a loop annotation?</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black">Cheers,</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black">Sjoerd.</span><o:p></o:p></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="x_divRplyFwdMsg">
<p class="xmsonormal"><b><span style="color:black">From:</span></b><span style="color:black"> Eli Friedman <<a href="mailto:efriedma@quicinc.com">efriedma@quicinc.com</a>><br>
<b>Sent:</b> 01 May 2020 19:30<br>
<b>To:</b> Sjoerd Meijer <<a href="mailto:Sjoerd.Meijer@arm.com">Sjoerd.Meijer@arm.com</a>>; llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a>><br>
<b>Subject:</b> RE: [llvm-dev] LV: predication</span> <o:p></o:p></p>
<div>
<p class="xmsonormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="xxmsonormal">The problem with your proposal, as written, is that the vectorizer is producing the intrinsic. Because we don’t impose any ordering on optimizations before codegen, every optimization pass in LLVM would have to be taught to preserve
any @llvm.set.loop.elements.i32 whenever it makes any change. This is completely impractical because the intrinsic isn’t related to anything optimizations would normally look for: it’s a random intrinsic in the middle of nowhere.<o:p></o:p></p>
<p class="xxmsonormal"> <o:p></o:p></p>
<p class="xxmsonormal">Probably the simplest path to get this working is to derive the number of elements in the backend (in HardwareLoops, or your tail predication pass). You should be able to figure it from the masks used in the llvm.masked.load/store instructions
in the loop.<o:p></o:p></p>
<p class="xxmsonormal"> <o:p></o:p></p>
<p class="xxmsonormal">-Eli<o:p></o:p></p>
<p class="xxmsonormal"> <o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="xxmsonormal"><b>From:</b> llvm-dev <<a href="mailto:llvm-dev-bounces@lists.llvm.org">llvm-dev-bounces@lists.llvm.org</a>>
<b>On Behalf Of </b>Sjoerd Meijer via llvm-dev<br>
<b>Sent:</b> Friday, May 1, 2020 3:50 AM<br>
<b>To:</b> <a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a><br>
<b>Subject:</b> [EXT] [llvm-dev] LV: predication<o:p></o:p></p>
</div>
</div>
<p class="xxmsonormal"> <o:p></o:p></p>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black">Hello,</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black">We are working on predication for our vector extension (MVE). Since quite a few people are working on predication and different forms of it (e.g. SVE, RISC-V, NEC), I thought I would share what
we would like to add to the loop vectoriser. Hopefully it's just a minor one and not intrusive, but could be interesting and useful for others, and feedback on this is welcome of course.</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black">TL;DR:</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black">We would like the loop vectoriser to emit a new IR intrinsic for certain loops:</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> void @llvm.set.loop.elements.i32(i32 )</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black">This represents the number of data elements processed by a vector loop, and will be emitted in the preheader block of the vector loop after querying TTI that the backend understands this intrinsic
and that it should be emitted for that loop. The vectoriser patch is available in D79100, and we pick this intrinsic up in the ARM backend here in D79175.</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black">Context:</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black">We are working on predication form that we call tail-predication: a vector hardwareloop has an implicit form of predication that sets active/inactive lanes for the last iteration of the vector
loop. Thus, the scalar epilogue loop (if there is one) is tail-folded and tail-predicated in the main vector body. And to support this, we need to know the number of data elements processed by the loop, which is used in the set up of a tail-predicated vector
loop. This new intrinsic communicates this information from the vectoriser to the codegen passes where we further lower these loops. In our case, we essentially let @llvm.set.loop.elements.i32 emit the trip count of the scalar loop, which represents the number
of data elements processed. Thus, we let the vectoriser emits both the scalar and vector loop trip count.</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black">Although in a different stage in the optimisation pipeline, this is exactly what the generic HardwareLoop pass is doing to communicate its information to target specific codegen passes; it emits
a few intrinsics to mark a hardware loop. To illustrate this and also the new intrinsic, this is the flow and life of a tail-predicated vector loop using some heavily edited/reduced examples. First, the vectoriser emits the number of elements processed, and
the loads/stores are masked because tail-folding is applied:</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> vector.ph:</span><o:p></o:p></p>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> call void @llvm.set.loop.elements.i32(i32 %N)</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> br label %vector.body</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> vector.body:</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> call <4 x i32> @llvm.masked.load</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> call <4 x i32> @llvm.masked.load</span><o:p></o:p></p>
</div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> call void @llvm.masked.store</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> br i1 %12, label %.*, label %vector.body</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black">After the HardwareLoop pass this is transformed into this, which adds the hardware loop intrinsics:</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> vector.ph:</span><o:p></o:p></p>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> call void @llvm.set.loop.elements.i32(i32 %N)</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> call void @llvm.set.loop.iterations.i32(i32 %5)</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> br label %vector.body</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> vector.body:</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> call <4 x i32> @llvm.masked.load</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> call <4 x i32> @llvm.masked.load</span><o:p></o:p></p>
</div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> call void @llvm.masked.store</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> call i32 @llvm.loop.decrement.reg</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> br i1 %12, label %.*, label %vector.body</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black">We then pick this up in our tail-predication pass, remove @llvm.set.loop.elements intrinsic, and add @vctp which is our intrinsic that generates the mask of active/inactive lanes:</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> vector.ph:</span><o:p></o:p></p>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> call void @llvm.set.loop.iterations.i32(i32 %5)</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> br label %vector.body</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> vector.body:</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> call <4 x i1> @llvm.arm.mve.vctp32</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> call <4 x i32> @llvm.masked.load</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> call <4 x i32> @llvm.masked.load</span><o:p></o:p></p>
</div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> call void @llvm.masked.store</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> call i32 @llvm.loop.decrement.reg</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> br i1 %12, label %.*, label %vector.body</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black">And this is then further lowered to a tail-predicted loop, or reverted to a 'normal' vector loop if some restrictions are not met.</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black">Cheers,</span><o:p></o:p></p>
</div>
<div>
<p class="xxmsonormal"><span style="font-size:12.0pt;color:black">Sjoerd.</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p>---------------------------------------------------------------------<br>
Intel Israel (74) Limited</p>
<p>This e-mail and any attachments may contain confidential material for<br>
the sole use of the intended recipient(s). Any review or distribution<br>
by others is strictly prohibited. If you are not the intended<br>
recipient, please contact the sender and delete all copies.</p></body>
</html>