<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Ayal,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<p style="font-size:11pt; font-family:Calibri,sans-serif; margin:0"><span style="color:black; font-size:12pt">>> %4 = icmp ule <4 x i32> %induction, %broadcast.splat</span></p>
<p style="font-size:11pt; font-family:Calibri,sans-serif; margin:0"><span style="color:black; font-size:12pt">>> call <4 x i16> @llvm.masked.load.v4i16.p0v4i16(<4 x i16>* %6, i32 2, <4 x i1> %4,</span></p>
<p style="font-size:11pt; font-family:Calibri,sans-serif; margin:0"><span style="color:black; font-size:12pt">>> </span></p>
<p style="font-size:11pt; font-family:Calibri,sans-serif; margin:0"><span style="color:black; font-size:12pt">>> </span></p>
<p style="font-size:11pt; font-family:Calibri,sans-serif; margin:0"><span style="color:black; font-size:12pt">>> Starting pattern matching from the icmp and %broadcast.splat, which is the BTC, we find %trip.count.minus.1 as the inner loop trip > count, but
we need to be certain about this.</span></p>
<p style="font-size:11pt; font-family:Calibri,sans-serif; margin:0"><span style="color:black; font-size:12pt">></span></p>
<p style="font-size:11pt; font-family:Calibri,sans-serif; margin:0"><span style="color:black; font-size:12pt">> - isn’t it enough to be certain that %induction is a VIV, and that %broadcast.splat is any uniform value that is invariant in the innermost loop?</span></p>
<p style="font-size:11pt; font-family:Calibri,sans-serif; margin:0"><span style="color:black; font-size:12pt">> From the perspective of an instruction being masked by an “if (VIV <= <X,X,…,X>)” compare, X is effectively “the BTC” as long as it is invariant,
right?</span></p>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks for pointing this out. This might be an alternative. I can't remember if we tried this and run into troubles, or if we missed this.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I still see advantages of the intrinsic:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
- It is still quite a bit more convenient to have this explicit, means jumping through less hoops, and a robust way to communicate this information through the optimisation pipeline.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
- It's good for explicit predication. Although that's a bit cheeky of me to say that, Simon is in a better position to comment.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
- Somewhat related I see advantages for users writing intrinsics as this allows a target independent way to express this, and with different vector extensions supporting this feature, that seems convenient and important.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
As get.active.lane.mask() in its current form seems a generally useful concept supporting different use-cases and targets, with no disadvantages, so this would still have my preference.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-size:12.0pt; color:black">> This leads to Simon’s question about having a vector load masked by an “if (VIV < X)” and an sdiv being unmasked, implying that X is the BTC for the load but not for the sdiv.
</span><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Correct me if I am wrong, but I still see this as something orthogonal to this intrinsic? And as I said before, agreed that explicit predication is a good thing. With auto-vectorisation, we rely on the vectoriser and its tail-folding technique: this creates
masked loads/stores and thus these are predicated, and leaves the data processing (vector) instruction unpredicated, that's the current model.
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
> <span style="font-size:12.0pt; color:black">Furthermore, a vector loop may in general have one instruction masked by an “if (VIV < X)” and another masked by an “if (VIV < Y)”; presumably only one (the larger?) can be taken care of using hardware predication,
leaving the other to perform the compare and masking in software. Also, if an instruction is masked by some combination involving both an “if (VIV < X)” and other parts, say “if (VIV < X) && (a[VIV] != 0)”, hardware predication can be used to take care of
the “if (VIV < X)” part, leaving the other parts to software?</span><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I haven't run many examples, but it looks like we either don't vectorise or tail-fold for these more complicated cases with different VIV conditions written in the input code.
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
But yes, I see that things could be made more general. As I wrote on the ticket, this intrinsic would be a good first step in this direction, and I don't think that this intrinsic proposal would inhibit any future extensions to this concept, or it would be
straightforward to adapt.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Cheers,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Sjoerd.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="appendonsend"></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Zaks, Ayal (Mobileye) <ayal.zaks@intel.com><br>
<b>Sent:</b> 23 May 2020 23:12<br>
<b>To:</b> Sjoerd Meijer <Sjoerd.Meijer@arm.com>; Eli Friedman <efriedma@quicinc.com>; simon.moll@emea.nec.com <simon.moll@emea.nec.com><br>
<b>Cc:</b> llvm-dev@lists.llvm.org <llvm-dev@lists.llvm.org><br>
<b>Subject:</b> RE: [llvm-dev] LV: predication</font>
<div> </div>
</div>
<div lang="EN-US">
<div class="x_WordSection1">
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">Thanks for the explanations.</span></p>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">Not sure I completely follow this part:</span></p>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">> %4 = icmp ule <4 x i32> %induction, %broadcast.splat</span></p>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">> call <4 x i16> @llvm.masked.load.v4i16.p0v4i16(<4 x i16>* %6, i32 2, <4 x i1> %4,</span></p>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">> </span></p>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">> </span></p>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">> Starting pattern matching from the icmp and %broadcast.splat, which is the BTC, we find %trip.count.minus.1 as the inner loop trip count, but we need to be certain about this.</span></p>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">- isn’t it enough to be certain that %induction is a VIV, and that %broadcast.splat is any uniform value that is invariant in the innermost loop?</span></p>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">From the perspective of an instruction being masked by an “if (VIV <= <X,X,…,X>)” compare, X is effectively “the BTC” as long as it is invariant, right?</span></p>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">This leads to Simon’s question about having a vector load masked by an “if (VIV < X)” and an sdiv being unmasked, implying that X is the BTC for the load but not for the sdiv. Furthermore, a vector loop may in general
have one instruction masked by an “if (VIV < X)” and another masked by an “if (VIV < Y)”; presumably only one (the larger?) can be taken care of using hardware predication, leaving the other to perform the compare and masking in software. Also, if an instruction
is masked by some combination involving both an “if (VIV < X)” and other parts, say “if (VIV < X) && (a[VIV] != 0)”, hardware predication can be used to take care of the “if (VIV < X)” part, leaving the other parts to software?</span></p>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">The intention here is to clarify the essential properties of the compares being sought, to hopefully simplify and generalize their recognition and that of “the BTC” for hardware predication. This alternative should
have several benefits mentioned earlier.</span></p>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
</p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0cm 0cm 0cm">
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<b>From:</b> Sjoerd Meijer <Sjoerd.Meijer@arm.com> <br>
<b>Sent:</b> Friday, May 22, 2020 00:14<br>
<b>To:</b> Zaks, Ayal (Mobileye) <ayal.zaks@intel.com>; Eli Friedman <efriedma@quicinc.com>; simon.moll@emea.nec.com<br>
<b>Cc:</b> llvm-dev@lists.llvm.org<br>
<b>Subject:</b> Re: [llvm-dev] LV: predication</p>
</div>
</div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
</p>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">> The compare of interest is clear, I think. It compares a Vector Induction Variable with a broadcasted loop invariant value, aka the BTC. Obtaining the latter operand is the goal, clearly, but to do so, the former
operand needs to be recognized as a VIV.</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">Yep, exactly that.</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">> What if this compare is not generated by LV’s fold-tail-by-masking transformation?</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">Not sure I completely follow this, because the whole point is that @llvm.get.active.lane.mask() will be generated by the LV and its tail-folding and masking. If a programmer is interfering and doing a manual fold,
it is okay if the LV tail folding doesn't get triggered (bad luck), but of course we shouldn't be emitting it incorrectly.</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">I feel that this is a slightly different discussion than yesterday. I.e., we don't need to discuss correctness because that should always be the case, and if we generate incorrect code than that is a problem, but
that is not a problem related to the proposed @llvm.get.active.lane.mask() intrinsic. But quickly checking the example we have:</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">%vec.ind = phi <4 x i32> [ <i32 0, i32 1, i32 2, i32 3>, %vector.ph ], [ %vec.ind.next, %vector.body ]</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">%4 = icmp ult <4 x i32> %vec.ind, <i32 998, i32 998, i32 998, i32 998></span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">and everything looks good here. Using the change D79100, we don't emit @llvm.get.active.lane.mask()</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">because we don't exactly have the case VIV <= BTC, here VIV is a vector phi and looks like this follows a different code path in the vectoriser. This is definitely a case we want to support too, but that's a different
story.</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">A previous concern was this inhibiting other things, but I don't see that. What we are changing is this original icmp:</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">%active.lane.mask =<4 x i1> icmp ult <4 x i32> %induction, <4 x i32> %broadcast.splat</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">%wide.masked.load = call <4 x i32> @llvm.masked.load.v4i32.p0v4i32(<4 x i32>* %3, i32 4, <4 x i1> %active.lane.mask, <4 x i32> undef)</span></p>
</div>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">with this:</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">%active.lane.mask = call <4 x i1> @llvm.get.active.lane.mask.v4i32(<4 x i32> %induction, <4 x i32> %broadcast.splat)</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">%wide.masked.load = call <4 x i32> @llvm.masked.load.v4i32.p0v4i32(<4 x i32>* %3, i32 4, <4 x i1> %active.lane.mask, <4 x i32> undef)</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">This @llvm.get.active.lane.mask() has very straightforward semantics, it is just a simple translation of the icmp, and also importantly, it is feeding another intrinsic: the masked load/store.
</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">As side node, after Simon's feedback we will change/simplify @llvm.get.active.lane.mask(), and let it operand on 2 scalar operands (the first induction value, and the scalar BTC, not splat), because that would help
other cases too, but none of this changes anything what I mentioned before, but could even simplify things more.</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p><span style="font-size:12.0pt; color:black">> If you can elaborate on the complicated cases and the difficulty they entail, possibly with concrete IR examples, a general robust way of recognizing the desired compare could perhaps be found. How are doubly
nested loops or reductions related here?</span><span style="color:black"></span></p>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">Our pattern match based approach breaks down for the following kind of examples. They are real-life filters and codes that we are looking at, and this one is slightly simplified example of that:</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> for (i = 0; i < N; i++) {</span></p>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> Sum = 0; </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> M = Size - i; </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> for (j = 0; j < M; j++)
</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> Sum += Input[j] * Input[j+i];
</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> Output[i] = Sum; </span></p>
</div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> }</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">We are vectorising the inner-loop and we need to know its BTC. Its loop upperbound M depends on outerloop i, which results in a recursive SCEV expression.</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> %trip.count.minus.1 = sub i32 %1, 1</span></p>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> %broadcast.splatinsert = insertelement <4 x i32> undef, i32 %trip.count.minus.1, i32 0</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> %broadcast.splat = shufflevector <4 x i32> %broadcast.splatinsert, <4 x i32> undef, <4 x i32> zeroinitializer</span></p>
</div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> br label %vector.body</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> vector.body: </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> ..</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> %4 = icmp ule <4 x i32> %induction, %broadcast.splat</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> call <4 x i16> @llvm.masked.load.v4i16.p0v4i16(<4 x i16>* %6, i32 2, <4 x i1> %4,</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">Starting pattern matching from the icmp and %broadcast.splat, which is the BTC, we find %trip.count.minus.1 as the inner loop trip count, but we need to be certain about this. This is where we use SCEV, and we check
whether the pattern matched IR definition matches with SCEV information. For simple 1d loops, the SCEV expression for the trip count has this general form:</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> (1 + ((-4 + (4 * ((3 + %Elems) /u 4))<nuw>) /u 4))<nuw><nsw></span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">where %Elems is the scalar trip count. For the example above that means if %Elems == %trip.count.minus.1, we know that we found the right trip count, and that's what we will use. But for this 2d example</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> (1 + ((-4 + (4 * ({(3 + %S),+,-1}<nw> /u 4))<nuw>) /u 4))<nuw><nsw></span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">First of all, this is a recursive SCEV, but problematic is that %S simply doesn't correspond to our inner loop trip count. We could still get it, but has all the disadvantages I mentioned earlier already, and it is
just a bit horrible, so we need something robust, and a simple way of passing this information on from the vectoriser to the backend.</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">I still might have skipped a few details here, but this is what it boils down to, and hopefully you've got a good impression of the problem.</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">Cheers,</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">Sjoerd.</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;text-align:center" align="center">
<hr width="98%" size="2" align="center">
</div>
<div id="x_divRplyFwdMsg">
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<b><span style="color:black">From:</span></b><span style="color:black"> Zaks, Ayal (Mobileye) <</span><a href="mailto:ayal.zaks@intel.com">ayal.zaks@intel.com</a><span style="color:black">><br>
<b>Sent:</b> 21 May 2020 18:44<br>
<b>To:</b> Sjoerd Meijer <</span><a href="mailto:Sjoerd.Meijer@arm.com">Sjoerd.Meijer@arm.com</a><span style="color:black">>; Eli Friedman <</span><a href="mailto:efriedma@quicinc.com">efriedma@quicinc.com</a><span style="color:black">><br>
<b>Cc:</b> </span><a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a><span style="color:black"> <</span><a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a><span style="color:black">><br>
<b>Subject:</b> RE: [llvm-dev] LV: predication</span> </p>
<div>
<p class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
</p>
</div>
</div>
<div>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt">> <span style="color:black">The compare that we are talking is the compare that compares the induction step and the backedge taken count, and this feeds the masked loads/stores.</span></span></p>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">The compare of interest is clear, I think. It compares a Vector Induction Variable with a broadcasted loop invariant value, aka the BTC. Obtaining the latter operand is the goal, clearly, but to do so, the former
operand needs to be recognized as a VIV.</span></p>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt"> </span></p>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt">What if this compare is not generated by LV’s fold-tail-by-masking transformation?</span></p>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt">For example, if a programmer folds the tail manually, by doing</span></p>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt"> for (int i=0; i<1000; i++) {</span></p>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt"> if (i<998) {</span></p>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt"> …</span></p>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt"> }</span></p>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt"> }</span></p>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt">which LV then vectorizes, generating the desired compare, treating it like any other compare, w/o recognizing that it’s the desired one?</span></p>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt">Or what if a vectorized loop with the desired compare originates from some front-end, such as OpenCL.</span></p>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt">Presumably, in all such cases this compare is still/equally of interest.</span></p>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt"> </span></p>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt">If you can elaborate on the complicated cases and the difficulty they entail, possibly with concrete IR examples, a general robust way of recognizing the desired compare could perhaps be found. How are doubly nested loops or reductions
related here?</span></p>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
</p>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
</p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0cm 0cm 0cm">
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<b>From:</b> Sjoerd Meijer <<a href="mailto:Sjoerd.Meijer@arm.com">Sjoerd.Meijer@arm.com</a>>
<br>
<b>Sent:</b> Thursday, May 21, 2020 01:17<br>
<b>To:</b> Zaks, Ayal (Mobileye) <<a href="mailto:ayal.zaks@intel.com">ayal.zaks@intel.com</a>>; Eli Friedman <<a href="mailto:efriedma@quicinc.com">efriedma@quicinc.com</a>><br>
<b>Cc:</b> <a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a><br>
<b>Subject:</b> Re: [llvm-dev] LV: predication</p>
</div>
</div>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
</p>
<div>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">Hi Ayal,</span></p>
</div>
<div>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">Let me start with commenting on this:</span></p>
</div>
<div>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">> A dedicated intrinsic that freezes the compare instruction, for no apparent reason, may potentially cripple subsequent passes from further optimizing the vectorized loop.</span></p>
</div>
<div>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">The point is we have a very good reason, which is that it passes on the right information on the backend, enabling opimisations as opposed to crippling them.</span></p>
</div>
<div>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">The compare that we are talking is the compare that compares the induction step and the backedge taken count, and this feeds the masked loads/stores. Thus, for example, we are not talking about the compare controlling
the backedge, and it is not affecting loop control. While it is undoubtedly true that there could optimisation that can't handle this particular icmp instruction, it is difficult to imagine for me at this point that being unable to analyse this icmp would
cripple things.</span></p>
</div>
<div>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">> Could you elaborate on these more complicated cases and the difficulty they entail?
</span></p>
</div>
<div>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">The problem that we are solving is that we need the scalar loop backedge taken count (BTC), or just the iteration count, of the original scalar loop for a given vector loop. Just to be clear, we do not only need the
vector iteration count, but again also the scalar loop Iteration Count (IC). We need this for a certain form of predication. This information, the scalar loop IC is produced by vectoriser, and is materialised in the form of the instructions that generate the
predicates for the masked loads/stores: this icmp with induction step and the scalar IC.</span></p>
</div>
<div>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">Our current approach works for simple cases, because we pattern match the IR, and look for the scalar IC in these icmps that feed masked loads/stores. To make sure we let's say don't accidentally pattern match a random
icmp, we compare this with SCEV information. Thus, we have to match up a SCEV expression with pattern matched IR. I could give IR examples, but hopefully it's easy to imagine that this pattern matching and matching up with SCEV info is becoming a bit horrible
for doubly nested loops or reductions. This icmp materliased as @llvm.get.active.lanes.mask(%IV, %BTC) avoids all of this, as we can just pick up %BTC in the backend.</span></p>
</div>
<div>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">As we are looking for the scalar loop iteration count, not the VIV, I don't think SCEV for vector loops is going to be helpful.</span></p>
</div>
<div>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">Please let me know if I can elaborate further, or if things are not clear.</span></p>
</div>
<div>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">Cheers,</span></p>
</div>
<div>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">Sjoerd.</span></p>
</div>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
</p>
<div>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;text-align:center" align="center">
<hr width="98%" size="2" align="center">
</div>
<div id="x_x_divRplyFwdMsg">
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<b><span style="color:black">From:</span></b><span style="color:black"> Zaks, Ayal (Mobileye) <</span><a href="mailto:ayal.zaks@intel.com">ayal.zaks@intel.com</a><span style="color:black">><br>
<b>Sent:</b> 20 May 2020 20:39<br>
<b>To:</b> Sjoerd Meijer <</span><a href="mailto:Sjoerd.Meijer@arm.com">Sjoerd.Meijer@arm.com</a><span style="color:black">>; Eli Friedman <</span><a href="mailto:efriedma@quicinc.com">efriedma@quicinc.com</a><span style="color:black">><br>
<b>Cc:</b> </span><a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a><span style="color:black"> <</span><a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a><span style="color:black">><br>
<b>Subject:</b> RE: [llvm-dev] LV: predication</span> </p>
<div>
<p class="x_xmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
</p>
</div>
</div>
<div>
<div>
<p class="x_xxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
I realize this discussion and <span style="font-size:12.0pt; color:black">D79100 </span>
have progressed, sorry, but could we revisit the “simplest path” of deriving the desired number?</p>
<p class="x_xxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
</p>
<p class="x_xxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
> This is what we are currently doing and works excellent for simpler cases. For the more complicated cases that we now what to handle as well, the pattern matching just becomes a bit too horrible, and it is fragile too.</p>
<p class="x_xxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
</p>
<p class="x_xxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
Could you elaborate on these more complicated cases and the difficulty they entail? Presumably a vector compare of a “Vector Induction Variable” with a broadcasted invariant value is sought, to be RAUW’d by a hardware configured mask. Is it the recognition
of VIV’s that’s becoming horrible and fragile? It may be generally useful to have a robust utility and/or analysis that identifies such VIV, effectively extending SCEV to reason about vector values, rather than complicating any backend pass. Middle-end passes
may find this information useful too, operating after LV, or on vector IR produced elsewhere.</p>
<p class="x_xxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
</p>
<p class="x_xxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
This is somewhat analogous to the argument about relying on a canonical induction variable versus employing SCEV to derive it, <a href="http://lists.llvm.org/pipermail/llvm-dev/2020-April/140572.html">http://lists.llvm.org/pipermail/llvm-dev/2020-April/140572.html</a>.</p>
<p class="x_xxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
</p>
<p class="x_xxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
A dedicated intrinsic that freezes the compare instruction, for no apparent reason, may potentially cripple subsequent passes from further optimizing the vectorized loop.</p>
<p class="x_xxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
</p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0cm 0cm 0cm">
<p class="x_xxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<b>From:</b> llvm-dev <<a href="mailto:llvm-dev-bounces@lists.llvm.org">llvm-dev-bounces@lists.llvm.org</a>>
<b>On Behalf Of </b>Sjoerd Meijer via llvm-dev<br>
<b>Sent:</b> Friday, May 01, 2020 21:54<br>
<b>To:</b> Eli Friedman <<a href="mailto:efriedma@quicinc.com">efriedma@quicinc.com</a>>; llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a>><br>
<b>Subject:</b> Re: [llvm-dev] LV: predication</p>
</div>
</div>
<p class="x_xxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
</p>
<div>
<p class="x_xxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">Hi Eli,</span></p>
</div>
<div>
<p class="x_xxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">> The problem with your proposal, as written, is that the vectorizer is producing the intrinsic. Because we don’t impose any ordering on optimizations before codegen, every optimization pass in LLVM would have to
be taught to preserve any @llvm.set.loop.elements.i32 whenever it makes any change. This is completely impractical because the intrinsic isn’t related to anything optimizations would normally look for: it’s a random intrinsic in the middle of nowhere.</span></p>
</div>
<div>
<p class="x_xxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">I do see that point. But is that also not the beauty of it? It just sits in the preheader, if gets removed, then so be it. And if it not recognised, then also no harm done?</span></p>
</div>
<div>
<p class="x_xxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">> Probably the simplest path to get this working is to derive the number of elements in the backend (in HardwareLoops, or your tail predication pass). You should be able to figure it from the masks used in the llvm.masked.load/store
instructions in the loop.</span></p>
</div>
<div>
<p class="x_xxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">This is what we are currently doing and works excellent for simpler cases. For the more complicated cases that we now what to handle as well, the pattern matching just becomes a bit too horrible, and it is fragile
too. All we need is the information that the vectoriser already has, and pass this on somehow.</span></p>
</div>
<div>
<p class="x_xxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">As I am really keen to simply our backend pass, would there be another way to pass this information on? If emitting an intrinsic is a blocker, could this be done with a loop annotation?</span></p>
</div>
<div>
<p class="x_xxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">Cheers,</span></p>
</div>
<div>
<p class="x_xxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">Sjoerd.</span></p>
</div>
<div class="x_MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;text-align:center" align="center">
<hr width="98%" size="2" align="center">
</div>
<div id="x_x_x_divRplyFwdMsg">
<p class="x_xxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<b><span style="color:black">From:</span></b><span style="color:black"> Eli Friedman <</span><a href="mailto:efriedma@quicinc.com">efriedma@quicinc.com</a><span style="color:black">><br>
<b>Sent:</b> 01 May 2020 19:30<br>
<b>To:</b> Sjoerd Meijer <</span><a href="mailto:Sjoerd.Meijer@arm.com">Sjoerd.Meijer@arm.com</a><span style="color:black">>; llvm-dev <</span><a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a><span style="color:black">><br>
<b>Subject:</b> RE: [llvm-dev] LV: predication</span> </p>
<div>
<p class="x_xxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
</p>
</div>
</div>
<div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
The problem with your proposal, as written, is that the vectorizer is producing the intrinsic. Because we don’t impose any ordering on optimizations before codegen, every optimization pass in LLVM would have to be taught to preserve any @llvm.set.loop.elements.i32
whenever it makes any change. This is completely impractical because the intrinsic isn’t related to anything optimizations would normally look for: it’s a random intrinsic in the middle of nowhere.</p>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
</p>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
Probably the simplest path to get this working is to derive the number of elements in the backend (in HardwareLoops, or your tail predication pass). You should be able to figure it from the masks used in the llvm.masked.load/store instructions in the loop.</p>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
</p>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
-Eli</p>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
</p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0cm 0cm 0cm">
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<b>From:</b> llvm-dev <<a href="mailto:llvm-dev-bounces@lists.llvm.org">llvm-dev-bounces@lists.llvm.org</a>>
<b>On Behalf Of </b>Sjoerd Meijer via llvm-dev<br>
<b>Sent:</b> Friday, May 1, 2020 3:50 AM<br>
<b>To:</b> <a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a><br>
<b>Subject:</b> [EXT] [llvm-dev] LV: predication</p>
</div>
</div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
</p>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">Hello,</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">We are working on predication for our vector extension (MVE). Since quite a few people are working on predication and different forms of it (e.g. SVE, RISC-V, NEC), I thought I would share what we would like to add
to the loop vectoriser. Hopefully it's just a minor one and not intrusive, but could be interesting and useful for others, and feedback on this is welcome of course.</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">TL;DR:</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">We would like the loop vectoriser to emit a new IR intrinsic for certain loops:</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> void @llvm.set.loop.elements.i32(i32 )</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">This represents the number of data elements processed by a vector loop, and will be emitted in the preheader block of the vector loop after querying TTI that the backend understands this intrinsic and that it should
be emitted for that loop. The vectoriser patch is available in D79100, and we pick this intrinsic up in the ARM backend here in D79175.</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">Context:</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">We are working on predication form that we call tail-predication: a vector hardwareloop has an implicit form of predication that sets active/inactive lanes for the last iteration of the vector loop. Thus, the scalar
epilogue loop (if there is one) is tail-folded and tail-predicated in the main vector body. And to support this, we need to know the number of data elements processed by the loop, which is used in the set up of a tail-predicated vector loop. This new intrinsic
communicates this information from the vectoriser to the codegen passes where we further lower these loops. In our case, we essentially let @llvm.set.loop.elements.i32 emit the trip count of the scalar loop, which represents the number of data elements processed.
Thus, we let the vectoriser emits both the scalar and vector loop trip count.</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">Although in a different stage in the optimisation pipeline, this is exactly what the generic HardwareLoop pass is doing to communicate its information to target specific codegen passes; it emits a few intrinsics to
mark a hardware loop. To illustrate this and also the new intrinsic, this is the flow and life of a tail-predicated vector loop using some heavily edited/reduced examples. First, the vectoriser emits the number of elements processed, and the loads/stores are
masked because tail-folding is applied:</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> vector.ph:</span></p>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> call void @llvm.set.loop.elements.i32(i32 %N)</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> br label %vector.body</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> vector.body:</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> call <4 x i32> @llvm.masked.load</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> call <4 x i32> @llvm.masked.load</span></p>
</div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> call void @llvm.masked.store</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> br i1 %12, label %.*, label %vector.body</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">After the HardwareLoop pass this is transformed into this, which adds the hardware loop intrinsics:</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> vector.ph:</span></p>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> call void @llvm.set.loop.elements.i32(i32 %N)</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> call void @llvm.set.loop.iterations.i32(i32 %5)</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> br label %vector.body</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> vector.body:</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> call <4 x i32> @llvm.masked.load</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> call <4 x i32> @llvm.masked.load</span></p>
</div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> call void @llvm.masked.store</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> call i32 @llvm.loop.decrement.reg</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> br i1 %12, label %.*, label %vector.body</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">We then pick this up in our tail-predication pass, remove @llvm.set.loop.elements intrinsic, and add @vctp which is our intrinsic that generates the mask of active/inactive lanes:</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> vector.ph:</span></p>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> call void @llvm.set.loop.iterations.i32(i32 %5)</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> br label %vector.body</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> vector.body:</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> call <4 x i1> @llvm.arm.mve.vctp32</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> call <4 x i32> @llvm.masked.load</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> call <4 x i32> @llvm.masked.load</span></p>
</div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> call void @llvm.masked.store</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> call i32 @llvm.loop.decrement.reg</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> br i1 %12, label %.*, label %vector.body</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">And this is then further lowered to a tail-predicted loop, or reverted to a 'normal' vector loop if some restrictions are not met.</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">Cheers,</span></p>
</div>
<div>
<p class="x_xxxmsonormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
<span style="font-size:12.0pt; color:black">Sjoerd.</span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p>---------------------------------------------------------------------<br>
Intel Israel (74) Limited</p>
<p>This e-mail and any attachments may contain confidential material for<br>
the sole use of the intended recipient(s). Any review or distribution<br>
by others is strictly prohibited. If you are not the intended<br>
recipient, please contact the sender and delete all copies.</p>
</div>
<p>---------------------------------------------------------------------<br>
Intel Israel (74) Limited</p>
<p>This e-mail and any attachments may contain confidential material for<br>
the sole use of the intended recipient(s). Any review or distribution<br>
by others is strictly prohibited. If you are not the intended<br>
recipient, please contact the sender and delete all copies.</p>
</div>
</body>
</html>