<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span
{mso-style-name:m_8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
.MsoPapDefault
{mso-style-type:export-only;
margin-left:46.2pt;
text-indent:-17.85pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:799957587;
mso-list-type:hybrid;
mso-list-template-ids:-1214880770 44585902 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><a name="_MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Ayal tried to enable interleave access, just by switching “false” to “true” and measured performance. He got significant performance
degradation on several benchmarks. Some benchmarks looked better, of course.<o:p></o:p></span></a></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Right now we concluded that interleaved access is not always beneficial for X86 or it requires additional target specific optimizations at least.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">First of all, we need more precise cost model that can estimate a real number of shuffles in each case. We also need to understand what to do with edge elements
in the vector if their loading is not required. We, probably, should issue a masked load in this case.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">As far as I remember, may be I’m wrong, vectorizer does not generate shuffles for interleave access. It generates a bunch of extracts and inserts that ought to
be coupled into shuffles after wise.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">It adds uncertainty to the cost modeling.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo1">
<![if !supportLists]><span style="font-family:"Calibri",sans-serif;color:#2F5496"><span style="mso-list:Ignore">-<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span dir="LTR"></span><b><i><span style="color:#2F5496"> Elena<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-right:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><a name="_____replyseparator"></a><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Michael Kuperstein [mailto:mkuper@google.com]
<br>
<b>Sent:</b> Friday, August 05, 2016 20:05<br>
<b>To:</b> Matthew Simpson <mssimpso@codeaurora.org><br>
<b>Cc:</b> Nema, Ashutosh <Ashutosh.Nema@amd.com>; Demikhovsky, Elena <elena.demikhovsky@intel.com>; Renato Golin <renato.golin@linaro.org>; Sanjay Patel <spatel@rotateright.com>; llvm-dev <llvm-dev@lists.llvm.org><br>
<b>Subject:</b> Re: [llvm-dev] enabling interleaved access loop vectorization<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:46.2pt;text-indent:-17.85pt">Regarding InterleavedAccessPass - sure, but proper strided/interleaved access optimization ought to have a positive impact even without target support.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:46.2pt;text-indent:-17.85pt">Case in point - Hal enabled it on PPC last September. An important difference vs. x86 seems to be that arbitrary shuffles are cheap on PPC, but, as I said below, I hope we can enable it on
x86 with a conservative cost function, and still get improvement.<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:46.2pt;text-indent:-17.85pt"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-left:46.2pt;text-indent:-17.85pt">On Fri, Aug 5, 2016 at 7:02 AM, Matthew Simpson <<a href="mailto:mssimpso@codeaurora.org" target="_blank">mssimpso@codeaurora.org</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-right:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Isn't our current interleaved access vectorization just a special case of the more general strided access proposal?
If so, from a development perspective, it might make sense to begin incorporating some of that work into the existing framework (with appropriate target hooks and costs). This could probably be done piecemeal rather than all at once.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Also, keep in mind that ARM/Aarch64 run an additional IR pass (InterleavedAccessPass) that matches the load/store
plus shuffle sequences that the vectorizer generates to target-specific instrinsics.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">-- Matt</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Nema, Ashutosh [mailto:<a href="mailto:Ashutosh.Nema@amd.com" target="_blank">Ashutosh.Nema@amd.com</a>]
<br>
<b>Sent:</b> Friday, August 05, 2016 7:21 AM<br>
<b>To:</b> Michael Kuperstein <<a href="mailto:mkuper@google.com" target="_blank">mkuper@google.com</a>>; Demikhovsky, Elena <<a href="mailto:elena.demikhovsky@intel.com" target="_blank">elena.demikhovsky@intel.com</a>><br>
<b>Cc:</b> Renato Golin <<a href="mailto:renato.golin@linaro.org" target="_blank">renato.golin@linaro.org</a>>; Sanjay Patel <<a href="mailto:spatel@rotateright.com" target="_blank">spatel@rotateright.com</a>>; Matthew Simpson <<a href="mailto:mssimpso@codeaurora.org" target="_blank">mssimpso@codeaurora.org</a>>;
llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>><br>
<b>Subject:</b> RE: [llvm-dev] enabling interleaved access loop vectorization</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi Michael,<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Sometime back I did some experiments with interleave vectorizer and did not found any degrade,<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">probably my tests/benchmarks are not extensive enough to cover much.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Elina is the right person to comment on it as she already experienced cases where it hinders performance.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">For interleave vectorizer on X86 we do not have any specific costing, it goes to BasicTTI where the costing is not appropriate(WRT X86).<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">It consider cost of extracts & inserts for extracting elements from a wide vector, which is really expensive.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">i.e. in your test case the cost of load associated with “in[i * 2]” is 10 (for VF4).<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Interleave vectorize will generate following instructions for it:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> %wide.vec = load <8 x i32>, <8 x i32>* %14, align 4, !tbaa !1, !alias.scope !5<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> %strided.vec = shufflevector <8 x i32> %wide.vec, <8 x i32> undef, <4 x i32> <i32 0, i32 2, i32 4, i32 6><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">For wide load it get cost as 2(as it has to generate 2 loads) but for extracting elements (shuffle operation) it get cost as 8 (4 for extract + 4 for insert).<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">The cost should be 3 here, 2 for loads & 1 for shuffle.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">To enable Interleave vectorizer on X86 we should implement a proper cost estimation.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Test you mentioned is indeed a candidate for Stride memory vectorization.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Regards,<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Ashutosh<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Michael Kuperstein [<a href="mailto:mkuper@google.com" target="_blank">mailto:mkuper@google.com</a>]
<br>
<b>Sent:</b> Friday, August 5, 2016 4:53 AM<br>
<b>To:</b> Demikhovsky, Elena <<a href="mailto:elena.demikhovsky@intel.com" target="_blank">elena.demikhovsky@intel.com</a>><br>
<b>Cc:</b> Renato Golin <<a href="mailto:renato.golin@linaro.org" target="_blank">renato.golin@linaro.org</a>>; Sanjay Patel <<a href="mailto:spatel@rotateright.com" target="_blank">spatel@rotateright.com</a>>; Nema, Ashutosh <<a href="mailto:Ashutosh.Nema@amd.com" target="_blank">Ashutosh.Nema@amd.com</a>>;
Matthew Simpson <<a href="mailto:mssimpso@codeaurora.org" target="_blank">mssimpso@codeaurora.org</a>>; llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>><br>
<b>Subject:</b> Re: [llvm-dev] enabling interleaved access loop vectorization</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi Elena,<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Circling back to this, do you know of any concrete cases where enabling interleaved access on x86 is unprofitable?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Right now, there are some cases where we lose significantly, because (a) we consider gathers (on architectures that don't have them) extremely expensive, so we won't vectorize them
at all without interleaved access, and (b) we have interleaved access turned off.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Consider something like this:<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">void foo(int *in, int *out) {<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> int i = 0;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> for (i = 0; i < 256; ++i) {<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> out[i] = in[i] + in[i + 1] + in[i + 2] + in[i * 2];<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> }<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">}<o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">We don't vectorize this loop at all, because we calculate the cost of the in[i * 2] gather to be 14 cycles per lane (!).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">This is an overestimate we need to fix, since the vectorized code is actually fairly decent - e.g. forcing vectorization, with SSE4.2, we get:<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">.LBB0_3: # %vector.body<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> # =>This Inner Loop Header: Depth=1<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">movdqu<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>(%rdi,%rax,4), %xmm3<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">movd<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>%xmm0, %rcx<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">movdqu<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>4(%rdi,%rcx,4), %xmm4<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">paddd<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>%xmm3, %xmm4<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">movdqu<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>8(%rdi,%rcx,4), %xmm3<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">paddd<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>%xmm4, %xmm3<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">movdqa<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>%xmm1, %xmm4<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">paddq<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>%xmm4, %xmm4<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">movdqa<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>%xmm0, %xmm5<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">paddq<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>%xmm5, %xmm5<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">movd<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>%xmm5, %rcx<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">pextrq<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>$1, %xmm5, %rdx<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">movd<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>%xmm4, %r8<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">pextrq<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>$1, %xmm4, %r9<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">movd<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>(%rdi,%rcx,4), %xmm4 # xmm4 = mem[0],zero,zero,zero<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">pinsrd<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>$1, (%rdi,%rdx,4), %xmm4<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">pinsrd<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>$2, (%rdi,%r8,4), %xmm4<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">pinsrd<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>$3, (%rdi,%r9,4), %xmm4<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">paddd<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>%xmm3, %xmm4<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">movdqu<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>%xmm4, (%rsi,%rax,4)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">addq<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>$4, %rax<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">paddq<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>%xmm2, %xmm0<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">paddq<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>%xmm2, %xmm1<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">cmpq<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>$256, %rax # imm = 0x100<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">jne<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>.LBB0_3<o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">But the real point is that with interleaved access enabled, we vectorize, and get:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">.LBB0_3: # %vector.body<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> # =>This Inner Loop Header: Depth=1<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">movdqu<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>(%rdi,%rcx), %xmm0<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">movdqu<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>4(%rdi,%rcx), %xmm1<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">movdqu<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>8(%rdi,%rcx), %xmm2<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">paddd<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>%xmm0, %xmm1<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">paddd<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>%xmm2, %xmm1<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">movdqu<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>(%rdi,%rcx,2), %xmm0<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">movdqu<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>16(%rdi,%rcx,2), %xmm2<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">pshufd<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>$132, %xmm2, %xmm2 # xmm2 = xmm2[0,1,0,2]<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">pshufd<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>$232, %xmm0, %xmm0 # xmm0 = xmm0[0,2,2,3]<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">pblendw<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>$240, %xmm2, %xmm0 # xmm0 = xmm0[0,1,2,3],xmm2[4,5,6,7]<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">paddd<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>%xmm1, %xmm0<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">movdqu<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>%xmm0, (%rsi,%rcx)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">cmpq<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>$992, %rcx # imm = 0x3E0<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">jne<span class="m8520143930005237927gmail-m-2363683590039465670m-6696729873126710771gmail-apple-tab-span">
</span>.LBB0_7<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">The performance I see out of the 3 versions (with a 500K-iteration outer loop):<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Scalar: 0m10.320s<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Vector (Non-interleaved): 0m8.054s<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Vector (Interleaved): 0m3.541s<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">This is far from being the perfect use case for interleaved access:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">1) There's no real interleaving, just one strided gather, so this would be better served by Ashutosh's full "strided access" proposal.<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">2) It looks like the actual move + shuffle sequence is not better, and even probably worse, than just inserting directly from memory - but it's still worthwhile because of how much
we save on the index computations.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Regardless of all that, the fact of the matter is that we get much better code by treating it as interleaved, and I think this may be a good enough motivation to enable it, unless
we significantly regress in other cases.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I was going to look at benchmarks to see if we get any regressions, but if you already have examples you're aware of, that would be great.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> Michael<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Thu, May 26, 2016 at 12:35 PM, Demikhovsky, Elena via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-right:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Interleaved access is not enabled on X86 yet.<br>
We looked at this feature and got into conclusion that interleaving (as loads + shuffles) is not always profitable on X86. We should provide the right cost which depends on number of shuffles. Number of shuffles depends on permutations (shuffle mask). And even
if we estimate the number of shuffles, the shuffles are not generated in-place. Vectorizer produces a long queue of "extracts" and "inserts" that hopefully will be coupled into shuffles on a later instcombine pass.<br>
<br>
- Elena<br>
<br>
<br>
>-----Original Message-----<br>
>From: Renato Golin [mailto:<a href="mailto:renato.golin@linaro.org" target="_blank">renato.golin@linaro.org</a>]<br>
>Sent: Thursday, May 26, 2016 21:25<br>
>To: Sanjay Patel <<a href="mailto:spatel@rotateright.com" target="_blank">spatel@rotateright.com</a>>; Demikhovsky, Elena<br>
><<a href="mailto:elena.demikhovsky@intel.com" target="_blank">elena.demikhovsky@intel.com</a>><br>
>Cc: llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>><br>
>Subject: Re: [llvm-dev] enabling interleaved access loop vectorization<br>
><br>
>On 26 May 2016 at 19:12, Sanjay Patel via llvm-dev <llvm-<br>
><a href="mailto:dev@lists.llvm.org" target="_blank">dev@lists.llvm.org</a>> wrote:<br>
>> Is there a compile-time and/or potential runtime cost that makes<br>
>> enableInterleavedAccessVectorization() default to 'false'?<br>
>><br>
>> I notice that this is set to true for ARM, AArch64, and PPC.<br>
>><br>
>> In particular, I'm wondering if there's a reason it's not enabled for<br>
>> x86 in relation to PR27881:<br>
>> <a href="https://llvm.org/bugs/show_bug.cgi?id=27881" target="_blank">https://llvm.org/bugs/show_bug.cgi?id=27881</a><br>
><br>
>Hi Sanjay,<br>
><br>
>The feature was originally developed for ARM's VLDn/VSTn instructions<br>
>and then extended to AArch64 and PPC, but not x86/64 yet.<br>
><br>
>I believe Elena was working on that, but needed to get the scatter/gather<br>
>intrinsics working first. I just copied her in case I'm wrong. :)<br>
><br>
>cheers,<br>
>--renato<br>
---------------------------------------------------------------------<br>
Intel Israel (74) Limited<br>
<br>
This e-mail and any attachments may contain confidential material for<br>
the sole use of the intended recipient(s). Any review or distribution<br>
by others is strictly prohibited. If you are not the intended<br>
recipient, please contact the sender and delete all copies.<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="margin-left:46.2pt;text-indent:-17.85pt"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</div>
<p>---------------------------------------------------------------------<br>
Intel Israel (74) Limited</p>
<p>This e-mail and any attachments may contain confidential material for<br>
the sole use of the intended recipient(s). Any review or distribution<br>
by others is strictly prohibited. If you are not the intended<br>
recipient, please contact the sender and delete all copies.</p></body>
</html>