<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
p.m-4988723288261583761msolistparagraph, li.m-4988723288261583761msolistparagraph, div.m-4988723288261583761msolistparagraph
{mso-style-name:m_-4988723288261583761msolistparagraph;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi Xin,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">The first phase of this work was uploaded for review in two parts:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><a href="https://reviews.llvm.org/D34393">https://reviews.llvm.org/D34393</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><a href="https://reviews.llvm.org/D34396">https://reviews.llvm.org/D34396</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">The reviews are currently pending.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Omer<o:p></o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></a></p>
<p class="MsoNormal"><a name="_____replyseparator"></a><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Xin Tong [mailto:trent.xin.tong@gmail.com]
<br>
<b>Sent:</b> Thursday, October 05, 2017 04:12<br>
<b>To:</b> Paparo Bivas, Omer <omer.paparo.bivas@intel.com><br>
<b>Cc:</b> llvm-dev <llvm-dev@lists.llvm.org>; Hal Finkel <hfinkel@anl.gov><br>
<b>Subject:</b> Re: [llvm-dev] RFC: Insertion of nops for performance stability<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Hi Paparo.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">I know this has been almost one year. I would like to know whether you have plan to contribute this back ? We have some performance instability that would benefit from a pass like this in Facebook. <o:p></o:p></p>
<div>
<p class="MsoNormal">Thanks.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">-Xin<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Mon, Nov 21, 2016 at 11:58 PM, Hal Finkel via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">
<hr size="3" width="100%" align="center" id="m_-4988723288261583761zwchr">
</span></div>
<blockquote style="border:none;border-left:solid #1010FF 1.5pt;padding:0cm 0cm 0cm 4.0pt;margin-left:3.75pt;margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-family:"Helvetica",sans-serif;color:black">From:
</span></b><span style="font-family:"Helvetica",sans-serif;color:black">"Paparo Bivas, Omer" <<a href="mailto:omer.paparo.bivas@intel.com" target="_blank">omer.paparo.bivas@intel.com</a>><br>
<b>To: </b>"Hal Finkel" <<a href="mailto:hfinkel@anl.gov" target="_blank">hfinkel@anl.gov</a>><br>
<b>Cc: </b><a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<b>Sent: </b>Monday, November 21, 2016 6:16:35 AM<br>
<b>Subject: </b>RE: [llvm-dev] RFC: Insertion of nops for performance stability<o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D">Hi Hal,</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D">Thanks for the reference. I’ve looked at <a name="m_-4988723288261583761__MailEndCompose">PPCBranchSelector and the PowerPC backend. It is very different
from the X86 architecture and unfortunately the way branch relaxation and alignment related issues are handled in PPC cannot be copied to X86. This is because:</a></span><span style="color:black"><o:p></o:p></span></p>
<p class="m-4988723288261583761msolistparagraph"><span style="font-family:"Helvetica",sans-serif;color:#1F497D">1.</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-family:"Helvetica",sans-serif;color:#1F497D">PPC instructions are of fixed length while X86 instructions are of variable length, and their length can change depending on features the target machine has. Not only that, but a relaxed
X86 instruction will differ in size from its non-relaxed version. The function TrargetInstrInfo::getInstSizeInBytes is not, and can’t be, implemented for X86. It follows that:</span><span style="font-family:"Helvetica",sans-serif;color:black"><o:p></o:p></span></p>
<p class="m-4988723288261583761msolistparagraph"><span style="font-family:"Helvetica",sans-serif;color:#1F497D">2.</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-family:"Helvetica",sans-serif;color:#1F497D">Target addresses of branches cannot be computed before the code emission in X86. That’s why X86 relies on the layout process in MCAssembler, which occurs after the encoding is done and (amongst
the rest) relaxes instructions that needs relaxation.</span><span style="font-family:"Helvetica",sans-serif;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D">For the same reasons, the nops insertion process also cannot occur until after the encoding phase.</span><span style="color:black"><o:p></o:p></span></p>
</div>
</blockquote>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">Okay, interesting. Thanks for explaining.<o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #1010FF 1.5pt;padding:0cm 0cm 0cm 4.0pt;margin-left:3.75pt;margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D">Regarding splitting blocks of code, that is essentially what I’m proposing, but at MC level (since this is the level I’m operating in). When needed,
a MCPerfNopFragment will be a separator between two MCFragments that contain code (e.g. MCDataFragments). However, a MCPerfNopFragment is not a MCAlignFragment and a block that follows a MCDataFragments will not (necessarily) be aligned as it could be wasteful
in terms of code size. Consider the example I provided. Aligning the fragment after the nops to 16B will cost one additional byte of nops in code size, and aligning it to 32B will cost 17 additional bytes of nops in code size. There’s no benefit in this extra
bytes.</span><span style="color:black"><o:p></o:p></span></p>
</div>
</blockquote>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">Makes sense.<o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #1010FF 1.5pt;padding:0cm 0cm 0cm 4.0pt;margin-left:3.75pt;margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D">As for filling the slots with other instructions, I think it won’t be right, because</span><span style="color:black"><o:p></o:p></span></p>
<p class="m-4988723288261583761msolistparagraph"><span style="font-family:"Helvetica",sans-serif;color:#1F497D">1.</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-family:"Helvetica",sans-serif;color:#1F497D">It will require some sort of branch prediction unit that will work in compilation time. Branch predicting is done in run time by the machine, and involves many parameters. In my opinion,
trying to imitate such a mechanism will create either very complex code or inaccurate predictions.</span><span style="font-family:"Helvetica",sans-serif;color:black"><o:p></o:p></span></p>
<p class="m-4988723288261583761msolistparagraph"><span style="font-family:"Helvetica",sans-serif;color:#1F497D">2.</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-family:"Helvetica",sans-serif;color:#1F497D">Execution of those other instructions can be very expensive compared to the execution of the nops. This means that wrong (compile-time) branch prediction will result in a performance penalty
that may even overshadow the benefit from the spacing those instructions provide.</span><span style="font-family:"Helvetica",sans-serif;color:black"><o:p></o:p></span></p>
</div>
</blockquote>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">We do have "compile-time branch prediction" in the form of static heuristics, and even better when available, profiling data. However, speculating in between two
branches to the same destination seems unlikely, so only the first situation would be relevant (DSB thrashing). There, however, just picking a different instruction ordering seems like a more-interesting option than speculation. Inserting nops, however, seems
like a reasonable general strategy regardless.<o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #1010FF 1.5pt;padding:0cm 0cm 0cm 4.0pt;margin-left:3.75pt;margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="m-4988723288261583761msolistparagraph"><span style="font-family:"Helvetica",sans-serif;color:#1F497D">3.</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-family:"Helvetica",sans-serif;color:#1F497D">As I mentioned, this process has to be done in the Assembler, after all the passes and the code emission. In my opinion, the Assembler phase is not a good time to change locations of instructions.</span><span style="font-family:"Helvetica",sans-serif;color:black"><o:p></o:p></span></p>
</div>
</blockquote>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">I agree with this. We shouldn't try to reschedule instructions in the assembler.<br>
<br>
-Hal<o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
<blockquote style="border:none;border-left:solid #1010FF 1.5pt;padding:0cm 0cm 0cm 4.0pt;margin-left:3.75pt;margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D">Thanks,</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D">Omer</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid windowtext 1.0pt;padding:3.0pt 0cm 0cm 0cm;border-color:-moz-use-text-color -moz-use-text-color">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><a name="m_-4988723288261583761______replyseparat"></a><b><span style="color:black">From:</span></b><span style="color:black"> Hal Finkel [mailto:<a href="mailto:hfinkel@anl.gov" target="_blank">hfinkel@anl.gov</a>]
<br>
<b>Sent:</b> Sunday, November 20, 2016 23:26<br>
<b>To:</b> Paparo Bivas, Omer <<a href="mailto:omer.paparo.bivas@intel.com" target="_blank">omer.paparo.bivas@intel.com</a>><br>
<b>Cc:</b> <a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<b>Subject:</b> Re: [llvm-dev] RFC: Insertion of nops for performance stability<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">
<hr size="3" width="100%" align="center">
</span></div>
<blockquote style="border:none;border-right:solid windowtext 1.5pt;padding:0cm 0cm 0cm 0cm;margin-left:3.75pt;margin-top:5.0pt;margin-bottom:5.0pt;border-color:-moz-use-text-color rgb(16,16,255) -moz-use-text-color -moz-use-text-color">
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><b><span style="font-family:"Helvetica",sans-serif;color:black">From:
</span></b><span style="font-family:"Helvetica",sans-serif;color:black">"Paparo Bivas, Omer" <<a href="mailto:omer.paparo.bivas@intel.com" target="_blank">omer.paparo.bivas@intel.com</a>><br>
<b>To: </b>"Hal Finkel" <<a href="mailto:hfinkel@anl.gov" target="_blank">hfinkel@anl.gov</a>><br>
<b>Cc: </b><a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<b>Sent: </b>Sunday, November 20, 2016 8:08:20 AM<br>
<b>Subject: </b>RE: [llvm-dev] RFC: Insertion of nops for performance stability</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D">Hi Hal,</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D">A pre-emit pass will indeed be preferable. I originally thought of it, too, however I could not figure out how can such a pass have an access to information
on instruction sizes and block alignments.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D">I know that for X86, at least, the branch relaxation is happening during the layout phase in the Assembler, where I plan to integrate the nop insertion
such that the new MCPerfNopFragment will just be yet another kind of fragment to handle when laying out all the fragments: layout for a relaxable branch is relaxing it if necessary, and layout for a MCPerfNopFragment will be computing the number of nops it
will contain.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D">Can you please refer me to a pre-emit pass that does something similar?</span><span style="color:black"><o:p></o:p></span></p>
</blockquote>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">Just as a quick example, the PowerPC branch relaxation pass, lib/Target/PowerPC/PPCBranchSelector.cpp,
uses TII->getInstSizeInBytes to determine instruction sizes and calculates alignment-based padding that will be inserted based on combining instruction-size information with the alignment information from each block (MBB.getAlignment()) and the containing
function (MBB.getParent()->getAlignment()). Perhaps I'm missing some complexity, but I'd think that nops could be inserted using a very similar algorithm.<br>
<br>
Two other thoughts:<br>
<br>
1. You can "insert nops" in some (many?) relevant cases by splitting the block and adjusting the block's alignment.<br>
2. You might want to fill the slots with other instructions which can be speculated from successor blocks.<br>
<br>
Thanks again,<br>
Hal</span><span style="color:black"><o:p></o:p></span></p>
<blockquote style="border:none;border-right:solid windowtext 1.5pt;padding:0cm 0cm 0cm 0cm;margin-left:3.75pt;margin-top:5.0pt;margin-bottom:5.0pt;border-color:-moz-use-text-color rgb(16,16,255) -moz-use-text-color -moz-use-text-color">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D">Thanks,</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D">Omer</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid windowtext 1.0pt;padding:3.0pt 0cm 0cm 0cm;border-color:-moz-use-text-color">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="color:black">From:</span></b><span style="color:black"> Hal Finkel [<a href="mailto:hfinkel@anl.gov" target="_blank">mailto:hfinkel@anl.gov</a>]
<br>
<b>Sent:</b> Friday, November 18, 2016 20:57<br>
<b>To:</b> Paparo Bivas, Omer <<a href="mailto:omer.paparo.bivas@intel.com" target="_blank">omer.paparo.bivas@intel.com</a>><br>
<b>Cc:</b> <a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<b>Subject:</b> Re: [llvm-dev] RFC: Insertion of nops for performance stability<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">
<hr size="3" width="100%" align="center">
</span></div>
<blockquote style="border:none;border-left:solid windowtext 1.5pt;padding:0cm 0cm 0cm 0cm;margin-left:3.75pt;margin-top:5.0pt;margin-bottom:5.0pt;border-color:-moz-use-text-color rgb(16,16,255) -moz-use-text-color -moz-use-text-color">
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><b><span style="font-family:"Helvetica",sans-serif;color:black">From:
</span></b><span style="font-family:"Helvetica",sans-serif;color:black">"Paparo Bivas, Omer via llvm-dev" <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>><br>
<b>To: </b><a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<b>Sent: </b>Thursday, November 17, 2016 3:55:43 AM<br>
<b>Subject: </b>[llvm-dev] RFC: Insertion of nops for performance stability</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black">Hi all,<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black">These days I am working on a feature designed to insert nops for IA code generation that will provide performance improvements and performance stability.
This feature will not affect other architectures. It will, however, set up an infrastructure for other architectures to do the same, if ever needed.<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black">Here are some examples for cases in which nops can improve performance:<o:p></o:p></span></p>
<p class="m-4988723288261583761msolistparagraph"><span style="font-family:"Helvetica",sans-serif;color:black">1.</span><span style="font-size:7.0pt;color:black">
</span><span style="font-family:"Helvetica",sans-serif;color:black">DSB (Decoded Stream Buffer) Thrashing.<o:p></o:p></span></p>
<p class="m-4988723288261583761msolistparagraph"><span style="font-family:"Helvetica",sans-serif;color:black">DSB is a cache for uops that have been decoded. It is limited to 3 ways per 32B window; 4 or more ways will cause a performance degradation. This can
be avoided by aligning with nops. See Zia Ansari's presentation for more information:
<a href="http://schd.ws/hosted_files/llvmdevelopersmeetingbay2016/d9/LLVM-Conference-US-2016-Code-Alignment.pdf" target="_blank">
http://schd.ws/hosted_files/llvmdevelopersmeetingbay2016/d9/LLVM-Conference-US-2016-Code-Alignment.pdf</a><o:p></o:p></span></p>
<p class="m-4988723288261583761msolistparagraph"><span style="font-family:"Helvetica",sans-serif;color:black">2.</span><span style="font-size:7.0pt;color:black">
</span><span style="font-family:"Helvetica",sans-serif;color:black">Two or more branches with the same target in a single instruction window may cause poor branch prediction.<o:p></o:p></span></p>
<p class="m-4988723288261583761msolistparagraph"><span style="font-family:"Helvetica",sans-serif;color:black">Our recommendation to programmers is to try and keep 2 branches out of the same 16B chunk if they both go to the same target. From the manual:<br>
“Avoid putting two conditional branch instructions in a loop so that both have the same branch target address and, at the same time, belong to (i.e. have their last bytes’ addresses within) the same 16-byte aligned code block.”<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black">Let’s see an example for the second case. Consider the following code:<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black">int y;</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black">int x;</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black">int foo(int i, int m, int p, int q, int *p1, int *p2)</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black">{</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> if (<span style="background:lime">i+*p1 == p</span> ||
<span style="background:aqua">i-*p2 > m</span> || <span style="background:silver">
i == q</span>) {</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> y++;</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> x++;</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> }</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> return 0;</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black">}</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black">The two last || clauses of the if will be translated to two conditional jumps to the same target. The generated code will look like so:<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black">foo:</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> 0: 48 8b 44 24 28 movq 40(%rsp), %rax</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> 5: 8b 00 movl (%rax), %eax</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> 7: 01 c8 addl %ecx, %eax</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> 9: 44 39 c0 cmpl %r8d, %eax</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black">
<span style="background:lime">c: 75 0f jne 15 <foo+0x1D></span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> e: ff 05 00 00 00 00 incl (%rip)</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> 14: ff 05 00 00 00 00 incl (%rip)</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> 1a: 31 c0 xorl %eax, %eax</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> 1c: c3 retq</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> 1d: 44 39 c9 cmpl %r9d, %ecx</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black">
<span style="background:silver">20: 74 ec je -20 <foo+0xE></span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> 22: 48 8b 44 24 30 movq 48(%rsp), %rax</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> 27: 2b 08 subl (%rax), %ecx</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> 29: 39 d1 cmpl %edx, %ecx</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black">
<span style="background:aqua">2b: 7f e1 jg -31 <foo+0xE></span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> 2d: 31 c0 xorl %eax, %eax</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> 2f: c3 retq</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black">Note: the first if clause jump is the jne instruction at 0x0C, the second if clause jump is the jg instruction at 0x2B and the third if clause jump is
the je instruction at 0x20. Also note that the jg and je share a 16 byte window, which is exactly the situation we wish to avoid (consider the case in which foo is called from inside a loop. This will cause performance penalty).<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black">The generated code after my changes will look like so:<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black">foo:</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> 0: 48 8b 44 24 28 movq 40(%rsp), %rax</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> 5: 8b 00 movl (%rax), %eax</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> 7: 01 c8 addl %ecx, %eax</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> 9: 44 39 c0 cmpl %r8d, %eax</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> c: 75 0f jne 15 <foo+0x1D></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> e: ff 05 00 00 00 00 incl (%rip)</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> 14: ff 05 00 00 00 00 incl (%rip)</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> 1a: 31 c0 xorl %eax, %eax</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> 1c: c3 retq</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> 1d: 44 39 c9 cmpl %r9d, %ecx</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> 20: 74 ec je -20 <foo+0xE></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> 22: 48 8b 44 24 30 movq 48(%rsp), %rax</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> 27: 2b 08 subl (%rax), %ecx</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> 29: 39 d1 cmpl %edx, %ecx</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black">
<span style="background:yellow">2b: 0f 1f 40 00 nopl (%rax)</span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> 2f: 7f dd jg -35 <foo+0xE></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> 31: 31 c0 xorl %eax, %eax</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New";color:black"> 33: c3 retq</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black">The only change is the nopl at 0x2B, before the jg, which pushes the jg instruction to the next 16 byte window, and thus avoids the undesired situation.
Note that, in this case, in order to push an instruction to the next window it is sufficient to push its last byte to the next window.<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black">Due to the nature of those performance nops, which often rely on the layout of the code (e.g. number of instructions in a 16/32B window) that is only known
in assembly time, the insertion is divided to two phases:<o:p></o:p></span></p>
<p class="m-4988723288261583761msolistparagraph"><span style="font-family:"Helvetica",sans-serif;color:black">1.</span><span style="font-size:7.0pt;color:black">
</span><span style="font-family:"Helvetica",sans-serif;color:black">Marking "suspicious" places while encoding, i.e. hotspots that may cause a performance penalty, during the encoding.<o:p></o:p></span></p>
<p class="m-4988723288261583761msolistparagraph"><span style="font-family:"Helvetica",sans-serif;color:black">2.</span><span style="font-size:7.0pt;color:black">
</span><span style="font-family:"Helvetica",sans-serif;color:black">Computing the actual number (zero or greater) of required nops in order to avoid a performance penalty.<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black">In order to achieve that, I am introducing a new kind of MCFragment named MCPerfNopFragment. It will be the connecting link between the two phases:<o:p></o:p></span></p>
<p class="m-4988723288261583761msolistparagraph"><span style="font-family:"Helvetica",sans-serif;color:black">1.</span><span style="font-size:7.0pt;color:black">
</span><span style="font-family:"Helvetica",sans-serif;color:black">During encoding, the object streamer (MCObjectStreamer) will query the backend for the need of a MCPerfNopFragment before every encoded instruction. If required, a MCPerfNopFragment will be
created and inserted to the MCSection.<o:p></o:p></span></p>
<p class="m-4988723288261583761msolistparagraph"><span style="font-family:"Helvetica",sans-serif;color:black">2.</span><span style="font-size:7.0pt;color:black">
</span><span style="font-family:"Helvetica",sans-serif;color:black">During the assembly phase:<o:p></o:p></span></p>
<p class="m-4988723288261583761msolistparagraph" style="margin-left:72.0pt"><span style="font-family:"Helvetica",sans-serif;color:black">a.</span><span style="font-size:7.0pt;color:black">
</span><span style="font-family:"Helvetica",sans-serif;color:black">When computing the fragment size in MCAssembler::computeFragmentSize the Assembler will consult the backend and will provide it the layout in order to determine the required size of the fragment.
Note that the computed size may change from call to call, similarly to the process of computing the size of a MCAlignFragment.<o:p></o:p></span></p>
<p class="m-4988723288261583761msolistparagraph" style="margin-left:72.0pt"><span style="font-family:"Helvetica",sans-serif;color:black">b.</span><span style="font-size:7.0pt;color:black">
</span><span style="font-family:"Helvetica",sans-serif;color:black">When writing the fragment the assembler will again use the backend in order to write the required number of nops.<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black">
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto" id="m_-4988723288261583761DWT5917">
<span style="color:black">Comments and questions are welcome.<o:p></o:p></span></p>
</blockquote>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">Do you need to insert the nops as such a late stage? We have other components that insert nops during post-RA
scheduling, for example. Even later, nops could be inserted during a pre-emit pass (similar to where other backends do branch relaxation/selection). The branch relaxation process also relies on knowing the layout of the code, but gathers this information from
the available information on instruction sizes, block alignments, etc.<br>
<br>
-Hal</span><span style="color:black"><o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid windowtext 1.5pt;padding:0cm 0cm 0cm 0cm;margin-left:3.75pt;margin-top:5.0pt;margin-bottom:5.0pt;border-color:-moz-use-text-color rgb(16,16,255) -moz-use-text-color -moz-use-text-color">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black">Omer<o:p></o:p></span></p>
<p><span style="font-family:"Helvetica",sans-serif;color:black">---------------------------------------------------------------------<br>
Intel Israel (74) Limited<o:p></o:p></span></p>
<p><span style="font-family:"Helvetica",sans-serif;color:black">This e-mail and any attachments may contain confidential material for<br>
the sole use of the intended recipient(s). Any review or distribution<br>
by others is strictly prohibited. If you are not the intended<br>
recipient, please contact the sender and delete all copies.<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Helvetica",sans-serif;color:black"><br>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a></span><span style="color:black"><o:p></o:p></span></p>
</blockquote>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"><br>
<br>
<br>
-- </span><span style="color:black"><o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">Hal Finkel<br>
Lead, Compiler Technology and Programming Languages<br>
Leadership Computing Facility<br>
Argonne National Laboratory</span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
<p><span style="font-family:"Helvetica",sans-serif;color:black">---------------------------------------------------------------------<br>
Intel Israel (74) Limited<o:p></o:p></span></p>
<p><span style="font-family:"Helvetica",sans-serif;color:black">This e-mail and any attachments may contain confidential material for<br>
the sole use of the intended recipient(s). Any review or distribution<br>
by others is strictly prohibited. If you are not the intended<br>
recipient, please contact the sender and delete all copies.<o:p></o:p></span></p>
</blockquote>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"><br>
<br>
<br>
-- </span><span style="color:black"><o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">Hal Finkel<br>
Lead, Compiler Technology and Programming Languages<br>
Leadership Computing Facility<br>
Argonne National Laboratory</span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
</div>
<p><span style="font-family:"Helvetica",sans-serif;color:black">---------------------------------------------------------------------<br>
Intel Israel (74) Limited<o:p></o:p></span></p>
<p><span style="font-family:"Helvetica",sans-serif;color:black">This e-mail and any attachments may contain confidential material for<br>
the sole use of the intended recipient(s). Any review or distribution<br>
by others is strictly prohibited. If you are not the intended<br>
recipient, please contact the sender and delete all copies.<o:p></o:p></span></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"><br>
<br>
<br>
-- <o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">Hal Finkel<br>
Lead, Compiler Technology and Programming Languages<br>
Leadership Computing Facility<br>
Argonne National Laboratory<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<p class="MsoNormal">Software Engineer - Compiler Toolchain<br>
Employee of Facebook Inc.<o:p></o:p></p>
</div>
</div>
</div>
</div>
<p>---------------------------------------------------------------------<br>
Intel Israel (74) Limited</p>
<p>This e-mail and any attachments may contain confidential material for<br>
the sole use of the intended recipient(s). Any review or distribution<br>
by others is strictly prohibited. If you are not the intended<br>
recipient, please contact the sender and delete all copies.</p></body>
</html>