<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.m5872875340768778888m-5486578834148932290m3162267875614969241m854065992349427772m1563025543004182786gmail-
{mso-style-name:m_5872875340768778888m_-5486578834148932290m_3162267875614969241m_854065992349427772m_1563025543004182786gmail-;}
span.hoenzb
{mso-style-name:hoenzb;}
span.m5872875340768778888hoenzb
{mso-style-name:m_5872875340768778888hoenzb;}
span.m5872875340768778888m-5486578834148932290hoenzb
{mso-style-name:m_5872875340768778888m_-5486578834148932290hoenzb;}
span.m5872875340768778888m-5486578834148932290m3162267875614969241m854065992349427772hoenzb
{mso-style-name:m_5872875340768778888m_-5486578834148932290m_3162267875614969241m_854065992349427772hoenzb;}
span.m5872875340768778888m-5486578834148932290m3162267875614969241m854065992349427772m1563025543004182786gmail-hoenzb
{mso-style-name:m_5872875340768778888m_-5486578834148932290m_3162267875614969241m_854065992349427772m_1563025543004182786gmail-hoenzb;}
span.EmailStyle24
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>Michael,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'> I have tried this patch with adaptation to our own linker and it works great. I would like to see this implemented on master.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>Please let me know if I can do anything to facilitate this.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>Sergei<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><b><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>From:</span></b><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'> llvm-dev [mailto:llvm-dev-bounces@lists.llvm.org] <b>On Behalf Of </b>Xinliang David Li via llvm-dev<br><b>Sent:</b> Thursday, June 15, 2017 5:38 PM<br><b>To:</b> Sean Silva <chisophugis@gmail.com><br><b>Cc:</b> LLVMDev <llvm-dev@lists.llvm.org>; haojiewang@google.com<br><b>Subject:</b> Re: [llvm-dev] [RFC] Profile guided section layout<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>On Thu, Jun 15, 2017 at 2:39 PM, Sean Silva <<a href="mailto:chisophugis@gmail.com" target="_blank">chisophugis@gmail.com</a>> wrote:<o:p></o:p></p><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>On Thu, Jun 15, 2017 at 2:33 PM, Xinliang David Li <<a href="mailto:xinliangli@gmail.com" target="_blank">xinliangli@gmail.com</a>> wrote:<o:p></o:p></p><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>On Thu, Jun 15, 2017 at 2:30 PM, Sean Silva <<a href="mailto:chisophugis@gmail.com" target="_blank">chisophugis@gmail.com</a>> wrote:<o:p></o:p></p><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>On Thu, Jun 15, 2017 at 11:09 AM, Xinliang David Li via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<o:p></o:p></p><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>On Thu, Jun 15, 2017 at 10:55 AM, Michael Spencer via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<o:p></o:p></p><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><div><div><div><div><p class=MsoNormal>On Thu, Jun 15, 2017 at 10:08 AM, Tobias Edler von Koch <<a href="mailto:tobias@codeaurora.org" target="_blank">tobias@codeaurora.org</a>> wrote:<o:p></o:p></p></div></div><div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><p class=MsoNormal>Hi Michael,<br><br>This is cool stuff, thanks for sharing!<br><br><span class="m5872875340768778888m-5486578834148932290m3162267875614969241m854065992349427772m1563025543004182786gmail-">On 06/15/2017 11:51 AM, Michael Spencer via llvm-dev wrote:<o:p></o:p></span></p><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><p class=MsoNormal>The first is a new llvm pass which uses branch frequency info to get counts for each call instruction and then adds a module flags metatdata table of function -> function edges along with their counts.<br><br>The second takes the module flags metadata and writes it into a .note.llvm.callgraph section in the object file. This currently just dumps it as text, but could save space by reusing the string table.<o:p></o:p></p></blockquote><p class=MsoNormal>Have you considered reading the profile in the linker and extracting that information directly from the profile? The profile should contain call sites and their sample counts and you could match these up with relocations (calls) in the section?<o:p></o:p></p></blockquote></div></div></div></blockquote><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>The main reason is that IPO transformations such as inlining and clonining will change the hotness of functions, so the original profile can not be directly for the purpose of function layout. There is a similar support in Gold plugin for Google GCC.<o:p></o:p></p></div></div></div></div></blockquote><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Will this cause issues with ThinLTO? E.g. the thinlto backends are doing inlining of imported functions. Do we have a mechanism for those decisions to be reflected in a global profile for the linker to look at?<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>In theory the thinlto backends can keep a history of their IPO decisions in metadata or something and emit a section for the linker to aggregate and reconstruct an accurate global profile, but that seems relatively invasive. <o:p></o:p></p></div></div></div></div></blockquote><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Yes, it will cause problems which is also known to GCC's LIPO. We have an intern working on that problem :)<o:p></o:p></p></div></div></div></div></blockquote><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Nice! What approach is being used to solve it?<o:p></o:p></p></div><div><p class=MsoNormal><span style='color:#888888'><o:p> </o:p></span></p></div></div></div></div></blockquote><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>The method and results will be shared at some point. One approach is what you mentioned which uses meta data and pass them to linker plugin. Another way to be explored is to let thin link phase to make a very quick cross module inlinining decisions globally and teach the backend inliner to honor the results . The global inline decisions are not intended to replace the backend inline analysis. The global analysis can for instance, 1) expose linker GC opportunities; 2) can focus on hot functions with few callsites or very few really hot callsites; 3) identify a single module that has the most # of calls to a function and assign that module to be be final info updater; 4) decide the post-link hotness for those functions whose decisions are made at thin-link time.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>David<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><div><div><div><div><p class=MsoNormal><span style='color:#888888'>-- Sean Silva<o:p></o:p></span></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><div><div><div><div><p class=MsoNormal><span style='color:#888888'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='color:#888888'>David<o:p></o:p></span></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><div><div><div><div><p class=MsoNormal><span style='color:#888888'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='color:#888888'>-- Sean Silva<o:p></o:p></span></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><div><div><div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>David<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><div><div><div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>I did this using IR PGO instead of sample PGO so the profile data can only be applied in the same place in the pipeline it is generated. Even for sample based this would be complicated as the linker would actually need to generate machine basic blocks from sections to be able to accurately match sample counts to relocations, as there may be cold calls in hot functions.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>It may be useful however for the linker to directly accept an externally generated call graph profile. The current approach can actually do this by embedding it into an extra object file.<o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><p class=MsoNormal style='margin-bottom:12.0pt'><span class="m5872875340768778888m-5486578834148932290m3162267875614969241m854065992349427772m1563025543004182786gmail-"><o:p> </o:p></span></p><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><p class=MsoNormal>It doesn't currently work for LTO as the llvm pass needs to be run after all inlining decisions have been made and LTO codegen has to be done with -ffunction-sections.<o:p></o:p></p></blockquote><p class=MsoNormal>So this is just an implementation issue, right? You can make LTO run with -ffunction-sections (by setting TargetOptions.FunctionSections=true) and insert your pass in the appropriate place in the pipeline.<o:p></o:p></p></blockquote><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Yeah, just an implementation issue. Just need to build the pass pipeline differently for LTO and add a way to do -ffunction-sections in lld.<o:p></o:p></p></div><div><p class=MsoNormal><span style='color:#888888'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='color:#888888'>- Michael Spencer<o:p></o:p></span></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><p class=MsoNormal style='margin-bottom:12.0pt'><br>Thanks,<br>Tobias<span style='color:#888888'><br><br><span class="m5872875340768778888m-5486578834148932290m3162267875614969241m854065992349427772m1563025543004182786gmail-hoenzb">-- </span><br><span class="m5872875340768778888m-5486578834148932290m3162267875614969241m854065992349427772m1563025543004182786gmail-hoenzb">Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,</span><br><span class="m5872875340768778888m-5486578834148932290m3162267875614969241m854065992349427772m1563025543004182786gmail-hoenzb">a Linux Foundation Collaborative Project.</span></span><o:p></o:p></p></blockquote></div><p class=MsoNormal><o:p> </o:p></p></div></div><p class=MsoNormal style='margin-bottom:12.0pt'><br>_______________________________________________<br>LLVM Developers mailing list<br><a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br><a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><o:p></o:p></p></blockquote></div><p class=MsoNormal><o:p> </o:p></p></div></div><p class=MsoNormal style='margin-bottom:12.0pt'><br>_______________________________________________<br>LLVM Developers mailing list<br><a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br><a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><o:p></o:p></p></blockquote></div><p class=MsoNormal><o:p> </o:p></p></div></div></blockquote></div><p class=MsoNormal><o:p> </o:p></p></div></div></blockquote></div><p class=MsoNormal><o:p> </o:p></p></div></div></blockquote></div><p class=MsoNormal><o:p> </o:p></p></div></div></div></body></html>