<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@DengXian";
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:-webkit-standard;
panose-1:2 11 6 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:929267221;
mso-list-type:hybrid;
mso-list-template-ids:-204174392 1957227358 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:\F0D8 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:20.25pt;
text-indent:-.25in;
font-family:Wingdings;
mso-fareast-font-family:DengXian;
mso-bidi-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:56.25pt;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:92.25pt;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:128.25pt;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:164.25pt;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:200.25pt;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:236.25pt;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:272.25pt;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:308.25pt;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">> <span style="font-family:"-webkit-standard",serif;color:black">
Do you have measurement how well SHT_LLVM_CALL_GRAPH_PROFILE optimizes?<br>
> My understanding is that with ThinLTO+PGO it is has very tiny benefit.</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">With PLO(BOLT) the benefit is zero, but with PGO and without PLO, it was still visible. We’ve seen ~1% RPS improvement in the past when measuring mysql.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">PGO does not cover profile guided function layout, either the linker or PLO need to take care of that part. We’ve been using standalone
<a href="https://research.fb.com/publications/optimizing-function-placement-for-large-scale-data-center-applications-2/">HFSort</a> on top of PGO, and we wanted to fold that into either PLO or linker – the profile guided function layout exists in lld, but we ran into bugs hence the RFC and Alexander’s fix.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
<p class="MsoNormal">Wenlei<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Alexander Yermolovich <ayermolo@fb.com><br>
<b>Date: </b>Thursday, May 27, 2021 at 5:44 PM<br>
<b>To: </b>maskray@google.com <maskray@google.com><br>
<b>Cc: </b>llvm-dev@lists.llvm.org <llvm-dev@lists.llvm.org>, Wenlei He <wenlei@fb.com>, Modi Mo <modimo@fb.com>, dblaikie@gmail.com <dblaikie@gmail.com><br>
<b>Subject: </b>Re: [RFC] Changing .llvm.call-graph-profile to use relocations<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:12.0pt;color:black">Replies inlined.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="divRplyFwdMsg">
<p class="MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> maskray@google.com <maskray@google.com><br>
<b>Sent:</b> Thursday, May 27, 2021 5:23 PM<br>
<b>To:</b> Alexander Yermolovich <ayermolo@fb.com><br>
<b>Cc:</b> llvm-dev@lists.llvm.org <llvm-dev@lists.llvm.org>; Wenlei He <wenlei@fb.com>; Modi Mo <modimo@fb.com>; dblaikie@gmail.com <dblaikie@gmail.com><br>
<b>Subject:</b> Re: [RFC] Changing .llvm.call-graph-profile to use relocations</span>
<o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">On 2021-05-27, Alexander Yermolovich wrote:<br>
>I was thinking keeping both structures for backward compatibility in case object files with old representation are fed in to new llvm-objdump, and even lld. For example if someone will use older clang release with lld/llvm-objdump after this change.<br>
>For just changing the value and keeping the name. Looks like it will leave a gap in this sequential sequence. If a new flag is added later down the road and in this case latest clang used with older lld/llvm-objdump this will also present a problem as older
tools will interpret new flag as SHT_LLVM_CALL_GRAPH_PROFILE.<br>
>Not sure if these are valid concerns, what do you think?<br>
>If we go for clean state, then maybe leave 0x6fff4c02 entry and change it to SHT_LLVM_CALL_GRAPH_PROFILE_DEPRICATED, and emit warning if objects with it are passed in.<br>
> SHT_LLVM_ODRTAB = 0x6fff4c00, // LLVM ODR table.<br>
> SHT_LLVM_LINKER_OPTIONS = 0x6fff4c01, // LLVM Linker Options.<br>
> SHT_LLVM_CALL_GRAPH_PROFILE = 0x6fff4c02, // LLVM Call Graph Profile.<br>
> SHT_LLVM_ADDRSIG = 0x6fff4c03, // List of address-significant symbols<br>
> // for safe ICF.<br>
> SHT_LLVM_DEPENDENT_LIBRARIES =<br>
> 0x6fff4c04, // LLVM Dependent Library Specifiers.<br>
> SHT_LLVM_SYMPART = 0x6fff4c05, // Symbol partition specification.<br>
> SHT_LLVM_PART_EHDR = 0x6fff4c06, // ELF header for loadable partition.<br>
> SHT_LLVM_PART_PHDR = 0x6fff4c07, // Phdrs for loadable partition.<br>
> SHT_LLVM_BB_ADDR_MAP = 0x6fff4c08, // LLVM Basic Block Address Map.<br>
<br>
Do you have measurement how well SHT_LLVM_CALL_GRAPH_PROFILE optimizes?<br>
My understanding is that with ThinLTO+PGO it is has very tiny benefit.<br>
[Alex] I do not. Wenlei mentioned he can dig up some numbers.<br>
<br>
The value reading old format is low. I don't expect users want to mix<br>
newer object files with old object files and want to have optimization<br>
from the old object files. SHT_LLVM_CALL_GRAPH_PROFILE_DEPRICATED is<br>
just adding maintenance cost.<br>
<br>
Changing the value but retaining the name is sufficient for the various<br>
compatibility issues.<br>
<br>
Reid's size concern is valid. Have you measured the size overhead?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">[Alex] Good point. I'll measure once I implement new approach with SHT_REL + <span style="color:black;background:white">R_X86_64_NONE.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
We can use the SHT_REL format for SHT_LLVM_CALL_GRAPH_PROFILE<br>
relocations, which brings down the per entry cost from 24 bytes to 16<br>
bytes for ELFCLASS64. (It is a bit unfortunate that the Linux kernel<br>
does not support ELFCLASS32 executables on a 64-bit architecture.<br>
For most usage we are using the small code model and don't benefit<br>
from 64-bit addresses/sizes.)<br>
<br>
>________________________________<br>
>From: maskray@google.com <maskray@google.com><br>
>Sent: Thursday, May 27, 2021 11:27 AM<br>
>To: Alexander Yermolovich <ayermolo@fb.com><br>
>Cc: llvm-dev@lists.llvm.org <llvm-dev@lists.llvm.org>; Wenlei He <wenlei@fb.com>; Modi Mo <modimo@fb.com>; dblaikie@gmail.com <dblaikie@gmail.com><br>
>Subject: Re: [RFC] Changing .llvm.call-graph-profile to use relocations<br>
><br>
>On 2021-05-27, Alexander Yermolovich wrote:<br>
>>Thank you for feedback, let me try to use R_X86_64_NONE, and introduce another type. . I thought R_X86_64_32 would be less impactful as structure of is preserved, but it is space wasteful.<br>
>>For type name: ELF::SHT_LLVM_CALL_GRAPH_PROFILE_RELA ?<br>
>><br>
>>Alex<br>
><br>
>Preversing the structure is not needed because the symbol representation<br>
>is changed anyway.<br>
><br>
>You just need to change the value in<br>
>llvm/llvm/include/llvm/BinaryFormat/ELF.h<br>
>The name doesn't need to change.<br>
><br>
>>From: maskray@google.com <maskray@google.com><br>
>>Sent: Wednesday, May 26, 2021 5:03 PM<br>
>>To: Alexander Yermolovich <ayermolo@fb.com><br>
>>Cc: llvm-dev@lists.llvm.org <llvm-dev@lists.llvm.org>; Wenlei He <wenlei@fb.com>; Modi Mo <modimo@fb.com>; dblaikie@gmail.com <dblaikie@gmail.com><br>
>>Subject: Re: [RFC] Changing .llvm.call-graph-profile to use relocations<br>
>><br>
>>On 2021-05-26, Alexander Yermolovich wrote:<br>
>>>Currently when .llvm.call-graph-profile is created by llvm it explicitly encodes the symbol indices. This section is basically a black box for post processing tools. For example, if we run strip -s on the object files the symbol table changes, but indices
in that section do not.<br>
>>><br>
>>>We propose to change this section to use relocations. The Frequency will still be in the .llvm.call-graph-profile, but symbol information will be in relocation section. In LLD information from both sections is used to reconstruct call graph profile. Relocations
themselves will never be applied.<br>
>><br>
>>Yes, a relocation based approach will be more robust and can fix the<br>
>>.llvm_addrsig issue <a href="https://sourceware.org/bugzilla/show_bug.cgi?id=23817">https://sourceware.org/bugzilla/show_bug.cgi?id=23817</a>
<br>
>><br>
>>The relocation type doesn't matter. Your implementation uses R_X86_64_32 and from/to/value/from/to/value/from/to/value<br>
>>An alternative design is R_X86_64_NONE + value/value/value, i.e. from/to do not occupy space in the content.<br>
>>We will get a 3x space saving.<br>
>><br>
>>We need to change the section type since changing representation is incompatible<br>
>>and the sections from old object files should be ignored.<br>
>>The new section type will be ignored by old LLVM tools as well.<br>
>><br>
>>>With this approach post processing tools that handle relocations correctly work for this section also.<br>
>>><br>
>>>One thing is section is marked with SHF_EXCLUDE.<br>
>>>From spec<br>
>>>"This section is excluded from input to the link-edit of an executable or shared object. This flag is ignored if the SHF_ALLOC flag is also set, or if relocations exist against the section."<br>
>>><br>
>>>So technically speaking it needs to be kept, and presumably relocations applied, but LLD follows gold and ld and discards sections marked as SHF_EXCLUDE even with relocations. So, I think this approach should be fine.
<a href="https://reviews.llvm.org/D24966">https://reviews.llvm.org/D24966</a>
<br>
>><br>
>>This is Solaris's spec (since 1996), not standard ELF's. It can be advisory but<br>
>>our behavior (mostly GNU ABI+Linux ABI+LLVM extensions) does not necessarily<br>
>>follow it. I cannot find a definition in the x86-64 psABI or a GNU ABI<br>
>>document.<br>
>><br>
>>I think the behavior as implemented in gold and LLD is more useful.<br>
>><br>
>>>Finally, this bug seems similar to <a href="https://sourceware.org/bugzilla/show_bug.cgi?id=23817">https://sourceware.org/bugzilla/show_bug.cgi?id=23817</a> . Proposed solution for that was also to use relocations.<br>
>>><br>
>>>Implementation: <a href="https://reviews.llvm.org/D103212">https://reviews.llvm.org/D103212</a>
<o:p></o:p></p>
</div>
</div>
</div>
</body>
</html>