<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none"><!-- p { margin-top: 0px; margin-bottom: 0px; } @font-face { font-family: Calibri; } @font-face { font-family: Tahoma; } p.MsoNormal, li.MsoNormal, div.MsoNormal { margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif; } a:link, span.MsoHyperlink { color: blue; text-decoration: underline; } a:visited, span.MsoHyperlinkFollowed { color: purple; text-decoration: underline; } span.EmailStyle18 { font-family: Calibri, sans-serif; color: rgb(31, 73, 125); } .MsoChpDefault { font-size: 10pt; } @page WordSection1 { margin: 1in; } div.WordSection1 { }--></style>
</head>
<body dir="ltr" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p><span style="color: rgb(31, 73, 125); font-family: Calibri, sans-serif; font-size: 11pt;">Thanks for answers, Paul !</span></p>
<p><span style="color: rgb(31, 73, 125); font-family: Calibri, sans-serif; font-size: 11pt;"><br>
</span></p>
<div style="">
<div style="">
<div class="WordSection1" style="">
<p class="MsoNormal" style="font-size: 16px; font-family: "Times New Roman", serif; background-color: rgb(255, 255, 255); color: rgb(33, 33, 33);">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">>So, I think whether type sections help or hurt will depend on how a particular project's build procedure is set up. Clang/LLVM are set up >to do lots of smaller compilations
and link them all together, in a fairly traditional model, and that is where type sections will provide the >most benefit. Your data, then, is essentially for a best-case scenario. Other kinds of projects will not benefit as much.</span></p>
<p class="MsoNormal" style="color: rgb(33, 33, 33);"><span style="font-size:11.0pt; font-family:"Calibri","sans-serif"; color:#1F497D"><br>
</span></p>
<p class="MsoNormal" style="color: rgb(33, 33, 33);"><span style="font-size:11.0pt; font-family:"Calibri","sans-serif"; color:#1F497D">This inspired me to do additional tests for LLVM binaries to see how much win they can have if we enable -fdebug-types-section.</span></p>
<p class="MsoNormal" style=""><font color="#1f497d" face="Calibri, sans-serif"><span style="font-size: 14.6667px;">(Full table with results is at the end of mail.) </span></font></p>
<p class="MsoNormal" style=""><font color="#1f497d" face="Calibri, sans-serif"><span style="font-size: 14.6667px;">During experiment I observed both object size penalies and a single final executable size penalty:</span></font></p>
<p class="MsoNormal" style=""><font color="#1f497d" face="Calibri, sans-serif"><span style="font-size: 14.6667px;">1) Size of .a files in LLVM/lib files inreases from 6.5GB to 7.7GB.</span></font></p>
<p class="MsoNormal" style=""><font color="#1f497d" face="Calibri, sans-serif"><span style="font-size: 14.6667px;">2) One binary which is llvm-PerfectShuffle </span></font><span style="font-size: 14.6667px; color: rgb(31, 73, 125); font-family: Calibri, sans-serif;">was
larger with flag, size changed from </span><span style="font-size: 14.6667px; color: rgb(31, 73, 125); font-family: Calibri, sans-serif;">120064</span><span style="font-size: 14.6667px; color: rgb(31, 73, 125); font-family: Calibri, sans-serif;"> to </span><span style="font-size: 14.6667px; color: rgb(31, 73, 125); font-family: Calibri, sans-serif;">124952</span><span style="font-size: 14.6667px; color: rgb(31, 73, 125); font-family: Calibri, sans-serif;">. </span></p>
<p class="MsoNormal" style=""><span style="font-size: 14.6667px; color: rgb(31, 73, 125); font-family: Calibri, sans-serif;"> For all others use of flag
<span style="color: rgb(31, 73, 125); font-family: Calibri, sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255);">
usually</span> grants noticable win (up to reduce of size by 41%).</span></p>
<p class="MsoNormal" style="color: rgb(33, 33, 33);"><span style="font-size:11.0pt; font-family:"Calibri","sans-serif"; color:#1F497D"><br>
</span></p>
<p class="MsoNormal" style="color: rgb(33, 33, 33);"><span style="font-size:11.0pt; font-family:"Calibri","sans-serif"; color:#1F497D">>Regarding DWARF 5 and emitting type sections into the .debug_info section rather than the .debug_types section: The work
to support >DWARF 5 in LLVM has not gotten very far yet. Conforming to the standard in this respect is certainly on my list, however there are other >features that Sony considers higher priority. If you or someone else wants to contribute that feature sooner,
that would be excellent! >Otherwise, we will get to it in due time.</span></p>
<p class="MsoNormal" style="color: rgb(33, 33, 33);"><span style="font-size:11.0pt; font-family:"Calibri","sans-serif"; color:#1F497D">>Thanks,</span></p>
<p class="MsoNormal" style="color: rgb(33, 33, 33);"><span style="font-size:11.0pt; font-family:"Calibri","sans-serif"; color:#1F497D">>--paulr</span></p>
<p class="MsoNormal" style="color: rgb(33, 33, 33);"><span style="font-size:11.0pt; font-family:"Calibri","sans-serif"; color:#1F497D"><br>
</span></p>
<p class="MsoNormal" style="color: rgb(33, 33, 33);"><span style="font-size:11.0pt; font-family:"Calibri","sans-serif"; color:#1F497D">I am going to look at it closer. At least I do not think LLD would work correctly with multiple</span></p>
<p class="MsoNormal" style="color: rgb(33, 33, 33);"><span style="font-size:11.0pt; font-family:"Calibri","sans-serif"; color:#1F497D">.debug_info right now for building .gdb_index. We expect to see unique .debug_info in a object file and</span></p>
<p class="MsoNormal" style="color: rgb(33, 33, 33);"><span style="font-size:11.0pt; font-family:"Calibri","sans-serif"; color:#1F497D">probably will do something wrong in another case. </span><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">Looks
</span><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">llvm/DebugInfo needs to be fixed first</span><span style="color: rgb(31, 73, 125); font-family: Calibri, sans-serif; font-size: 11pt;">, which</span></p>
<p class="MsoNormal" style="color: rgb(33, 33, 33);"><span style="color: rgb(31, 73, 125); font-family: Calibri, sans-serif; font-size: 11pt;">also affects tools lile llvm-dwarfdump and p</span><span style="color: rgb(31, 73, 125); font-family: Calibri, sans-serif; font-size: 11pt;">robably
</span><span style="color: rgb(31, 73, 125); font-family: Calibri, sans-serif; font-size: 11pt;">something else</span><span style="color: rgb(31, 73, 125); font-family: Calibri, sans-serif; font-size: 11pt;">. Going to investigate all of that.</span></p>
<p class="MsoNormal" style="color: rgb(33, 33, 33);"><br>
</p>
<p class="MsoNormal" style="color: rgb(33, 33, 33);"><font face="Calibri, Arial, Helvetica, sans-serif">Testing results:</font><br>
</p>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);">
----------------------------------------------------------------<br>
</div>
<div>Name Size change (-g / -g -fdebug-types-section)</div>
<div>----------------------------------------------------------------</div>
<div>arcmt-test 461644608 / 322758048 = 1.430x</div>
<div>bugpoint 938191280 / 552402624 = 1.698x</div>
<div>c-arcmt-test 20968 / 20968 = 1.000x</div>
<div>c-index-test 941325408 / 613643776 = 1.533x</div>
<div>clang-6.0 1697417824 / 1025908400 = 1.654x</div>
<div>clang-check 1440335448 / 864954472 = 1.665x</div>
<div>clang-diff 422183328 / 293650384 = 1.437x</div>
<div>clang-format 67763352 / 51596584 = 1.313x</div>
<div>clang-func-mapping 423746376 / 294311536 = 1.439x</div>
<div>clang-import-test 611477912 / 410019056 = 1.491x</div>
<div>clang-offload-bundler 76254024 / 61321152 = 1.243x</div>
<div>clang-refactor 448153976 / 311549496 = 1.438x</div>
<div>clang-rename 441661264 / 307777416 = 1.435x</div>
<div>clang-tblgen 17489504 / 16802744 = 1.040x</div>
<div>count 18392 / 18392 = 1.000x</div>
<div>diagtool 415912688 / 289701512 = 1.435x</div>
<div>FileCheck 6681280 / 6489896 = 1.029x</div>
<div>llc 903308048 / 529531864 = 1.705x</div>
<div>lld 1009754992 / 620445232 = 1.627x</div>
<div>lli 419682176 / 270912680 = 1.549x</div>
<div>lli-child-target 77237632 / 63011888 = 1.225x</div>
<div>llvm-ar 131787624 / 102692104 = 1.283x</div>
<div>llvm-as 72916752 / 57792456 = 1.261x</div>
<div>llvm-bcanalyzer 6464984 / 6259992 = 1.032x</div>
<div>llvm-cat 73318016 / 57999784 = 1.264x</div>
<div>llvm-cfi-verify 160259072 / 125738440 = 1.274x</div>
<div>llvm-config 5947768 / 5776752 = 1.029x</div>
<div>llvm-cov 80728632 / 65663448 = 1.229x</div>
<div>llvm-c-test 843631952 / 498768912 = 1.691x</div>
<div>llvm-cvtres 72163840 / 58065104 = 1.242x</div>
<div>llvm-cxxdump 74284720 / 59261168 = 1.253x</div>
<div>llvm-cxxfilt 7046752 / 6865368 = 1.026x</div>
<div>llvm-demangle-fuzzer 70156288 / 55760784 = 1.258x</div>
<div>llvm-diff 58551832 / 46506104 = 1.259x</div>
<div>llvm-dis 52982824 / 42252624 = 1.253x</div>
<div>llvm-dsymutil 883071928 / 517877728 = 1.705x</div>
<div>llvm-dwarfdump 121679064 / 95079960 = 1.279x</div>
<div>llvm-dwp 879362280 / 514570584 = 1.708x</div>
<div>llvm-extract 115790888 / 87646504 = 1.321x</div>
<div>llvm-isel-fuzzer 887217736 / 519910464 = 1.706x</div>
<div>llvm-link 79158192 / 62087976 = 1.274x</div>
<div>llvm-lto 932838656 / 553536912 = 1.685x</div>
<div>llvm-lto2 926319416 / 550018696 = 1.684x</div>
<div>llvm-mc 118139784 / 89656216 = 1.317x</div>
<div>llvm-mcmarkup 5974664 / 5775368 = 1.034x</div>
<div>llvm-modextract 68740776 / 54352208 = 1.264x</div>
<div>llvm-mt 6749720 / 6440088 = 1.048x</div>
<div>llvm-nm 131633536 / 102825080 = 1.280x</div>
<div>llvm-objcopy 73991272 / 60029840 = 1.232x</div>
<div>llvm-objdump 150270880 / 118629456 = 1.266x</div>
<div>llvm-opt-fuzzer 891258608 / 527493664 = 1.689x</div>
<div>llvm-opt-report 8814368 / 8585952 = 1.026x</div>
<div>llvm-pdbutil 110919744 / 93010704 = 1.192x</div>
<div>llvm-PerfectShuffle 120064 / 124952 = 0.960x</div>
<div>llvm-profdata 41889560 / 32957976 = 1.270x</div>
<div>llvm-rc 8954768 / 8551192 = 1.047x</div>
<div>llvm-readobj 85723040 / 70542776 = 1.215x</div>
<div>llvm-rtdyld 138255056 / 108085992 = 1.279x</div>
<div>llvm-size 71567376 / 57589872 = 1.259x</div>
<div>llvm-split 125299816 / 95063600 = 1.318x</div>
<div>llvm-stress 46366576 / 37211688 = 1.246x</div>
<div>llvm-strings 5746216 / 5563216 = 1.032x</div>
<div>llvm-symbolizer 87280568 / 71248216 = 1.225x</div>
<div>llvm-tblgen 49304088 / 42580848 = 1.157x</div>
<div>llvm-xray 93953928 / 77434112 = 1.213x</div>
<div>not 5495536 / 5325816 = 1.031x</div>
<div>obj2yaml 97146752 / 81415480 = 1.193x</div>
<div>opt 955386696 / 564492184 = 1.692x</div>
<div>sancov 146145680 / 114837520 = 1.272x</div>
<div>sanstats 87031832 / 71004312 = 1.225x</div>
<div>scan-build 53444 / 53444 = 1.000x</div>
<div>scan-view 4504 / 4504 = 1.000x</div>
<div>verify-uselistorder 73506560 / 58211520 = 1.262x</div>
<div>yaml2obj 27882712 / 26506184 = 1.051x</div>
<div>yaml-bench 7001024 / 6763952 = 1.035x<br>
</div>
<p class="MsoNormal" style="color: rgb(33, 33, 33);"><span style="font-size:11.0pt; font-family:"Calibri","sans-serif"; color:#1F497D"><br>
</span></p>
<p class="MsoNormal" style="color: rgb(33, 33, 33);"><a name="_MailEndCompose"><span style="font-size:11.0pt; font-family:"Calibri","sans-serif"; color:#1F497D"><br>
</span></a></p>
<p class="MsoNormal" style="color: rgb(33, 33, 33);"><a name="_MailEndCompose"><span style="font-size:11.0pt; font-family:"Calibri","sans-serif"; color:#1F497D"><br>
</span></a></p>
<div style="color: rgb(33, 33, 33); border-top: none; border-right: none; border-bottom: none; border-image: initial; border-left: 1.5pt solid blue; padding: 0in 0in 0in 4pt;">
<div>
<div style="border:none; border-top:solid #B5C4DF 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt; font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt; font-family:"Tahoma","sans-serif""> llvm-dev [mailto:llvm-dev-bounces@lists.llvm.org]
<b>On Behalf Of </b>George Rimar via llvm-dev<br>
<b>Sent:</b> Monday, December 04, 2017 7:11 AM<br>
<b>To:</b> llvm-dev@lists.llvm.org<br>
<b>Subject:</b> [llvm-dev] [RFC] - Deduplication of debug information in linkers (LLD).</span></p>
</div>
</div>
<p class="MsoNormal"> </p>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">Hi all !</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">We have an issue with LLD, it is "relocation R_X86_64_32 out of range" (PR31109)</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">which occurs during resolving relocations in debug sections. It looks happens</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">because .debug_info section can be too large sometimes and 32x relocation is not enough</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">to represent the value. One of possible solutions looks to be to deduplicate information</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">to reduce .debug_info size.</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">The rest of mail contains information about experiments I did, the obtained results and</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">some questions and suggestions as well.</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">I was investigating idea to deduplicate debug types information. Idea is described at</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">p276 of DWARF4 specification (<a href="http://www.dwarfstd.org/doc/DWARF4.pdf">http://www.dwarfstd.org/doc/DWARF4.pdf</a>). It suggests</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">to split types information out of .debug_info and emit multiple .debug_types sections</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">with use of COMDATs. Both clang and gcc I tested implements -fdebug-types-section flag for that:</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">-fdebug-types-section, -fno-debug-types-section</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">Place debug types in their own section (ELF Only)</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">gcc's description is here:
<a href="https://gcc.gnu.org/onlinedocs/gcc-6.4.0/gcc/Debugging-Options.html#Debugging-Options">
https://gcc.gnu.org/onlinedocs/gcc-6.4.0/gcc/Debugging-Options.html#Debugging-Options</a>.</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">This flag is disabled by default. I compared clang binaries to see the difference</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">with and without the linker side optimisation.</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">1) Clang built with -g has size of 1.7 GB, .debug_info section size is 894.5 Mb.</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">2) Clang built with -g -fdebug-types-section has size of 1.0 GB.</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black"> .debug_types size is 26.267 MB, .debug_info size is 227.7 MB.</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">Difference is huge and I believe shows (though probably for most of readers here it was</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">already obvious) that optimization can be useful. Though
<span style="background:white">-fdebug-types-section</span> is disabled by default.</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">Looks it was initially disabled because not all of DWARF consumers were aware of .debug_types section.</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">Now in 2017 situation is different. I think most of DWARF consumers knows about .debug_types, but:</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">1) DWARF5 specification explicitly eliminates the .debug_types section introduced in DWARF4:</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black"> p8, "1.4 Changes from Version 4 to Version 5"
<a href="http://dwarfstd.org/doc/DWARF5.pdf">http://dwarfstd.org/doc/DWARF5.pdf</a></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">2) Instead of emiting multiple .debug_types it suggests to emit multiple .debug_info COMDAT</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black"> sections. (p375, p376).</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">And it seems currently there is no way to make clang to emit multiple .debug_info with type information</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">like DWARF5 suggests. I tried command line below:</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">-g -fdebug-types-section -gdwarf-5</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">It still emits .debug_types and does not look there is a flag for emiting multiple .debug_info.</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">Looking at whole LLVM code (lib/mc, lib/CodeGen) actually it seems it is just always assumed .debug_info is</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">a unique section in object.</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">(also not sure why clang emits .debug_types when -gdwarf-5 flag is set, as this section is incompatible with v5,</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">probably it is a bug).</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">So my questions are following:</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">1) Do we want to try to implement multiple .debug_info approach ? As it seems can be very useful sometimes.</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">2) For now in LLD may be we may want to extend our error message from "relocation X out of range" to something</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black"> suggesting to use -fdebug-types-section (only for relocations in debug sections) ?</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif"; color:black">3) Why -fdebug-types-section is disabled by default ?</span></p>
</div>
<p><span style="font-family:"Calibri","sans-serif"; color:black"></span></p>
<div id="Signature">
<div name="divtagdefaultwrapper">
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Calibri","sans-serif"; color:black">Best regards,<br>
George | Developer | </span><span style="font-size:10.0pt; font-family:"Calibri","sans-serif"; color:#212121; background:white">Access Softek, Inc</span><span style="font-size:10.0pt; font-family:"Calibri","sans-serif"; color:black"></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>