<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Yes, <a href="https://bugs.llvm.org/show_bug.cgi?id=50404">https://bugs.llvm.org/show_bug.cgi?id=50404</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">-Brian<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Justin Cady <desk@justincady.com> <br>
<b>Sent:</b> Friday, May 21, 2021 9:25 AM<br>
<b>To:</b> Brian Cain <bcain@quicinc.com><br>
<b>Cc:</b> llvm-dev@lists.llvm.org<br>
<b>Subject:</b> [EXT] Re: [llvm-dev] llvm-objcopy --keep-symbols performance<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Unfortunately I do not have an answer or workaround, but I want to add that I am also hitting this issue. The performance of --keep-symbols with a large list of symbols is quite slow.<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Have you filed a bug to track this?<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">(For other reasons in my specific case I cannot fall back to GNU objcopy, so this performance penalty must be paid today.)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Justin<o:p></o:p></p>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On May 18, 2021, at 12:11 AM, Brian Cain via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">While investigating an issue with android build performance, I found that llvm-objcopy performance is far from parity with GNU objcopy for the “--keep-symbols” case, when the keep symbols list is very large.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">I can see that llvm objcopy creates a list of NameOrPattern-s to represent the keep symbol list, but it looks like it exhaustively compares each symbol in the keep list against each symbol in the input file. Not sure what GNU objcopy does
but merging the non-pattern symbol names into a sorted list would make searching much faster.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Below is a script to reproduce the problem and its output. The test case below demonstrates a significant performance difference. The android build failure case reported had a shared obj with ~700k symbols and the keep list was ~300k
symbols and it took ~8 minutes to execute.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">~~~<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">#!/bin/bash<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">set -euo pipefail<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">echo creating init file<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">echo <<EOF > input.S<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">.section .text<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">EOF<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">echo '' > keep_syms.txt<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">for i in $(seq 0 100000)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">do<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> echo -e "sym_${i}:\nnop\n" >> input.S<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> if [[ ${i} -lt 30000 ]]; then<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> echo "sym_${i}" >> keep_syms.txt<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> fi<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">done<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">echo <<EOF >> input.S<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">.section .ballast<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">nop<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">EOF<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">echo creating obj file<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">llvm-mc -triple arm-linux-androideabi -filetype=obj input.S -o out.o<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">echo creating shared obj file<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">ld.lld -shared out.o -o libtestcase.so<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">echo performing llvm objcopy<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">\time llvm-objcopy -S --remove-section .ballast \<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> --keep-symbols=keep_syms.txt \<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> libtestcase.so libtestcase_smaller.so<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">echo performing GNU objcopy<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">\time arm-linux-androideabi-objcopy -S --remove-section .ballast \<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> --keep-symbols=keep_syms.txt \<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> libtestcase.so libtestcase_smaller.so<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">~~~<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Here’s the output I get when I run it on ToT-within-last-week-or-so:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal">$ PATH=$PWD/bin:$PATH ../../tmp/qt66370/32/objcopy_perf.sh<o:p></o:p></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal">creating init file<o:p></o:p></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal">creating obj file<o:p></o:p></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal">creating shared obj file<o:p></o:p></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal">ld.lld: warning: lld uses blx instruction, no object with architecture supporting feature detected<o:p></o:p></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal">performing llvm objcopy<o:p></o:p></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal">12.34user 0.00system 0:12.35elapsed 99%CPU (0avgtext+0avgdata 20228maxresident)k<o:p></o:p></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal">0inputs+2288outputs (0major+3815minor)pagefaults 0swaps<o:p></o:p></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal">performing GNU objcopy<o:p></o:p></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal">0.03user 0.00system 0:00.04elapsed 97%CPU (0avgtext+0avgdata 20396maxresident)k<o:p></o:p></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal">0inputs+2288outputs (0major+5895minor)pagefaults 0swaps<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Alternate access to the script above:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://gist.github.com/androm3da/83560d92f3fe637b58aa115ba6b68456"><span style="color:#0563C1">https://gist.github.com/androm3da/83560d92f3fe637b58aa115ba6b68456</span></a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">-Brian<o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">_______________________________________________<br>
LLVM Developers mailing list<br>
</span><a href="mailto:llvm-dev@lists.llvm.org"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:#0563C1">llvm-dev@lists.llvm.org</span></a><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
</span><a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:#0563C1">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</span></a><o:p></o:p></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</body>
</html>