<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p><br>
</p>
<div class="moz-cite-prefix">On 01.09.2020 20:18, James Y Knight
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAA2zVHrP-2+WVi2EmhMTqY6KDE7nUzA8Qg4RNQVYLTSKTDs4fQ@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div dir="ltr"><br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Aug 31, 2020 at
11:06 AM Alexey <<a href="mailto:avl.lapshin@gmail.com"
moz-do-not-send="true">avl.lapshin@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div> Hi James,<br>
<br>
Thank you for the comments. <br>
<br>
>I think we're not terribly far from that ideal, now,
for ELF. Maybe only these three things need to be done? --<br>
> 1. Teach lld how to emit a separated debuginfo
output file directly, without requiring an objcopy step.<br>
> 2. Integrate DWARFLinker into lld.<br>
> 3. Create a new tool which takes the separated
debuginfo and DWO/DWP files and uses DWARFLinker library <br>
> to create a new (dwarf-linked) separated-debug file,
that doesn't depend on DWO/DWP files.<br>
<br>
The three goals which you`ve described are our far goals.
<br>
Indeed, the best solution would be to create valid
optimized debug info without additional <br>
stages and additional modifications of resulting binaries.
<br>
<br>
There was an attempt to use DWARFLinker from the lld - <a
href="https://reviews.llvm.org/D74169" target="_blank"
moz-do-not-send="true">https://reviews.llvm.org/D74169</a><br>
It did not receive enough support to be integrated yet.
There are fair reasons for that:<br>
<br>
1. Execution time. The time required by DWARFLinker for
processing clang binary is 8x bigger<br>
than the usual linking time. Linking clang binary with
DWARFLinker takes 72 sec, <br>
linking with the only lld takes 9 sec.<br>
<br>
2. "Removing obsolete debug info" could not be switched
off. Thus, lld could not use DWARFLinker for<br>
other tasks(like generation of index tables - .gdb_index,
.debug_names) without significant performance <br>
degradation.<br>
<br>
3. DWARFLinker does not support split dwarf at the moment.<br>
<br>
All these reasons are not blockers. And I believe
implementation from D74169 might be integrated and <br>
incrementally improved if there would be agreement on
that.</div>
</blockquote>
<div><br>
</div>
<div>Those do sound like absolutely critical issues for
deploying this for real -- whether as a separate tool or
integrated with lld. But possibly not critical enough to
prevent adding this behind an experimental flag, and working
on the code incrementally in-tree. However (without having
looked at the code in question), I wonder if the reported 8x
regression in link-time is even going to be salvageable just
by incremental optimizations, or if it might require a
complete re-architecting of the DwarfLinker code.</div>
</div>
</div>
</blockquote>
<p><br>
</p>
That would be more like complete re-architecturing of the
DWARFLinker code. <br>
Current dsymutil implementation does "analyzing" and "cloning"
stages in parallel.<br>
i.e. it sequentially analyzes all object files and sequentially
clones them. Speed up <br>
ratio of parallelization is 2x. Changing this scenario to process
compilation units in <br>
parallel might speed up execution time. Supporting that scenario
would require <br>
huge refactoring. The advantage could be speedup execution time and
reducing<br>
memory usage. I am planning to make a prototype of it to prove the
fact that <br>
such a refactoring will have these benefits.<br>
<p><br>
</p>
<blockquote type="cite"
cite="mid:CAA2zVHrP-2+WVi2EmhMTqY6KDE7nUzA8Qg4RNQVYLTSKTDs4fQ@mail.gmail.com">
<div dir="ltr">
<div class="gmail_quote">
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>Using DWARFLinker from llvm-dwarfutil is another
possibility to use and improve it. <br>
</div>
</blockquote>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div> When finally implemented - llvm-dwarfutil should solve
the above three issues and there <br>
would probably be more reasons to include DWARFLinker into
lld.</div>
</blockquote>
<div><br>
</div>
<div>Is it the case that if the code is built to support the
"read an executable, output a new better executable"
use-case, it will actually be what's needed for the "output
an optimized executable while linking object files"
use-case? I worry that those could have enough different
requirements that you really need to be developing the
linker-integrated version from the very beginning in order
to get a good result, rather than trying to shoehorn it in
as an afterthought.</div>
</div>
</div>
</blockquote>
There are already exist working prototypes of the "read an
executable, output a new better executable" use case -
<a class="moz-txt-link-freetext" href="https://reviews.llvm.org/D86539">https://reviews.llvm.org/D86539</a>. And "output an optimized executable
while linking object files" <a class="moz-txt-link-freetext" href="https://reviews.llvm.org/D74169">https://reviews.llvm.org/D74169</a>. They
share the most of DWARFLinker. The differences exist but they could
be managed. <br>
<br>
The major problem of D86539 is that it loads all dies into the
memory(since we have only one source file). <br>
For clang it requires approx 30G of memory. D74169 does not have
such a problem since it loads/frees dies per object file. Changing
processing from "per source file" into "per compilation unit" should
help with this problem.<br>
<br>
So it looks like DWARFLinker refactored to parallel "per compilation
unit" scenario fits quite well for both of these tasks.<br>
<br>
At current moment I do not see other problems which would prevent
from using <br>
the same DWARFLinker library for both of these tasks. There could be
unseen some, of course.<br>
<br>
<br>
<blockquote type="cite"
cite="mid:CAA2zVHrP-2+WVi2EmhMTqY6KDE7nUzA8Qg4RNQVYLTSKTDs4fQ@mail.gmail.com">
<div dir="ltr">
<div class="gmail_quote">
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>Even if we would have the best solution - it is still
useful to have a tool like llvm-dwarfutil</div>
</blockquote>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div> for cases when it is necessary to process already
created binaries. </div>
</blockquote>
<div><br>
</div>
<div>Sure -- I just think that should be considered as a
secondary use-case, and not the primary goal.</div>
<div><br>
</div>
</div>
</div>
</blockquote>
</body>
</html>