[llvm-dev] [Proposal][Debuginfo] dsymutil-like tool for ELF.

[Alexey via llvm-dev]

On 01.09.2020 20:18, James Y Knight wrote:
>
>
> On Mon, Aug 31, 2020 at 11:06 AM Alexey <avl.lapshin at gmail.com 
> <mailto:avl.lapshin at gmail.com>> wrote:
>
>     Hi James,
>
>     Thank you for the comments.
>
>     >I think we're not terribly far from that ideal, now, for ELF.
>     Maybe only these three things need to be done? --
>     >  1. Teach lld how to emit a separated debuginfo output file
>     directly, without requiring an objcopy step.
>     >  2. Integrate DWARFLinker into lld.
>     >  3. Create a new tool which takes the separated debuginfo and
>     DWO/DWP files and uses DWARFLinker library
>     > to create a new (dwarf-linked) separated-debug file, that
>     doesn't depend on DWO/DWP files.
>
>     The three goals which you`ve described are our far goals.
>     Indeed, the best solution would be to create valid optimized debug
>     info without additional
>     stages and additional modifications of resulting binaries.
>
>     There was an attempt to use DWARFLinker from the lld -
>     https://reviews.llvm.org/D74169
>     It did not receive enough support to be integrated yet. There are
>     fair reasons for that:
>
>     1. Execution time. The time required by DWARFLinker for processing
>     clang binary is 8x bigger
>     than the usual linking time. Linking clang binary with DWARFLinker
>     takes 72 sec,
>     linking with the only lld takes 9 sec.
>
>     2. "Removing obsolete debug info" could not be switched off. Thus,
>     lld could not use DWARFLinker for
>     other tasks(like generation of index tables - .gdb_index,
>     .debug_names) without significant performance
>     degradation.
>
>     3. DWARFLinker does not support split dwarf at the moment.
>
>     All these reasons are not blockers. And I believe implementation
>     from D74169 might be integrated and
>     incrementally improved if there would be agreement on that.
>
>
> Those do sound like absolutely critical issues for deploying this for 
> real -- whether as a separate tool or integrated with lld. But 
> possibly not critical enough to prevent adding this behind an 
> experimental flag, and working on the code incrementally in-tree. 
> However (without having looked at the code in question), I wonder if 
> the reported 8x regression in link-time is even going to be 
> salvageable just by incremental optimizations, or if it might require 
> a complete re-architecting of the DwarfLinker code.


That would be more like complete re-architecturing of the DWARFLinker code.
Current dsymutil implementation does "analyzing" and "cloning" stages in 
parallel.
i.e. it sequentially analyzes all object files and sequentially clones 
them. Speed up
ratio of parallelization is 2x. Changing this scenario to process 
compilation units in
parallel might speed up execution time. Supporting that scenario would 
require
huge refactoring. The advantage could be speedup execution time and reducing
memory usage. I am planning to make a prototype of it to prove the fact 
that
such a refactoring will have these benefits.


>
>     Using DWARFLinker from llvm-dwarfutil is another possibility to
>     use and improve it.
>
>     When finally implemented - llvm-dwarfutil should solve the above
>     three issues and there
>     would probably be more reasons to include DWARFLinker into lld.
>
>
> Is it the case that if the code is built to support the "read an 
> executable, output a new better executable" use-case, it will actually 
> be what's needed for the "output an optimized executable while linking 
> object files" use-case? I worry that those could have enough different 
> requirements that you really need to be developing the 
> linker-integrated version from the very beginning in order to get a 
> good result, rather than trying to shoehorn it in as an afterthought.
There are already exist working prototypes of the "read an executable, 
output a new better executable" use case - 
https://reviews.llvm.org/D86539. And "output an optimized executable 
while linking object files" https://reviews.llvm.org/D74169. They share 
the most of DWARFLinker. The differences exist but they could be managed.

The major problem of D86539 is that it loads all dies into the 
memory(since we have only one source file).
For clang it requires approx 30G of memory. D74169 does not have such a 
problem since it loads/frees dies per object file. Changing processing 
from "per source file" into "per compilation unit" should help with this 
problem.

So it looks like DWARFLinker refactored to parallel "per compilation 
unit" scenario fits quite well for both of these tasks.

At current moment I do not see other problems which would prevent from 
using
the same DWARFLinker library for both of these tasks. There could be 
unseen some, of course.


>
>     Even if we would have the best solution - it is still useful to
>     have a tool like llvm-dwarfutil
>
>     for cases when it is necessary to process already created binaries.
>
>
> Sure -- I just think that should be considered as a secondary 
> use-case, and not the primary goal.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200901/ee48ad1a/attachment.html>