[llvm-dev] [RFC] - Deduplication of debug information in linkers (LLD).

[Rui Ueyama via llvm-dev]

On Mon, Dec 4, 2017 at 10:49 AM, Robinson, Paul via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> Thanks for providing the experimental data!  It clearly shows the value of
> type sections in DWARF.
>
> Regarding why type sections are off by default, aside from the issue of
> consumers needing to understand them, there is a size penalty to type
> sections that becomes more evident in smaller projects (meaning, fewer
> compilation units).  The size penalty can be balanced against the amount of
> deduplication for a net win, if you have enough duplicates that you can
> eliminate.  But it is a tradeoff.
>

By a size penalty, which do you mean, the size of the final executable or
the intermediate object files? If it is a size penalty of object files, how
much is that? I wonder if the current situation is a reasonable trade-off.

In Sony's case, it is not uncommon for studios to do what are called
> "unity" builds, where you have basically one master .cpp file that does
> #include of each other .cpp file, giving you an LTO-like build.  In this
> case the debug-info production will automatically produce only one copy of
> each type, and so using type sections would probably make the net debug
> info bigger.  And of course an LTO build will deduplicate type info at the
> metadata level, with a similar effect.
>
> So, I think whether type sections help or hurt will depend on how a
> particular project's build procedure is set up.  Clang/LLVM are set up to
> do lots of smaller compilations and link them all together, in a fairly
> traditional model, and that is where type sections will provide the most
> benefit.  Your data, then, is essentially for a best-case scenario.  Other
> kinds of projects will not benefit as much.
>
>
>
> Regarding DWARF 5 and emitting type sections into the .debug_info section
> rather than the .debug_types section:  The work to support DWARF 5 in LLVM
> has not gotten very far yet.  Conforming to the standard in this respect is
> certainly on my list, however there are other features that Sony considers
> higher priority.  If you or someone else wants to contribute that feature
> sooner, that would be excellent!  Otherwise, we will get to it in due time.
>
> Thanks,
>
> --paulr
>
>
>
> *From:* llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] *On Behalf Of *George
> Rimar via llvm-dev
> *Sent:* Monday, December 04, 2017 7:11 AM
> *To:* llvm-dev at lists.llvm.org
> *Subject:* [llvm-dev] [RFC] - Deduplication of debug information in
> linkers (LLD).
>
>
>
> Hi all !
>
>
>
> We have an issue with LLD, it is  "relocation R_X86_64_32 out of range"
> (PR31109)
>
> which occurs during resolving relocations in debug sections. It looks
> happens
>
> because .debug_info section can be too large sometimes and 32x relocation
> is not enough
>
> to represent the value. One of possible solutions looks to be to
> deduplicate information
>
> to reduce .debug_info size.
>
> The rest of mail contains information about experiments I did, the
> obtained results and
>
> some questions and suggestions as well.
>
>
>
> I was investigating idea to deduplicate debug types information. Idea is
> described at
>
> p276 of DWARF4 specification (http://www.dwarfstd.org/doc/DWARF4.pdf). It
> suggests
>
> to split types information out of .debug_info and emit multiple
> .debug_types sections
>
> with use of COMDATs. Both clang and gcc I tested implements
> -fdebug-types-section flag for that:
>
>
>
> -fdebug-types-section, -fno-debug-types-section
>
> Place debug types in their own section (ELF Only)
>
> gcc's description is here: https://gcc.gnu.org/onlinedocs/gcc-6.4.0/gcc/
> Debugging-Options.html#Debugging-Options.
>
>
>
> This flag is disabled by default. I compared clang binaries to see the
> difference
>
> with and without the linker side optimisation.
>
> 1) Clang built with -g has size of 1.7 GB, .debug_info section size is
> 894.5 Mb.
>
> 2) Clang built with -g -fdebug-types-section has size of 1.0 GB.
>
>    .debug_types size is 26.267 MB, .debug_info size is 227.7 MB.
>
>
>
> Difference is huge and I believe shows (though probably for most of
> readers here it was
>
> already obvious) that optimization can be useful. Though
> -fdebug-types-section is disabled by default.
>
> Looks it was initially disabled because not all of DWARF consumers were
> aware of .debug_types section.
>
>
>
> Now in 2017 situation is different. I think most of DWARF consumers knows
> about .debug_types, but:
>
> 1) DWARF5 specification explicitly eliminates the .debug_types section
> introduced in DWARF4:
>
>    p8, "1.4 Changes from Version 4 to Version 5" http://dwarfstd.org/doc/
> DWARF5.pdf
>
> 2) Instead of emiting multiple .debug_types it suggests to emit multiple
> .debug_info COMDAT
>
>    sections. (p375, p376).
>
>
>
> And it seems currently there is no way to make clang to emit multiple
> .debug_info with type information
>
> like DWARF5 suggests. I tried command line below:
>
> -g -fdebug-types-section -gdwarf-5
>
> It still emits .debug_types and does not look there is a flag for emiting
> multiple .debug_info.
>
> Looking at whole LLVM code (lib/mc, lib/CodeGen) actually it seems it is
> just always assumed .debug_info is
>
> a unique section in object.
>
> (also not sure why clang emits .debug_types when -gdwarf-5 flag is set, as
> this section is incompatible with v5,
>
> probably it is a bug).
>
>
>
> So my questions are following:
>
> 1) Do we want to try to implement multiple .debug_info approach ? As it
> seems can be very useful sometimes.
>
> 2) For now in LLD may be we may want to extend our error message from
> "relocation X out of range" to something
>
>    suggesting to use -fdebug-types-section (only for relocations in debug
> sections) ?
>
> 3) Why -fdebug-types-section is disabled by default ?
>
> ​
>
> Best regards,
> George | Developer | Access Softek, Inc
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20171204/e9960766/attachment.html>