[llvm-dev] [RFC] - Deduplication of debug information in linkers (LLD).

Thu Dec 7 19:30:42 PST 2017

On Thu, Dec 7, 2017 at 12:20 PM David Blaikie <dblaikie at gmail.com> wrote:

> On Thu, Dec 7, 2017 at 4:47 AM George Rimar <grimar at accesssoftek.com>
> wrote:
>
>> >*nod* That's been the historic ELF+DWARF approach, but both MacOS (with
>> dsyms+DWARF) and Windows
>> >(COFF+CodeView+PDB) don't do it that way, and instead involve the linker
>> to a degree.
>> >Mostly I'm wondering if it'd be reasonable to (and if anyone would be
>> interested in doing it) do
>> >something more like the PDB support - fully debug-aware linking.
>>
>> Honestly saying I only know how ELF linker works and may be my thoughts
>> below are silly for some reason or duplicating
>> some already existent approach. Looking at what .dwp do, looks there are
>> two main things reducing size debug data:
>> 1) "It must allow for the removal of duplicate type units".
>> 2) "It must allow for the removal of duplicate strings".
>>
>
> Yeah, DWPs are mostly the same as a linker linking debug info without
> knowing anything aobut it. Except instead of relocations, it uses the
> cu/tu_index section (& str_index section). Otherwise the DWP packaging tool
> doesn't know anything about the debug info (it doesn't need to parse many
> DIEs, etc).
>
> This is still simple/coarse grained compared to Windows PDBs or MacOS
> dsyms.
>
>
>
>> Linker already deduplicates strings by itself, though it can delegate it
>> to some API for debug sections.
>> And what it could probably do is call some library API. Linker could give
>> it a some set (or all of)
>> .debug_* sections so this library would rebuild and optimize the dwarf
>> data, eliminate duplicates, and
>> return optimized debug sections back to linker. Then linker would perform
>> relocations and emit the result to output.
>>
>> That way library can be used for stand alone post proccessing tool
>> probably
>> and linker should be able to work with data on a sections level only and
>> be not DWARF aware.
>>
>
> Postprocessing (ie: running a tool on the fully linked binary with the
> debug info we have today, and having the tool reprocess the debug info to
> make it more compact) is an option, but wouldn't help address the problem
> you started with - that the output can't fit the large offsets, so the
> output is invalid/broken. So that output would be broken before the
> postprocessing step could run to compact things.
>
>
>>
>> >Sure - but it works/is supported/is implemented. If someone wants to
>> implement the newer thing, that's cool, but I don't have any
>> >personal motivation to do so for example. (& honestly we've been
>> throwing around some ideas about how to further generalize the
>> >debug_info contributions to reduce some of the overhead of isolating
>> types - so maybe if we're lazy enough, we might leapfrog
>> >this particular state and just implement that future better thing)
>>
>> I see. Basing on all comments in this thread I am inclined to agree that
>> implementing newer thing does not make much sence atm.
>> For now I prepared patch to error out when LLD faces objects with
>> multiple .debug_* sections for cases when we do not support it.
>> (D40950). (In LLD we are supporting deduplicating COMDATs, so generally
>> such object is not a problem as already supported,
>> but for error reporting purposes and for --gdb-index we assume debug
>> sections are unique in object,
>> so in that case we looks want to error out).
>>
>> Have last thoughts/question about this though :)
>>
>> Currently clang -gdwarf-5 -fdebug-types-section works. And so linker can
>> deduplicate types. Though that probably violates
>> specification saying there is no more .debug_type sections. But behavior
>> is convinent for users of -fdebug-types-section.
>> I do not know how transition from v4 to v5 will happen/happens (or how
>> transition between dwarf standarts usually happens).
>> I suppose one day clang just will start to produce v5 debug data by
>> default.
>> And at the same time multiple .debug_info sections mentioned in DWARF5
>> spec as an optimization, so it should not be a mandatory
>> thing to implement. If so it just seems that either we will need to
>> implement this optimization before switching to v5 by default or allow
>> -gdwarf-5 -fdebug-types-section to support existent use case. And since
>> it is already works and already allowed in releases it probably means it is
>> acceptable to keep (and use) this behavior ? (If so, attempt to leapfrog
>> can be nice strategy IMO).
>>
>> >>>>I think Paul covered some of the reasons type units might not be a
>> reasonable default.
>> >>>
>> >>>One additional reason is that if you use Split DWARF (another great
>> way to massively reduce the amount of debug info going to the linker)
>> >>>type units are mostly /just/ overhead in the .dwo files: since the
>> debug info is not linked, there's no opportunity to remove the
>> >>>duplication anyway (unless you're making a DWP - like a >dsym file)
>> >>
>> >>Yeah. Looks -gsplit-dwarf and -fdebug-types-section are harmfull
>> together. Probably it worth to restrict using of them together or
>> >>emit a warning (both clang and gcc silently allows the combination and
>> output has size penalty you describing).
>> >
>> >Nah, only if you're not producing a DWP at the end (
>> https://gcc.gnu.org/wiki/DebugFissionDWP ).
>>
>> Sure DWP do great job here it seems, but even for DWP use case flow it
>> does not look make sence to force compiler to do excessive job
>> to produce types sections, because DWP producing tools probably should
>> have no benefit from larger .dwo files with .debug_types at all I think.
>>
>
> The current DWP tools (one in binutils, one in LLVM) don't do DWARF-aware
> debug info compaction. They just concatenate the sections together,
> deduplicate strings, deduplicate type units.
>
> So, yes, to have a smaller DWP file in the end it's beneficial to use type
> units (be they in debug_types or debug_info).
>
> But a fancier DWP tool that would process all the DWARF and compact the
> result wouldn't need explicit type units & could avoid that overhead.
>
>

Prior art is "dwz" written by Jakub Jelinek :)

-eric

> I can only imagine now that somebody could use -gsplit-dwarf and
>> -fdebug-types-section together so that can parse .debug_types.dwo
>> instead of parsing .debug_info.dwo to look for types in a bit more
>> convinent way, but that looks too synthetic case.
>>
>> >In short, I probably wouldn't change any of LLVM's defaults. But there
>> are certainly flags people can use to reduce their debug info size.
>> >
>> >You mentioned starting with this because LLVM's defaults mean the DWARF
>> is too large to link with DWARF 32 bit? How does gold cope with this?
>> >I haven't seen failures/error messages/etc from either gold or lld
>> related to this? (though I mostly use Split DWARF myself)
>>
>> I posted some results earlier here:
>> https://bugs.llvm.org//show_bug.cgi?id=31109#c3,
>> in short: gold 2.26.1 silently ignored this (probably produced broken
>> output), and
>> newer versions of gold are able to report and catch the same error.
>>
>> I think it is simply still not common to have such a large debug
>> sections, we had only single bug about this so far. And hopefully
>> DWARF64 can be a solution, though it can just hide the issue, looks would
>> be nice to reduce amount of debug data we produce still.
>>
>> Best regards,
>> George | Developer | Access Softek, Inc
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20171208/8e5a4b2f/attachment.html>