[PATCH] D117853: [ELF] Parallelize --compress-debug-sections=zlib

[Fangrui Song via Phabricator via llvm-commits]

MaskRay added a comment.

In D117853#3268063 <https://reviews.llvm.org/D117853#3268063>, @dblaikie wrote:

> Yeah, might be interesting to know where peak linker memory usage is - if this isn't at the peak point, that's fair - less to worry about.
>
>>>>> Another direction to go could be to do compressed data concatenation - if the compression algorithm supports concatenation, you could lose some size benefits and gain speed (like lld's sliding scale of string deduplication) by just concatenating the compressed sections together - predictable size and you could write the updated compressed section header based on the input sections headers.
>>>>
>>>> The concatenation approach is what used here :)
>>>
>>> Ah, sorry, I meant concatenation of the input sections - no need to decompress or recompress, but that only applies if there are no relocations or other changes to apply to the data.
>>
>> Oh, you mean compressing input sections individually and than concatenating them.
>> I've thought about this.
>> One big issue is that initializating zlib data structures takes time.
>> If we create z_stream one for every input section, the overhead may be too high.
>
> Ah, sorry, no, I meant taking the already-compressed input sections and writing them straight to the output without the linker ever decompressing or compressing this data. Which, yeah, only applies if there are no relocations to apply - which is more relevant with dwp (where I mostly have in mind) than with lld (if you're using Split DWARF - if you're not using Split DWARF but you are using DWARFv5, there might be more opportunities for DWARF sections that have no relocations), though some sections even with Split DWARF have no relocations, like .debug_rnglists for instance.

OK, got it:) Strip the zlib header and the trailer of a compressed input section and concatenate the data part.
This is what https://github.com/madler/zlib/blob/master/examples/gzjoin.c#L34 does. It does not re-compress the output but needs to uncompresses input to get the final block marker (BFINAL).
The implementation is a bit involved and more, the compressed data may not be retained (see D52917 <https://reviews.llvm.org/D52917> for `data()`). `--gdb-index` needs to uncompress `.debug_info`.
If we want to leverage this optimization (the output will be larger because the default 32KiB window size is essentially shrunk to the input section size), there would be quite involved changes....


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D117853/new/

https://reviews.llvm.org/D117853