[llvm-dev] Default compression level for -compress-debug-info=zlib?

Thu Aug 2 14:05:38 PDT 2018

> As to (3), in most cases, I believe it is rare to distribute executables
with debug info widely
>  I think it is at least less important than (1).

Agreed.

> I think (1) is definitely the case, and that's also true for a
distributed build system with which a lot of object files are copied
between machines.
> My suggestion was to use compression level 9 when both -O2 and
-compress-debug-section=zlib are specified.

Ok great, I'm less concerned if it still requires an explicit
-compress-debug-section=zlib
even with -O2 (I thought you were proposing to add to O2)

Still for informational / advisory purposes it would be good for us to
produce link time vs compression level vs total exe size, ideally with a
couple of different storage types (at least PCIe SSD vs spinning) and CPUs.

Thanks,

Simon

On Thu, Aug 2, 2018 at 1:17 PM, Rui Ueyama <ruiu at google.com> wrote:

> On Thu, Aug 2, 2018 at 10:24 AM Simon Whittaker via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Hi Rui,
>>
>> What's the intended advantage for compressing the debug sections? - (i)
>> Improved link time through smaller IO / (ii) Improved Load / startup time
>> for the debugger / (iii) Smaller exe with debug info for distribution /
>> disk space?
>>
>
> I think (1) is definitely the case, and that's also true for a distributed
> build system with which a lot of object files are copied between machines.
>
> I doubt (2) is true. Does compressing debug sections improve debugger load
> time? Of course, as you mentioned, it depends on the ratio of CPU speed and
> IO speed, but since linked debug info isn't as large as the total of input
> files, I think it is at least less important than (1).
>
> As to (3), in most cases, I believe it is rare to distribute executables
> with debug info widely. Only developers need debug info.
>
> For i) and ii) how much this is worth it depends on balance of storage
>> bandwidth to compression (i) / decompression (ii) bandwidth. For spinning
>> drives it *might* be a win but for SATA and especially PCIe / NVMe SSD it
>> could be a CPU bottleneck? Though we should also bear in mind that
>> compression can be pipelined with writes in i) and debug info loading could
>> be lazy in ii)
>>
>> (e.g. for highly compressible data we've generally seen ~10MiB/s output
>> bandwidth on single thread i7 @3.2GHz memory to memory for zlib9 with 32KiB
>> window, that doesn't stack up well against modern IO)
>>
>> How is the compression implemented in lld? Is it chunked and therefore
>> paralellizable (and able to be pipelined with IO) or more serial?
>>
>> I think the intention is i) so we'd be happy to link a few of our game
>> titles with varying compression levels vs storage types and let you know
>> the results. Might be a couple of weeks...
>>
>> > I wonder what is the best compression level when -O2 is passed to lld.
>>
>> Just to check, if the default is changed to compress at -O2 we'll still
>> be able to override to disable compression with -compress-debug-section=none
>> ?
>>
>
> My suggestion was to use compression level 9 when both -O2 and
> -compress-debug-section=zlib are specified.
>
> Thanks,
>>
>> Simon
>>
>>
>> On Thu, Aug 2, 2018 at 7:00 AM, via llvm-dev <llvm-dev at lists.llvm.org>
>> wrote:
>>
>>> More data on different compression levels will be good.  In this case
>>> we're compressing fairly consistent looking input data (a DWARF section) so
>>> I think we stand a good chance of being able to pick a very reasonable
>>> level.
>>>
>>> I cringe at the thought of yet another user-facing knob, though.
>>>
>>> --paulr
>>>
>>>
>>>
>>> *From:* llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] *On Behalf Of
>>> *James Henderson via llvm-dev
>>> *Sent:* Thursday, August 02, 2018 6:32 AM
>>> *To:* Pavel Labath
>>> *Cc:* LLVM Dev
>>> *Subject:* Re: [llvm-dev] Default compression level for
>>> -compress-debug-info=zlib?
>>>
>>>
>>>
>>> Also not an expert, but would it make sense for this to be configurable
>>> at a fine-grained level, perhaps with another option, or an extension to
>>> the compress-debug-sections switch interface? That way users who care about
>>> the finer details can configure it themselves. And we should pick sensible
>>> options for the default.
>>>
>>>
>>>
>>> James
>>>
>>>
>>>
>>> On 2 August 2018 at 11:08, Pavel Labath via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>> I don't claim to be an expert, but I did some zlib compression
>>> benchmarks in the past. IIRC, my conclusion from that was that the
>>> "DEFAULT" zlib level (6) is indeed a very good default for a lot of
>>> cases -- it does not generate much larger outputs, while being
>>> significantly faster than the max level. This all depends on the data
>>> set and what you intend to do with the resulting data, of course, but
>>> I guess my point is you don't have to choose only between 1 and 9. I
>>> think it would be interesting to at least get the data for the default
>>> level before making choice.
>>>
>>> cheers,
>>> pl
>>>
>>> On Thu, 2 Aug 2018 at 01:57, Rui Ueyama via llvm-dev
>>> <llvm-dev at lists.llvm.org> wrote:
>>> >
>>> > Folks,
>>> >
>>> > I'd like to get expert's opinion on which compression level is
>>> suitable for lld's -compress-debug-section=zlib option, which let the
>>> linker compress .debug_* sections using zlib.
>>> >
>>> > Currently, lld uses compression level 9 which produces the smallest
>>> output in exchange for a longer link time. My question is, is this what
>>> people actually want? We didn't consciously choose compression level 9.
>>> That was just the default compression level for zlib::compress function.
>>> >
>>> > For an experiment, I created a patch to use compression level 1
>>> instead of 9 and linked clang using that modified lld. By default, lld
>>> takes 1m4s to link clang with --compress-debug-sections=zlib. With that
>>> patch, it took only 31s.
>>> >
>>> > Here is a comparison of clang executable size with various
>>> configurations:
>>> >
>>> > no debug sections:    275 MB
>>> > level 9 compression:  855 MB
>>> > level 1 compression:  922 MB
>>> > no compression:      2044 MB
>>> >
>>> > Given that the best compression takes significantly longer time than
>>> the fastest compression, we probably should change the default to level 1.
>>> Any objections?
>>> >
>>> > I wonder what is the best compression level when -O2 is passed to lld.
>>> We could use level 9 when -O2 is passed, but is there any reason to
>>> compress debug sections that hard in the first place?
>>>
>>> > _______________________________________________
>>> > LLVM Developers mailing list
>>> > llvm-dev at lists.llvm.org
>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>>
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180802/139721c0/attachment.html>