[llvm-dev] Default compression level for -compress-debug-info=zlib?

Thu Aug 2 15:21:46 PDT 2018

On Thu, Aug 2, 2018 at 2:05 PM Simon Whittaker <simon.f.whittaker at gmail.com>
wrote:

>
> > As to (3), in most cases, I believe it is rare to distribute
> executables with debug info widely
> >  I think it is at least less important than (1).
>
> Agreed.
>
> > I think (1) is definitely the case, and that's also true for a
> distributed build system with which a lot of object files are copied
> between machines.
> > My suggestion was to use compression level 9 when both -O2 and
> -compress-debug-section=zlib are specified.
>
> Ok great, I'm less concerned if it still requires an explicit   -compress-debug-section=zlib
> even with -O2 (I thought you were proposing to add to O2)
>
> Still for informational / advisory purposes it would be good for us to
> produce link time vs compression level vs total exe size, ideally with a
> couple of different storage types (at least PCIe SSD vs spinning) and CPUs.
>

Debug sections are compresssed using zlib, so I think such benchmark would
be testing the performance of zlib itself on various conditions.

Thanks,
>
> Simon
>
> On Thu, Aug 2, 2018 at 1:17 PM, Rui Ueyama <ruiu at google.com> wrote:
>
>> On Thu, Aug 2, 2018 at 10:24 AM Simon Whittaker via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> Hi Rui,
>>>
>>> What's the intended advantage for compressing the debug sections? - (i)
>>> Improved link time through smaller IO / (ii) Improved Load / startup time
>>> for the debugger / (iii) Smaller exe with debug info for distribution /
>>> disk space?
>>>
>>
>> I think (1) is definitely the case, and that's also true for a
>> distributed build system with which a lot of object files are copied
>> between machines.
>>
>> I doubt (2) is true. Does compressing debug sections improve debugger
>> load time? Of course, as you mentioned, it depends on the ratio of CPU
>> speed and IO speed, but since linked debug info isn't as large as the total
>> of input files, I think it is at least less important than (1).
>>
>> As to (3), in most cases, I believe it is rare to distribute executables
>> with debug info widely. Only developers need debug info.
>>
>> For i) and ii) how much this is worth it depends on balance of storage
>>> bandwidth to compression (i) / decompression (ii) bandwidth. For spinning
>>> drives it *might* be a win but for SATA and especially PCIe / NVMe SSD it
>>> could be a CPU bottleneck? Though we should also bear in mind that
>>> compression can be pipelined with writes in i) and debug info loading could
>>> be lazy in ii)
>>>
>>> (e.g. for highly compressible data we've generally seen ~10MiB/s output
>>> bandwidth on single thread i7 @3.2GHz memory to memory for zlib9 with 32KiB
>>> window, that doesn't stack up well against modern IO)
>>>
>>> How is the compression implemented in lld? Is it chunked and therefore
>>> paralellizable (and able to be pipelined with IO) or more serial?
>>>
>>> I think the intention is i) so we'd be happy to link a few of our game
>>> titles with varying compression levels vs storage types and let you know
>>> the results. Might be a couple of weeks...
>>>
>>> > I wonder what is the best compression level when -O2 is passed to lld.
>>>
>>> Just to check, if the default is changed to compress at -O2 we'll still
>>> be able to override to disable compression with -compress-debug-section=none
>>> ?
>>>
>>
>> My suggestion was to use compression level 9 when both -O2 and
>> -compress-debug-section=zlib are specified.
>>
>> Thanks,
>>>
>>> Simon
>>>
>>>
>>> On Thu, Aug 2, 2018 at 7:00 AM, via llvm-dev <llvm-dev at lists.llvm.org>
>>> wrote:
>>>
>>>> More data on different compression levels will be good.  In this case
>>>> we're compressing fairly consistent looking input data (a DWARF section) so
>>>> I think we stand a good chance of being able to pick a very reasonable
>>>> level.
>>>>
>>>> I cringe at the thought of yet another user-facing knob, though.
>>>>
>>>> --paulr
>>>>
>>>>
>>>>
>>>> *From:* llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] *On Behalf
>>>> Of *James Henderson via llvm-dev
>>>> *Sent:* Thursday, August 02, 2018 6:32 AM
>>>> *To:* Pavel Labath
>>>> *Cc:* LLVM Dev
>>>> *Subject:* Re: [llvm-dev] Default compression level for
>>>> -compress-debug-info=zlib?
>>>>
>>>>
>>>>
>>>> Also not an expert, but would it make sense for this to be configurable
>>>> at a fine-grained level, perhaps with another option, or an extension to
>>>> the compress-debug-sections switch interface? That way users who care about
>>>> the finer details can configure it themselves. And we should pick sensible
>>>> options for the default.
>>>>
>>>>
>>>>
>>>> James
>>>>
>>>>
>>>>
>>>> On 2 August 2018 at 11:08, Pavel Labath via llvm-dev <
>>>> llvm-dev at lists.llvm.org> wrote:
>>>>
>>>> I don't claim to be an expert, but I did some zlib compression
>>>> benchmarks in the past. IIRC, my conclusion from that was that the
>>>> "DEFAULT" zlib level (6) is indeed a very good default for a lot of
>>>> cases -- it does not generate much larger outputs, while being
>>>> significantly faster than the max level. This all depends on the data
>>>> set and what you intend to do with the resulting data, of course, but
>>>> I guess my point is you don't have to choose only between 1 and 9. I
>>>> think it would be interesting to at least get the data for the default
>>>> level before making choice.
>>>>
>>>> cheers,
>>>> pl
>>>>
>>>> On Thu, 2 Aug 2018 at 01:57, Rui Ueyama via llvm-dev
>>>> <llvm-dev at lists.llvm.org> wrote:
>>>> >
>>>> > Folks,
>>>> >
>>>> > I'd like to get expert's opinion on which compression level is
>>>> suitable for lld's -compress-debug-section=zlib option, which let the
>>>> linker compress .debug_* sections using zlib.
>>>> >
>>>> > Currently, lld uses compression level 9 which produces the smallest
>>>> output in exchange for a longer link time. My question is, is this what
>>>> people actually want? We didn't consciously choose compression level 9.
>>>> That was just the default compression level for zlib::compress function.
>>>> >
>>>> > For an experiment, I created a patch to use compression level 1
>>>> instead of 9 and linked clang using that modified lld. By default, lld
>>>> takes 1m4s to link clang with --compress-debug-sections=zlib. With that
>>>> patch, it took only 31s.
>>>> >
>>>> > Here is a comparison of clang executable size with various
>>>> configurations:
>>>> >
>>>> > no debug sections:    275 MB
>>>> > level 9 compression:  855 MB
>>>> > level 1 compression:  922 MB
>>>> > no compression:      2044 MB
>>>> >
>>>> > Given that the best compression takes significantly longer time than
>>>> the fastest compression, we probably should change the default to level 1.
>>>> Any objections?
>>>> >
>>>> > I wonder what is the best compression level when -O2 is passed to
>>>> lld. We could use level 9 when -O2 is passed, but is there any reason to
>>>> compress debug sections that hard in the first place?
>>>>
>>>> > _______________________________________________
>>>> > LLVM Developers mailing list
>>>> > llvm-dev at lists.llvm.org
>>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180802/e7bbf7f2/attachment.html>