[llvm-dev] Default compression level for -compress-debug-info=zlib?

Fri Aug 3 10:43:35 PDT 2018

On Thu, Aug 2, 2018 at 4:38 PM Simon Whittaker <simon.f.whittaker at gmail.com>
wrote:

> Not really, as well as some sensitivity to the input data overall
> performance of the link with compression will depend on how this is
> implemented in lld - how is it parallelized?  How is it chunked? Is it
> effectively pipelined with IO?
>

In order to compress a section, lld creates an in-memory image of the
section and then passes it to zlib. No parallelization and no chunking.

Or, I wouldn't feel comfortable being able to make a recommendation to our
> end-users on whether to use this option or not based on my existing
> extensive benchmarking of zlib in isolation. It's necessary to test in real
> conditions.
>

Are you already using the option?

> Thanks,
>
> Simon
>
>
> On Thu, 2 Aug 2018, 15:22 Rui Ueyama, <ruiu at google.com> wrote:
>
>> On Thu, Aug 2, 2018 at 2:05 PM Simon Whittaker <
>> simon.f.whittaker at gmail.com> wrote:
>>
>>>
>>> > As to (3), in most cases, I believe it is rare to distribute
>>> executables with debug info widely
>>> >  I think it is at least less important than (1).
>>>
>>> Agreed.
>>>
>>> > I think (1) is definitely the case, and that's also true for a
>>> distributed build system with which a lot of object files are copied
>>> between machines.
>>> > My suggestion was to use compression level 9 when both -O2 and
>>> -compress-debug-section=zlib are specified.
>>>
>>> Ok great, I'm less concerned if it still requires an explicit   -compress-debug-section=zlib
>>> even with -O2 (I thought you were proposing to add to O2)
>>>
>>> Still for informational / advisory purposes it would be good for us to
>>> produce link time vs compression level vs total exe size, ideally with a
>>> couple of different storage types (at least PCIe SSD vs spinning) and CPUs.
>>>
>>
>> Debug sections are compresssed using zlib, so I think such benchmark
>> would be testing the performance of zlib itself on various conditions.
>>
>> Thanks,
>>>
>>> Simon
>>>
>>> On Thu, Aug 2, 2018 at 1:17 PM, Rui Ueyama <ruiu at google.com> wrote:
>>>
>>>> On Thu, Aug 2, 2018 at 10:24 AM Simon Whittaker via llvm-dev <
>>>> llvm-dev at lists.llvm.org> wrote:
>>>>
>>>>> Hi Rui,
>>>>>
>>>>> What's the intended advantage for compressing the debug sections? -
>>>>> (i) Improved link time through smaller IO / (ii) Improved Load / startup
>>>>> time for the debugger / (iii) Smaller exe with debug info for distribution
>>>>> / disk space?
>>>>>
>>>>
>>>> I think (1) is definitely the case, and that's also true for a
>>>> distributed build system with which a lot of object files are copied
>>>> between machines.
>>>>
>>>> I doubt (2) is true. Does compressing debug sections improve debugger
>>>> load time? Of course, as you mentioned, it depends on the ratio of CPU
>>>> speed and IO speed, but since linked debug info isn't as large as the total
>>>> of input files, I think it is at least less important than (1).
>>>>
>>>> As to (3), in most cases, I believe it is rare to distribute
>>>> executables with debug info widely. Only developers need debug info.
>>>>
>>>> For i) and ii) how much this is worth it depends on balance of storage
>>>>> bandwidth to compression (i) / decompression (ii) bandwidth. For spinning
>>>>> drives it *might* be a win but for SATA and especially PCIe / NVMe SSD it
>>>>> could be a CPU bottleneck? Though we should also bear in mind that
>>>>> compression can be pipelined with writes in i) and debug info loading could
>>>>> be lazy in ii)
>>>>>
>>>>> (e.g. for highly compressible data we've generally seen ~10MiB/s
>>>>> output bandwidth on single thread i7 @3.2GHz memory to memory for zlib9
>>>>> with 32KiB window, that doesn't stack up well against modern IO)
>>>>>
>>>>> How is the compression implemented in lld? Is it chunked and therefore
>>>>> paralellizable (and able to be pipelined with IO) or more serial?
>>>>>
>>>>> I think the intention is i) so we'd be happy to link a few of our game
>>>>> titles with varying compression levels vs storage types and let you know
>>>>> the results. Might be a couple of weeks...
>>>>>
>>>>> > I wonder what is the best compression level when -O2 is passed to
>>>>> lld.
>>>>>
>>>>> Just to check, if the default is changed to compress at -O2 we'll
>>>>> still be able to override to disable compression with -compress-debug-section=none
>>>>> ?
>>>>>
>>>>
>>>> My suggestion was to use compression level 9 when both -O2 and
>>>> -compress-debug-section=zlib are specified.
>>>>
>>>> Thanks,
>>>>>
>>>>> Simon
>>>>>
>>>>>
>>>>> On Thu, Aug 2, 2018 at 7:00 AM, via llvm-dev <llvm-dev at lists.llvm.org>
>>>>> wrote:
>>>>>
>>>>>> More data on different compression levels will be good.  In this case
>>>>>> we're compressing fairly consistent looking input data (a DWARF section) so
>>>>>> I think we stand a good chance of being able to pick a very reasonable
>>>>>> level.
>>>>>>
>>>>>> I cringe at the thought of yet another user-facing knob, though.
>>>>>>
>>>>>> --paulr
>>>>>>
>>>>>>
>>>>>>
>>>>>> *From:* llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] *On Behalf
>>>>>> Of *James Henderson via llvm-dev
>>>>>> *Sent:* Thursday, August 02, 2018 6:32 AM
>>>>>> *To:* Pavel Labath
>>>>>> *Cc:* LLVM Dev
>>>>>> *Subject:* Re: [llvm-dev] Default compression level for
>>>>>> -compress-debug-info=zlib?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Also not an expert, but would it make sense for this to be
>>>>>> configurable at a fine-grained level, perhaps with another option, or an
>>>>>> extension to the compress-debug-sections switch interface? That way users
>>>>>> who care about the finer details can configure it themselves. And we should
>>>>>> pick sensible options for the default.
>>>>>>
>>>>>>
>>>>>>
>>>>>> James
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 2 August 2018 at 11:08, Pavel Labath via llvm-dev <
>>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>>
>>>>>> I don't claim to be an expert, but I did some zlib compression
>>>>>> benchmarks in the past. IIRC, my conclusion from that was that the
>>>>>> "DEFAULT" zlib level (6) is indeed a very good default for a lot of
>>>>>> cases -- it does not generate much larger outputs, while being
>>>>>> significantly faster than the max level. This all depends on the data
>>>>>> set and what you intend to do with the resulting data, of course, but
>>>>>> I guess my point is you don't have to choose only between 1 and 9. I
>>>>>> think it would be interesting to at least get the data for the default
>>>>>> level before making choice.
>>>>>>
>>>>>> cheers,
>>>>>> pl
>>>>>>
>>>>>> On Thu, 2 Aug 2018 at 01:57, Rui Ueyama via llvm-dev
>>>>>> <llvm-dev at lists.llvm.org> wrote:
>>>>>> >
>>>>>> > Folks,
>>>>>> >
>>>>>> > I'd like to get expert's opinion on which compression level is
>>>>>> suitable for lld's -compress-debug-section=zlib option, which let the
>>>>>> linker compress .debug_* sections using zlib.
>>>>>> >
>>>>>> > Currently, lld uses compression level 9 which produces the smallest
>>>>>> output in exchange for a longer link time. My question is, is this what
>>>>>> people actually want? We didn't consciously choose compression level 9.
>>>>>> That was just the default compression level for zlib::compress function.
>>>>>> >
>>>>>> > For an experiment, I created a patch to use compression level 1
>>>>>> instead of 9 and linked clang using that modified lld. By default, lld
>>>>>> takes 1m4s to link clang with --compress-debug-sections=zlib. With that
>>>>>> patch, it took only 31s.
>>>>>> >
>>>>>> > Here is a comparison of clang executable size with various
>>>>>> configurations:
>>>>>> >
>>>>>> > no debug sections:    275 MB
>>>>>> > level 9 compression:  855 MB
>>>>>> > level 1 compression:  922 MB
>>>>>> > no compression:      2044 MB
>>>>>> >
>>>>>> > Given that the best compression takes significantly longer time
>>>>>> than the fastest compression, we probably should change the default to
>>>>>> level 1. Any objections?
>>>>>> >
>>>>>> > I wonder what is the best compression level when -O2 is passed to
>>>>>> lld. We could use level 9 when -O2 is passed, but is there any reason to
>>>>>> compress debug sections that hard in the first place?
>>>>>>
>>>>>> > _______________________________________________
>>>>>> > LLVM Developers mailing list
>>>>>> > llvm-dev at lists.llvm.org
>>>>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>> _______________________________________________
>>>>>> LLVM Developers mailing list
>>>>>> llvm-dev at lists.llvm.org
>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> LLVM Developers mailing list
>>>>>> llvm-dev at lists.llvm.org
>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> LLVM Developers mailing list
>>>>> llvm-dev at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180803/8d23f356/attachment.html>