[llvm-dev] Default compression level for -compress-debug-info=zlib?

Thu Aug 2 16:38:04 PDT 2018

Not really, as well as some sensitivity to the input data overall
performance of the link with compression will depend on how this is
implemented in lld - how is it parallelized?  How is it chunked? Is it
effectively pipelined with IO?

Or, I wouldn't feel comfortable being able to make a recommendation to our
end-users on whether to use this option or not based on my existing
extensive benchmarking of zlib in isolation. It's necessary to test in real
conditions.

Thanks,

Simon

On Thu, 2 Aug 2018, 15:22 Rui Ueyama, <ruiu at google.com> wrote:

> On Thu, Aug 2, 2018 at 2:05 PM Simon Whittaker <
> simon.f.whittaker at gmail.com> wrote:
>
>>
>> > As to (3), in most cases, I believe it is rare to distribute
>> executables with debug info widely
>> >  I think it is at least less important than (1).
>>
>> Agreed.
>>
>> > I think (1) is definitely the case, and that's also true for a
>> distributed build system with which a lot of object files are copied
>> between machines.
>> > My suggestion was to use compression level 9 when both -O2 and
>> -compress-debug-section=zlib are specified.
>>
>> Ok great, I'm less concerned if it still requires an explicit   -compress-debug-section=zlib
>> even with -O2 (I thought you were proposing to add to O2)
>>
>> Still for informational / advisory purposes it would be good for us to
>> produce link time vs compression level vs total exe size, ideally with a
>> couple of different storage types (at least PCIe SSD vs spinning) and CPUs.
>>
>
> Debug sections are compresssed using zlib, so I think such benchmark would
> be testing the performance of zlib itself on various conditions.
>
> Thanks,
>>
>> Simon
>>
>> On Thu, Aug 2, 2018 at 1:17 PM, Rui Ueyama <ruiu at google.com> wrote:
>>
>>> On Thu, Aug 2, 2018 at 10:24 AM Simon Whittaker via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> Hi Rui,
>>>>
>>>> What's the intended advantage for compressing the debug sections? - (i)
>>>> Improved link time through smaller IO / (ii) Improved Load / startup time
>>>> for the debugger / (iii) Smaller exe with debug info for distribution /
>>>> disk space?
>>>>
>>>
>>> I think (1) is definitely the case, and that's also true for a
>>> distributed build system with which a lot of object files are copied
>>> between machines.
>>>
>>> I doubt (2) is true. Does compressing debug sections improve debugger
>>> load time? Of course, as you mentioned, it depends on the ratio of CPU
>>> speed and IO speed, but since linked debug info isn't as large as the total
>>> of input files, I think it is at least less important than (1).
>>>
>>> As to (3), in most cases, I believe it is rare to distribute executables
>>> with debug info widely. Only developers need debug info.
>>>
>>> For i) and ii) how much this is worth it depends on balance of storage
>>>> bandwidth to compression (i) / decompression (ii) bandwidth. For spinning
>>>> drives it *might* be a win but for SATA and especially PCIe / NVMe SSD it
>>>> could be a CPU bottleneck? Though we should also bear in mind that
>>>> compression can be pipelined with writes in i) and debug info loading could
>>>> be lazy in ii)
>>>>
>>>> (e.g. for highly compressible data we've generally seen ~10MiB/s output
>>>> bandwidth on single thread i7 @3.2GHz memory to memory for zlib9 with 32KiB
>>>> window, that doesn't stack up well against modern IO)
>>>>
>>>> How is the compression implemented in lld? Is it chunked and therefore
>>>> paralellizable (and able to be pipelined with IO) or more serial?
>>>>
>>>> I think the intention is i) so we'd be happy to link a few of our game
>>>> titles with varying compression levels vs storage types and let you know
>>>> the results. Might be a couple of weeks...
>>>>
>>>> > I wonder what is the best compression level when -O2 is passed to
>>>> lld.
>>>>
>>>> Just to check, if the default is changed to compress at -O2 we'll still
>>>> be able to override to disable compression with -compress-debug-section=none
>>>> ?
>>>>
>>>
>>> My suggestion was to use compression level 9 when both -O2 and
>>> -compress-debug-section=zlib are specified.
>>>
>>> Thanks,
>>>>
>>>> Simon
>>>>
>>>>
>>>> On Thu, Aug 2, 2018 at 7:00 AM, via llvm-dev <llvm-dev at lists.llvm.org>
>>>> wrote:
>>>>
>>>>> More data on different compression levels will be good.  In this case
>>>>> we're compressing fairly consistent looking input data (a DWARF section) so
>>>>> I think we stand a good chance of being able to pick a very reasonable
>>>>> level.
>>>>>
>>>>> I cringe at the thought of yet another user-facing knob, though.
>>>>>
>>>>> --paulr
>>>>>
>>>>>
>>>>>
>>>>> *From:* llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] *On Behalf
>>>>> Of *James Henderson via llvm-dev
>>>>> *Sent:* Thursday, August 02, 2018 6:32 AM
>>>>> *To:* Pavel Labath
>>>>> *Cc:* LLVM Dev
>>>>> *Subject:* Re: [llvm-dev] Default compression level for
>>>>> -compress-debug-info=zlib?
>>>>>
>>>>>
>>>>>
>>>>> Also not an expert, but would it make sense for this to be
>>>>> configurable at a fine-grained level, perhaps with another option, or an
>>>>> extension to the compress-debug-sections switch interface? That way users
>>>>> who care about the finer details can configure it themselves. And we should
>>>>> pick sensible options for the default.
>>>>>
>>>>>
>>>>>
>>>>> James
>>>>>
>>>>>
>>>>>
>>>>> On 2 August 2018 at 11:08, Pavel Labath via llvm-dev <
>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>
>>>>> I don't claim to be an expert, but I did some zlib compression
>>>>> benchmarks in the past. IIRC, my conclusion from that was that the
>>>>> "DEFAULT" zlib level (6) is indeed a very good default for a lot of
>>>>> cases -- it does not generate much larger outputs, while being
>>>>> significantly faster than the max level. This all depends on the data
>>>>> set and what you intend to do with the resulting data, of course, but
>>>>> I guess my point is you don't have to choose only between 1 and 9. I
>>>>> think it would be interesting to at least get the data for the default
>>>>> level before making choice.
>>>>>
>>>>> cheers,
>>>>> pl
>>>>>
>>>>> On Thu, 2 Aug 2018 at 01:57, Rui Ueyama via llvm-dev
>>>>> <llvm-dev at lists.llvm.org> wrote:
>>>>> >
>>>>> > Folks,
>>>>> >
>>>>> > I'd like to get expert's opinion on which compression level is
>>>>> suitable for lld's -compress-debug-section=zlib option, which let the
>>>>> linker compress .debug_* sections using zlib.
>>>>> >
>>>>> > Currently, lld uses compression level 9 which produces the smallest
>>>>> output in exchange for a longer link time. My question is, is this what
>>>>> people actually want? We didn't consciously choose compression level 9.
>>>>> That was just the default compression level for zlib::compress function.
>>>>> >
>>>>> > For an experiment, I created a patch to use compression level 1
>>>>> instead of 9 and linked clang using that modified lld. By default, lld
>>>>> takes 1m4s to link clang with --compress-debug-sections=zlib. With that
>>>>> patch, it took only 31s.
>>>>> >
>>>>> > Here is a comparison of clang executable size with various
>>>>> configurations:
>>>>> >
>>>>> > no debug sections:    275 MB
>>>>> > level 9 compression:  855 MB
>>>>> > level 1 compression:  922 MB
>>>>> > no compression:      2044 MB
>>>>> >
>>>>> > Given that the best compression takes significantly longer time than
>>>>> the fastest compression, we probably should change the default to level 1.
>>>>> Any objections?
>>>>> >
>>>>> > I wonder what is the best compression level when -O2 is passed to
>>>>> lld. We could use level 9 when -O2 is passed, but is there any reason to
>>>>> compress debug sections that hard in the first place?
>>>>>
>>>>> > _______________________________________________
>>>>> > LLVM Developers mailing list
>>>>> > llvm-dev at lists.llvm.org
>>>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>> _______________________________________________
>>>>> LLVM Developers mailing list
>>>>> llvm-dev at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> LLVM Developers mailing list
>>>>> llvm-dev at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>
>>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180802/a19f2e4f/attachment.html>