[llvm-dev] [RFC] - Deduplication of debug information in linkers (LLD)
Eric Christopher via llvm-dev
llvm-dev at lists.llvm.org
Tue Dec 5 15:18:28 PST 2017
And, IMO, a good time for a post processing tool similar to dsymutil or dwz.
On Mon, Dec 4, 2017, 8:59 PM Rui Ueyama via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Jon,
>
> I think what George suggested is different from making lld to parse,
> deduplicate and rewrite the DWARF debug info. What he suggested is to make
> the compiler emit multiple debug sections so that the linker can eliminate
> them just like it does for, for example, inline functions. The elimination
> is done by (essentially) section name, so it should be quite fast. Parsing
> all debug info and reconstructing it is completely different IMO.
>
> On Mon, Dec 4, 2017 at 3:58 PM, Jon Chesterfield via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> At least one proprietary linker put a lot of effort into deduplicating
>> and rewriting debug information. This took up the majority of the link time
>> despite serious engineering time on performance optimisation. For example,
>> some sections were written from scratch by the linker because that proved
>> faster than parsing the input. Teaching LLD to dedup DWARF should be
>> expected to dramatically slow it down (when enabled, ideally not when
>> disabled).
>>
>> Is a more incremental approach viable? In particular, are there IR passes
>> that fold debug strings etc that could be deployed before feeding
>> everything into a linker?
>>
>>
>> On Mon, Dec 4, 2017 at 10:59 PM, via llvm-dev <llvm-dev at lists.llvm.org>
>> wrote:
>>
>>> Send llvm-dev mailing list submissions to
>>> llvm-dev at lists.llvm.org
>>>
>>> To subscribe or unsubscribe via the World Wide Web, visit
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>> or, via email, send a message with subject or body 'help' to
>>> llvm-dev-request at lists.llvm.org
>>>
>>> You can reach the person managing the list at
>>> llvm-dev-owner at lists.llvm.org
>>>
>>> When replying, please edit your Subject line so it is more specific
>>> than "Re: Contents of llvm-dev digest..."
>>>
>>>
>>> Today's Topics:
>>>
>>> 1. Re: [RFC] Half-Precision Support in the Arm Backends
>>> (Friedman, Eli via llvm-dev)
>>> 2. LLVM Weekly - #205, Dec 4th 2017 (Alex Bradbury via llvm-dev)
>>> 3. Re: [RFC] - Deduplication of debug information in linkers
>>> (LLD). (UE US via llvm-dev)
>>> 4. Re: [RFC] - Deduplication of debug information in linkers
>>> (LLD). (UE US via llvm-dev)
>>> 5. Adding a string-argument Intrinsic (Dounia Khaldi via llvm-dev)
>>> 6. Re: Adding a string-argument Intrinsic
>>> (Jonathan Roelofs via llvm-dev)
>>> 7. Passes to add/validate synthetic debug info
>>> (Vedant Kumar via llvm-dev)
>>> 8. Re: Passes to add/validate synthetic debug info
>>> (Davide Italiano via llvm-dev)
>>> 9. Re: [RFC] - Deduplication of debug information in linkers
>>> (LLD). (Eric Christopher via llvm-dev)
>>>
>>>
>>> ----------------------------------------------------------------------
>>>
>>> Message: 1
>>> Date: Mon, 4 Dec 2017 12:20:11 -0800
>>> From: "Friedman, Eli via llvm-dev" <llvm-dev at lists.llvm.org>
>>> To: Sjoerd Meijer <Sjoerd.Meijer at arm.com>, "llvm-dev at lists.llvm.org"
>>> <llvm-dev at lists.llvm.org>
>>> Subject: Re: [llvm-dev] [RFC] Half-Precision Support in the Arm
>>> Backends
>>> Message-ID: <71e3a663-1b80-8c77-c495-9b5a30ade76a at codeaurora.org>
>>> Content-Type: text/plain; charset="windows-1252"; Format="flowed"
>>>
>>> On 12/4/2017 6:44 AM, Sjoerd Meijer via llvm-dev wrote:
>>> >
>>> > Custom Lowering
>>> > -------------------------
>>> >
>>> > Making f16 legal and not having native load/stores instructions
>>> available,
>>> > (no FullFP16 support) means custom lowering loads/stores:
>>> > 1) Since we don't have FP16 load/store instructions available, we
>>> create
>>> > integer half-word loads. I unfortunately need the FP16_TO_FP node
>>> here,
>>> > because that "models" creating an integer value, which is what we
>>> need
>>> > to create a "truncating i16" integer load instructions. Instead, of
>>> > using
>>> > FP16_TO_FP, I have tried BITCASTs, but this can lead to code
>>> generation
>>> > to stack loads/stores which I don't want.
>>> > 2) Custom lowering f16 stores is very similar, and creates truncating
>>> > half-word integer stores.
>>>
>>> Technically, there are no f16 load/store instructions, yes, but we can
>>> use NEON vdl1 and vst1 to get something roughly equivalent, right?
>>>
>>> You probably want to custom-lower BITCAST instructions; the generic
>>> sequence emitted by the legalizer is pretty inefficient in most cases.
>>>
>>> ---
>>>
>>> Overall, I think your approach makes sense.
>>>
>>> -Eli
>>>
>>> --
>>> Employee of Qualcomm Innovation Center, Inc.
>>> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a
>>> Linux Foundation Collaborative Project
>>>
>>> -------------- next part --------------
>>> An HTML attachment was scrubbed...
>>> URL: <
>>> http://lists.llvm.org/pipermail/llvm-dev/attachments/20171204/4fc9092b/attachment-0001.html
>>> >
>>>
>>> ------------------------------
>>>
>>> Message: 2
>>> Date: Mon, 4 Dec 2017 21:14:19 +0000
>>> From: Alex Bradbury via llvm-dev <llvm-dev at lists.llvm.org>
>>> To: llvm-dev <llvm-dev at lists.llvm.org>
>>> Subject: [llvm-dev] LLVM Weekly - #205, Dec 4th 2017
>>> Message-ID:
>>> <
>>> CA+wH295CPJHG_trD4wbiqStUiN+AvhpwvMdNjcCtZuCWkONY_A at mail.gmail.com>
>>> Content-Type: text/plain; charset="UTF-8"
>>>
>>> LLVM Weekly - #205, Dec 4th 2017
>>> ================================
>>>
>>> If you prefer, you can read a HTML version of this email at
>>> <http://llvmweekly.org/issue/205>.
>>>
>>> Welcome to the two hundred and fifth issue of LLVM Weekly, a weekly
>>> newsletter
>>> (published every Monday) covering developments in LLVM, Clang, and
>>> related
>>> projects. LLVM Weekly is brought to you by [Alex
>>> Bradbury](https://www.linkedin.com/in/alex-bradbury/). Subscribe to
>>> future
>>> issues at <http://llvmweekly.org> and pass it on to anyone else you
>>> think may
>>> be interested. Please send any tips or feedback to <asb at asbradbury.org>,
>>> or
>>> @llvmweekly or @asbradbury on Twitter.
>>>
>>> If you weren't able to make it to the 7th RISC-V Workshop in Milpitas
>>> last
>>> week, you might find my
>>> [blog](
>>> http://www.lowrisc.org/blog/2017/11/seventh-risc-v-workshop-day-one/)
>>> [summary](
>>> http://www.lowrisc.org/blog/2017/11/seventh-risc-v-workshop-day-two/)
>>> on lowrisc.org useful.
>>>
>>>
>>> ## News and articles from around the web
>>>
>>> LLVM/Clang 5.0.1-rc2 [has been
>>> tagged](
>>> http://lists.llvm.org/pipermail/llvm-dev/2017-November/119383.html).
>>>
>>> The call for papers has been
>>> [issued](
>>> http://lists.llvm.org/pipermail/llvm-dev/2017-November/119411.html)
>>> for the LLVM Performance Workshop at CGO 2018. This will take place on
>>> February 24th in Vienna, Austria.
>>>
>>>
>>> ## On the mailing lists
>>>
>>> * Alexey Kukanov has published an RFC on [contributing Intel's Parallel
>>> STL
>>> implementation to
>>> libc++](
>>> http://lists.llvm.org/pipermail/cfe-dev/2017-November/056135.html).
>>>
>>> * Artem Dergachev has [started to
>>> document](
>>> http://lists.llvm.org/pipermail/cfe-dev/2017-November/056095.html) a
>>> number of issues preventing inlining in C++ code.
>>>
>>> * Mohammed Agabaria
>>> [proposes](
>>> http://lists.llvm.org/pipermail/llvm-dev/2017-November/119364.html)
>>> adding a `no-overflow` keyword to the sdiv and udiv IR instructions.
>>>
>>> * Greg Clayton has responded to a user question with a [handy summary of
>>> how
>>> to remotely launch a new process in
>>> LLDB](http://lists.llvm.org/pipermail/lldb-dev/2017-November/012979.html
>>> ).
>>>
>>> * Jonas Thiem is [looking for
>>> answers](
>>> http://lists.llvm.org/pipermail/llvm-dev/2017-November/119269.html)
>>> to a licensing question about the runtime libraries, and [further
>>> elaborates](
>>> http://lists.llvm.org/pipermail/llvm-dev/2017-November/119334.html)
>>> in this follow-up.
>>>
>>>
>>> ## LLVM commits
>>>
>>> * A new SpeculateAroundPHIs pass has been introduced. It's stunningly
>>> well
>>> documented, so well worth a ready.
>>> [r319164](http://reviews.llvm.org/rL319164).
>>>
>>> * The new `-stack-size-section` flag causes metadata to be emitted in an
>>> ELF
>>> section with information on function stack sizes.
>>> [r319430](http://reviews.llvm.org/rL319430).
>>>
>>> * MergeConsecutiveStores is now run a second time, just before
>>> instruction
>>> selection. This allows lowered intrinsics to be merged as well.
>>> [r319036](http://reviews.llvm.org/rL319036).
>>>
>>> * The MachineVerifier PHI and register operand checking has been
>>> improved.
>>> [r319140](http://reviews.llvm.org/rL319140),
>>> [r319141](http://reviews.llvm.org/rL319141).
>>>
>>> * You can now use `-ppc-reg-with-percent-prefix` to get more readable PPC
>>> assembler output. [r319381](http://reviews.llvm.org/rL319381).
>>>
>>> * The ARC backend can now assemble a subset of ARC instructions.
>>> [r319609](http://reviews.llvm.org/rL319609).
>>>
>>>
>>> ## Clang commits
>>>
>>> * clang-format gained options to control sorting of include blocks,
>>> allowing
>>> groups to be sorted individually, merged and sorted, and merged and
>>> resplit.
>>> [r319024](http://reviews.llvm.org/rL319024).
>>>
>>> * A number of math libcalls/builtins are now converted to equivalent LLVM
>>> intrinsics. [r319593](http://reviews.llvm.org/rL319593).
>>>
>>> * clangd gained a fuzzy match scorer.
>>> [r319557](http://reviews.llvm.org/rL319557).
>>>
>>>
>>> ## Other project commits
>>>
>>> * LLD gained support for range extension thunks on AArch64.
>>> [r319307](http://reviews.llvm.org/rL319307).
>>>
>>> * LLD has reduced the size of the .gnu.hash section.
>>> [r319503](http://reviews.llvm.org/rL319503).
>>>
>>> * LLDB remote debugging documentation has been updated.
>>> [r319213](http://reviews.llvm.org/rL319213).
>>>
>>>
>>> ## Review corner
>>>
>>> The LLVM Weekly review corner serves to highlight patches that are stuck
>>> waiting awaiting review, or work from first-time contributors. See
>>> [here](http://llvmweekly.org/reviewcorner) for more information and how
>>> to
>>> submit you work for inclusion. Of course the hope is that highlighting
>>> these
>>> patches will enable LLVM Weekly readers will step up and help to get them
>>> merged. I'll be reporting back each week on any activity generated on
>>> these
>>> patches, as well as sharing a new batch. If you want your patch included
>>> you
>>> must submit it via the linked form.
>>>
>>> After a quiet period, we've had some new patch submissions this week.
>>> Three
>>> were submitted, though [D38778](https://reviews.llvm.org/D38778)
>>> (support for
>>> the PowerPC Signal Processing Engine) has actually had reviews since
>>> being
>>> submitted. Thanks Nemanja Ivanovic!
>>>
>>> * "Allows pretty printing of declarations through the libclang (C) API.
>>> This
>>> is useful for e.g. tooltips in an IDE - hover a function call and get the
>>> corresponding declaration as tooltip."
>>> [D39903](https://reviews.llvm.org/D39903), patch by Nikolai Kosjar.
>>>
>>> * "This patch extends clang's -verify option to support prefixes similar
>>> to
>>> FileCheck's --check-prefixes. While -verify plus preprocessor directives
>>> could
>>> already accomplish the same goal, source is often easier to read/maintain
>>> without preprocessor directives or duplicate passages of code (see
>>> updated
>>> clang tests in patch)." [D39694](https://reviews.llvm.org/D39694),
>>> patch by
>>> Joel E. Denny.
>>>
>>>
>>> ------------------------------
>>>
>>> Message: 3
>>> Date: Mon, 4 Dec 2017 15:20:53 -0600
>>> From: UE US via llvm-dev <llvm-dev at lists.llvm.org>
>>> To: George Rimar <grimar at accesssoftek.com>
>>> Cc: "llvm-dev at lists.llvm.org" <llvm-dev at lists.llvm.org>
>>> Subject: Re: [llvm-dev] [RFC] - Deduplication of debug information in
>>> linkers (LLD).
>>> Message-ID:
>>> <
>>> CABQ7ENJjzD8i+YnZmK796-t9w06OhN45v5Jv_3mcy+Bo_MbxEw at mail.gmail.com>
>>> Content-Type: text/plain; charset="utf-8"
>>>
>>> Funny, I just filed a bug on that last night. Your solutions look like
>>> they'll help me extensively as cutting the size if half will prevent my
>>> 80GB make install issues.
>>> https://bugs.llvm.org/show_bug.cgi?id=35512
>>>
>>> I'll leave the bug open for tracking purposes.
>>>
>>> GNOMETOYS
>>>
>>> On Mon, Dec 4, 2017 at 9:11 AM, George Rimar via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>> > Hi all !
>>> >
>>> > We have an issue with LLD, it is "relocation R_X86_64_32 out of range"
>>> > (PR31109)
>>> > which occurs during resolving relocations in debug sections. It looks
>>> > happens
>>> > because .debug_info section can be too large sometimes and 32x
>>> relocation
>>> > is not enough
>>> > to represent the value. One of possible solutions looks to be to
>>> > deduplicate information
>>> > to reduce .debug_info size.
>>> > The rest of mail contains information about experiments I did, the
>>> > obtained results and
>>> > some questions and suggestions as well.
>>> >
>>> > I was investigating idea to deduplicate debug types information. Idea
>>> is
>>> > described at
>>> > p276 of DWARF4 specification (http://www.dwarfstd.org/doc/DWARF4.pdf).
>>> It
>>> > suggests
>>> > to split types information out of .debug_info and emit multiple
>>> > .debug_types sections
>>> > with use of COMDATs. Both clang and gcc I tested implements
>>> > -fdebug-types-section flag for that:
>>> >
>>> > -fdebug-types-section, -fno-debug-types-section
>>> > Place debug types in their own section (ELF Only)
>>> > gcc's description is here:
>>> https://gcc.gnu.org/onlinedocs/gcc-6.4.0/gcc/
>>> > Debugging-Options.html#Debugging-Options.
>>> >
>>> > This flag is disabled by default. I compared clang binaries to see the
>>> > difference
>>> > with and without the linker side optimisation.
>>> > 1) Clang built with -g has size of 1.7 GB, .debug_info section size is
>>> > 894.5 Mb.
>>> > 2) Clang built with -g -fdebug-types-section has size of 1.0 GB.
>>> > .debug_types size is 26.267 MB, .debug_info size is 227.7 MB.
>>> >
>>> > Difference is huge and I believe shows (though probably for most of
>>> > readers here it was
>>> > already obvious) that optimization can be useful. Though
>>> > -fdebug-types-section is disabled by default.
>>> > Looks it was initially disabled because not all of DWARF consumers were
>>> > aware of .debug_types section.
>>> >
>>> > Now in 2017 situation is different. I think most of DWARF consumers
>>> knows
>>> > about .debug_types, but:
>>> > 1) DWARF5 specification explicitly eliminates the .debug_types section
>>> > introduced in DWARF4:
>>> > p8, "1.4 Changes from Version 4 to Version 5"
>>> http://dwarfstd.org/doc/
>>> > DWARF5.pdf
>>> > 2) Instead of emiting multiple .debug_types it suggests to emit
>>> multiple
>>> > .debug_info COMDAT
>>> > sections. (p375, p376).
>>> >
>>> > And it seems currently there is no way to make clang to emit multiple
>>> > .debug_info with type information
>>> > like DWARF5 suggests. I tried command line below:
>>> > -g -fdebug-types-section -gdwarf-5
>>> > It still emits .debug_types and does not look there is a flag for
>>> emiting
>>> > multiple .debug_info.
>>> > Looking at whole LLVM code (lib/mc, lib/CodeGen) actually it seems it
>>> is
>>> > just always assumed .debug_info is
>>> > a unique section in object.
>>> > (also not sure why clang emits .debug_types when -gdwarf-5 flag is
>>> set, as
>>> > this section is incompatible with v5,
>>> > probably it is a bug).
>>> >
>>> > So my questions are following:
>>> > 1) Do we want to try to implement multiple .debug_info approach ? As it
>>> > seems can be very useful sometimes.
>>> > 2) For now in LLD may be we may want to extend our error message from
>>> > "relocation X out of range" to something
>>> > suggesting to use -fdebug-types-section (only for relocations in
>>> debug
>>> > sections) ?
>>> > 3) Why -fdebug-types-section is disabled by default ?
>>> >
>>> >
>>> > Best regards,
>>> > George | Developer | Access Softek, Inc
>>> >
>>> > _______________________________________________
>>> > LLVM Developers mailing list
>>> > llvm-dev at lists.llvm.org
>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>> >
>>> >
>>> -------------- next part --------------
>>> An HTML attachment was scrubbed...
>>> URL: <
>>> http://lists.llvm.org/pipermail/llvm-dev/attachments/20171204/1e34aba3/attachment-0001.html
>>> >
>>>
>>> ------------------------------
>>>
>>> Message: 4
>>> Date: Mon, 4 Dec 2017 15:23:44 -0600
>>> From: UE US via llvm-dev <llvm-dev at lists.llvm.org>
>>> To: "Robinson, Paul" <paul.robinson at sony.com>
>>> Cc: "llvm-dev at lists.llvm.org" <llvm-dev at lists.llvm.org>, George Rimar
>>> <grimar at accesssoftek.com>
>>> Subject: Re: [llvm-dev] [RFC] - Deduplication of debug information in
>>> linkers (LLD).
>>> Message-ID:
>>> <CABQ7ENL+eZDSRJe9Y2AHJHwNaW+icHDgu_HhbxuHQsM=
>>> MvbMGw at mail.gmail.com>
>>> Content-Type: text/plain; charset="utf-8"
>>>
>>> An old co-worker told me that writing a dwarf support library was the
>>> most
>>> painful experience of his life due to the confusing standards documents,
>>> so
>>> it's not surprising DWARF5 is going slow.
>>>
>>> GNOMETOYS
>>>
>>> On Mon, Dec 4, 2017 at 12:49 PM, Robinson, Paul via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>> > Thanks for providing the experimental data! It clearly shows the
>>> value of
>>> > type sections in DWARF.
>>> >
>>> > Regarding why type sections are off by default, aside from the issue of
>>> > consumers needing to understand them, there is a size penalty to type
>>> > sections that becomes more evident in smaller projects (meaning, fewer
>>> > compilation units). The size penalty can be balanced against the
>>> amount of
>>> > deduplication for a net win, if you have enough duplicates that you can
>>> > eliminate. But it is a tradeoff.
>>> >
>>> > In Sony's case, it is not uncommon for studios to do what are called
>>> > "unity" builds, where you have basically one master .cpp file that does
>>> > #include of each other .cpp file, giving you an LTO-like build. In
>>> this
>>> > case the debug-info production will automatically produce only one
>>> copy of
>>> > each type, and so using type sections would probably make the net debug
>>> > info bigger. And of course an LTO build will deduplicate type info at
>>> the
>>> > metadata level, with a similar effect.
>>> >
>>> > So, I think whether type sections help or hurt will depend on how a
>>> > particular project's build procedure is set up. Clang/LLVM are set up
>>> to
>>> > do lots of smaller compilations and link them all together, in a fairly
>>> > traditional model, and that is where type sections will provide the
>>> most
>>> > benefit. Your data, then, is essentially for a best-case scenario.
>>> Other
>>> > kinds of projects will not benefit as much.
>>> >
>>> >
>>> >
>>> > Regarding DWARF 5 and emitting type sections into the .debug_info
>>> section
>>> > rather than the .debug_types section: The work to support DWARF 5 in
>>> LLVM
>>> > has not gotten very far yet. Conforming to the standard in this
>>> respect is
>>> > certainly on my list, however there are other features that Sony
>>> considers
>>> > higher priority. If you or someone else wants to contribute that
>>> feature
>>> > sooner, that would be excellent! Otherwise, we will get to it in due
>>> time.
>>> >
>>> > Thanks,
>>> >
>>> > --paulr
>>> >
>>> >
>>> >
>>> > *From:* llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] *On Behalf
>>> Of *George
>>> > Rimar via llvm-dev
>>> > *Sent:* Monday, December 04, 2017 7:11 AM
>>> > *To:* llvm-dev at lists.llvm.org
>>> > *Subject:* [llvm-dev] [RFC] - Deduplication of debug information in
>>> > linkers (LLD).
>>> >
>>> >
>>> >
>>> > Hi all !
>>> >
>>> >
>>> >
>>> > We have an issue with LLD, it is "relocation R_X86_64_32 out of range"
>>> > (PR31109)
>>> >
>>> > which occurs during resolving relocations in debug sections. It looks
>>> > happens
>>> >
>>> > because .debug_info section can be too large sometimes and 32x
>>> relocation
>>> > is not enough
>>> >
>>> > to represent the value. One of possible solutions looks to be to
>>> > deduplicate information
>>> >
>>> > to reduce .debug_info size.
>>> >
>>> > The rest of mail contains information about experiments I did, the
>>> > obtained results and
>>> >
>>> > some questions and suggestions as well.
>>> >
>>> >
>>> >
>>> > I was investigating idea to deduplicate debug types information. Idea
>>> is
>>> > described at
>>> >
>>> > p276 of DWARF4 specification (http://www.dwarfstd.org/doc/DWARF4.pdf).
>>> It
>>> > suggests
>>> >
>>> > to split types information out of .debug_info and emit multiple
>>> > .debug_types sections
>>> >
>>> > with use of COMDATs. Both clang and gcc I tested implements
>>> > -fdebug-types-section flag for that:
>>> >
>>> >
>>> >
>>> > -fdebug-types-section, -fno-debug-types-section
>>> >
>>> > Place debug types in their own section (ELF Only)
>>> >
>>> > gcc's description is here:
>>> https://gcc.gnu.org/onlinedocs/gcc-6.4.0/gcc/
>>> > Debugging-Options.html#Debugging-Options.
>>> >
>>> >
>>> >
>>> > This flag is disabled by default. I compared clang binaries to see the
>>> > difference
>>> >
>>> > with and without the linker side optimisation.
>>> >
>>> > 1) Clang built with -g has size of 1.7 GB, .debug_info section size is
>>> > 894.5 Mb.
>>> >
>>> > 2) Clang built with -g -fdebug-types-section has size of 1.0 GB.
>>> >
>>> > .debug_types size is 26.267 MB, .debug_info size is 227.7 MB.
>>> >
>>> >
>>> >
>>> > Difference is huge and I believe shows (though probably for most of
>>> > readers here it was
>>> >
>>> > already obvious) that optimization can be useful. Though
>>> > -fdebug-types-section is disabled by default.
>>> >
>>> > Looks it was initially disabled because not all of DWARF consumers were
>>> > aware of .debug_types section.
>>> >
>>> >
>>> >
>>> > Now in 2017 situation is different. I think most of DWARF consumers
>>> knows
>>> > about .debug_types, but:
>>> >
>>> > 1) DWARF5 specification explicitly eliminates the .debug_types section
>>> > introduced in DWARF4:
>>> >
>>> > p8, "1.4 Changes from Version 4 to Version 5"
>>> http://dwarfstd.org/doc/
>>> > DWARF5.pdf
>>> >
>>> > 2) Instead of emiting multiple .debug_types it suggests to emit
>>> multiple
>>> > .debug_info COMDAT
>>> >
>>> > sections. (p375, p376).
>>> >
>>> >
>>> >
>>> > And it seems currently there is no way to make clang to emit multiple
>>> > .debug_info with type information
>>> >
>>> > like DWARF5 suggests. I tried command line below:
>>> >
>>> > -g -fdebug-types-section -gdwarf-5
>>> >
>>> > It still emits .debug_types and does not look there is a flag for
>>> emiting
>>> > multiple .debug_info.
>>> >
>>> > Looking at whole LLVM code (lib/mc, lib/CodeGen) actually it seems it
>>> is
>>> > just always assumed .debug_info is
>>> >
>>> > a unique section in object.
>>> >
>>> > (also not sure why clang emits .debug_types when -gdwarf-5 flag is
>>> set, as
>>> > this section is incompatible with v5,
>>> >
>>> > probably it is a bug).
>>> >
>>> >
>>> >
>>> > So my questions are following:
>>> >
>>> > 1) Do we want to try to implement multiple .debug_info approach ? As it
>>> > seems can be very useful sometimes.
>>> >
>>> > 2) For now in LLD may be we may want to extend our error message from
>>> > "relocation X out of range" to something
>>> >
>>> > suggesting to use -fdebug-types-section (only for relocations in
>>> debug
>>> > sections) ?
>>> >
>>> > 3) Why -fdebug-types-section is disabled by default ?
>>> >
>>> >
>>> >
>>> > Best regards,
>>> > George | Developer | Access Softek, Inc
>>> >
>>> > _______________________________________________
>>> > LLVM Developers mailing list
>>> > llvm-dev at lists.llvm.org
>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>> >
>>> >
>>> -------------- next part --------------
>>> An HTML attachment was scrubbed...
>>> URL: <
>>> http://lists.llvm.org/pipermail/llvm-dev/attachments/20171204/07993994/attachment-0001.html
>>> >
>>>
>>> ------------------------------
>>>
>>> Message: 5
>>> Date: Mon, 4 Dec 2017 15:29:48 -0600
>>> From: Dounia Khaldi via llvm-dev <llvm-dev at lists.llvm.org>
>>> To: llvm-dev <llvm-dev at lists.llvm.org>
>>> Subject: [llvm-dev] Adding a string-argument Intrinsic
>>> Message-ID:
>>> <CAEL=PfEZA0PYBVhLk8aEiD-+=
>>> DgGfpwK70ahaLFw2HQUSL4Tqw at mail.gmail.com>
>>> Content-Type: text/plain; charset="utf-8"
>>>
>>> hello,
>>>
>>> Is there a way to add an intrinsic that takes a string as argument?
>>>
>>> If yes, what should be the second value provided to the macro //
>>> BUILTIN(__builtin_dummy, "v?", "n")
>>> in tools/clang/include/clang/Basic/BuiltinsX86.def
>>>
>>> Also, what should be the type passed in include/llvm/IR/IntrinsicsX86.td
>>> def dummy : GCCBuiltin<"__builtin_dummy">,
>>> Intrinsic<[], [llvm_???_ty], [IntrArgMemOnly] /* , "llvm.dummy" */>;
>>>
>>>
>>> Thanks,
>>> Dounia
>>> -------------- next part --------------
>>> An HTML attachment was scrubbed...
>>> URL: <
>>> http://lists.llvm.org/pipermail/llvm-dev/attachments/20171204/b89dafd0/attachment-0001.html
>>> >
>>>
>>> ------------------------------
>>>
>>> Message: 6
>>> Date: Mon, 4 Dec 2017 14:38:15 -0700
>>> From: Jonathan Roelofs via llvm-dev <llvm-dev at lists.llvm.org>
>>> To: Dounia Khaldi <khaldi.dounia at gmail.com>, llvm-dev
>>> <llvm-dev at lists.llvm.org>
>>> Subject: Re: [llvm-dev] Adding a string-argument Intrinsic
>>> Message-ID: <b3b4af89-74f6-f330-7aac-96364490e6b7 at codesourcery.com>
>>> Content-Type: text/plain; charset="utf-8"; format=flowed
>>>
>>>
>>>
>>> On 12/4/17 2:29 PM, Dounia Khaldi via llvm-dev wrote:
>>> > hello,
>>> >
>>> > Is there a way to add an intrinsic that takes a string as argument?
>>>
>>> Have a look at __builtin_annotation
>>>
>>>
>>> Jon
>>>
>>> >
>>> > If yes, what should be the second value provided to the macro //
>>> > BUILTIN(__builtin_dummy, "v?", "n")
>>> > in tools/clang/include/clang/Basic/BuiltinsX86.def
>>> >
>>> > Also, what should be the type passed in
>>> include/llvm/IR/IntrinsicsX86.td
>>> > def dummy : GCCBuiltin<"__builtin_dummy">,
>>> > Intrinsic<[], [llvm_???_ty], [IntrArgMemOnly] /* , "llvm.dummy"
>>> */>;
>>> >
>>> >
>>> > Thanks,
>>> > Dounia
>>> >
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > LLVM Developers mailing list
>>> > llvm-dev at lists.llvm.org
>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>> >
>>>
>>> --
>>> Jon Roelofs
>>> jonathan at codesourcery.com
>>> CodeSourcery / Mentor Embedded / Siemens
>>>
>>>
>>> ------------------------------
>>>
>>> Message: 7
>>> Date: Mon, 04 Dec 2017 13:44:15 -0800
>>> From: Vedant Kumar via llvm-dev <llvm-dev at lists.llvm.org>
>>> To: llvm-dev <llvm-dev at lists.llvm.org>
>>> Subject: [llvm-dev] Passes to add/validate synthetic debug info
>>> Message-ID: <5851F5DA-4917-44DB-AB14-32EC70E6E591 at apple.com>
>>> Content-Type: text/plain; charset="utf-8"
>>>
>>> Hi,
>>>
>>> I've prototyped a pair of passes which 1) add synthetic debug info to a
>>> Module and 2) determine how much of that info is lost. The idea is to make
>>> it easier to write targeted test cases for debug info preservation. For
>>> example, here is a quick way to test that Mem2Reg doesn't drop debug info
>>> from one test input:
>>>
>>> > ; RUN: opt < %s -debugify -mem2reg -check-debugify -S 2>&1 | FileCheck
>>> %s
>>> > ; CHECK: CheckDebugify: PASS
>>>
>>> So far, I've used this utility to identify problems in LSR and
>>> instcombine (haven't gotten around to filing bugs yet). There's some more
>>> discussion about this utility on the thread "RFC: [GlobalISel] Towards a
>>> generic MI combiner framework", where Amara and Daniel suggested trying
>>> something similar at the MI level.
>>>
>>> I've put the passes up for review here https://reviews.llvm.org/D40512 <
>>> https://reviews.llvm.org/D40512>. I just wanted to ping llvm-dev to
>>> raise awareness about the proposed passes, and to see if anyone in the
>>> community has concerns/feedback.
>>>
>>> thanks,
>>> vedant
>>> -------------- next part --------------
>>> An HTML attachment was scrubbed...
>>> URL: <
>>> http://lists.llvm.org/pipermail/llvm-dev/attachments/20171204/4b0f6f51/attachment-0001.html
>>> >
>>>
>>> ------------------------------
>>>
>>> Message: 8
>>> Date: Mon, 4 Dec 2017 14:18:37 -0800
>>> From: Davide Italiano via llvm-dev <llvm-dev at lists.llvm.org>
>>> To: Vedant Kumar <vsk at apple.com>
>>> Cc: llvm-dev <llvm-dev at lists.llvm.org>
>>> Subject: Re: [llvm-dev] Passes to add/validate synthetic debug info
>>> Message-ID:
>>> <CACYV=-E=
>>> a-XRyE-Mi0SbT_WiTR6wdaGwvUetffDeVGzd0pYUXw at mail.gmail.com>
>>> Content-Type: text/plain; charset="UTF-8"
>>>
>>> On Mon, Dec 4, 2017 at 1:44 PM, Vedant Kumar <vsk at apple.com> wrote:
>>> > Hi,
>>> >
>>> > I've prototyped a pair of passes which 1) add synthetic debug info to a
>>> > Module and 2) determine how much of that info is lost. The idea is to
>>> make
>>> > it easier to write targeted test cases for debug info preservation. For
>>> > example, here is a quick way to test that Mem2Reg doesn't drop debug
>>> info
>>> > from one test input:
>>> >
>>> > ; RUN: opt < %s -debugify -mem2reg -check-debugify -S 2>&1 | FileCheck
>>> %s
>>> > ; CHECK: CheckDebugify: PASS
>>> >
>>> >
>>> > So far, I've used this utility to identify problems in LSR and
>>> instcombine
>>> > (haven't gotten around to filing bugs yet). There's some more
>>> discussion
>>> > about this utility on the thread "RFC: [GlobalISel] Towards a generic
>>> MI
>>> > combiner framework", where Amara and Daniel suggested trying something
>>> > similar at the MI level.
>>> >
>>> > I've put the passes up for review here https://reviews.llvm.org/D40512.
>>> I
>>> > just wanted to ping llvm-dev to raise awareness about the proposed
>>> passes,
>>> > and to see if anyone in the community has concerns/feedback.
>>> >
>>> > thanks,
>>> > vedant
>>>
>>> After you addressed all my comments, this is pretty close to me. I'll
>>> take another look today or tomorrow but I'm confident we can iterate
>>> in tree.
>>> In other words, +1 from me (you may want to wait a week or so before
>>> committing so that people have a chance to take a look).
>>>
>>>
>>> --
>>> Davide
>>>
>>> "There are no solved problems; there are only problems that are more
>>> or less solved" -- Henri Poincare
>>>
>>>
>>> ------------------------------
>>>
>>> Message: 9
>>> Date: Mon, 04 Dec 2017 22:59:22 +0000
>>> From: Eric Christopher via llvm-dev <llvm-dev at lists.llvm.org>
>>> To: UE US <uexplorer666 at gmail.com>
>>> Cc: "llvm-dev at lists.llvm.org" <llvm-dev at lists.llvm.org>, George Rimar
>>> <grimar at accesssoftek.com>
>>> Subject: Re: [llvm-dev] [RFC] - Deduplication of debug information in
>>> linkers (LLD).
>>> Message-ID:
>>> <
>>> CALehDX6_O7Vw9uftv5Hkpkaje1iVq6H66NZDDD+efGvHFpvOaw at mail.gmail.com>
>>> Content-Type: text/plain; charset="utf-8"
>>>
>>> This isn't a particularly productive email - especially as a number of
>>> people on this list are current contributors to the standard. Mostly
>>> dwarf5
>>> support is lined up behind one of us having the spare cycles to implement
>>> it rather than anything else FWIW :)
>>>
>>> That said, if you have specific feedback about confusing items I'm
>>> definitely happy to help figure out:
>>>
>>> a) some better way to say it,
>>> b) some other implementation to avoid it being confusing
>>>
>>> Having partially implemented a couple of readers and writers at this
>>> point
>>> I agree that it's not the friendliest of documents, but sometimes being
>>> inside of it makes it harder to see where it's causing issues.
>>>
>>> Thanks!
>>>
>>> -eric
>>>
>>> On Mon, Dec 4, 2017 at 1:23 PM UE US via llvm-dev <
>>> llvm-dev at lists.llvm.org>
>>> wrote:
>>>
>>> > An old co-worker told me that writing a dwarf support library was the
>>> most
>>> > painful experience of his life due to the confusing standards
>>> documents, so
>>> > it's not surprising DWARF5 is going slow.
>>> >
>>> > GNOMETOYS
>>> >
>>> > On Mon, Dec 4, 2017 at 12:49 PM, Robinson, Paul via llvm-dev <
>>> > llvm-dev at lists.llvm.org> wrote:
>>> >
>>> >> Thanks for providing the experimental data! It clearly shows the
>>> value
>>> >> of type sections in DWARF.
>>> >>
>>> >> Regarding why type sections are off by default, aside from the issue
>>> of
>>> >> consumers needing to understand them, there is a size penalty to type
>>> >> sections that becomes more evident in smaller projects (meaning, fewer
>>> >> compilation units). The size penalty can be balanced against the
>>> amount of
>>> >> deduplication for a net win, if you have enough duplicates that you
>>> can
>>> >> eliminate. But it is a tradeoff.
>>> >>
>>> >> In Sony's case, it is not uncommon for studios to do what are called
>>> >> "unity" builds, where you have basically one master .cpp file that
>>> does
>>> >> #include of each other .cpp file, giving you an LTO-like build. In
>>> this
>>> >> case the debug-info production will automatically produce only one
>>> copy of
>>> >> each type, and so using type sections would probably make the net
>>> debug
>>> >> info bigger. And of course an LTO build will deduplicate type info
>>> at the
>>> >> metadata level, with a similar effect.
>>> >>
>>> >> So, I think whether type sections help or hurt will depend on how a
>>> >> particular project's build procedure is set up. Clang/LLVM are set
>>> up to
>>> >> do lots of smaller compilations and link them all together, in a
>>> fairly
>>> >> traditional model, and that is where type sections will provide the
>>> most
>>> >> benefit. Your data, then, is essentially for a best-case scenario.
>>> Other
>>> >> kinds of projects will not benefit as much.
>>> >>
>>> >>
>>> >>
>>> >> Regarding DWARF 5 and emitting type sections into the .debug_info
>>> section
>>> >> rather than the .debug_types section: The work to support DWARF 5 in
>>> LLVM
>>> >> has not gotten very far yet. Conforming to the standard in this
>>> respect is
>>> >> certainly on my list, however there are other features that Sony
>>> considers
>>> >> higher priority. If you or someone else wants to contribute that
>>> feature
>>> >> sooner, that would be excellent! Otherwise, we will get to it in due
>>> time.
>>> >>
>>> >> Thanks,
>>> >>
>>> >> --paulr
>>> >>
>>> >>
>>> >>
>>> >> *From:* llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] *On Behalf
>>> Of *George
>>> >> Rimar via llvm-dev
>>> >> *Sent:* Monday, December 04, 2017 7:11 AM
>>> >> *To:* llvm-dev at lists.llvm.org
>>> >> *Subject:* [llvm-dev] [RFC] - Deduplication of debug information in
>>> >> linkers (LLD).
>>> >>
>>> >>
>>> >>
>>> >> Hi all !
>>> >>
>>> >>
>>> >>
>>> >> We have an issue with LLD, it is "relocation R_X86_64_32 out of
>>> range"
>>> >> (PR31109)
>>> >>
>>> >> which occurs during resolving relocations in debug sections. It looks
>>> >> happens
>>> >>
>>> >> because .debug_info section can be too large sometimes and 32x
>>> relocation
>>> >> is not enough
>>> >>
>>> >> to represent the value. One of possible solutions looks to be to
>>> >> deduplicate information
>>> >>
>>> >> to reduce .debug_info size.
>>> >>
>>> >> The rest of mail contains information about experiments I did, the
>>> >> obtained results and
>>> >>
>>> >> some questions and suggestions as well.
>>> >>
>>> >>
>>> >>
>>> >> I was investigating idea to deduplicate debug types information. Idea
>>> is
>>> >> described at
>>> >>
>>> >> p276 of DWARF4 specification (http://www.dwarfstd.org/doc/DWARF4.pdf
>>> ).
>>> >> It suggests
>>> >>
>>> >> to split types information out of .debug_info and emit multiple
>>> >> .debug_types sections
>>> >>
>>> >> with use of COMDATs. Both clang and gcc I tested implements
>>> >> -fdebug-types-section flag for that:
>>> >>
>>> >>
>>> >>
>>> >> -fdebug-types-section, -fno-debug-types-section
>>> >>
>>> >> Place debug types in their own section (ELF Only)
>>> >>
>>> >> gcc's description is here:
>>> >>
>>> https://gcc.gnu.org/onlinedocs/gcc-6.4.0/gcc/Debugging-Options.html#Debugging-Options
>>> >> .
>>> >>
>>> >>
>>> >>
>>> >> This flag is disabled by default. I compared clang binaries to see the
>>> >> difference
>>> >>
>>> >> with and without the linker side optimisation.
>>> >>
>>> >> 1) Clang built with -g has size of 1.7 GB, .debug_info section size is
>>> >> 894.5 Mb.
>>> >>
>>> >> 2) Clang built with -g -fdebug-types-section has size of 1.0 GB.
>>> >>
>>> >> .debug_types size is 26.267 MB, .debug_info size is 227.7 MB.
>>> >>
>>> >>
>>> >>
>>> >> Difference is huge and I believe shows (though probably for most of
>>> >> readers here it was
>>> >>
>>> >> already obvious) that optimization can be useful. Though
>>> >> -fdebug-types-section is disabled by default.
>>> >>
>>> >> Looks it was initially disabled because not all of DWARF consumers
>>> were
>>> >> aware of .debug_types section.
>>> >>
>>> >>
>>> >>
>>> >> Now in 2017 situation is different. I think most of DWARF consumers
>>> knows
>>> >> about .debug_types, but:
>>> >>
>>> >> 1) DWARF5 specification explicitly eliminates the .debug_types section
>>> >> introduced in DWARF4:
>>> >>
>>> >> p8, "1.4 Changes from Version 4 to Version 5"
>>> >> http://dwarfstd.org/doc/DWARF5.pdf
>>> >>
>>> >> 2) Instead of emiting multiple .debug_types it suggests to emit
>>> multiple
>>> >> .debug_info COMDAT
>>> >>
>>> >> sections. (p375, p376).
>>> >>
>>> >>
>>> >>
>>> >> And it seems currently there is no way to make clang to emit multiple
>>> >> .debug_info with type information
>>> >>
>>> >> like DWARF5 suggests. I tried command line below:
>>> >>
>>> >> -g -fdebug-types-section -gdwarf-5
>>> >>
>>> >> It still emits .debug_types and does not look there is a flag for
>>> emiting
>>> >> multiple .debug_info.
>>> >>
>>> >> Looking at whole LLVM code (lib/mc, lib/CodeGen) actually it seems it
>>> is
>>> >> just always assumed .debug_info is
>>> >>
>>> >> a unique section in object.
>>> >>
>>> >> (also not sure why clang emits .debug_types when -gdwarf-5 flag is
>>> set,
>>> >> as this section is incompatible with v5,
>>> >>
>>> >> probably it is a bug).
>>> >>
>>> >>
>>> >>
>>> >> So my questions are following:
>>> >>
>>> >> 1) Do we want to try to implement multiple .debug_info approach ? As
>>> it
>>> >> seems can be very useful sometimes.
>>> >>
>>> >> 2) For now in LLD may be we may want to extend our error message from
>>> >> "relocation X out of range" to something
>>> >>
>>> >> suggesting to use -fdebug-types-section (only for relocations in
>>> debug
>>> >> sections) ?
>>> >>
>>> >> 3) Why -fdebug-types-section is disabled by default ?
>>> >>
>>> >>
>>> >>
>>> >> Best regards,
>>> >> George | Developer | Access Softek, Inc
>>> >>
>>> >> _______________________________________________
>>> >> LLVM Developers mailing list
>>> >> llvm-dev at lists.llvm.org
>>> >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>> >>
>>> >>
>>> > _______________________________________________
>>> > LLVM Developers mailing list
>>> > llvm-dev at lists.llvm.org
>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>> >
>>> -------------- next part --------------
>>> An HTML attachment was scrubbed...
>>> URL: <
>>> http://lists.llvm.org/pipermail/llvm-dev/attachments/20171204/f7d97e3d/attachment.html
>>> >
>>>
>>> ------------------------------
>>>
>>> Subject: Digest Footer
>>>
>>> _______________________________________________
>>> llvm-dev mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>>
>>> ------------------------------
>>>
>>> End of llvm-dev Digest, Vol 162, Issue 10
>>> *****************************************
>>>
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20171205/946282db/attachment-0001.html>
More information about the llvm-dev
mailing list