[llvm-dev] ld.lld "Don't let __start_/__stop_ retain C identifier name sections" && Swift

[Fāng-ruì Sòng via llvm-dev]

On Mon, Feb 22, 2021 at 8:08 AM <paul.robinson at sony.com> wrote:

> What this proposes is really at the very edge of my understanding of ELF
> sections, but I have a side project that makes me think the “drop the rule
> entirely (D96914)” part will be a problem for people.
>
>
>
> My side project is to enhance the googletest infrastructure to detect
> un-executed test assertions.  When using Clang as the build compiler, my
> tactics depend on __start/__stop references to C identifier name sections
> retaining everything in those sections.  The data allocated to the section
> does not define any globals so there are no other GC roots.  (I could *
> *almost** get the same tactic to work with GCC as the build compiler, but
> there is one GCC quirk related to inline functions that got in the way, so
> I do something more complicated and ugly there.)
>
>
>
> In researching how to make this work, it appears that depending on this
> behavior of __start/__stop is a not-uncommon tactic; it is fairly well
> known to work with GNU linkers and LLD.  In effect you can allocate static
> data elements to the section at arbitrary points, and the __start/__stop
> symbols let you treat the entire thing as an array.  It is impractical to
> generate unique global symbols for the data elements, and even if you do,
> it is not possible to generate references to those global symbols from
> elsewhere.  And in general, you do **not** want the static elements to be
> GC’d; it defeats the purpose of allocating them in the first place.
> There’s no use of SHF_LINK_ORDER or SHF_GROUP here; these are normal static
> variables allocated to a custom section.  In my case, I can’t depend on the
> order of elements anyway, and macro invocations can’t tell whether they’re
> invoked inside templates so I can’t use SHF_GROUP either.  I end up sorting
> and deduplicating data manually when it’s time to look at everything.
>
>
>
> I see the idea for adding a new Clang attribute to “retain” something, but
> mainly what that does is create work for anyone depending on the historical
> behavior; we have to conditionalize the set of attributes based on whether
> Clang understands “retain” and then cross our fingers hoping we don’t end
> up in a situation with a pre-retain Clang and a post-retain LLD, because
> that will break everything.
>
>
>
> I hope this is clear enough, let me know if my explanation doesn’t make
> any sense.
>
> Thanks,
>
> --paulr
>

On https://reviews.llvm.org/D96838#2585171

> Aha; attribute used *by itself* is not sufficient to preserve sections in
the output. But the __start_/__stop_ symbols implicitly create a reference
to each of the named sections, and that implicit reference can preserve
them in the output (assuming gc roots etc). So, the idea is that attribute
retain can be used *instead* of the __start_/__stop_ symbols, to preserve
sections in the output (with the advantage that it will work even for
sections that do not have a C-identifier name).

> Thanks for helping me understand this from a user perspective. That will
be important when you go to write the release note for this new attribute.

I dug up the history a bit. gold had this behavior in 2010.
GNU ld got a workaround in 2010 partly because glibc refused to fix the
issue (facepalm) https://sourceware.org/bugzilla/show_bug.cgi?id=3400
Before 2015, the GNU ld behavior only applied to sections in the same .o of
the __start_/__stop_ references. In 2015-10, the behavior finally applied
to other .o files.

2015-10 is relatively new, so I don't think there are many applications
depending on the behavior.
But there are indeed some applications.

I submitted a GNU ld patch for -z start-stop-gc which has been accepted by
Alan Modra (https://sourceware.org/bugzilla/show_bug.cgi?id=27451).
I think at some point (14.0.0?) we can still switch the default.


>
>
> *From:* llvm-dev <llvm-dev-bounces at lists.llvm.org> *On Behalf Of *James
> Henderson via llvm-dev
> *Sent:* Monday, February 22, 2021 5:35 AM
> *To:* Fangrui Song <maskray at google.com>
> *Cc:* LLVM Dev <llvm-dev at lists.llvm.org>
> *Subject:* Re: [llvm-dev] ld.lld "Don't let __start_/__stop_ retain C
> identifier name sections" && Swift
>
>
>
> I've filed an internal issue tracker for us to investigate the impact of
> this proposal, although I don't know when we'll get a chance to schedule
> the work at this point. Also, it's worth noting that we can't test all
> downstream codebases that potentially could use this feature, so we'll
> likely want some potential source code or switch that will allow our users
> to keep their unused sections.
>
>
>
> On Sat, 20 Feb 2021 at 00:01, Fangrui Song via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>
> tl;dr With --gc-sections, I think the rule "__start_foo/__stop_foo
> references from live sections retains all non-SHF_LINK_ORDER input sections
> foo" does not cary its weight, so I'd like to drop it entirely in
> https://reviews.llvm.org/D96914
> <https://urldefense.com/v3/__https:/reviews.llvm.org/D96914__;!!JmoZiZGBv3RvKRSx!pAsf5XQa2GgpqsyGaZGjOEeqTeliL8ikbNK-wdQ1KCaKgZDaTWVo-EyJNUPFMdsL_g$>
>
> I have done a large-scale internal test with huge amount of OSS usage and
> spotted two issues:
>
> (1) Linking systemd.
> https://github.com/systemd/systemd/blob/main/src/libsystemd/sd-bus/bus-error.h#L33
> <https://urldefense.com/v3/__https:/github.com/systemd/systemd/blob/main/src/libsystemd/sd-bus/bus-error.h*L33__;Iw!!JmoZiZGBv3RvKRSx!pAsf5XQa2GgpqsyGaZGjOEeqTeliL8ikbNK-wdQ1KCaKgZDaTWVo-EyJNUNdvKsIKw$>
> there will be an `undefined symbol: __start_SYSTEMD_BUS_ERROR_MAP` error.
> Supposedly it can be trivially fixed by using undefined weak symbols on
> __start_/__stop_.
> (2) Linking Swift. There will be errors like `undefined hidden symbol:
> __start_swift5_protocols`.
>
> https://github.com/apple/swift/blob/main/stdlib/public/runtime/SwiftRT-ELF.cpp
> <https://urldefense.com/v3/__https:/github.com/apple/swift/blob/main/stdlib/public/runtime/SwiftRT-ELF.cpp__;!!JmoZiZGBv3RvKRSx!pAsf5XQa2GgpqsyGaZGjOEeqTeliL8ikbNK-wdQ1KCaKgZDaTWVo-EyJNUPUQLE7WQ$>
>     It seems that trivially making `extern const char __start_##name` does
> not work.
>     The code relies on some `swift5_*` input sections being GC root.
>     (If someone can file an issue to Swift, I'd appreciate that.)
>     (If Swift folks can fix it, I'll give my big thanks:)
>
> This can still potentially break some propritary code so I am sending this
> heads-up.
> I'll place rationale below (it is complicated).
>
>
>
> The current rule is:
>
>    __start_/__stop_ references retains all non-SHF_LINK_ORDER C identifier
> name sections.
>
> After https://reviews.llvm.org/D96753
> <https://urldefense.com/v3/__https:/reviews.llvm.org/D96753__;!!JmoZiZGBv3RvKRSx!pAsf5XQa2GgpqsyGaZGjOEeqTeliL8ikbNK-wdQ1KCaKgZDaTWVo-EyJNUOp7Zgkwg$>
> , it will become
>
>    __start_/__stop_ references retains all non-SHF_LINK_ORDER
> non-SHF_GROUP C identifier name sections.
>
> (The section group special case is to allow garbage collecting
> __llvm_prf_* sections for -fprofile-generate/-fprofile-instr-generate. The
> saving is huge.)
>
> Personally I'd drop the rule entirely (D96914) (get support from
> jhenderson and phosek), i.e.
>
>    __start_/__stop_ references do not retain C identifier name sections.
>
> and hope folks can fix Swift/systemd to not rely on the original rule.
>
> ---
>
> I have placed more details in
> https://maskray.me/blog/2021-01-31-metadata-sections-comdat-and-shf-link-order
> <https://urldefense.com/v3/__https:/maskray.me/blog/2021-01-31-metadata-sections-comdat-and-shf-link-order__;!!JmoZiZGBv3RvKRSx!pAsf5XQa2GgpqsyGaZGjOEeqTeliL8ikbNK-wdQ1KCaKgZDaTWVo-EyJNUPBUkT5Qg$>
> discussing why the rule gets in the away and why SHF_LINK_ORDER is not a
> solution.
> (Section groups have size overhead for single metadata section.)
>
> I'll paste the relevant paragraph here for your convenience.
> (I may edit my article to make it clear)
>
> This is a common usage of metadata sections: each text section references
> a metadata section.
> The metadata sections have a C identifier name to allow the runtime to
> collect them via `__start_`/`__stop_` symbols.
>
> Since `__start_`/`__stop_` references are always present from live
> sections, the C identifier name sections appear like GC roots, which means
> they cannot be discarded by `ld --gc-sections`.
>
> For users who want to keep GC for these metadata sections, they can set
> the `SHF_LINK_ORDER` flag or make the metadata section a member of a
> section group.
> (GNU ld does not implement the `SHF_LINK_ORDER` rule yet. <
> https://sourceware.org/bugzilla/show_bug.cgi?id=27259
> <https://urldefense.com/v3/__https:/sourceware.org/bugzilla/show_bug.cgi?id=27259__;!!JmoZiZGBv3RvKRSx!pAsf5XQa2GgpqsyGaZGjOEeqTeliL8ikbNK-wdQ1KCaKgZDaTWVo-EyJNUNYIjk40Q$>
> >)
> (In LLD, some folks have concluded that this rule does not cary its
> weight, so possibly it would be nice it we can drop it ([D96914](
> https://reviews.llvm.org/D96914))
> <https://urldefense.com/v3/__https:/reviews.llvm.org/D96914))__;!!JmoZiZGBv3RvKRSx!pAsf5XQa2GgpqsyGaZGjOEeqTeliL8ikbNK-wdQ1KCaKgZDaTWVo-EyJNUMtgSFzag$>
> .)
>
> Now, let's walk through an `SHF_LINK_ORDER` example when inlining can
> cause problems.
>
> ```asm
> # Monolithic meta.
> .globl _start
> _start:
>    leaq __start_meta(%rip), %rdi
>    leaq __stop_meta(%rip), %rsi
>
> .section .text.foo,"ax", at progbits
> .globl foo
> foo:
>    leaq .Lmeta.foo(%rip), %rax
>    ret
>
> .section .text.bar,"ax", at progbits
> .globl bar
> bar:
>    call foo
>    leaq .Lmeta.bar(%rip), %rax
>    ret
>
> .section meta,"a", at progbits
> .Lmeta.foo:
>    .byte 0
> .Lmeta.bar:
>    .byte 1
> ```
>
> The monolithic `meta` does not enable precise garbage collection.
> It may be attempting to make `meta` separate and add the `SHF_LINK_ORDER`
> flag (to defeat the "C identifier name sections appear like GC roots" rule):
>
> ```asm
> .section meta,"ao", at progbits,foo
> .Lmeta.foo:
>    .byte 0
>
> .section meta,"ao", at progbits,bar
> .Lmeta.bar:
>    .byte 1
> ```
>
> However, due to inlining (foo into bar), the `meta` for `.text.foo` may
> now get a reference from another text section `.text.bar`, breaking an
> implicit assumption of `SHF_LINK_ORDER`: such a section can only be
> referenced from its linked-to section.
> ```asm
> # Both .text.foo and .text.bar reference meta.
> .section .text.foo,"ax", at progbits
> .globl foo
> foo:
>    leaq .Lmeta.foo(%rip), %rax
>    ret
>
> .section .text.bar,"ax", at progbits
> .globl bar
> bar:
>    leaq .Lmeta.foo(%rip), %rax
>    leaq .Lmeta.bar(%rip), %rax
>    ret
> ```
>
> If `.text.bar` (caller) is retained while `.text.foo` (callee) is
> discarded, the `meta` for `foo` will link to a discarded section: an
> invalid state.
> LLD has an error: `{{.*}}:(meta): sh_link points to discarded section
> {{.*}}:(.text.foo)`.
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> <https://urldefense.com/v3/__https:/lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev__;!!JmoZiZGBv3RvKRSx!pAsf5XQa2GgpqsyGaZGjOEeqTeliL8ikbNK-wdQ1KCaKgZDaTWVo-EyJNUNDA2XUCA$>
>
>

-- 
宋方睿
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210301/5fc0ec30/attachment.html>