[llvm-dev] RFC: ELF Autolinking

Thu Mar 14 10:27:47 PDT 2019

On Thu, Mar 14, 2019 at 9:59 AM Rui Ueyama via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> On Thu, Mar 14, 2019 at 9:28 AM Reid Kleckner via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Hi,
>>
>> I guess I agree it would be best to remove the objfile linker option
>> support and replace it with just auto-linking. We already have a mechanism
>> for adding new features to object files: .note sections. Linkers already
>> know to ignore ones that they don't understand. If, in the future, we want
>> to add a new feature that could be handled by embedding linker flags, we
>> can instead implement it with a new .note section that other linkers and
>> old versions of LLD will know to ignore.
>>
>> On top of that, the generic ABI group has previously rejected proposals
>> to embed linker options in object files (
>> https://groups.google.com/forum/#!topic/generic-abi/iS_-m-X5ZwQ).
>>
>> Given how ELF has done things in the past, maybe the section name should
>> be ".note.autolink". We could also be like GCC and namespace our extensions
>> as ".note.LLVM.autolink", but maybe that's a step too far.
>>
>
> A .note section consists of a series of type-length-value records. My
> understanding is that the static linker aggregates them to a single
> location and put it into PT_NOTE segment, and the records can still be read
> by the loader even after the section table is stripped from an executable.
> For the proposed purpose, the note section header would not be useful or
> meaningful, so a plain section that just contains an ASCII string would be
> simpler.
>

I agree that this should not be a note section because the section is
meaningless in a final linked executable.

Note that the proposal is limited to adding -l flags only. I would not
expect this particular section to be expanded in the future to cover other
use cases; rather, we would likely invent more section types.

Peter

> Reid
>>
>> On Thu, Mar 14, 2019 at 6:08 AM bd1976 llvm via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> At Sony we offer autolinking as a feature in our ELF toolchain. We would
>>> like to see full support for this feature upstream as there is anecdotal
>>> evidence that it would find use beyond Sony.
>>>
>>> In general autolinking (https://en.wikipedia.org/wiki/Auto-linking)
>>> allows developers to specify inputs to the linker in their source code.
>>> LLVM and Clang already have support for autolinking on ELF via embedding
>>> strings, which specify linker behavior, into a .linker-options section in
>>> relocatable object files, see:
>>>
>>> RFC - http://lists.llvm.org/pipermail/llvm-dev/2018-January/120101.html
>>> LLVM -
>>> https://llvm.org/docs/Extensions.html#linker-options-section-linker-options,
>>> https://reviews.llvm.org/D40849
>>> Clang -
>>> https://clang.llvm.org/docs/LanguageExtensions.html#specifying-linker-options-on-elf-targets,
>>> https://reviews.llvm.org/D42758
>>>
>>> However, although support was added to Clang and LLVM, no support has
>>> been implemented in LLD; and, I get the sense, from reading the reviews,
>>> that there wasn't agreement on the implementation when the changes landed.
>>> The original motivation seems to have been to remove the "autolink-extract"
>>> mechanism used by Swift to workaround the lack of autolinking support for
>>> ELF. However, looking at the Swift source code, Swift still seems to be
>>> using the "autolink-extract" method.
>>>
>>> So my first question: Are there any users of the current implementation
>>> for ELF?
>>>
>>> Assuming that no one is using the current code, I would like to suggest
>>> a different mechanism for autolinking.
>>>
>>> For ELF we need limited autolinking support. Specifically, we only need
>>> support for "comment lib" pragmas (
>>> https://docs.microsoft.com/en-us/cpp/preprocessor/comment-c-cpp?view=vs-2017)
>>> in C/C++ e.g. #pragma comment(lib, "foo"). My suggestion that we keep the
>>> implementation as lean as possible.
>>>
>>> Principles to guide the implementation:
>>> - Developers should be able to easily understand autolinking behavior.
>>> - Developers should be able to override autolinking from the linker
>>> command line.
>>> - Inputs specified via pragmas should be handled in a general way to
>>> allow the same source code to work in different environments.
>>>
>>> I would like to propose that we focus on autolinking exclusively and
>>> that we divorce the implementation from the idea of "linker options" which,
>>> by nature, would tie source code to the vagaries of particular linkers. I
>>> don't see much value in supporting other linker operations so I suggest
>>> that the binary representation be a mergable string section (SHF_MERGE,
>>> SHF_STRINGS), called .autolink, with custom type SHT_LLVM_AUTOLINK
>>> (0x6fff4c04), and SHF_EXCLUDE set (to avoid the contents appearing in the
>>> output). The compiler can form this section by concatenating the arguments
>>> of the "comment lib" pragmas in the order they are encountered. Partial
>>> (-r, -Ur) links can be handled by concatenating .autolink sections with the
>>> normal mergeable string section rules. The current .linker-options can
>>> remain (or be removed); but, "comment lib" pragmas for ELF should be
>>> lowered to .autolink not to .linker-options. This makes sense as there is
>>> no linker option that "comment lib" pragmas map directly to. As an example,
>>> #pragma comment(lib, "foo") would result in:
>>>
>>> .section ".autolink","eMS", at llvm_autolink,1
>>>         .asciz "foo"
>>>
>>> For LTO, equivalent information to the contents of a the .autolink
>>> section will be written to the IRSymtab so that it is available to the
>>> linker for symbol resolution.
>>>
>>> The linker will process the .autolink strings in the following way:
>>>
>>> 1. Inputs from the .autolink sections of a relocatable object file are
>>> added when the linker decides to include that file (which could itself be
>>> in a library) in the link. Autolinked inputs behave as if they were
>>> appended to the command line as a group after all other options. As a
>>> consequence the set of autolinked libraries are searched last to resolve
>>> symbols.
>>> 2. It is an error if a file cannot be found for a given string.
>>> 3. Any command line options in effect at the end of the command line
>>> parsing apply to autolinked inputs, e.g. --whole-archive.
>>> 4. Duplicate autolinked inputs are ignored.
>>> 5. The linker tries to add a library or relocatable object file from
>>> each of the strings in a .autolink section by; first, handling the string
>>> as if it was specified on the commandline; second, by looking for the
>>> string in each of the library search paths in turn; third, by looking for a
>>> lib<string>.a or lib<string>.so (depending on the current mode of the
>>> linker) in each of the library search paths.
>>> 6. A new command line option --no-llvm-autolink will tell LLD to ignore
>>> the .autolink sections.
>>>
>>> Rationale for the above points:
>>>
>>> 1. Adding the autolinked inputs last makes the process simple to
>>> understand from a developers perspective. All linkers are able to implement
>>> this scheme.
>>> 2. Error-ing for libraries that are not found seems like better behavior
>>> than failing the link during symbol resolution.
>>> 3. It seems useful for the user to be able to apply command line options
>>> which will affect all of the autolinked input files. There is a potential
>>> problem of surprise for developers, who might not realize that these
>>> options would apply to the "invisible" autolinked input files; however,
>>> despite the potential for surprise, this is easy for developers to reason
>>> about and gives developers the control that they may require.
>>> 4. Unlike on the command line it is probably easy to include the same
>>> input file twice via pragmas and might be a pain to fix; think of
>>> Third-party libraries supplied as binaries.
>>> 5. This algorithm takes into account all of the different ways that ELF
>>> linkers find input files. The different search methods are tried by the
>>> linker in most obvious to least obvious order.
>>> 6. I considered adding finer grained control over which .autolink inputs
>>> were ignored (e.g. MSVC has /nodefaultlib:<library>); however, I concluded
>>> that this is not necessary: if finer control is required developers can
>>> recreate the same effect autolinking would have had using command line
>>> options.
>>>
>>> Thoughts?
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>

-- 
-- 
Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190314/019c717a/attachment-0001.html>