[llvm-dev] lld: ELF/COFF main() interface

Thu Jan 7 15:59:15 PST 2016

> Why do you want to use that as a library? I don't think "because LLVM and
Clang allow that" is not a compelling argument since the linker and the
compiler are pretty different programs. I clearly see many reasons to use
LLVM and part of Clang as libraries, but they are not directly applicable
to LLD. The new linker is more like a command which is built on top of the
libraries that LLVM provides.

My specific use-case is that of a compiler with an in-process linker that
also runs tests (like Eric mentioned) - so basically a complete
ahead-of-time compilation suite in a single process. In my case I *think*
it's not prohibitively expensive to invoke a process - it complicates
things but does not make them impossible. I started down this path because
LLVM and lld allowed that. I had some plans that involved using similar
infrastructure in a JIT-like manner (e.g. a REPL that would link the binary
and load it), but this is probably better served using the actual JIT.

I think there are environments where invoking a process or exit(1)-ing is
absolutely prohibitive. I personally can solve these issues by
fork/exec-ing the driver to perfom linking, but I don't quite understand
how AMDGPU target can be useful in the new ELF linker since I had the
impression that AMD basically uses ELF for the shader binary format, and
compilation has to happen inside OpenGL driver (where you *want* to run
shader compilation from multiple threads and definitely don't want a
linking issue to exit the process in case of a user-mode driver).

Arseny

On Thu, Jan 7, 2016 at 9:06 AM, Rui Ueyama <ruiu at google.com> wrote:

> Designing it a command makes things simpler because you can safely assume
> that most functions always success, or otherwise the entire process
> terminates. You can defer the operating system to clean up all resources
> that was used by the process however it fails (with a few exceptions such
> as temporary files.) I actually like to use the linker as an external
> command since the operating system provides good isolation between my
> process and the linker which could fail by an unknown bug or something.
> Rewriting all of them as functions that return ErrorOr<Something> is
> technically doable, but it needs strong justification, so I guess that
> unlikely to change.
>
> Why do you want to use that as a library? I don't think "because LLVM and
> Clang allow that" is not a compelling argument since the linker and the
> compiler are pretty different programs. I clearly see many reasons to use
> LLVM and part of Clang as libraries, but they are not directly applicable
> to LLD. The new linker is more like a command which is built on top of the
> libraries that LLVM provides.
>
> On Thu, Jan 7, 2016 at 8:28 AM, Arseny Kapoulkine <
> arseny.kapoulkine at gmail.com> wrote:
>
>> This is really unfortunate.
>>
>> I've read the discussion threads for the atom/chunk controversy and I
>> feel like I understand the reasons for rewriting the linker, but this does
>> not seem to have anything to do with whether the linker is usable as a
>> library or not.
>>
>> As it stands, not only does lld have two completely different linkers
>> (I'm treating COFF/ELF2 as one since they are really two different
>> implementations of the same concept, AFAIU), but one is usable as a library
>> (and even does not require round-tripping generated code through an object
>> file! I was really happy to use that) and the other one isn't. Not sure
>> what the future plans are for Mach-O linker (at this point it seems logical
>> to rewrite that using the new designs but I'm not sure if it ever happens),
>> so maybe at some point we'll just have one linker application instead of a
>> library and an application.
>>
>> Anyway, since linker is the only missing piece for full compilation stack
>> (source language to runnable executable), it's sad to see this specific
>> part of LLVM not working as a library when everything else does.
>>
>> Are there specific concerns in terms of implementation that prevent new
>> lld from being a library? I understand that using global variables and
>> error() function is simpler, but the rest of LLVM does not do that and the
>> codebase there is significantly larger. Am I missing any other issues
>> except for the ones I mentioned in my original e-mail that will come up in
>> a library-like usage scenario?
>>
>> Arseny
>>
>> On Thu, Jan 7, 2016 at 7:17 AM, Rui Ueyama <ruiu at google.com> wrote:
>>
>>> On Thu, Jan 7, 2016 at 7:03 AM, Arseny Kapoulkine via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> In the process of migrating from old lld ELF linker to new (previously
>>>> ELF2) I noticed the interface lost several important features (ordered by
>>>> importance for my use case):
>>>>
>>>> 1. Detecting errors in the first place. New linker seems to call
>>>> exit(1) for any error.
>>>>
>>>> 2. Reporting messages to non-stderr outputs. Previously all link
>>>> functions had a raw_ostream argument so it was possible to delay the error
>>>> output, aggregate it for multiple linked files, output via a different
>>>> format, etc.
>>>>
>>>> 3. Linking multiple outputs in parallel (useful for test drivers) in a
>>>> single process. Not really an interface issue but there are at least two
>>>> global pointers (Config & Driver) that refer to stack variables and are
>>>> used in various places in the code.
>>>>
>>>> All of this seems to indicate a departure from the linker being useable
>>>> as a library. To maintain the previous behavior you'd have to use a linker
>>>> binary & popen.
>>>>
>>>> Is this a conscious design decision or a temporary limitation?
>>>>
>>>
>>> That the new ELF and COFF linkers are designed as commands instead of
>>> libraries is very much an intended design change.
>>>
>>>
>>>> If it's a limitation, how would one go about fixing this? I'm not too
>>>> familiar with the idiomatical error handling in LLVM. Normally in this
>>>> situation I'd just throw from error() but lld is probably compiled without
>>>> exceptions. Is ErrorOr the established practice? How does LLVM generally
>>>> deal with error handling for parsers (like linker script parser) where it's
>>>> a lot of mutually recursive functions where every single one can fail?
>>>>
>>>
>>> Since it's as designed, please run the linker as an external command
>>> using fork/exec (or wrappers of them) instead of trying to use that inside
>>> the same process.
>>>
>>>
>>>> Arseny
>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160107/99b04ae9/attachment.html>