[llvm-dev] lld: ELF/COFF main() interface

Fri Jan 22 11:21:18 PST 2016

We can provide a fall-back function that writes files to disk and starts
the linker. My point is that we can provide a drop-in replacement for the
old ELF's main function rather than just turning down requests from users
by just saying that "we'll get to the point later."

Since the new linker is much faster than the old one, I expect that will be
usually faster even with the cost of new process invocation. In order to
use the old linker's main, you had to write files to disks, so we don't
lose anything about that.

On Fri, Jan 22, 2016 at 10:57 AM, Alex Rosenberg <alexr at leftfield.org>
wrote:

> Most embedded development (game consoles included) is cross-developed from
> Windows. A lot of that development is targeting ELF.
>
> Alex
>
> On Jan 22, 2016, at 10:06 AM, Rui Ueyama via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> If you want to link ELF object files, you are likely to be using a Unix
> machine. I'm not trying to address all possible problems but suggesting a
> practical solution.
>
> On Fri, Jan 22, 2016 at 9:49 AM, Yaron Keren <yaron.keren at gmail.com>
> wrote:
>
>> On Windows fork() is not available. If exec() is used instead, process
>> creation time is several times slower than Linux. This may be or not a
>> problem dependeing on how lld is used. In general, on Windows the best
>> solution is multi-threading.
>>
>>
>> 2016-01-22 18:31 GMT+02:00 Rui Ueyama via llvm-dev <
>> llvm-dev at lists.llvm.org>:
>>
>>> I think I have an idea to cover your need and possibly other people's on
>>> this thread. It provides the "main() as a library function" feature,
>>> input/output files wouldn't go through disks nor file systems, and it
>>> doesn't require any major design changes. Sounds too good?
>>>
>>> That is, we can provide a function that takes command line parameters,
>>> do fork, and call the linker's main function.
>>>
>>> This may sound too simple, but I think that is fairly powerful. Because
>>> the child process has a copy(-on-write) of the parent's memory, it can read
>>> parent's in-memory object files directly with no overhead. The child can
>>> pass the resulting file back to the parent through a shared memory object
>>> (which we can obtain using shm_open or something like that). In addition to
>>> that, your main process gets a protection from linker's bugs thanks to the
>>> operating system's memory protection. But from the user's point of view,
>>> that is just a linker's main function that you can call and that works as
>>> expected.
>>>
>>> Even if we want to call exec for whatever reason, we can copy in-memory
>>> objects to shared memory objects and exec the linker, so the basic design
>>> should work in such case too.
>>>
>>> The function signature would be something like:
>>>
>>>   bool link(ArrayRef<LinkerArg> CommandLineArgs, MemoryBuffer
>>> &OutputFile, std::string &ErrorMsg);
>>>
>>> where the return value indicates success/failure. LinkerArg is a union
>>> type of StringRef and MemoryBufferRef. The result is returned as OutputFile
>>> memory buffer. If it prints out any message, ErrorMsg will hold it.
>>>
>>> (I want to point out that the function "bool link(ArrayRef<const char*>
>>> args, raw_ostream& diagnostics, ArrayRef<unique_ptr<MemoryBuffer>>
>>> inputs)" doesn't work because the order of command line parameters matters.
>>> Some command line parameters, such as --whole-archive/--no-whole-archive,
>>> affects how files in between will be interpreted, so you can't separate
>>> command line parameters from a list of files.)
>>>
>>> I think this is a practical solution that we can do now. I can implement
>>> this for you.
>>>
>>> On Thu, Jan 21, 2016 at 9:28 PM, Arseny Kapoulkine <
>>> arseny.kapoulkine at gmail.com> wrote:
>>>
>>>> > In any case, I have simply wasted too much time on a thread with
>>>> someone with no patches on the new elf linker. It is really annoying that
>>>> you don't put effort into it and seem entitled to dictate its direction.
>>>>
>>>> Sorry about that. I was initially planning to work on a patch to
>>>> enhance the interface for new lld - hence my questions in the original
>>>> post. Since I learned that people writing the code for lld are hostile to
>>>> the idea of linker-as-a-library, error_code is treated as spaghetti (which
>>>> would be fine if LLVM used exceptions which it does not) and patches, even
>>>> if submitted, will not actually be reviewed in a timely manner, I'll try to
>>>> adapt my code to either not use lld or use lld-as-a-binary.
>>>>
>>>> I'm disappointed by all of this but obviously it's not my project so I
>>>> should not have a say in this.
>>>>
>>>> Thank you for your time.
>>>>
>>>> Arseny
>>>>
>>>> On Thu, Jan 21, 2016 at 8:44 PM, Rafael Espíndola <
>>>> rafael.espindola at gmail.com> wrote:
>>>>
>>>>> > Also, one of the other possible motivations of using LLD directly
>>>>> from Clang would be to avoid process overhead on operating systems where
>>>>> that is a much more significant part of the compile time cost. We could
>>>>> today actually take the fork out of the Clang driver because the Clang
>>>>> frontend *is* designed in this way. But we would also need LLD to work in
>>>>> this way.
>>>>>
>>>>> Then go change clang and send a patch for lld once you are done. It
>>>>> will be interested to see if you can measure a single fork in an entire
>>>>> build.
>>>>>
>>>>> Even better, please finish the new pass manager before working on
>>>>> clang forking cc1.
>>>>>
>>>>> In any case, I have simply wasted too much time on a thread with
>>>>> someone with no patches on the new elf linker. It is really annoying that
>>>>> you don't put effort into it and seem entitled to dictate its direction.
>>>>>
>>>>> If you want to kick us out of the llvm project, please start a thread
>>>>> on llvm-dev.
>>>>>
>>>>> If you want lld to be a library, figure out how to do it without
>>>>> sacrificing lld's productivity,  error reporting and performance (no
>>>>> error_code spaghetti) and write a patch. Just don't expect it to be
>>>>> reviewed while we have actual missing features.
>>>>>
>>>>> I will go back to implementing the linker.
>>>>>
>>>>> Rafael
>>>>>
>>>>
>>>>
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>>
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160122/654baff6/attachment-0001.html>