[llvm-dev] Make LLD output COFF relocatable object file (like ELF's -r does). How much work is required to implement this?

Tue Oct 10 11:00:14 PDT 2017

Thank you for your detailed explanation!

On Tue, Oct 10, 2017 at 1:36 AM, kyra <kyrab at mail.ru> wrote:

> TL;DR:
> I'm trying to evaluate if LLD can be used with GHC (Glasgow Haskell
> Compiler) on Windows.
>
> Haskell binary code is usually deployed in "packages". A package typically
> provides static library(ies) and optionally – shared library(ies) and/or
> prelinked ('ld -r') object file. The latter is the best way to satisfy GHC
> runtime linker, since it requires no separate compile/link pass (as shared
> library requires), and is much faster to consume by GHC runtime linker than
> a static library.
>

I'm not sure if I understand correctly. If my understanding is correct, you
are saying that GHC can link either .o or .so at runtime, which sounds a
bit odd because .o is not designed for dynamic linking. Am I missing
something?

I also do not understand why only static libraries need "compile/link pass"
-- they at least don't need a compile pass, as they contain compiled .o
files, and they indeed need a link pass, but that's also true for a single
big .o file generated by -r, no? After all, in order to link against a .a
file, I think you need to pull out a .o file from a .a and do whatever you
need to do to link a single big .o file.

Long story:
>
> To prevent linking unused code GHC have always been supported splitting
> intermediate assembly which is horribly slow when compiling. Now GHC
> supports a direct analogue of '-ffunction-sections' ('-split-sections' in
> GHC parlance), which dramatically improves compile times, but now BFD
> linker is horribly slow on the files with a *lot* of sections. In the *nix
> world they have gold linker, in the windows world we have nothing other
> than GNU BFD ld ATM.
>
> GHC on Windows uses Mingw tools and LLD doesn't fit into Mingw ecosystem
> yet (I know that some support have creeped into LLD recently, but it is
> still far from being complete), moreover, when assemling GHC native codegen
> output, GNU assembler produces peculiar non-standard COFF files (with 0x11
> relocations), and finally binutils doesn't (and probably never would)
> support bigobj extension in the 32-bit case.
>
> Windows GHC relies heavily on GCC, especially it's runtime system's code
> is full of gnu-isms, but Clang has a unique ability to combine gnu-ish
> frontend with ms-ish backend, I've experimented a bit and have concluded
> that replacing GCC as a C compiler/system assembler with Clang in GHC on
> Windows is very much doable.
>
> GHC uses object file combining ('ld -r') when C stubs/wrappers generation
> is triggered, these stubs/wrappers are compiled with gcc and are linked
> back into the 'main' object file. In the MS world this use case can easily
> be satisfied by packing the object files into a library since MS linker
> looks into libraries both when linking final exe/dll *and/or* creating
> another library (i.e. when creating another library it unpacks all object
> files from all libraries it is fed with, and repacks them into the output
> library, llvm-lib doesn't support this ATM, and AFAIR LLVM developers are
> aware of this).
>

I have an in-progress patch to add the feature to llvm-lib. I didn't have
time to finish it, but it is on the table, and needs to be done for
compatibility with MSVC lib.exe.

> But my question is motivated by another important use-case: when packaging
> compiled Haskell code it is very desirable to provide not only a static
> library, but also to partially link this library's object modules into the
> one big object file, which can further be consumed by GHC runtime linker.
> GHC runtime linker can link binary code in any form, but linking static
> library is much slower than linking the single object file.

IIUC, GHC is faster when handling .a files compared to a prelinked big .o
file, even if they contain the same binary code/data. But it sounds like an
artifact of the current implementation of GHC, because, in theory, there's
no reason the former is much inefficient than the latter. If that's the
case, doesn't it make more sense to improve GHC?

> On 10/10/2017 2:39 AM, Rui Ueyama wrote:
>
>> As far as I know, no one has ever tried to add the -r option to the lld
>> COFF linker. It shouldn't be super hard to add it to the COFF linker, but
>> from our experience of implementing it to lld ELF linker, I can say that it
>> was tricky and somewhat fragile. We had to add a number of small pieces of
>> code here and there.
>>
>> We wanted to support it in the ELF linker because that's an existing
>> feature and people are actually using it. Otherwise, we wouldn't have added
>> it. So, what is the motivation of adding the feature to the COFF linker? I
>> don't think MSVC linker supports it.
>>
>> (For those who are not familiar with -r,  the option is to make the
>> linker emit a .o file instead of an executable or a shared library. With
>> the option, you can combine multiple object files into one object file.)
>>
>> On Mon, Oct 9, 2017 at 8:02 AM, kyra via llvm-dev <
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>>
>>     Hi,
>>
>>     How far are we from having '-r' in the LLD COFF linker?
>>     I'd try to implement this if not too much effort is required.
>>     Any suggestions and/or pointers?
>>
>>     Cheers,
>>     Kyra
>>
>>
>>     _______________________________________________
>>     LLVM Developers mailing list
>>     llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>>     http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>     <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20171010/47cf488c/attachment-0001.html>