[llvm-dev] RFC: A new llvm-dlltool driver and llvm-lib driver improvements

[Rui Ueyama via llvm-dev]

The code in LLD is somewhat tightly coupled with the rest of the linker.
Probably it is better to create a new library rather than trying to split
LLD into two. I believe this would result in a better API. Of course you
can reuse code, but what we need is a clean interface.

The library shouldn't be complicated. In fact, it would export only one
function.

  std::vector<uint8_t> createImportLibrary(std::vector<ExportSymbol>)

I.e. it takes a list of DLLExported symbols and returns a .lib file.
ExportSymbol type contains information about DLLExported symbols.

On Mon, Feb 13, 2017 at 6:06 PM, Martell Malone <martellmalone at gmail.com>
wrote:

> Well that means it would just be the only plan then.
> I assume the first step would be to move the code from lld into this new
> library and lld can use that? I can then re add the extra functionality for
> weak externals?
> We can then have lib and dlltool use this then.
>
> On Tue 14 Feb 2017 at 01:58, Rui Ueyama <ruiu at google.com> wrote:
>
>> On Mon, Feb 13, 2017 at 5:56 PM, Martell Malone <martellmalone at gmail.com>
>> wrote:
>>
>> Yes, that would be the long term plan.
>> llvm-link ideally would share the same library.
>> I'm not sure where it should live though.
>>
>>
>> What is the short-term plan? I don't think you can submit this
>> llvm-dlltool as-is because it has to much duplicate code with LLD.
>>
>>
>>
>> On Tue 14 Feb 2017 at 01:49, Rui Ueyama <ruiu at google.com> wrote:
>>
>> I took a look at the code. Looks like you need a library to create import
>> library files in LLVM and use that from llvm-dlltool and LLD. Is that what
>> you are planning?
>>
>> On Mon, Feb 13, 2017 at 5:37 PM, Martell Malone <martellmalone at gmail.com>
>> wrote:
>>
>> Ohh nice.
>> With that method I can support it without upsetting ld users by
>> introducing an api breakage.
>>
>> On Tue 14 Feb 2017 at 01:32, Rui Ueyama <ruiu at google.com> wrote:
>>
>> On Mon, Feb 13, 2017 at 5:26 PM, Peter Collingbourne <peter at pcc.me.uk>
>> wrote:
>>
>> On Mon, Feb 13, 2017 at 5:20 PM, Rui Ueyama via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>> On Mon, Feb 13, 2017 at 4:46 PM, Martell Malone <martellmalone at gmail.com>
>> wrote:
>>
>> No, I meant an even thinner wrapper which textually translates arguments.
>> For example, the wrapper would translates "/out:foo.exe foo.obj" to "-o
>> foo.exe foo.obj" and then call lld::COFFF:link(). It doesn't do anything
>> with Config object nor LinkerDriver::run and have absolutely zero knowledge
>> on the internals of LLD.
>>
>> Ohh okay I misunderstood.
>>
>> (Why I'm asking this is because I want to sure that there's no difference
>> between the regular Windows linker and mingw. If all differences are
>> superficial, the command line translator should just work. If not, there's
>> some semantic difference.)
>>
>> The libraries can be used with MSVC link.exe directly assuming the
>> correct arguments are passed to the linker.
>> I tested this after creating a working llvm-dlltool.
>> The only change I had to make to support this was alter ming-w64 crt to
>> change all references from __image_base__ to _ImageBase to support that
>>
>>
>> You may be able to define __ImageBase as a weak external symbol to
>> __image_base__. Then, if __ImageBase is not defined, all references against
>> __ImageBase will be resolved using __image_base__.
>>
>>
>> Or you could have your wrapper driver pass "/alternatename:__image_base__
>> =__ImageBase".
>>
>>
>> Ah, that's much better indeed.
>>
>>
>> Peter
>>
>>
>>
>>
>> That should be fine.
>>
>> Great
>>
>> On Mon, Feb 13, 2017 at 9:34 PM, Rui Ueyama <ruiu at google.com> wrote:
>>
>> On Mon, Feb 13, 2017 at 1:16 PM, Martell Malone <martellmalone at gmail.com>
>> wrote:
>>
>> I wonder if it can be a wrapper for LLD/COFF. It can be an in-process
>> wrapper, meaning that you could add a main function for mingw which
>> translates all arguments into the MSVC style and then invoke the COFF
>> linker's main function.
>>
>> That should work, if I am understanding this correctly I can create an
>> argument parser (probably partially based on the ELF one) then I can set
>> the data in the COFF Config object and call COFF LinkerDriver::run etc
>> directly from that?
>>
>>
>> No, I meant an even thinner wrapper which textually translates arguments.
>> For example, the wrapper would translates "/out:foo.exe foo.obj" to "-o
>> foo.exe foo.obj" and then call lld::COFFF:link(). It doesn't do anything
>> with Config object nor LinkerDriver::run and have absolutely zero knowledge
>> on the internals of LLD.
>>
>> (Why I'm asking this is because I want to sure that there's no difference
>> between the regular Windows linker and mingw. If all differences are
>> superficial, the command line translator should just work. If not, there's
>> some semantic difference.)
>>
>> This proposal is for generally moving code from lld into llvm that could
>> be part of lib.exe / dlltool, what are your thoughts here?
>>
>>
>> That should be fine.
>>
>>
>> On Mon, Feb 13, 2017 at 8:57 PM, Rui Ueyama <ruiu at google.com> wrote:
>>
>> On Mon, Feb 13, 2017 at 12:52 PM, Martell Malone <martellmalone at gmail.com
>> > wrote:
>>
>> Also you need to make a change to LLD/COFF to accept GNU command
>> arguments, right? (Looks like you already have that patch locally.)
>>
>> Yes
>>
>>
>> I wonder if it can be a wrapper for LLD/COFF. It can be an in-process
>> wrapper, meaning that you could add a main function for mingw which
>> translates all arguments into the MSVC style and then invoke the COFF
>> linker's main function.
>>
>>
>>
>> My patch to hack lld into accepting some very basic gnu front end
>> arguments was enough to get all the above working which was enough to
>> develop further.
>>
>>
>>
>> On Mon, Feb 13, 2017 at 8:41 PM, Rui Ueyama <ruiu at google.com> wrote:
>>
>> On Mon, Feb 13, 2017 at 12:33 PM, Martell Malone <martellmalone at gmail.com
>> > wrote:
>>
>> Hey Rui,
>>
>> I wonder how llvm-dlltool would fit in the entire picture of mingw
>> support. I don't think dlltool is the last missing piece. What do you need
>> to do other than that to fully support mingw using LLVM toolchain?
>>
>> Other then changing `lib/MC/WinCOFFStreamer.cpp` to not use -aligncomm
>> within the EmitCommonSymbol function and a single patch for mingw-w64
>> itself to pre-populate it's .ctors and .dtors list, so llvm-dlltool is
>> infact the only missing part really.
>>
>>
>> Also you need to make a change to LLD/COFF to accept GNU command
>> arguments, right? (Looks like you already have that patch locally.)
>>
>>
>> This gives us a fully working clang based mingw-w64 C compiler.
>>
>> C++ and exception handling is a different story.
>>
>> libc++ is somewhat working with the following test results
>>
>> Expected Passes    : 2188
>> Expected Failures  : 44
>> Unsupported Tests  : 588
>> Unexpected Failures: 2816
>>
>> I was able to build it with exceptions disabled and I actually managed to
>> bootstrap llvm and clang itself.
>> It didn't run very well though as you can imagine based on the above
>> tests.
>>
>> It's not a performance issue but a code maintenance issue. The initial
>> patches to support mingw was trying to add a new linker driver and a
>> different linkin semantics to the COFF linker which seemed too complicated
>> to me. IIRC, I suggested adding a shim which translates GNU command line
>> arguments to MSVC linker arguments. Didn't it work?
>>
>> There were just so many differences between link and ld I never followed
>> down that path. I pushed forward with the short import library support
>> based on your later suggestions. My patch to hack lld into accepting some
>> very basic gnu front end arguments was enough to get all the above working
>> which was enough to develop further.
>>
>> Best,
>> Martell
>>
>> On Mon, Feb 13, 2017 at 8:08 PM, Rui Ueyama <ruiu at google.com> wrote:
>>
>> Hi Martell,
>>
>> I wonder how llvm-dlltool would fit in the entire picture of mingw
>> support. I don't think dlltool is the last missing piece. What do you need
>> to do other than that to fully support mingw using LLVM toolchain?
>>
>> On Mon, Feb 13, 2017 at 8:56 AM, Martell Malone via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>> Hey llvm'ers,
>>
>> I have been working on a dlltool replacement for llvm.
>> Here is my initial differential https://reviews.llvm.org/D29892
>> It is based on some functionality that already exists in lld.
>> I added functionality to support, PE COFF Weak Externals and of course a
>> front end to actually use it.
>> I believe the work here can also be used for llvm-lib and lessen the load
>> on lld.
>> I would like some comments about how this could be be structured to live
>> in llvm with a shared code base across lib ar and dlltool.
>> I also have a section below called "Difference from lib" which is
>> somewhat of a rationale for the tool.
>>
>> Many Thanks,
>> Martell
>>
>>
>> Context
>> ==========
>>
>> Awhile back I talked to various llvm'ers about getting mingw-w64 support
>> for lld.
>> There were a few issues raised but the main issue was that mingw-w64
>> should try best to comply with the PECOFF spec, adding support for custom
>> sections and various binutils/mingw hacks would impact the performance of
>> the COFF linker and in general is not something that lld should support.
>>
>>
>> It's not a performance issue but a code maintenance issue. The initial
>> patches to support mingw was trying to add a new linker driver and a
>> different linkin semantics to the COFF linker which seemed too complicated
>> to me. IIRC, I suggested adding a shim which translates GNU command line
>> arguments to MSVC linker arguments. Didn't it work?
>>
>>
>> Motivation
>> ==========
>>
>> The main motivation was because dlltool and ld did not comply with PECOFF
>> Spec.
>> It has some custom formatting and uses the assembler for some reason to
>> generate import libraries, it did not use the short import library format
>> at all.
>>
>> There has been many work arounds for the problem this creates such as the
>> creation of the `reimp` tool. Which imports MSVC built libs creates a def
>> and uses dlltool so that the binutils linker can use it.
>>
>> We should just be using the short import format and the linker should
>> support that.
>> Thus llvm-dlltool was born.
>>
>>
>> Difference from lib
>> ===================
>>
>> Using PE COFF spec (section 8, Import Library Format) should be self
>> explanatory.
>> lib.exe is able to accept def files and create libraries using this
>> format.
>>
>> example
>>
>>   `LIBRARY "user32.dll"
>>    EXPORTS
>>    MessageBoxA`
>>
>> LIB.exe can create a user32.lib with the function MessageBoxA from the
>> above definition.
>>
>> Mingw-w64 is different MSVC in that we need to compile the runtime.
>> MS provide us with their crt prebuilt so lib.exe doesn't have support for
>> external function aliasing.
>> We often use aliases for posix naming reasons as well as avoid using the
>> MS version of a function.
>>
>> example
>>
>>   `LIBRARY "user32.dll"
>>    EXPORTS
>>    MessageBoxA
>>    MessageBoxW==MessageBoxA`
>>
>> LIB.exe doesn't not have support for this but dlltool does.
>> The best fit for this within the spec is PE COFF spec (Aux Format 3:
>> Weak Externals)
>>
>> Mingw-w64 also uses lib prefix and .a file extensions for the library
>> name so the driver should be different. The above example would be
>> libuser32.a, for when shared and static versions of the same library exist
>> there is the .dll.a variant.
>>
>>
>> Issues
>> =======
>>
>> Like more of the gnu suite dlltool was not designed in one build multi
>> target manner.
>> We would have to introduce custom arguments to specify the target,
>> "i686", "x86_64", "thumb2pe" etc.
>> This will most likely break some level of backwards compatibility with
>> the binutils version.
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
>>
>>
>> --
>> --
>> Peter
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170213/5d877e21/attachment-0001.html>