[llvm-dev] RFC: A new llvm-dlltool driver and llvm-lib driver improvements

Rui Ueyama via llvm-dev llvm-dev at lists.llvm.org
Mon Feb 13 17:31:58 PST 2017 

On Mon, Feb 13, 2017 at 5:26 PM, Peter Collingbourne <peter at pcc.me.uk>
wrote:

> On Mon, Feb 13, 2017 at 5:20 PM, Rui Ueyama via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> On Mon, Feb 13, 2017 at 4:46 PM, Martell Malone <martellmalone at gmail.com>
>> wrote:
>>
>>> No, I meant an even thinner wrapper which textually translates
>>>> arguments. For example, the wrapper would translates "/out:foo.exe foo.obj"
>>>> to "-o foo.exe foo.obj" and then call lld::COFFF:link(). It doesn't do
>>>> anything with Config object nor LinkerDriver::run and have absolutely zero
>>>> knowledge on the internals of LLD.
>>>
>>> Ohh okay I misunderstood.
>>>
>>> (Why I'm asking this is because I want to sure that there's no
>>>> difference between the regular Windows linker and mingw. If all differences
>>>> are superficial, the command line translator should just work. If not,
>>>> there's some semantic difference.)
>>>
>>> The libraries can be used with MSVC link.exe directly assuming the
>>> correct arguments are passed to the linker.
>>> I tested this after creating a working llvm-dlltool.
>>> The only change I had to make to support this was alter ming-w64 crt to
>>> change all references from __image_base__ to _ImageBase to support that
>>>
>>
>> You may be able to define __ImageBase as a weak external symbol to
>> __image_base__. Then, if __ImageBase is not defined, all references against
>> __ImageBase will be resolved using __image_base__.
>>
>
> Or you could have your wrapper driver pass "/alternatename:__image_base__
> =__ImageBase".
>

Ah, that's much better indeed.


> Peter
>
>
>>
>>
>>> That should be fine.
>>>
>>> Great
>>>
>>> On Mon, Feb 13, 2017 at 9:34 PM, Rui Ueyama <ruiu at google.com> wrote:
>>>
>>>> On Mon, Feb 13, 2017 at 1:16 PM, Martell Malone <
>>>> martellmalone at gmail.com> wrote:
>>>>
>>>>> I wonder if it can be a wrapper for LLD/COFF. It can be an in-process
>>>>>> wrapper, meaning that you could add a main function for mingw which
>>>>>> translates all arguments into the MSVC style and then invoke the COFF
>>>>>> linker's main function.
>>>>>
>>>>> That should work, if I am understanding this correctly I can create an
>>>>> argument parser (probably partially based on the ELF one) then I can set
>>>>> the data in the COFF Config object and call COFF LinkerDriver::run etc
>>>>> directly from that?
>>>>>
>>>>
>>>> No, I meant an even thinner wrapper which textually translates
>>>> arguments. For example, the wrapper would translates "/out:foo.exe foo.obj"
>>>> to "-o foo.exe foo.obj" and then call lld::COFFF:link(). It doesn't do
>>>> anything with Config object nor LinkerDriver::run and have absolutely zero
>>>> knowledge on the internals of LLD.
>>>>
>>>> (Why I'm asking this is because I want to sure that there's no
>>>> difference between the regular Windows linker and mingw. If all differences
>>>> are superficial, the command line translator should just work. If not,
>>>> there's some semantic difference.)
>>>>
>>>> This proposal is for generally moving code from lld into llvm that
>>>>> could be part of lib.exe / dlltool, what are your thoughts here?
>>>>>
>>>>
>>>> That should be fine.
>>>>
>>>>
>>>>> On Mon, Feb 13, 2017 at 8:57 PM, Rui Ueyama <ruiu at google.com> wrote:
>>>>>
>>>>>> On Mon, Feb 13, 2017 at 12:52 PM, Martell Malone <
>>>>>> martellmalone at gmail.com> wrote:
>>>>>>
>>>>>>> Also you need to make a change to LLD/COFF to accept GNU command
>>>>>>>> arguments, right? (Looks like you already have that patch locally.)
>>>>>>>
>>>>>>> Yes
>>>>>>>
>>>>>>
>>>>>> I wonder if it can be a wrapper for LLD/COFF. It can be an in-process
>>>>>> wrapper, meaning that you could add a main function for mingw which
>>>>>> translates all arguments into the MSVC style and then invoke the COFF
>>>>>> linker's main function.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> My patch to hack lld into accepting some very basic gnu front end
>>>>>>>> arguments was enough to get all the above working which was enough to
>>>>>>>> develop further.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Feb 13, 2017 at 8:41 PM, Rui Ueyama <ruiu at google.com> wrote:
>>>>>>>
>>>>>>>> On Mon, Feb 13, 2017 at 12:33 PM, Martell Malone <
>>>>>>>> martellmalone at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hey Rui,
>>>>>>>>>
>>>>>>>>>> I wonder how llvm-dlltool would fit in the entire picture of
>>>>>>>>>> mingw support. I don't think dlltool is the last missing piece. What do you
>>>>>>>>>> need to do other than that to fully support mingw using LLVM toolchain?
>>>>>>>>>
>>>>>>>>> Other then changing `lib/MC/WinCOFFStreamer.cpp` to not use
>>>>>>>>> -aligncomm within the EmitCommonSymbol function and a single patch for
>>>>>>>>> mingw-w64 itself to pre-populate it's .ctors and .dtors list, so
>>>>>>>>> llvm-dlltool is infact the only missing part really.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Also you need to make a change to LLD/COFF to accept GNU command
>>>>>>>> arguments, right? (Looks like you already have that patch locally.)
>>>>>>>>
>>>>>>>>
>>>>>>>>> This gives us a fully working clang based mingw-w64 C compiler.
>>>>>>>>>
>>>>>>>>> C++ and exception handling is a different story.
>>>>>>>>>
>>>>>>>>> libc++ is somewhat working with the following test results
>>>>>>>>>
>>>>>>>>> Expected Passes    : 2188
>>>>>>>>> Expected Failures  : 44
>>>>>>>>> Unsupported Tests  : 588
>>>>>>>>> Unexpected Failures: 2816
>>>>>>>>>
>>>>>>>>> I was able to build it with exceptions disabled and I actually
>>>>>>>>> managed to bootstrap llvm and clang itself.
>>>>>>>>> It didn't run very well though as you can imagine based on the
>>>>>>>>> above tests.
>>>>>>>>>
>>>>>>>>> It's not a performance issue but a code maintenance issue. The
>>>>>>>>>> initial patches to support mingw was trying to add a new linker driver and
>>>>>>>>>> a different linkin semantics to the COFF linker which seemed too
>>>>>>>>>> complicated to me. IIRC, I suggested adding a shim which translates GNU
>>>>>>>>>> command line arguments to MSVC linker arguments. Didn't it work?
>>>>>>>>>
>>>>>>>>> There were just so many differences between link and ld I never
>>>>>>>>> followed down that path. I pushed forward with the short import library
>>>>>>>>> support based on your later suggestions. My patch to hack lld into
>>>>>>>>> accepting some very basic gnu front end arguments was enough to get all the
>>>>>>>>> above working which was enough to develop further.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Martell
>>>>>>>>>
>>>>>>>>> On Mon, Feb 13, 2017 at 8:08 PM, Rui Ueyama <ruiu at google.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Martell,
>>>>>>>>>>
>>>>>>>>>> I wonder how llvm-dlltool would fit in the entire picture of
>>>>>>>>>> mingw support. I don't think dlltool is the last missing piece. What do you
>>>>>>>>>> need to do other than that to fully support mingw using LLVM toolchain?
>>>>>>>>>>
>>>>>>>>>> On Mon, Feb 13, 2017 at 8:56 AM, Martell Malone via llvm-dev <
>>>>>>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hey llvm'ers,
>>>>>>>>>>>
>>>>>>>>>>> I have been working on a dlltool replacement for llvm.
>>>>>>>>>>> Here is my initial differential https://reviews.llvm.org/D29892
>>>>>>>>>>> It is based on some functionality that already exists in lld.
>>>>>>>>>>> I added functionality to support, PE COFF Weak Externals and of
>>>>>>>>>>> course a front end to actually use it.
>>>>>>>>>>> I believe the work here can also be used for llvm-lib and lessen
>>>>>>>>>>> the load on lld.
>>>>>>>>>>> I would like some comments about how this could be be structured
>>>>>>>>>>> to live in llvm with a shared code base across lib ar and dlltool.
>>>>>>>>>>> I also have a section below called "Difference from lib" which
>>>>>>>>>>> is somewhat of a rationale for the tool.
>>>>>>>>>>>
>>>>>>>>>>> Many Thanks,
>>>>>>>>>>> Martell
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Context
>>>>>>>>>>> ==========
>>>>>>>>>>>
>>>>>>>>>>> Awhile back I talked to various llvm'ers about getting mingw-w64
>>>>>>>>>>> support for lld.
>>>>>>>>>>> There were a few issues raised but the main issue was that
>>>>>>>>>>> mingw-w64 should try best to comply with the PECOFF spec, adding support
>>>>>>>>>>> for custom sections and various binutils/mingw hacks would impact the
>>>>>>>>>>> performance of the COFF linker and in general is not something that lld
>>>>>>>>>>> should support.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> It's not a performance issue but a code maintenance issue. The
>>>>>>>>>> initial patches to support mingw was trying to add a new linker driver and
>>>>>>>>>> a different linkin semantics to the COFF linker which seemed too
>>>>>>>>>> complicated to me. IIRC, I suggested adding a shim which translates GNU
>>>>>>>>>> command line arguments to MSVC linker arguments. Didn't it work?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Motivation
>>>>>>>>>>> ==========
>>>>>>>>>>>
>>>>>>>>>>> The main motivation was because dlltool and ld did not comply
>>>>>>>>>>> with PECOFF Spec.
>>>>>>>>>>> It has some custom formatting and uses the assembler for some
>>>>>>>>>>> reason to generate import libraries, it did not use the short import
>>>>>>>>>>> library format at all.
>>>>>>>>>>>
>>>>>>>>>>> There has been many work arounds for the problem this creates
>>>>>>>>>>> such as the creation of the `reimp` tool. Which imports MSVC built libs
>>>>>>>>>>> creates a def and uses dlltool so that the binutils linker can use it.
>>>>>>>>>>>
>>>>>>>>>>> We should just be using the short import format and the linker
>>>>>>>>>>> should support that.
>>>>>>>>>>> Thus llvm-dlltool was born.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Difference from lib
>>>>>>>>>>> ===================
>>>>>>>>>>>
>>>>>>>>>>> Using PE COFF spec (section 8, Import Library Format) should be
>>>>>>>>>>> self explanatory.
>>>>>>>>>>> lib.exe is able to accept def files and create libraries using
>>>>>>>>>>> this format.
>>>>>>>>>>>
>>>>>>>>>>> example
>>>>>>>>>>>
>>>>>>>>>>>   `LIBRARY "user32.dll"
>>>>>>>>>>>    EXPORTS
>>>>>>>>>>>    MessageBoxA`
>>>>>>>>>>>
>>>>>>>>>>> LIB.exe can create a user32.lib with the function MessageBoxA
>>>>>>>>>>> from the above definition.
>>>>>>>>>>>
>>>>>>>>>>> Mingw-w64 is different MSVC in that we need to compile the
>>>>>>>>>>> runtime.
>>>>>>>>>>> MS provide us with their crt prebuilt so lib.exe doesn't have
>>>>>>>>>>> support for external function aliasing.
>>>>>>>>>>> We often use aliases for posix naming reasons as well as avoid
>>>>>>>>>>> using the MS version of a function.
>>>>>>>>>>>
>>>>>>>>>>> example
>>>>>>>>>>>
>>>>>>>>>>>   `LIBRARY "user32.dll"
>>>>>>>>>>>    EXPORTS
>>>>>>>>>>>    MessageBoxA
>>>>>>>>>>>    MessageBoxW==MessageBoxA`
>>>>>>>>>>>
>>>>>>>>>>> LIB.exe doesn't not have support for this but dlltool does.
>>>>>>>>>>> The best fit for this within the spec is PE COFF spec (Aux
>>>>>>>>>>> Format 3: Weak Externals)
>>>>>>>>>>>
>>>>>>>>>>> Mingw-w64 also uses lib prefix and .a file extensions for the
>>>>>>>>>>> library name so the driver should be different. The above example would be
>>>>>>>>>>> libuser32.a, for when shared and static versions of the same library exist
>>>>>>>>>>> there is the .dll.a variant.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Issues
>>>>>>>>>>> =======
>>>>>>>>>>>
>>>>>>>>>>> Like more of the gnu suite dlltool was not designed in one build
>>>>>>>>>>> multi target manner.
>>>>>>>>>>> We would have to introduce custom arguments to specify the
>>>>>>>>>>> target, "i686", "x86_64", "thumb2pe" etc.
>>>>>>>>>>> This will most likely break some level of backwards
>>>>>>>>>>> compatibility with the binutils version.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> LLVM Developers mailing list
>>>>>>>>>>> llvm-dev at lists.llvm.org
>>>>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
>
>
> --
> --
> Peter
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170213/e79dc480/attachment.html>

More information about the llvm-dev mailing list