[llvm-dev] RFC: A new llvm-dlltool driver and llvm-lib driver improvements

Rui Ueyama via llvm-dev llvm-dev at lists.llvm.org
Mon Feb 13 17:20:20 PST 2017 

On Mon, Feb 13, 2017 at 4:46 PM, Martell Malone <martellmalone at gmail.com>
wrote:

> No, I meant an even thinner wrapper which textually translates arguments.
>> For example, the wrapper would translates "/out:foo.exe foo.obj" to "-o
>> foo.exe foo.obj" and then call lld::COFFF:link(). It doesn't do anything
>> with Config object nor LinkerDriver::run and have absolutely zero knowledge
>> on the internals of LLD.
>
> Ohh okay I misunderstood.
>
> (Why I'm asking this is because I want to sure that there's no difference
>> between the regular Windows linker and mingw. If all differences are
>> superficial, the command line translator should just work. If not, there's
>> some semantic difference.)
>
> The libraries can be used with MSVC link.exe directly assuming the correct
> arguments are passed to the linker.
> I tested this after creating a working llvm-dlltool.
> The only change I had to make to support this was alter ming-w64 crt to
> change all references from __image_base__ to _ImageBase to support that
>

You may be able to define __ImageBase as a weak external symbol to
__image_base__. Then, if __ImageBase is not defined, all references against
__ImageBase will be resolved using __image_base__.


> That should be fine.
>
> Great
>
> On Mon, Feb 13, 2017 at 9:34 PM, Rui Ueyama <ruiu at google.com> wrote:
>
>> On Mon, Feb 13, 2017 at 1:16 PM, Martell Malone <martellmalone at gmail.com>
>> wrote:
>>
>>> I wonder if it can be a wrapper for LLD/COFF. It can be an in-process
>>>> wrapper, meaning that you could add a main function for mingw which
>>>> translates all arguments into the MSVC style and then invoke the COFF
>>>> linker's main function.
>>>
>>> That should work, if I am understanding this correctly I can create an
>>> argument parser (probably partially based on the ELF one) then I can set
>>> the data in the COFF Config object and call COFF LinkerDriver::run etc
>>> directly from that?
>>>
>>
>> No, I meant an even thinner wrapper which textually translates arguments.
>> For example, the wrapper would translates "/out:foo.exe foo.obj" to "-o
>> foo.exe foo.obj" and then call lld::COFFF:link(). It doesn't do anything
>> with Config object nor LinkerDriver::run and have absolutely zero knowledge
>> on the internals of LLD.
>>
>> (Why I'm asking this is because I want to sure that there's no difference
>> between the regular Windows linker and mingw. If all differences are
>> superficial, the command line translator should just work. If not, there's
>> some semantic difference.)
>>
>> This proposal is for generally moving code from lld into llvm that could
>>> be part of lib.exe / dlltool, what are your thoughts here?
>>>
>>
>> That should be fine.
>>
>>
>>> On Mon, Feb 13, 2017 at 8:57 PM, Rui Ueyama <ruiu at google.com> wrote:
>>>
>>>> On Mon, Feb 13, 2017 at 12:52 PM, Martell Malone <
>>>> martellmalone at gmail.com> wrote:
>>>>
>>>>> Also you need to make a change to LLD/COFF to accept GNU command
>>>>>> arguments, right? (Looks like you already have that patch locally.)
>>>>>
>>>>> Yes
>>>>>
>>>>
>>>> I wonder if it can be a wrapper for LLD/COFF. It can be an in-process
>>>> wrapper, meaning that you could add a main function for mingw which
>>>> translates all arguments into the MSVC style and then invoke the COFF
>>>> linker's main function.
>>>>
>>>>
>>>>>
>>>>> My patch to hack lld into accepting some very basic gnu front end
>>>>>> arguments was enough to get all the above working which was enough to
>>>>>> develop further.
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Feb 13, 2017 at 8:41 PM, Rui Ueyama <ruiu at google.com> wrote:
>>>>>
>>>>>> On Mon, Feb 13, 2017 at 12:33 PM, Martell Malone <
>>>>>> martellmalone at gmail.com> wrote:
>>>>>>
>>>>>>> Hey Rui,
>>>>>>>
>>>>>>>> I wonder how llvm-dlltool would fit in the entire picture of mingw
>>>>>>>> support. I don't think dlltool is the last missing piece. What do you need
>>>>>>>> to do other than that to fully support mingw using LLVM toolchain?
>>>>>>>
>>>>>>> Other then changing `lib/MC/WinCOFFStreamer.cpp` to not use
>>>>>>> -aligncomm within the EmitCommonSymbol function and a single patch for
>>>>>>> mingw-w64 itself to pre-populate it's .ctors and .dtors list, so
>>>>>>> llvm-dlltool is infact the only missing part really.
>>>>>>>
>>>>>>
>>>>>> Also you need to make a change to LLD/COFF to accept GNU command
>>>>>> arguments, right? (Looks like you already have that patch locally.)
>>>>>>
>>>>>>
>>>>>>> This gives us a fully working clang based mingw-w64 C compiler.
>>>>>>>
>>>>>>> C++ and exception handling is a different story.
>>>>>>>
>>>>>>> libc++ is somewhat working with the following test results
>>>>>>>
>>>>>>> Expected Passes    : 2188
>>>>>>> Expected Failures  : 44
>>>>>>> Unsupported Tests  : 588
>>>>>>> Unexpected Failures: 2816
>>>>>>>
>>>>>>> I was able to build it with exceptions disabled and I actually
>>>>>>> managed to bootstrap llvm and clang itself.
>>>>>>> It didn't run very well though as you can imagine based on the above
>>>>>>> tests.
>>>>>>>
>>>>>>> It's not a performance issue but a code maintenance issue. The
>>>>>>>> initial patches to support mingw was trying to add a new linker driver and
>>>>>>>> a different linkin semantics to the COFF linker which seemed too
>>>>>>>> complicated to me. IIRC, I suggested adding a shim which translates GNU
>>>>>>>> command line arguments to MSVC linker arguments. Didn't it work?
>>>>>>>
>>>>>>> There were just so many differences between link and ld I never
>>>>>>> followed down that path. I pushed forward with the short import library
>>>>>>> support based on your later suggestions. My patch to hack lld into
>>>>>>> accepting some very basic gnu front end arguments was enough to get all the
>>>>>>> above working which was enough to develop further.
>>>>>>>
>>>>>>> Best,
>>>>>>> Martell
>>>>>>>
>>>>>>> On Mon, Feb 13, 2017 at 8:08 PM, Rui Ueyama <ruiu at google.com> wrote:
>>>>>>>
>>>>>>>> Hi Martell,
>>>>>>>>
>>>>>>>> I wonder how llvm-dlltool would fit in the entire picture of mingw
>>>>>>>> support. I don't think dlltool is the last missing piece. What do you need
>>>>>>>> to do other than that to fully support mingw using LLVM toolchain?
>>>>>>>>
>>>>>>>> On Mon, Feb 13, 2017 at 8:56 AM, Martell Malone via llvm-dev <
>>>>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>>>>
>>>>>>>>> Hey llvm'ers,
>>>>>>>>>
>>>>>>>>> I have been working on a dlltool replacement for llvm.
>>>>>>>>> Here is my initial differential https://reviews.llvm.org/D29892
>>>>>>>>> It is based on some functionality that already exists in lld.
>>>>>>>>> I added functionality to support, PE COFF Weak Externals and of
>>>>>>>>> course a front end to actually use it.
>>>>>>>>> I believe the work here can also be used for llvm-lib and lessen
>>>>>>>>> the load on lld.
>>>>>>>>> I would like some comments about how this could be be structured
>>>>>>>>> to live in llvm with a shared code base across lib ar and dlltool.
>>>>>>>>> I also have a section below called "Difference from lib" which is
>>>>>>>>> somewhat of a rationale for the tool.
>>>>>>>>>
>>>>>>>>> Many Thanks,
>>>>>>>>> Martell
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Context
>>>>>>>>> ==========
>>>>>>>>>
>>>>>>>>> Awhile back I talked to various llvm'ers about getting mingw-w64
>>>>>>>>> support for lld.
>>>>>>>>> There were a few issues raised but the main issue was that
>>>>>>>>> mingw-w64 should try best to comply with the PECOFF spec, adding support
>>>>>>>>> for custom sections and various binutils/mingw hacks would impact the
>>>>>>>>> performance of the COFF linker and in general is not something that lld
>>>>>>>>> should support.
>>>>>>>>>
>>>>>>>>
>>>>>>>> It's not a performance issue but a code maintenance issue. The
>>>>>>>> initial patches to support mingw was trying to add a new linker driver and
>>>>>>>> a different linkin semantics to the COFF linker which seemed too
>>>>>>>> complicated to me. IIRC, I suggested adding a shim which translates GNU
>>>>>>>> command line arguments to MSVC linker arguments. Didn't it work?
>>>>>>>>
>>>>>>>>
>>>>>>>>> Motivation
>>>>>>>>> ==========
>>>>>>>>>
>>>>>>>>> The main motivation was because dlltool and ld did not comply with
>>>>>>>>> PECOFF Spec.
>>>>>>>>> It has some custom formatting and uses the assembler for some
>>>>>>>>> reason to generate import libraries, it did not use the short import
>>>>>>>>> library format at all.
>>>>>>>>>
>>>>>>>>> There has been many work arounds for the problem this creates such
>>>>>>>>> as the creation of the `reimp` tool. Which imports MSVC built libs creates
>>>>>>>>> a def and uses dlltool so that the binutils linker can use it.
>>>>>>>>>
>>>>>>>>> We should just be using the short import format and the linker
>>>>>>>>> should support that.
>>>>>>>>> Thus llvm-dlltool was born.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Difference from lib
>>>>>>>>> ===================
>>>>>>>>>
>>>>>>>>> Using PE COFF spec (section 8, Import Library Format) should be
>>>>>>>>> self explanatory.
>>>>>>>>> lib.exe is able to accept def files and create libraries using
>>>>>>>>> this format.
>>>>>>>>>
>>>>>>>>> example
>>>>>>>>>
>>>>>>>>>   `LIBRARY "user32.dll"
>>>>>>>>>    EXPORTS
>>>>>>>>>    MessageBoxA`
>>>>>>>>>
>>>>>>>>> LIB.exe can create a user32.lib with the function MessageBoxA from
>>>>>>>>> the above definition.
>>>>>>>>>
>>>>>>>>> Mingw-w64 is different MSVC in that we need to compile the runtime.
>>>>>>>>> MS provide us with their crt prebuilt so lib.exe doesn't have
>>>>>>>>> support for external function aliasing.
>>>>>>>>> We often use aliases for posix naming reasons as well as avoid
>>>>>>>>> using the MS version of a function.
>>>>>>>>>
>>>>>>>>> example
>>>>>>>>>
>>>>>>>>>   `LIBRARY "user32.dll"
>>>>>>>>>    EXPORTS
>>>>>>>>>    MessageBoxA
>>>>>>>>>    MessageBoxW==MessageBoxA`
>>>>>>>>>
>>>>>>>>> LIB.exe doesn't not have support for this but dlltool does.
>>>>>>>>> The best fit for this within the spec is PE COFF spec (Aux Format
>>>>>>>>> 3: Weak Externals)
>>>>>>>>>
>>>>>>>>> Mingw-w64 also uses lib prefix and .a file extensions for the
>>>>>>>>> library name so the driver should be different. The above example would be
>>>>>>>>> libuser32.a, for when shared and static versions of the same library exist
>>>>>>>>> there is the .dll.a variant.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Issues
>>>>>>>>> =======
>>>>>>>>>
>>>>>>>>> Like more of the gnu suite dlltool was not designed in one build
>>>>>>>>> multi target manner.
>>>>>>>>> We would have to introduce custom arguments to specify the target,
>>>>>>>>> "i686", "x86_64", "thumb2pe" etc.
>>>>>>>>> This will most likely break some level of backwards compatibility
>>>>>>>>> with the binutils version.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> LLVM Developers mailing list
>>>>>>>>> llvm-dev at lists.llvm.org
>>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170213/c45a1dda/attachment-0001.html>

More information about the llvm-dev mailing list