[llvm-dev] RFC: A new llvm-dlltool driver and llvm-lib driver improvements

Peter Collingbourne via llvm-dev llvm-dev at lists.llvm.org
Mon Feb 13 17:26:06 PST 2017 

On Mon, Feb 13, 2017 at 5:20 PM, Rui Ueyama via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> On Mon, Feb 13, 2017 at 4:46 PM, Martell Malone <martellmalone at gmail.com>
> wrote:
>
>> No, I meant an even thinner wrapper which textually translates arguments.
>>> For example, the wrapper would translates "/out:foo.exe foo.obj" to "-o
>>> foo.exe foo.obj" and then call lld::COFFF:link(). It doesn't do anything
>>> with Config object nor LinkerDriver::run and have absolutely zero knowledge
>>> on the internals of LLD.
>>
>> Ohh okay I misunderstood.
>>
>> (Why I'm asking this is because I want to sure that there's no difference
>>> between the regular Windows linker and mingw. If all differences are
>>> superficial, the command line translator should just work. If not, there's
>>> some semantic difference.)
>>
>> The libraries can be used with MSVC link.exe directly assuming the
>> correct arguments are passed to the linker.
>> I tested this after creating a working llvm-dlltool.
>> The only change I had to make to support this was alter ming-w64 crt to
>> change all references from __image_base__ to _ImageBase to support that
>>
>
> You may be able to define __ImageBase as a weak external symbol to
> __image_base__. Then, if __ImageBase is not defined, all references against
> __ImageBase will be resolved using __image_base__.
>

Or you could have your wrapper driver pass
"/alternatename:__image_base__=__ImageBase".

Peter


>
>
>> That should be fine.
>>
>> Great
>>
>> On Mon, Feb 13, 2017 at 9:34 PM, Rui Ueyama <ruiu at google.com> wrote:
>>
>>> On Mon, Feb 13, 2017 at 1:16 PM, Martell Malone <martellmalone at gmail.com
>>> > wrote:
>>>
>>>> I wonder if it can be a wrapper for LLD/COFF. It can be an in-process
>>>>> wrapper, meaning that you could add a main function for mingw which
>>>>> translates all arguments into the MSVC style and then invoke the COFF
>>>>> linker's main function.
>>>>
>>>> That should work, if I am understanding this correctly I can create an
>>>> argument parser (probably partially based on the ELF one) then I can set
>>>> the data in the COFF Config object and call COFF LinkerDriver::run etc
>>>> directly from that?
>>>>
>>>
>>> No, I meant an even thinner wrapper which textually translates
>>> arguments. For example, the wrapper would translates "/out:foo.exe foo.obj"
>>> to "-o foo.exe foo.obj" and then call lld::COFFF:link(). It doesn't do
>>> anything with Config object nor LinkerDriver::run and have absolutely zero
>>> knowledge on the internals of LLD.
>>>
>>> (Why I'm asking this is because I want to sure that there's no
>>> difference between the regular Windows linker and mingw. If all differences
>>> are superficial, the command line translator should just work. If not,
>>> there's some semantic difference.)
>>>
>>> This proposal is for generally moving code from lld into llvm that could
>>>> be part of lib.exe / dlltool, what are your thoughts here?
>>>>
>>>
>>> That should be fine.
>>>
>>>
>>>> On Mon, Feb 13, 2017 at 8:57 PM, Rui Ueyama <ruiu at google.com> wrote:
>>>>
>>>>> On Mon, Feb 13, 2017 at 12:52 PM, Martell Malone <
>>>>> martellmalone at gmail.com> wrote:
>>>>>
>>>>>> Also you need to make a change to LLD/COFF to accept GNU command
>>>>>>> arguments, right? (Looks like you already have that patch locally.)
>>>>>>
>>>>>> Yes
>>>>>>
>>>>>
>>>>> I wonder if it can be a wrapper for LLD/COFF. It can be an in-process
>>>>> wrapper, meaning that you could add a main function for mingw which
>>>>> translates all arguments into the MSVC style and then invoke the COFF
>>>>> linker's main function.
>>>>>
>>>>>
>>>>>>
>>>>>> My patch to hack lld into accepting some very basic gnu front end
>>>>>>> arguments was enough to get all the above working which was enough to
>>>>>>> develop further.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Feb 13, 2017 at 8:41 PM, Rui Ueyama <ruiu at google.com> wrote:
>>>>>>
>>>>>>> On Mon, Feb 13, 2017 at 12:33 PM, Martell Malone <
>>>>>>> martellmalone at gmail.com> wrote:
>>>>>>>
>>>>>>>> Hey Rui,
>>>>>>>>
>>>>>>>>> I wonder how llvm-dlltool would fit in the entire picture of mingw
>>>>>>>>> support. I don't think dlltool is the last missing piece. What do you need
>>>>>>>>> to do other than that to fully support mingw using LLVM toolchain?
>>>>>>>>
>>>>>>>> Other then changing `lib/MC/WinCOFFStreamer.cpp` to not use
>>>>>>>> -aligncomm within the EmitCommonSymbol function and a single patch for
>>>>>>>> mingw-w64 itself to pre-populate it's .ctors and .dtors list, so
>>>>>>>> llvm-dlltool is infact the only missing part really.
>>>>>>>>
>>>>>>>
>>>>>>> Also you need to make a change to LLD/COFF to accept GNU command
>>>>>>> arguments, right? (Looks like you already have that patch locally.)
>>>>>>>
>>>>>>>
>>>>>>>> This gives us a fully working clang based mingw-w64 C compiler.
>>>>>>>>
>>>>>>>> C++ and exception handling is a different story.
>>>>>>>>
>>>>>>>> libc++ is somewhat working with the following test results
>>>>>>>>
>>>>>>>> Expected Passes    : 2188
>>>>>>>> Expected Failures  : 44
>>>>>>>> Unsupported Tests  : 588
>>>>>>>> Unexpected Failures: 2816
>>>>>>>>
>>>>>>>> I was able to build it with exceptions disabled and I actually
>>>>>>>> managed to bootstrap llvm and clang itself.
>>>>>>>> It didn't run very well though as you can imagine based on the
>>>>>>>> above tests.
>>>>>>>>
>>>>>>>> It's not a performance issue but a code maintenance issue. The
>>>>>>>>> initial patches to support mingw was trying to add a new linker driver and
>>>>>>>>> a different linkin semantics to the COFF linker which seemed too
>>>>>>>>> complicated to me. IIRC, I suggested adding a shim which translates GNU
>>>>>>>>> command line arguments to MSVC linker arguments. Didn't it work?
>>>>>>>>
>>>>>>>> There were just so many differences between link and ld I never
>>>>>>>> followed down that path. I pushed forward with the short import library
>>>>>>>> support based on your later suggestions. My patch to hack lld into
>>>>>>>> accepting some very basic gnu front end arguments was enough to get all the
>>>>>>>> above working which was enough to develop further.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Martell
>>>>>>>>
>>>>>>>> On Mon, Feb 13, 2017 at 8:08 PM, Rui Ueyama <ruiu at google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Martell,
>>>>>>>>>
>>>>>>>>> I wonder how llvm-dlltool would fit in the entire picture of mingw
>>>>>>>>> support. I don't think dlltool is the last missing piece. What do you need
>>>>>>>>> to do other than that to fully support mingw using LLVM toolchain?
>>>>>>>>>
>>>>>>>>> On Mon, Feb 13, 2017 at 8:56 AM, Martell Malone via llvm-dev <
>>>>>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>>>>>
>>>>>>>>>> Hey llvm'ers,
>>>>>>>>>>
>>>>>>>>>> I have been working on a dlltool replacement for llvm.
>>>>>>>>>> Here is my initial differential https://reviews.llvm.org/D29892
>>>>>>>>>> It is based on some functionality that already exists in lld.
>>>>>>>>>> I added functionality to support, PE COFF Weak Externals and of
>>>>>>>>>> course a front end to actually use it.
>>>>>>>>>> I believe the work here can also be used for llvm-lib and lessen
>>>>>>>>>> the load on lld.
>>>>>>>>>> I would like some comments about how this could be be structured
>>>>>>>>>> to live in llvm with a shared code base across lib ar and dlltool.
>>>>>>>>>> I also have a section below called "Difference from lib" which is
>>>>>>>>>> somewhat of a rationale for the tool.
>>>>>>>>>>
>>>>>>>>>> Many Thanks,
>>>>>>>>>> Martell
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Context
>>>>>>>>>> ==========
>>>>>>>>>>
>>>>>>>>>> Awhile back I talked to various llvm'ers about getting mingw-w64
>>>>>>>>>> support for lld.
>>>>>>>>>> There were a few issues raised but the main issue was that
>>>>>>>>>> mingw-w64 should try best to comply with the PECOFF spec, adding support
>>>>>>>>>> for custom sections and various binutils/mingw hacks would impact the
>>>>>>>>>> performance of the COFF linker and in general is not something that lld
>>>>>>>>>> should support.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> It's not a performance issue but a code maintenance issue. The
>>>>>>>>> initial patches to support mingw was trying to add a new linker driver and
>>>>>>>>> a different linkin semantics to the COFF linker which seemed too
>>>>>>>>> complicated to me. IIRC, I suggested adding a shim which translates GNU
>>>>>>>>> command line arguments to MSVC linker arguments. Didn't it work?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Motivation
>>>>>>>>>> ==========
>>>>>>>>>>
>>>>>>>>>> The main motivation was because dlltool and ld did not comply
>>>>>>>>>> with PECOFF Spec.
>>>>>>>>>> It has some custom formatting and uses the assembler for some
>>>>>>>>>> reason to generate import libraries, it did not use the short import
>>>>>>>>>> library format at all.
>>>>>>>>>>
>>>>>>>>>> There has been many work arounds for the problem this creates
>>>>>>>>>> such as the creation of the `reimp` tool. Which imports MSVC built libs
>>>>>>>>>> creates a def and uses dlltool so that the binutils linker can use it.
>>>>>>>>>>
>>>>>>>>>> We should just be using the short import format and the linker
>>>>>>>>>> should support that.
>>>>>>>>>> Thus llvm-dlltool was born.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Difference from lib
>>>>>>>>>> ===================
>>>>>>>>>>
>>>>>>>>>> Using PE COFF spec (section 8, Import Library Format) should be
>>>>>>>>>> self explanatory.
>>>>>>>>>> lib.exe is able to accept def files and create libraries using
>>>>>>>>>> this format.
>>>>>>>>>>
>>>>>>>>>> example
>>>>>>>>>>
>>>>>>>>>>   `LIBRARY "user32.dll"
>>>>>>>>>>    EXPORTS
>>>>>>>>>>    MessageBoxA`
>>>>>>>>>>
>>>>>>>>>> LIB.exe can create a user32.lib with the function MessageBoxA
>>>>>>>>>> from the above definition.
>>>>>>>>>>
>>>>>>>>>> Mingw-w64 is different MSVC in that we need to compile the
>>>>>>>>>> runtime.
>>>>>>>>>> MS provide us with their crt prebuilt so lib.exe doesn't have
>>>>>>>>>> support for external function aliasing.
>>>>>>>>>> We often use aliases for posix naming reasons as well as avoid
>>>>>>>>>> using the MS version of a function.
>>>>>>>>>>
>>>>>>>>>> example
>>>>>>>>>>
>>>>>>>>>>   `LIBRARY "user32.dll"
>>>>>>>>>>    EXPORTS
>>>>>>>>>>    MessageBoxA
>>>>>>>>>>    MessageBoxW==MessageBoxA`
>>>>>>>>>>
>>>>>>>>>> LIB.exe doesn't not have support for this but dlltool does.
>>>>>>>>>> The best fit for this within the spec is PE COFF spec (Aux
>>>>>>>>>> Format 3: Weak Externals)
>>>>>>>>>>
>>>>>>>>>> Mingw-w64 also uses lib prefix and .a file extensions for the
>>>>>>>>>> library name so the driver should be different. The above example would be
>>>>>>>>>> libuser32.a, for when shared and static versions of the same library exist
>>>>>>>>>> there is the .dll.a variant.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Issues
>>>>>>>>>> =======
>>>>>>>>>>
>>>>>>>>>> Like more of the gnu suite dlltool was not designed in one build
>>>>>>>>>> multi target manner.
>>>>>>>>>> We would have to introduce custom arguments to specify the
>>>>>>>>>> target, "i686", "x86_64", "thumb2pe" etc.
>>>>>>>>>> This will most likely break some level of backwards compatibility
>>>>>>>>>> with the binutils version.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> LLVM Developers mailing list
>>>>>>>>>> llvm-dev at lists.llvm.org
>>>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>


-- 
-- 
Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170213/70e0cfb0/attachment.html>

More information about the llvm-dev mailing list