[llvm-dev] [lldb-dev] Trying out lld to link windows binaries (using msvc as a compiler)

Sun Jan 28 14:20:03 PST 2018

This part is only for objects with /Z7 debug information in them right? I
think most of the third parties are either: .lib/obj without debug
information, the same with information on pdb files. Rewriting all
.lib/.obj with /Z7 information seems doable with a small python script, the
pdb one is going to be more work, but I always wanted to know how a pdb
file is structured so "fun" times ahead. But yeah printing it out, and
timing it might be very useful indeed.

Did anyone tried to compile/link lld-link.exe with LTO+PGO to see how much
faster can it get? I might try that as well, as 10% speed improvement might
be handy.

On Sun, Jan 28, 2018 at 11:14 PM, Zachary Turner <zturner at google.com> wrote:

> Look for this code in lld/coff/pdb.cpp
>
>
> if (Config->DebugGHashes) {
> ArrayRef<GloballyHashedType> Hashes;
> std::vector<GloballyHashedType> OwnedHashes;
> if (Optional<ArrayRef<uint8_t>> DebugH = getDebugH(File))
> Hashes = getHashesFromDebugH(*DebugH);
> else {
> OwnedHashes = GloballyHashedType::hashTypes(Types);
> Hashes = OwnedHashes;
> }
>
> In the else block there, add a log message that says “synthesizing
> .debug$h section for “ + Obj->Name
>
> See how many of these you get. When I build chrome + all third party
> libraries this way i get about 100, which is small enough to still see
> large performance gains.
>
> If you have many 3rd party libraries, it may be necessary to rewrite the
> .lib files too, not just the .obj files. Eventually I’ll get around to
> implementing all of this as well, as well as better heuristics in lld-link
> to disable ghash if it’s going to be slow
> On Sun, Jan 28, 2018 at 1:51 PM Leonardo Santagada <santagada at gmail.com>
> wrote:
>
>> Ok I went for kind of middle ground solution, I patch in the obj files,
>> but as adding a new section didn't seem to work, I add a "shadow" section,
>> by editing the pointer to line number and the virtual size on the .debug$T
>> section. Although technically broken, both link.exe and lld-link.exe don't
>> seem to mind the alterations and as the shadow .debug$H is not really a
>> section anymore (its just some bytes at the end of the file) it doesn't
>> change anything else that does matter. With that I could do my first test
>> with a subset of our code base, and the results are not good. I found one
>> of our sources that break the ghash computation, I will get more info on
>> this and post a proper bug report, but I guess its type information that is
>> generated only by msvc. The other more alarming problem is that linking is
>> way slower with the ghahes... my guess is that we have a bunch of pdb files
>> for some third party libraries and calculating those ghashes takes more
>> time than actual linking of this small part of the source (it links in 4s
>> in both link.exe and lld-link.exe without ghashes).
>>
>> On Fri, Jan 26, 2018 at 8:52 PM, Leonardo Santagada <santagada at gmail.com>
>> wrote:
>>
>>> We don't generate any .lib as those don't work well with incremental
>>> linking (and give zero advantages when linking AFAIK), and it would be
>>> pretty easy to have a modern format for having a .ghash for multiple files,
>>> something simple like size prefixed name and then size prefixed ghash blobs.
>>>
>>> On Fri, Jan 26, 2018 at 8:44 PM, Zachary Turner <zturner at google.com>
>>> wrote:
>>>
>>>> We considered that early on, but most object files actually end up in
>>>> .lib files so unless there were a way to connect the objects in the .lib to
>>>> the corresponding .ghash files, this would disable ghash usage for a large
>>>> amount of inputs. Supporting both is an option, but it adds a bit of
>>>> complexity and I’m not totally convinced it’s worth it
>>>>
>>>> On Fri, Jan 26, 2018 at 11:38 AM Leonardo Santagada <
>>>> santagada at gmail.com> wrote:
>>>>
>>>>> it does.
>>>>>
>>>>> I just had an epiphany: why not just write a .ghash file and have lld
>>>>> read those if they exist for an .obj file?
>>>>>
>>>>> Seem much simpler than trying to wire up a 20 year old file format. I
>>>>> will try to do this, is something like this acceptable for LLD? The cool
>>>>> thing is that I can generate .ghash for .lib or any obj lying around (maybe
>>>>> even for pdb in the future).
>>>>>
>>>>> On Fri, Jan 26, 2018 at 8:32 PM, Zachary Turner <zturner at google.com>
>>>>> wrote:
>>>>>
>>>>>> In general, we should be able to accept any MSVC .obj file to LLD.
>>>>>> At the very least, we're not aware of any cases that don't work.
>>>>>>
>>>>>> Does your MSVC .obj file link fine before you add the .debug$H?
>>>>>>
>>>>>> On Fri, Jan 26, 2018 at 11:23 AM Leonardo Santagada <
>>>>>> santagada at gmail.com> wrote:
>>>>>>
>>>>>>> Okay, apparently coff2yaml and yaml2coff are not in a great place as
>>>>>>> they both don't deal well with the fact that you can have overlapping
>>>>>>> sections, which seems to be what clang-cl produces (the .data section
>>>>>>> points to the same place as a later section). Which is not a big big
>>>>>>> problem for me particularly because msvc doesn't even generate .data
>>>>>>> sections in .obj.
>>>>>>>
>>>>>>> I'm trying to put support for .bss sections in both coff2yaml and
>>>>>>> yaml2coff... but I still can link just fine with my transformations
>>>>>>> clang-cl generated files... what does give me problems is msvc .obj files.
>>>>>>> Have you tried to link one of these?
>>>>>>>
>>>>>>> On Fri, Jan 26, 2018 at 8:05 PM, Leonardo Santagada <
>>>>>>> santagada at gmail.com> wrote:
>>>>>>>
>>>>>>>> yeah, apparently .bss has a flag of unitialized data that is not
>>>>>>>> being respected on the layout of the coff files (it should skip those
>>>>>>>> sections) but I dunno what to do with .data as it doesn't have a size.
>>>>>>>>
>>>>>>>> (resending as apparently my pastes generated a ton of hidden html
>>>>>>>> data and this message hit the mailinglist limit of 100k)
>>>>>>>> --
>>>>>>>>
>>>>>>>> Leonardo Santagada
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Leonardo Santagada
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Leonardo Santagada
>>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Leonardo Santagada
>>>
>>
>>
>>
>> --
>>
>> Leonardo Santagada
>>
>

-- 

Leonardo Santagada
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180128/30d8fa51/attachment.html>