[llvm-dev] [lldb-dev] Trying out lld to link windows binaries (using msvc as a compiler)
Zachary Turner via llvm-dev
llvm-dev at lists.llvm.org
Fri Jan 26 09:49:46 PST 2018
(Ignore the fact that my hashes are 8 byte in the "good" file, this is due
to some local changes I've been experimenting with)
On Fri, Jan 26, 2018 at 9:48 AM Zachary Turner <zturner at google.com> wrote:
> I did this:
>
> // a.cpp
> static int x = 0;
> void b(int);
> void a(int) {
> if (x)
> b(x);
> }
> int main(int argc, char **argv) {
> a(argc);
> return x;
> }
>
>
> clang-cl /Z7 /c a.cpp /Foa.noghash.obj
> clang-cl /Z7 /c a.cpp -mllvm -emit-codeview-ghash-section
> /Foa.ghash.good.obj
> llvm-objcopy a.noghash.obj a.ghash.bad.obj
> obj2yaml a.ghash.good.obj > a.ghash.good.yaml
> obj2yaml a.ghash.bad.obj > a.ghash.bad.yaml
>
> Then open these 2 yaml files up in a diff viewer. It looks like the
> hashes aren't getting emitted at all. For example, in the good yaml file I
> see this:
>
> - Name: '.debug$H'
> Characteristics: [ IMAGE_SCN_CNT_INITIALIZED_DATA,
> IMAGE_SCN_MEM_DISCARDABLE, IMAGE_SCN_MEM_READ ]
> Alignment: 4
> SectionData:
> C5C93301000001005549419E78044E3896D45CD7009428758BE4A1E2B3E022BA267DEE221F5C42B17BCA182AF84584814A8B5E7E3FB17B397A9E3DEA75CD5627
> GlobalHashes:
> Version: 0
> HashAlgorithm: 1
> HashValues:
> - 5549419E78044E38
> - 96D45CD700942875
> - 8BE4A1E2B3E022BA
> - 267DEE221F5C42B1
> - 7BCA182AF8458481
> - 4A8B5E7E3FB17B39
> - 7A9E3DEA75CD5627
> - Name: .pdata
>
> And in the bad yaml file I see this:
> - Name: '.debug$H'
> Characteristics: [ IMAGE_SCN_CNT_INITIALIZED_DATA,
> IMAGE_SCN_MEM_DISCARDABLE, IMAGE_SCN_MEM_READ ]
> Alignment: 4
> SectionData: C5C9330100000000
> GlobalHashes:
> Version: 0
> HashAlgorithm: 0
> - Name: .pdata
>
> Don't focus too much on trying to figure out weird linker errors. Just
> get the output of obj2yaml to be identical when run under a diff utility,
> then everything should work fine.
>
> On Fri, Jan 26, 2018 at 7:27 AM Leonardo Santagada <santagada at gmail.com>
> wrote:
>
>> I'm so close I can almost smell it :)
>>
>> I know how bad the code looks, I don't intend to submit this, but if you
>> want to try it out its at:
>> https://gist.github.com/santagada/544136b1ee143bf31653b1158ac6829e
>>
>> I'm seeing: lld-link.exe: error: duplicate symbol: "<redacted_unmangled>"
>> (<redacted>) in <internal> and in <redacted_filename>.obj, looking at the
>> .yaml dump the symbols are all similar to this:
>>
>> - Name: <redacted>
>> Value: 0
>> SectionNumber: 0
>> SimpleType: IMAGE_SYM_TYPE_NULL
>> ComplexType: IMAGE_SYM_DTYPE_FUNCTION
>> StorageClass: IMAGE_SYM_CLASS_WEAK_EXTERNAL
>> WeakExternal:
>> TagIndex: 134
>> Characteristics: IMAGE_WEAK_EXTERN_SEARCH_LIBRARY
>>
>> On Thu, Jan 25, 2018 at 8:01 PM, Zachary Turner <zturner at google.com>
>> wrote:
>>
>>> I haven't really dabbled in this part of the COFF format personally, so
>>> hopefully I'm not leading you astray :)
>>>
>>> But I checked the code for coff2yaml, and I see this:
>>>
>>> } else if (Symbol.isSectionDefinition()) {
>>> // This symbol represents a section definition.
>>> assert(Symbol.getNumberOfAuxSymbols() == 1 &&
>>> "Expected a single aux symbol to describe this section!");
>>> const object::coff_aux_section_definition *ObjSD =
>>> reinterpret_cast<const object::coff_aux_section_definition
>>> *>(
>>> AuxData.data());
>>>
>>> So it looks like you need exactly 1 aux symbol for each section symbol.
>>>
>>> I then scrolled up in this function to figure out where AuxData comes
>>> from, and it comes from COFFObjectFile::getSymbolAuxData. I think that
>>> function holds the clue to what you need to do. It looks like you need to
>>> set coff::symbol::NumberOfAuxSymbols to 1, and then there is a comment in
>>> getSymbolAuxData which says:
>>>
>>> // AUX data comes immediately after the symbol in COFF
>>> Aux = reinterpret_cast<const uint8_t *>(Symbol.getRawPtr()) +
>>> SymbolSize;
>>>
>>> So I think you just need to write the bytes immediately after the
>>> coff::symbol. The thing you need to write looks like a
>>> coff::coff_aux_section_definition structure.
>>>
>>> For the CheckSum, look at WinCOFFObjectWriter::writeSection. It looks
>>> like its a CRC32 of the actual section contents, which you can generate
>>> with a couple of lines of code:
>>>
>>> JamCRC JC(/*Init=*/0);
>>> JC.update(DebugHContents);
>>> AuxSymbol.CheckSum = JC.getCRC();
>>>
>>> Hope this helps
>>>
>>> On Thu, Jan 25, 2018 at 10:46 AM Leonardo Santagada <santagada at gmail.com>
>>> wrote:
>>>
>>>>
>>>> I see that there is an auxsymbol per section symbol, and also on the
>>>> yaml representation there is a checksum, selection and unused all of them I
>>>> have no idea how to fill in, also this aux symbol might have some important
>>>> information for me to patch on the other symbols. Can you find the part in
>>>> llvm that it writes those? because at least for auxsymbol the yaml part of
>>>> the code threats as a binary blob so there is no info on what they should
>>>> be.
>>>>
>>>> On Thu, Jan 25, 2018 at 7:15 PM, Zachary Turner <zturner at google.com>
>>>> wrote:
>>>>
>>>>> If you run obj2yaml against a very simple object file, you'll see
>>>>> something like this at the end:
>>>>> ```
>>>>> symbols:
>>>>> - Name: '@comp.id'
>>>>> Value: 17130443
>>>>> SectionNumber: -1
>>>>> SimpleType: IMAGE_SYM_TYPE_NULL
>>>>> ComplexType: IMAGE_SYM_DTYPE_NULL
>>>>> StorageClass: IMAGE_SYM_CLASS_STATIC
>>>>> - Name: '@feat.00'
>>>>> Value: 2147484048 <(21)%204748-4048>
>>>>> SectionNumber: -1
>>>>> SimpleType: IMAGE_SYM_TYPE_NULL
>>>>> ComplexType: IMAGE_SYM_DTYPE_NULL
>>>>> StorageClass: IMAGE_SYM_CLASS_STATIC
>>>>> - Name: .drectve
>>>>> Value: 0
>>>>> SectionNumber: 1
>>>>> SimpleType: IMAGE_SYM_TYPE_NULL
>>>>> ComplexType: IMAGE_SYM_DTYPE_NULL
>>>>> StorageClass: IMAGE_SYM_CLASS_STATIC
>>>>> SectionDefinition:
>>>>> Length: 47
>>>>> NumberOfRelocations: 0
>>>>> NumberOfLinenumbers: 0
>>>>> CheckSum: 0
>>>>> Number: 0
>>>>> ...
>>>>> ```
>>>>>
>>>>> There's a structure called coff::symbol which basically represents
>>>>> each one of these records. It looks like this:
>>>>>
>>>>> ```
>>>>> struct symbol {
>>>>> char Name[NameSize];
>>>>> uint32_t Value;
>>>>> int32_t SectionNumber;
>>>>> uint16_t Type;
>>>>> uint8_t StorageClass;
>>>>> uint8_t NumberOfAuxSymbols;
>>>>> };
>>>>> ```
>>>>>
>>>>> So you'll need to create one for the debug$H section and stick it into
>>>>> the list. This particular list doesn't have to be in any special order, so
>>>>> you can just put it at the end (although it's probably not that much harder
>>>>> to insert into the middle, and it will make for a good test that you've
>>>>> done it right. The output can be diffed against clang-cl object file and
>>>>> be identical this way). So write all the normal symbols as you probably
>>>>> already are, then write one for the .debug$H section. Initialize the
>>>>> fields to the same thing that you see when you run obj2yaml against an
>>>>> object file generated by clang-cl for the .debug$H section.
>>>>>
>>>>> This structure doesn't contain any kind of file pointers or offsets,
>>>>> so all you really need to fix up are the "SectionNumber" fields. Basically
>>>>> as you are writing the existing symbols, you would do somethign like:
>>>>>
>>>>> for (const auto &Sym : ObjFile.symbols()) {
>>>>> if (Symbol->SectionNumber >= DebugHInsertionIndex)
>>>>> ++Symbol->SectionNumber;
>>>>> writeSymbol(Sym);
>>>>> }
>>>>> writeSymbol(DebugHSym);
>>>>>
>>>>>
>>>>> On Thu, Jan 25, 2018 at 9:57 AM Leonardo Santagada <
>>>>> santagada at gmail.com> wrote:
>>>>>
>>>>>> Any idea on how to create this new symbol there? I saw that there is
>>>>>> a symbol pointing to each section, but didn't understand the format, and
>>>>>> yaml2obj doesn't check it or do anything with the list.
>>>>>>
>>>>>> On Thu, Jan 25, 2018 at 6:56 PM, Leonardo Santagada <
>>>>>> santagada at gmail.com> wrote:
>>>>>>
>>>>>>> YES, THANK YOU... I WAS THINKING THIS BUT COMPLETELY FORGOT.
>>>>>>>
>>>>>>> sorry for the caps... long day of working on this, and using vs
>>>>>>> 2017, which adds a new section type .chks64 that I couldn't find
>>>>>>> documentation anywhere was difficult. I highly recommend everyone to just
>>>>>>> not using vs 2017 until 15.8 or something, our internal bug list is
>>>>>>> gigantic.
>>>>>>>
>>>>>>> On Thu, Jan 25, 2018 at 6:52 PM, Zachary Turner <zturner at google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Actually I already have a theory that even though you are adding
>>>>>>>> the section to the section table, you might not be adding a *symbol* for
>>>>>>>> the section to the symbol table. So the existing symbols (which reference
>>>>>>>> sections by index) will all be wrong because you've inserted a new
>>>>>>>> section. Still though, obj2yaml would expose that.
>>>>>>>>
>>>>>>>> On Thu, Jan 25, 2018 at 9:50 AM Zachary Turner <zturner at google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Yea as long as you compare clang-cl object file with automatically
>>>>>>>>> generated .debug$H section against clang-cl object file without .debug$H
>>>>>>>>> but added after the fact with llvm-objcopy, that should expose the problem
>>>>>>>>> I think when you run obj2yaml on them.
>>>>>>>>>
>>>>>>>>> On Thu, Jan 25, 2018 at 9:49 AM Leonardo Santagada <
>>>>>>>>> santagada at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> I did reorder my sections, so that .debug$H is in the correct
>>>>>>>>>> place, but now I get some errors on dubplicate symbols, I created a folder
>>>>>>>>>> with examples:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> https://www.dropbox.com/sh/nmvzi44pi0boe76/AAA0f47O5PCJ9JiUc6wVuwBra?dl=0
>>>>>>>>>>
>>>>>>>>>> t.obj is generated by vs 2015 and it links fine with
>>>>>>>>>> lld-link.exe, but tout.obj gives this errors:
>>>>>>>>>>
>>>>>>>>>> lld-link.exe /DEBUG:GHASH tout.obj
>>>>>>>>>> LLD-LINK.EXE: error: duplicate symbol:
>>>>>>>>>> __local_stdio_printf_options in tout.obj and in
>>>>>>>>>> LIBCMT.lib(default_local_stdio_options.obj)
>>>>>>>>>> LLD-LINK.EXE: error: duplicate symbol:
>>>>>>>>>> __local_stdio_printf_options in tout.obj and in
>>>>>>>>>> libvcruntime.lib(undname.obj)
>>>>>>>>>>
>>>>>>>>>> I'm using PEView from http://wjradburn.com/software/ to look at
>>>>>>>>>> the files and can't see anything wrong, except some valid differences in
>>>>>>>>>> the offsets being used for the data (so pointer to data is different
>>>>>>>>>> between them).
>>>>>>>>>>
>>>>>>>>>> I will look into yaml2obj now to see if I see anything else weird
>>>>>>>>>> going on.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Jan 25, 2018 at 6:41 PM, Zachary Turner <
>>>>>>>>>> zturner at google.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> I'm pretty confident that cl is not putting anything strange in
>>>>>>>>>>> the .debug$T sections. We've done a lot of testing and never seen anything
>>>>>>>>>>> except CodeView type records in a .debug$T. My hunch is that your objcopy
>>>>>>>>>>> patch is probably not doing the right thing in one or more of the section
>>>>>>>>>>> headers, and this is confusing the linker.
>>>>>>>>>>>
>>>>>>>>>>> One idea might be to build a simple object file with clang-cl
>>>>>>>>>>> but without the magic -mllvm -emit-codeview-ghash-section, then run your
>>>>>>>>>>> llvm-objcopy on it. Then build the same object file passing -mllvm
>>>>>>>>>>> -emit-codeview-ghash-section. Then run obj2yaml on both and diff the
>>>>>>>>>>> results. They should be byte-for-byte identical. That should give you a
>>>>>>>>>>> clue about if objcopy is doing something wrong.
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Jan 25, 2018 at 2:21 AM Leonardo Santagada <
>>>>>>>>>>> santagada at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Don't worry, I definetly want to perfect this to generate legal
>>>>>>>>>>>> obj files, this is just to speed up testing.
>>>>>>>>>>>>
>>>>>>>>>>>> Now after patching all the obj files I get this errors when
>>>>>>>>>>>> linking a small part of our code base (msvc 2017 15.5.3, lld and
>>>>>>>>>>>> llvm-objcopy 7.0.0):
>>>>>>>>>>>> lld-link.exe : error : relocation against symbol in discarded
>>>>>>>>>>>> section: $LN8
>>>>>>>>>>>> lld-link.exe : error : relocation against symbol in discarded
>>>>>>>>>>>> section: $LN43
>>>>>>>>>>>> lld-link.exe : error : relocation against symbol in discarded
>>>>>>>>>>>> section: $LN37
>>>>>>>>>>>>
>>>>>>>>>>>> I'm starting to guess that cl.exe might be putting some random
>>>>>>>>>>>> comdat or other discardable symbols in the .debug$T and clang doesn't? I
>>>>>>>>>>>> will try to debug this and see what more I can uncover.
>>>>>>>>>>>>
>>>>>>>>>>>> Linking works perfectly without my llvm-objcopy pass to add
>>>>>>>>>>>> .debug$H?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Jan 25, 2018 at 1:53 AM, Zachary Turner <
>>>>>>>>>>>> zturner at google.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> It might not influence LLD, but at the same time we don't want
>>>>>>>>>>>>> to upstream something that is producing technically illegal COFF files.
>>>>>>>>>>>>> Also good to hear about the planned changes to your header files. Looking
>>>>>>>>>>>>> forward to hearing about your experiences with clang-cl.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Jan 24, 2018 at 10:41 AM Leonardo Santagada <
>>>>>>>>>>>>> santagada at gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I finally got my first .obj file patched with .debug$H to
>>>>>>>>>>>>>> look somewhat right. I added the new section at the end of the file so I
>>>>>>>>>>>>>> don't have to recalculate all sections (although now I probably could
>>>>>>>>>>>>>> position it in the middle, knowing that each section is: SizeOfRawData +
>>>>>>>>>>>>>> (last.Header.NumberOfRelocations * (4+4+2)) and the $H needs to come right
>>>>>>>>>>>>>> after $T in the file). That although illegal based on the coff specs
>>>>>>>>>>>>>> doesn't seem its going to influence lld.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Also we talked and we are probably going to do something
>>>>>>>>>>>>>> similar to a bunch of windows defines and a check for our own define (to
>>>>>>>>>>>>>> guarantee that no one imported windows.h before win32.h) and drop the
>>>>>>>>>>>>>> namespace and the conflicting names.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Jan 23, 2018 at 12:46 AM, Zachary Turner <
>>>>>>>>>>>>>> zturner at google.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> That's very possible that a 3rd party indirect header
>>>>>>>>>>>>>>> include is involved. One idea might be like I suggested where you #define
>>>>>>>>>>>>>>> _WINDOWS_ in win32.h and guarantee that it's always included first. Then
>>>>>>>>>>>>>>> those other headers won't be able to #include <windows.h>. but it will
>>>>>>>>>>>>>>> probably greatly expand the amount of stuff you have to add to win32.h, as
>>>>>>>>>>>>>>> you will probably find some callers of functions that aren't yet in your
>>>>>>>>>>>>>>> win32.h that you'd have to add.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Jan 22, 2018 at 3:28 PM Leonardo Santagada <
>>>>>>>>>>>>>>> santagada at gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Ok some information was lost on getting this example to
>>>>>>>>>>>>>>>> you, I'm sorry for not being clear.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We have a huge code base, let's say 90% of it doesn't
>>>>>>>>>>>>>>>> include either header, 9% include win32.h and 1% includes both, I will try
>>>>>>>>>>>>>>>> to discover why, but my guess is they include both a third party that
>>>>>>>>>>>>>>>> includes windows.h and some of our libs that use win32.h.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I will try to fully understand this tomorrow.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I guess clang will not implement this ever so finishing the
>>>>>>>>>>>>>>>> object copier is the best solution until all code is ported to clang.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 23 Jan 2018 00:02, "Zachary Turner" <zturner at google.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> You said win32.h doesn't include windows.h, but main.cpp
>>>>>>>>>>>>>>>>> does. So what's the disadvantage of just including it in win32.h anyway,
>>>>>>>>>>>>>>>>> since it's already going to be in every translation unit? (Unless you
>>>>>>>>>>>>>>>>> didn't mean to #include it in main.cpp)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I guess all I can do is warn you how bad of an idea this
>>>>>>>>>>>>>>>>> is. For starters, I already found a bug in your code ;-)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> // stdint.h
>>>>>>>>>>>>>>>>> typedef int int32_t;
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> // winnt.h
>>>>>>>>>>>>>>>>> typedef long LONG;
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> // windef.h
>>>>>>>>>>>>>>>>> typedef struct tagPOINT
>>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>>> LONG x; // long x
>>>>>>>>>>>>>>>>> LONG y; // long y
>>>>>>>>>>>>>>>>> } POINT, *PPOINT, NEAR *NPPOINT, FAR *LPPOINT;
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> // win32.h
>>>>>>>>>>>>>>>>> typedef int32_t LONG;
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> struct POINT
>>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>>> LONG x; // int x
>>>>>>>>>>>>>>>>> LONG y; // int y
>>>>>>>>>>>>>>>>> };
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> So POINT is defined two different ways. In your minimal
>>>>>>>>>>>>>>>>> interface, it's declared as 2 int32's, which are int. In the actual
>>>>>>>>>>>>>>>>> Windows header files, it's declared as 2 longs.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This might seem like a unimportant bug since int and long
>>>>>>>>>>>>>>>>> are the same size, but int and long also mangle differently and affect
>>>>>>>>>>>>>>>>> overload resolution, so you could have weird linker errors or call the
>>>>>>>>>>>>>>>>> wrong function overload.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Plus, it illustrates the fact that this struct *actually
>>>>>>>>>>>>>>>>> is* a different type from the one in the windows header.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> You said at the end that you never intentionally import
>>>>>>>>>>>>>>>>> win32.h and windows.h from the same translation unit. But then in this
>>>>>>>>>>>>>>>>> example you did. I wonder if you could enforce that by doing this:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> // win32.h
>>>>>>>>>>>>>>>>> #pragma once
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> // Error if windows.h was included before us.
>>>>>>>>>>>>>>>>> #if defined(_WINDOWS_)
>>>>>>>>>>>>>>>>> #error "You're including win32.h after having already
>>>>>>>>>>>>>>>>> included windows.h. Don't do this!"
>>>>>>>>>>>>>>>>> #endif
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> // And also make sure windows.h can't get included after us
>>>>>>>>>>>>>>>>> #define _WINDOWS_
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> For the record, I tried the test case you linked when
>>>>>>>>>>>>>>>>> windows.h is not included in main.cpp and it works (but still has the bug
>>>>>>>>>>>>>>>>> about int and long).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Jan 22, 2018 at 2:23 PM Leonardo Santagada <
>>>>>>>>>>>>>>>>> santagada at gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> It is super gross, but we copy parts of windows.h because
>>>>>>>>>>>>>>>>>> having all of it if both gigantic and very very messy. So our win32.h has a
>>>>>>>>>>>>>>>>>> couple thousands of lines and not 30k+ for windows.h and we try to have
>>>>>>>>>>>>>>>>>> zero macros. Win32.h doesn't include windows.h so using ::BOOL wouldn't
>>>>>>>>>>>>>>>>>> work. We don't want to create a namespace, we just want a cleaner interface
>>>>>>>>>>>>>>>>>> to windows api. The namespace with c linkage is the way to trick cl into
>>>>>>>>>>>>>>>>>> allowing us to in some files have both windows.h and Win32.h. I really
>>>>>>>>>>>>>>>>>> don't see any way for us to have this Win32.h without this cl support, so
>>>>>>>>>>>>>>>>>> maybe we should either put windows.h in a compiled header somewhere and not
>>>>>>>>>>>>>>>>>> care that it is infecting everything or just have one place we can call to
>>>>>>>>>>>>>>>>>> clean up after including windows.h (a massive set of undefs).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> So using can't work, because we never intentionally
>>>>>>>>>>>>>>>>>> import windows.h and win32.h on the same translation unit.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mon, Jan 22, 2018 at 7:08 PM, Zachary Turner <
>>>>>>>>>>>>>>>>>> zturner at google.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> This is pretty gross, honestly :)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Can't you just use using declarations?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> namespace Win32 {
>>>>>>>>>>>>>>>>>>> extern "C" {
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> using ::BOOL;
>>>>>>>>>>>>>>>>>>> using ::LONG;
>>>>>>>>>>>>>>>>>>> using ::POINT;
>>>>>>>>>>>>>>>>>>> using ::LPPOINT;
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> using ::GetCursorPos;
>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> This works with clang-cl.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, Jan 22, 2018 at 5:39 AM Leonardo Santagada <
>>>>>>>>>>>>>>>>>>> santagada at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Here it is a minimal example, we do this so we don't
>>>>>>>>>>>>>>>>>>>> have to import the whole windows api everywhere.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> https://gist.github.com/santagada/7977e929d31c629c4bf18ebb987f6be3
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Sun, Jan 21, 2018 at 2:31 AM, Zachary Turner <
>>>>>>>>>>>>>>>>>>>> zturner at google.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Clang-cl maintains compatibility with msvc even in
>>>>>>>>>>>>>>>>>>>>> cases where it’s non standards compliant (eg 2 phase name lookup), but we
>>>>>>>>>>>>>>>>>>>>> try to keep these cases few and far between.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> To help me understand your case, do you mean you copy
>>>>>>>>>>>>>>>>>>>>> windows.h and modify it? How does this lead to the same struct being
>>>>>>>>>>>>>>>>>>>>> defined twice? If i were to write this:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> struct Foo {};
>>>>>>>>>>>>>>>>>>>>> struct Foo {};
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Is this a small repro of the issue you’re talking
>>>>>>>>>>>>>>>>>>>>> about?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 3:44 PM Leonardo Santagada <
>>>>>>>>>>>>>>>>>>>>> santagada at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I can totally see something like incremental linking
>>>>>>>>>>>>>>>>>>>>>> with a simple padding between obj and a mapping file (which can also help
>>>>>>>>>>>>>>>>>>>>>> with edit and continue, something we also would love to have).
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> We have another developer doing the port to support
>>>>>>>>>>>>>>>>>>>>>> clang-cl, but although most of our code also goes trough a version of
>>>>>>>>>>>>>>>>>>>>>> clang, migrating the rest to clang-cl has been a fight. From what I heard
>>>>>>>>>>>>>>>>>>>>>> the main problem is that we have a copy of parts of windows.h (so not to
>>>>>>>>>>>>>>>>>>>>>> bring the awful parts of it like lower case macros) and that totally works
>>>>>>>>>>>>>>>>>>>>>> on cl, but clang (at least 6.0) complains about two struct/vars with the
>>>>>>>>>>>>>>>>>>>>>> same name, even though they are exactly the same. Making clang-cl as broken
>>>>>>>>>>>>>>>>>>>>>> as cl.exe is not an option I suppose? I would love to turn on a flag
>>>>>>>>>>>>>>>>>>>>>> --accept-that-cl-made-bad-decisions-and-live-with-it and have this at least
>>>>>>>>>>>>>>>>>>>>>> until this is completely fixed in our code base.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> the biggest win with moving to cl would be a better
>>>>>>>>>>>>>>>>>>>>>> more standards compliant compiler, no 1 minute compiles on heavily
>>>>>>>>>>>>>>>>>>>>>> templated files and maybe the holy grail of ThinLTO.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 10:56 PM, Zachary Turner <
>>>>>>>>>>>>>>>>>>>>>> zturner at google.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> 10-15s will be hard without true incremental linking.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> At some point that's going to be the only way to get
>>>>>>>>>>>>>>>>>>>>>>> any faster, but incremental linking is hard (putting it lightly), and since
>>>>>>>>>>>>>>>>>>>>>>> our full links are already really fast we think we can get reasonably close
>>>>>>>>>>>>>>>>>>>>>>> to link.exe incremental speeds with full links. But it's never enough and
>>>>>>>>>>>>>>>>>>>>>>> I will always want it to be faster, so you may see incremental linking in
>>>>>>>>>>>>>>>>>>>>>>> the future after we hit a performance wall with full link speed :)
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> In any case, I'm definitely interested in seeing
>>>>>>>>>>>>>>>>>>>>>>> what kind of numbers you get with /debug:ghash after you get this
>>>>>>>>>>>>>>>>>>>>>>> llvm-objcopy feature implemented. So keep me updated :)
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> As an aside, have you tried building with clang
>>>>>>>>>>>>>>>>>>>>>>> instead of cl? If you build with clang you wouldn't even have to do this
>>>>>>>>>>>>>>>>>>>>>>> llvm-objcopy work, because it would "just work". If you've tried but ran
>>>>>>>>>>>>>>>>>>>>>>> into issues I'm interested in hearing about those too. On the other hand,
>>>>>>>>>>>>>>>>>>>>>>> it's also reasonable to only switch one thing at a time.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 1:34 PM Leonardo Santagada <
>>>>>>>>>>>>>>>>>>>>>>> santagada at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> if we get to < 30s I think most users would prefer
>>>>>>>>>>>>>>>>>>>>>>>> it to link.exe, just hopping there is still some more optimizations to get
>>>>>>>>>>>>>>>>>>>>>>>> closer to ELF linking times (around 10-15s here).
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 9:50 PM, Zachary Turner <
>>>>>>>>>>>>>>>>>>>>>>>> zturner at google.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Generally speaking a good rule of thumb is that
>>>>>>>>>>>>>>>>>>>>>>>>> /debug:ghash will be close to or faster than /debug:fastlink, but with none
>>>>>>>>>>>>>>>>>>>>>>>>> of the penalties like slow debug time
>>>>>>>>>>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 12:44 PM Zachary Turner <
>>>>>>>>>>>>>>>>>>>>>>>>> zturner at google.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Chrome is actually one of my exact benchmark
>>>>>>>>>>>>>>>>>>>>>>>>>> cases. When building blink_core.dll and browser_tests.exe, i get anywhere
>>>>>>>>>>>>>>>>>>>>>>>>>> from a 20-40% reduction in link time. We have some other optimizations in
>>>>>>>>>>>>>>>>>>>>>>>>>> the pipeline but not upstream yet.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> My best time so far (including other
>>>>>>>>>>>>>>>>>>>>>>>>>> optimizations not yet upstream) is 28s on blink_core.dll, compared to 110s
>>>>>>>>>>>>>>>>>>>>>>>>>> with /debug
>>>>>>>>>>>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 12:28 PM Leonardo
>>>>>>>>>>>>>>>>>>>>>>>>>> Santagada <santagada at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 9:05 PM, Zachary Turner
>>>>>>>>>>>>>>>>>>>>>>>>>>> <zturner at google.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> You probably don't want to go down the same
>>>>>>>>>>>>>>>>>>>>>>>>>>>> route that clang goes through to write the object file. If you think
>>>>>>>>>>>>>>>>>>>>>>>>>>>> yaml2coff is convoluted, the way clang does it will just give you a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> headache. There are multiple abstractions involved to account for
>>>>>>>>>>>>>>>>>>>>>>>>>>>> different object file formats (ELF, COFF, MachO) and output formats
>>>>>>>>>>>>>>>>>>>>>>>>>>>> (Assembly, binary file). At least with yaml2coff
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I think your phrase got cut there, but yeah I
>>>>>>>>>>>>>>>>>>>>>>>>>>> just found AsmPrinter.cpp and it is convoluted.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's true that yaml2coff is using the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> COFFParser structure, but if you look at the writeCOFF
>>>>>>>>>>>>>>>>>>>>>>>>>>>> function in yaml2coff it's pretty bare-metal. The logic you need will be
>>>>>>>>>>>>>>>>>>>>>>>>>>>> almost identical, except that instead of checking the COFFParser for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> various fields, you'll check the existing COFFObjectFile, which should have
>>>>>>>>>>>>>>>>>>>>>>>>>>>> similar fields.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> The only thing you need to different is when
>>>>>>>>>>>>>>>>>>>>>>>>>>>> writing the section table and section contents, to insert a new entry. Since
>>>>>>>>>>>>>>>>>>>>>>>>>>>> you're injecting a section into the middle, you'll also probably need to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> push back the file pointer of all subsequent sections so that they don't
>>>>>>>>>>>>>>>>>>>>>>>>>>>> overlap. (e.g. if the original sections are 1, 2, 3, 4, 5 and you insert
>>>>>>>>>>>>>>>>>>>>>>>>>>>> between 2 and 3, then the original sections 3, 4, and 5 would need to have
>>>>>>>>>>>>>>>>>>>>>>>>>>>> their FilePointerToRawData offset by the size of the new section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I have the PE/COFF spec open here and I'm happy
>>>>>>>>>>>>>>>>>>>>>>>>>>> that I read a bit of it so I actually know what you are talking about...
>>>>>>>>>>>>>>>>>>>>>>>>>>> yeah it doesn't seem too complicated.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> If you need to know what values to put for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> other fields in a section header, run `dumpbin /headers foo.obj` on a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> clang-generated object file that has a .debug$H section already (e.g. run
>>>>>>>>>>>>>>>>>>>>>>>>>>>> clang with -emit-codeview-ghash-section, and look at the properties of the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> .debug$H section and use the same values).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks I will do that and then also look at how
>>>>>>>>>>>>>>>>>>>>>>>>>>> the CodeView part of the code does it if I can't understand some of it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> The only invariant that needs to be maintained
>>>>>>>>>>>>>>>>>>>>>>>>>>>> is that Section[N]->FilePointerOfRawData ==
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Section[N-1]->FilePointerOfRawData + Section[N-1]->SizeOfRawData
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Well, that and all the sections need to be on
>>>>>>>>>>>>>>>>>>>>>>>>>>> the final file... But I'm hopeful.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Anyone has times on linking a big project like
>>>>>>>>>>>>>>>>>>>>>>>>>>> chrome with this so that at least I know what kind of performance to expect?
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> My numbers are something like:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 pdb per obj file: link.exe takes ~15 minutes
>>>>>>>>>>>>>>>>>>>>>>>>>>> and 16GB of ram, lld-link.exe takes 2:30 minutes and ~8GB of ram
>>>>>>>>>>>>>>>>>>>>>>>>>>> around 10 pdbs per folder: link.exe takes 1
>>>>>>>>>>>>>>>>>>>>>>>>>>> minute and 2-3GB of ram, lld-link.exe takes 1:30 minutes and ~6GB of ram
>>>>>>>>>>>>>>>>>>>>>>>>>>> faslink: link.exe takes 40 seconds, but then 20
>>>>>>>>>>>>>>>>>>>>>>>>>>> seconds of loading at the first break point in the debugger and we lost DIA
>>>>>>>>>>>>>>>>>>>>>>>>>>> support for listing symbols.
>>>>>>>>>>>>>>>>>>>>>>>>>>> incremental: link.exe takes 8 seconds, but it
>>>>>>>>>>>>>>>>>>>>>>>>>>> only happens when very minor changes happen.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> We have an non negligible number of symbols used
>>>>>>>>>>>>>>>>>>>>>>>>>>> on some runtime systems.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 11:52 AM Leonardo
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Santagada <santagada at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the tips, I now have something that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reads the obj file, finds .debug$T sections and global hashes it (proof of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> concept kind of code). What I can't find is: how does clang itself writes
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the coff files with global hashes, as that might help me understand how to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> create the .debug$H section, how to update the file section count and how
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to properly write this back.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The code on yaml2coff is expecting to be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> working on the yaml COFFParser struct and I'm having quite a bit of a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> headache turning the COFFObjectFile into a COFFParser object or
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compatible... Tomorrow I might try the very non efficient path of coff2yaml
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and then yaml2coff with the hashes header... but it seems way too
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inefficient and convoluted.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jan 19, 2018 at 10:38 PM, Zachary
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Turner <zturner at google.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jan 19, 2018 at 1:02 PM Leonardo
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Santagada <santagada at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jan 19, 2018 at 9:44 PM, Zachary
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Turner <zturner at google.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jan 19, 2018 at 12:29 PM Leonardo
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Santagada <santagada at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> No I didn't, I used cl.exe from the visual
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> studio toolchain. What I'm proposing is a tool for processing .obj files in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> COFF format, reading them and generating the GHASH part.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> To make our build faster we use hundreds
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of unity build files (.cpp's with a lot of other .cpp's in them aka munch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> files) but still have a lot of single .cpp's as well (in total something
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like 3.4k .obj files).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ps: sorry for sending to the wrong list, I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> was reading about llvm mailing lists and jumped when I saw what I thought
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> was a lld exclusive list.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> A tool like this would be useful, yes.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We've talked about it internally as well and agreed it would be useful, we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just haven't prioritized it. If you're interested in submitting a patch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> along those lines though, I think it would be a good addition.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm not sure what the best place for it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> would be. llvm-readobj and llvm-objdump seem like obvious choices, but
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> they are intended to be read-only, so perhaps they wouldn't be a good fit.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> llvm-pdbutil is kind of a hodgepodge of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> everything else related to PDBs and symbols, so I wouldn't be opposed to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> making a new subcommand there called "ghash" or something that could
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> process an object file and output a new object file with a .debug$H section.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> A third option would be to make a new tool
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I don't htink it would be that hard to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> write. If you're interested in trying to make a patch for this, I can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> offer some guidance on where to look in the code. Otherwise it's something
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that we'll probably get to, I'm just not sure when.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would love to write it and contribute it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> back, please do tell, I did find some of the code of ghash in lld, but in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fuzzy on the llvm codeview part of it and never seen llvm-readobj/objdump
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> or llvm-pdbutil, but I'm not afraid to look :)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Luckily all of the important code is hidden
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behind library calls, and it should already just do the right thing, so I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspect you won't need to know much about CodeView to do this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think Peter has the right idea about
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> putting this in llvm-objcopy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> You can look at one of the existing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CopyBinary functions there, which currently only work for ELF, but you can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just make a new overload that accepts a COFFObjectFile.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would probably start by iterating over each
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the sections (getNumberOfSections / getSectionName) looking for .debug$T
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and .debug$H sections.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If you find a .debug$H section then you can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just skip that object file.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If you find a .debug$T but not a .debug$H,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> then basically do the same thing that LLD does in PDBLinker::mergeDebugT
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (create a CVTypeArray, and pass it to GloballyHashedType::hashTypes. That
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will return an array of hash values. (the format of .debug$H is the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> header, followed by the hash values). Then when you're writing the list of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sections, just add in the .debug$H section right after the .debug$T section.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently llvm-objcopy only writes ELF files,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> so it would need to be taught to write COFF files. We have code to do this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the yaml2obj utility (specifically, in yaml2coff.cpp in the function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> writeCOFF). There may be a way to move this code to somewhere else
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (llvm/Object/COFF.h?) so that it can be re-used by both yaml2coff and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> llvm-objcopy, but in the worst case scenario you could copy the code and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> re-write it to work with these new structures.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lastly, you'll probably want to put all of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this behind an option in llvm-objcopy such as -add-codeview-ghash-section
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Leonardo Santagada
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Leonardo Santagada
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Leonardo Santagada
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Leonardo Santagada
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Leonardo Santagada
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Leonardo Santagada
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Leonardo Santagada
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>>
>>>>>>>>>>>> Leonardo Santagada
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>> Leonardo Santagada
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Leonardo Santagada
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Leonardo Santagada
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Leonardo Santagada
>>>>
>>>
>>
>>
>> --
>>
>> Leonardo Santagada
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180126/23654a9e/attachment-0001.html>
More information about the llvm-dev
mailing list