[llvm-dev] [lldb-dev] Trying out lld to link windows binaries (using msvc as a compiler)
Zachary Turner via llvm-dev
llvm-dev at lists.llvm.org
Fri Jan 26 09:48:35 PST 2018
I did this:
// a.cpp
static int x = 0;
void b(int);
void a(int) {
if (x)
b(x);
}
int main(int argc, char **argv) {
a(argc);
return x;
}
clang-cl /Z7 /c a.cpp /Foa.noghash.obj
clang-cl /Z7 /c a.cpp -mllvm -emit-codeview-ghash-section
/Foa.ghash.good.obj
llvm-objcopy a.noghash.obj a.ghash.bad.obj
obj2yaml a.ghash.good.obj > a.ghash.good.yaml
obj2yaml a.ghash.bad.obj > a.ghash.bad.yaml
Then open these 2 yaml files up in a diff viewer. It looks like the hashes
aren't getting emitted at all. For example, in the good yaml file I see
this:
- Name: '.debug$H'
Characteristics: [ IMAGE_SCN_CNT_INITIALIZED_DATA,
IMAGE_SCN_MEM_DISCARDABLE, IMAGE_SCN_MEM_READ ]
Alignment: 4
SectionData:
C5C93301000001005549419E78044E3896D45CD7009428758BE4A1E2B3E022BA267DEE221F5C42B17BCA182AF84584814A8B5E7E3FB17B397A9E3DEA75CD5627
GlobalHashes:
Version: 0
HashAlgorithm: 1
HashValues:
- 5549419E78044E38
- 96D45CD700942875
- 8BE4A1E2B3E022BA
- 267DEE221F5C42B1
- 7BCA182AF8458481
- 4A8B5E7E3FB17B39
- 7A9E3DEA75CD5627
- Name: .pdata
And in the bad yaml file I see this:
- Name: '.debug$H'
Characteristics: [ IMAGE_SCN_CNT_INITIALIZED_DATA,
IMAGE_SCN_MEM_DISCARDABLE, IMAGE_SCN_MEM_READ ]
Alignment: 4
SectionData: C5C9330100000000
GlobalHashes:
Version: 0
HashAlgorithm: 0
- Name: .pdata
Don't focus too much on trying to figure out weird linker errors. Just get
the output of obj2yaml to be identical when run under a diff utility, then
everything should work fine.
On Fri, Jan 26, 2018 at 7:27 AM Leonardo Santagada <santagada at gmail.com>
wrote:
> I'm so close I can almost smell it :)
>
> I know how bad the code looks, I don't intend to submit this, but if you
> want to try it out its at:
> https://gist.github.com/santagada/544136b1ee143bf31653b1158ac6829e
>
> I'm seeing: lld-link.exe: error: duplicate symbol: "<redacted_unmangled>"
> (<redacted>) in <internal> and in <redacted_filename>.obj, looking at the
> .yaml dump the symbols are all similar to this:
>
> - Name: <redacted>
> Value: 0
> SectionNumber: 0
> SimpleType: IMAGE_SYM_TYPE_NULL
> ComplexType: IMAGE_SYM_DTYPE_FUNCTION
> StorageClass: IMAGE_SYM_CLASS_WEAK_EXTERNAL
> WeakExternal:
> TagIndex: 134
> Characteristics: IMAGE_WEAK_EXTERN_SEARCH_LIBRARY
>
> On Thu, Jan 25, 2018 at 8:01 PM, Zachary Turner <zturner at google.com>
> wrote:
>
>> I haven't really dabbled in this part of the COFF format personally, so
>> hopefully I'm not leading you astray :)
>>
>> But I checked the code for coff2yaml, and I see this:
>>
>> } else if (Symbol.isSectionDefinition()) {
>> // This symbol represents a section definition.
>> assert(Symbol.getNumberOfAuxSymbols() == 1 &&
>> "Expected a single aux symbol to describe this section!");
>> const object::coff_aux_section_definition *ObjSD =
>> reinterpret_cast<const object::coff_aux_section_definition *>(
>> AuxData.data());
>>
>> So it looks like you need exactly 1 aux symbol for each section symbol.
>>
>> I then scrolled up in this function to figure out where AuxData comes
>> from, and it comes from COFFObjectFile::getSymbolAuxData. I think that
>> function holds the clue to what you need to do. It looks like you need to
>> set coff::symbol::NumberOfAuxSymbols to 1, and then there is a comment in
>> getSymbolAuxData which says:
>>
>> // AUX data comes immediately after the symbol in COFF
>> Aux = reinterpret_cast<const uint8_t *>(Symbol.getRawPtr()) +
>> SymbolSize;
>>
>> So I think you just need to write the bytes immediately after the
>> coff::symbol. The thing you need to write looks like a
>> coff::coff_aux_section_definition structure.
>>
>> For the CheckSum, look at WinCOFFObjectWriter::writeSection. It looks
>> like its a CRC32 of the actual section contents, which you can generate
>> with a couple of lines of code:
>>
>> JamCRC JC(/*Init=*/0);
>> JC.update(DebugHContents);
>> AuxSymbol.CheckSum = JC.getCRC();
>>
>> Hope this helps
>>
>> On Thu, Jan 25, 2018 at 10:46 AM Leonardo Santagada <santagada at gmail.com>
>> wrote:
>>
>>>
>>> I see that there is an auxsymbol per section symbol, and also on the
>>> yaml representation there is a checksum, selection and unused all of them I
>>> have no idea how to fill in, also this aux symbol might have some important
>>> information for me to patch on the other symbols. Can you find the part in
>>> llvm that it writes those? because at least for auxsymbol the yaml part of
>>> the code threats as a binary blob so there is no info on what they should
>>> be.
>>>
>>> On Thu, Jan 25, 2018 at 7:15 PM, Zachary Turner <zturner at google.com>
>>> wrote:
>>>
>>>> If you run obj2yaml against a very simple object file, you'll see
>>>> something like this at the end:
>>>> ```
>>>> symbols:
>>>> - Name: '@comp.id'
>>>> Value: 17130443
>>>> SectionNumber: -1
>>>> SimpleType: IMAGE_SYM_TYPE_NULL
>>>> ComplexType: IMAGE_SYM_DTYPE_NULL
>>>> StorageClass: IMAGE_SYM_CLASS_STATIC
>>>> - Name: '@feat.00'
>>>> Value: 2147484048 <(21)%204748-4048>
>>>> SectionNumber: -1
>>>> SimpleType: IMAGE_SYM_TYPE_NULL
>>>> ComplexType: IMAGE_SYM_DTYPE_NULL
>>>> StorageClass: IMAGE_SYM_CLASS_STATIC
>>>> - Name: .drectve
>>>> Value: 0
>>>> SectionNumber: 1
>>>> SimpleType: IMAGE_SYM_TYPE_NULL
>>>> ComplexType: IMAGE_SYM_DTYPE_NULL
>>>> StorageClass: IMAGE_SYM_CLASS_STATIC
>>>> SectionDefinition:
>>>> Length: 47
>>>> NumberOfRelocations: 0
>>>> NumberOfLinenumbers: 0
>>>> CheckSum: 0
>>>> Number: 0
>>>> ...
>>>> ```
>>>>
>>>> There's a structure called coff::symbol which basically represents each
>>>> one of these records. It looks like this:
>>>>
>>>> ```
>>>> struct symbol {
>>>> char Name[NameSize];
>>>> uint32_t Value;
>>>> int32_t SectionNumber;
>>>> uint16_t Type;
>>>> uint8_t StorageClass;
>>>> uint8_t NumberOfAuxSymbols;
>>>> };
>>>> ```
>>>>
>>>> So you'll need to create one for the debug$H section and stick it into
>>>> the list. This particular list doesn't have to be in any special order, so
>>>> you can just put it at the end (although it's probably not that much harder
>>>> to insert into the middle, and it will make for a good test that you've
>>>> done it right. The output can be diffed against clang-cl object file and
>>>> be identical this way). So write all the normal symbols as you probably
>>>> already are, then write one for the .debug$H section. Initialize the
>>>> fields to the same thing that you see when you run obj2yaml against an
>>>> object file generated by clang-cl for the .debug$H section.
>>>>
>>>> This structure doesn't contain any kind of file pointers or offsets, so
>>>> all you really need to fix up are the "SectionNumber" fields. Basically as
>>>> you are writing the existing symbols, you would do somethign like:
>>>>
>>>> for (const auto &Sym : ObjFile.symbols()) {
>>>> if (Symbol->SectionNumber >= DebugHInsertionIndex)
>>>> ++Symbol->SectionNumber;
>>>> writeSymbol(Sym);
>>>> }
>>>> writeSymbol(DebugHSym);
>>>>
>>>>
>>>> On Thu, Jan 25, 2018 at 9:57 AM Leonardo Santagada <santagada at gmail.com>
>>>> wrote:
>>>>
>>>>> Any idea on how to create this new symbol there? I saw that there is a
>>>>> symbol pointing to each section, but didn't understand the format, and
>>>>> yaml2obj doesn't check it or do anything with the list.
>>>>>
>>>>> On Thu, Jan 25, 2018 at 6:56 PM, Leonardo Santagada <
>>>>> santagada at gmail.com> wrote:
>>>>>
>>>>>> YES, THANK YOU... I WAS THINKING THIS BUT COMPLETELY FORGOT.
>>>>>>
>>>>>> sorry for the caps... long day of working on this, and using vs 2017,
>>>>>> which adds a new section type .chks64 that I couldn't find documentation
>>>>>> anywhere was difficult. I highly recommend everyone to just not using vs
>>>>>> 2017 until 15.8 or something, our internal bug list is gigantic.
>>>>>>
>>>>>> On Thu, Jan 25, 2018 at 6:52 PM, Zachary Turner <zturner at google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Actually I already have a theory that even though you are adding the
>>>>>>> section to the section table, you might not be adding a *symbol* for the
>>>>>>> section to the symbol table. So the existing symbols (which reference
>>>>>>> sections by index) will all be wrong because you've inserted a new
>>>>>>> section. Still though, obj2yaml would expose that.
>>>>>>>
>>>>>>> On Thu, Jan 25, 2018 at 9:50 AM Zachary Turner <zturner at google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Yea as long as you compare clang-cl object file with automatically
>>>>>>>> generated .debug$H section against clang-cl object file without .debug$H
>>>>>>>> but added after the fact with llvm-objcopy, that should expose the problem
>>>>>>>> I think when you run obj2yaml on them.
>>>>>>>>
>>>>>>>> On Thu, Jan 25, 2018 at 9:49 AM Leonardo Santagada <
>>>>>>>> santagada at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> I did reorder my sections, so that .debug$H is in the correct
>>>>>>>>> place, but now I get some errors on dubplicate symbols, I created a folder
>>>>>>>>> with examples:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://www.dropbox.com/sh/nmvzi44pi0boe76/AAA0f47O5PCJ9JiUc6wVuwBra?dl=0
>>>>>>>>>
>>>>>>>>> t.obj is generated by vs 2015 and it links fine with lld-link.exe,
>>>>>>>>> but tout.obj gives this errors:
>>>>>>>>>
>>>>>>>>> lld-link.exe /DEBUG:GHASH tout.obj
>>>>>>>>> LLD-LINK.EXE: error: duplicate symbol:
>>>>>>>>> __local_stdio_printf_options in tout.obj and in
>>>>>>>>> LIBCMT.lib(default_local_stdio_options.obj)
>>>>>>>>> LLD-LINK.EXE: error: duplicate symbol:
>>>>>>>>> __local_stdio_printf_options in tout.obj and in
>>>>>>>>> libvcruntime.lib(undname.obj)
>>>>>>>>>
>>>>>>>>> I'm using PEView from http://wjradburn.com/software/ to look at
>>>>>>>>> the files and can't see anything wrong, except some valid differences in
>>>>>>>>> the offsets being used for the data (so pointer to data is different
>>>>>>>>> between them).
>>>>>>>>>
>>>>>>>>> I will look into yaml2obj now to see if I see anything else weird
>>>>>>>>> going on.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Jan 25, 2018 at 6:41 PM, Zachary Turner <
>>>>>>>>> zturner at google.com> wrote:
>>>>>>>>>
>>>>>>>>>> I'm pretty confident that cl is not putting anything strange in
>>>>>>>>>> the .debug$T sections. We've done a lot of testing and never seen anything
>>>>>>>>>> except CodeView type records in a .debug$T. My hunch is that your objcopy
>>>>>>>>>> patch is probably not doing the right thing in one or more of the section
>>>>>>>>>> headers, and this is confusing the linker.
>>>>>>>>>>
>>>>>>>>>> One idea might be to build a simple object file with clang-cl but
>>>>>>>>>> without the magic -mllvm -emit-codeview-ghash-section, then run your
>>>>>>>>>> llvm-objcopy on it. Then build the same object file passing -mllvm
>>>>>>>>>> -emit-codeview-ghash-section. Then run obj2yaml on both and diff the
>>>>>>>>>> results. They should be byte-for-byte identical. That should give you a
>>>>>>>>>> clue about if objcopy is doing something wrong.
>>>>>>>>>>
>>>>>>>>>> On Thu, Jan 25, 2018 at 2:21 AM Leonardo Santagada <
>>>>>>>>>> santagada at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Don't worry, I definetly want to perfect this to generate legal
>>>>>>>>>>> obj files, this is just to speed up testing.
>>>>>>>>>>>
>>>>>>>>>>> Now after patching all the obj files I get this errors when
>>>>>>>>>>> linking a small part of our code base (msvc 2017 15.5.3, lld and
>>>>>>>>>>> llvm-objcopy 7.0.0):
>>>>>>>>>>> lld-link.exe : error : relocation against symbol in discarded
>>>>>>>>>>> section: $LN8
>>>>>>>>>>> lld-link.exe : error : relocation against symbol in discarded
>>>>>>>>>>> section: $LN43
>>>>>>>>>>> lld-link.exe : error : relocation against symbol in discarded
>>>>>>>>>>> section: $LN37
>>>>>>>>>>>
>>>>>>>>>>> I'm starting to guess that cl.exe might be putting some random
>>>>>>>>>>> comdat or other discardable symbols in the .debug$T and clang doesn't? I
>>>>>>>>>>> will try to debug this and see what more I can uncover.
>>>>>>>>>>>
>>>>>>>>>>> Linking works perfectly without my llvm-objcopy pass to add
>>>>>>>>>>> .debug$H?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Jan 25, 2018 at 1:53 AM, Zachary Turner <
>>>>>>>>>>> zturner at google.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> It might not influence LLD, but at the same time we don't want
>>>>>>>>>>>> to upstream something that is producing technically illegal COFF files.
>>>>>>>>>>>> Also good to hear about the planned changes to your header files. Looking
>>>>>>>>>>>> forward to hearing about your experiences with clang-cl.
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Jan 24, 2018 at 10:41 AM Leonardo Santagada <
>>>>>>>>>>>> santagada at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I finally got my first .obj file patched with .debug$H to look
>>>>>>>>>>>>> somewhat right. I added the new section at the end of the file so I don't
>>>>>>>>>>>>> have to recalculate all sections (although now I probably could position it
>>>>>>>>>>>>> in the middle, knowing that each section is: SizeOfRawData +
>>>>>>>>>>>>> (last.Header.NumberOfRelocations * (4+4+2)) and the $H needs to come right
>>>>>>>>>>>>> after $T in the file). That although illegal based on the coff specs
>>>>>>>>>>>>> doesn't seem its going to influence lld.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Also we talked and we are probably going to do something
>>>>>>>>>>>>> similar to a bunch of windows defines and a check for our own define (to
>>>>>>>>>>>>> guarantee that no one imported windows.h before win32.h) and drop the
>>>>>>>>>>>>> namespace and the conflicting names.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Jan 23, 2018 at 12:46 AM, Zachary Turner <
>>>>>>>>>>>>> zturner at google.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> That's very possible that a 3rd party indirect header include
>>>>>>>>>>>>>> is involved. One idea might be like I suggested where you #define
>>>>>>>>>>>>>> _WINDOWS_ in win32.h and guarantee that it's always included first. Then
>>>>>>>>>>>>>> those other headers won't be able to #include <windows.h>. but it will
>>>>>>>>>>>>>> probably greatly expand the amount of stuff you have to add to win32.h, as
>>>>>>>>>>>>>> you will probably find some callers of functions that aren't yet in your
>>>>>>>>>>>>>> win32.h that you'd have to add.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Jan 22, 2018 at 3:28 PM Leonardo Santagada <
>>>>>>>>>>>>>> santagada at gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Ok some information was lost on getting this example to you,
>>>>>>>>>>>>>>> I'm sorry for not being clear.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We have a huge code base, let's say 90% of it doesn't
>>>>>>>>>>>>>>> include either header, 9% include win32.h and 1% includes both, I will try
>>>>>>>>>>>>>>> to discover why, but my guess is they include both a third party that
>>>>>>>>>>>>>>> includes windows.h and some of our libs that use win32.h.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I will try to fully understand this tomorrow.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I guess clang will not implement this ever so finishing the
>>>>>>>>>>>>>>> object copier is the best solution until all code is ported to clang.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 23 Jan 2018 00:02, "Zachary Turner" <zturner at google.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> You said win32.h doesn't include windows.h, but main.cpp
>>>>>>>>>>>>>>>> does. So what's the disadvantage of just including it in win32.h anyway,
>>>>>>>>>>>>>>>> since it's already going to be in every translation unit? (Unless you
>>>>>>>>>>>>>>>> didn't mean to #include it in main.cpp)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I guess all I can do is warn you how bad of an idea this
>>>>>>>>>>>>>>>> is. For starters, I already found a bug in your code ;-)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> // stdint.h
>>>>>>>>>>>>>>>> typedef int int32_t;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> // winnt.h
>>>>>>>>>>>>>>>> typedef long LONG;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> // windef.h
>>>>>>>>>>>>>>>> typedef struct tagPOINT
>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>> LONG x; // long x
>>>>>>>>>>>>>>>> LONG y; // long y
>>>>>>>>>>>>>>>> } POINT, *PPOINT, NEAR *NPPOINT, FAR *LPPOINT;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> // win32.h
>>>>>>>>>>>>>>>> typedef int32_t LONG;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> struct POINT
>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>> LONG x; // int x
>>>>>>>>>>>>>>>> LONG y; // int y
>>>>>>>>>>>>>>>> };
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> So POINT is defined two different ways. In your minimal
>>>>>>>>>>>>>>>> interface, it's declared as 2 int32's, which are int. In the actual
>>>>>>>>>>>>>>>> Windows header files, it's declared as 2 longs.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This might seem like a unimportant bug since int and long
>>>>>>>>>>>>>>>> are the same size, but int and long also mangle differently and affect
>>>>>>>>>>>>>>>> overload resolution, so you could have weird linker errors or call the
>>>>>>>>>>>>>>>> wrong function overload.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Plus, it illustrates the fact that this struct *actually
>>>>>>>>>>>>>>>> is* a different type from the one in the windows header.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> You said at the end that you never intentionally import
>>>>>>>>>>>>>>>> win32.h and windows.h from the same translation unit. But then in this
>>>>>>>>>>>>>>>> example you did. I wonder if you could enforce that by doing this:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> // win32.h
>>>>>>>>>>>>>>>> #pragma once
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> // Error if windows.h was included before us.
>>>>>>>>>>>>>>>> #if defined(_WINDOWS_)
>>>>>>>>>>>>>>>> #error "You're including win32.h after having already
>>>>>>>>>>>>>>>> included windows.h. Don't do this!"
>>>>>>>>>>>>>>>> #endif
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> // And also make sure windows.h can't get included after us
>>>>>>>>>>>>>>>> #define _WINDOWS_
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> For the record, I tried the test case you linked when
>>>>>>>>>>>>>>>> windows.h is not included in main.cpp and it works (but still has the bug
>>>>>>>>>>>>>>>> about int and long).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Jan 22, 2018 at 2:23 PM Leonardo Santagada <
>>>>>>>>>>>>>>>> santagada at gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It is super gross, but we copy parts of windows.h because
>>>>>>>>>>>>>>>>> having all of it if both gigantic and very very messy. So our win32.h has a
>>>>>>>>>>>>>>>>> couple thousands of lines and not 30k+ for windows.h and we try to have
>>>>>>>>>>>>>>>>> zero macros. Win32.h doesn't include windows.h so using ::BOOL wouldn't
>>>>>>>>>>>>>>>>> work. We don't want to create a namespace, we just want a cleaner interface
>>>>>>>>>>>>>>>>> to windows api. The namespace with c linkage is the way to trick cl into
>>>>>>>>>>>>>>>>> allowing us to in some files have both windows.h and Win32.h. I really
>>>>>>>>>>>>>>>>> don't see any way for us to have this Win32.h without this cl support, so
>>>>>>>>>>>>>>>>> maybe we should either put windows.h in a compiled header somewhere and not
>>>>>>>>>>>>>>>>> care that it is infecting everything or just have one place we can call to
>>>>>>>>>>>>>>>>> clean up after including windows.h (a massive set of undefs).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> So using can't work, because we never intentionally import
>>>>>>>>>>>>>>>>> windows.h and win32.h on the same translation unit.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Jan 22, 2018 at 7:08 PM, Zachary Turner <
>>>>>>>>>>>>>>>>> zturner at google.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This is pretty gross, honestly :)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Can't you just use using declarations?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> namespace Win32 {
>>>>>>>>>>>>>>>>>> extern "C" {
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> using ::BOOL;
>>>>>>>>>>>>>>>>>> using ::LONG;
>>>>>>>>>>>>>>>>>> using ::POINT;
>>>>>>>>>>>>>>>>>> using ::LPPOINT;
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> using ::GetCursorPos;
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This works with clang-cl.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mon, Jan 22, 2018 at 5:39 AM Leonardo Santagada <
>>>>>>>>>>>>>>>>>> santagada at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Here it is a minimal example, we do this so we don't
>>>>>>>>>>>>>>>>>>> have to import the whole windows api everywhere.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> https://gist.github.com/santagada/7977e929d31c629c4bf18ebb987f6be3
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Sun, Jan 21, 2018 at 2:31 AM, Zachary Turner <
>>>>>>>>>>>>>>>>>>> zturner at google.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Clang-cl maintains compatibility with msvc even in
>>>>>>>>>>>>>>>>>>>> cases where it’s non standards compliant (eg 2 phase name lookup), but we
>>>>>>>>>>>>>>>>>>>> try to keep these cases few and far between.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> To help me understand your case, do you mean you copy
>>>>>>>>>>>>>>>>>>>> windows.h and modify it? How does this lead to the same struct being
>>>>>>>>>>>>>>>>>>>> defined twice? If i were to write this:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> struct Foo {};
>>>>>>>>>>>>>>>>>>>> struct Foo {};
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Is this a small repro of the issue you’re talking about?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 3:44 PM Leonardo Santagada <
>>>>>>>>>>>>>>>>>>>> santagada at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I can totally see something like incremental linking
>>>>>>>>>>>>>>>>>>>>> with a simple padding between obj and a mapping file (which can also help
>>>>>>>>>>>>>>>>>>>>> with edit and continue, something we also would love to have).
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> We have another developer doing the port to support
>>>>>>>>>>>>>>>>>>>>> clang-cl, but although most of our code also goes trough a version of
>>>>>>>>>>>>>>>>>>>>> clang, migrating the rest to clang-cl has been a fight. From what I heard
>>>>>>>>>>>>>>>>>>>>> the main problem is that we have a copy of parts of windows.h (so not to
>>>>>>>>>>>>>>>>>>>>> bring the awful parts of it like lower case macros) and that totally works
>>>>>>>>>>>>>>>>>>>>> on cl, but clang (at least 6.0) complains about two struct/vars with the
>>>>>>>>>>>>>>>>>>>>> same name, even though they are exactly the same. Making clang-cl as broken
>>>>>>>>>>>>>>>>>>>>> as cl.exe is not an option I suppose? I would love to turn on a flag
>>>>>>>>>>>>>>>>>>>>> --accept-that-cl-made-bad-decisions-and-live-with-it and have this at least
>>>>>>>>>>>>>>>>>>>>> until this is completely fixed in our code base.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> the biggest win with moving to cl would be a better
>>>>>>>>>>>>>>>>>>>>> more standards compliant compiler, no 1 minute compiles on heavily
>>>>>>>>>>>>>>>>>>>>> templated files and maybe the holy grail of ThinLTO.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 10:56 PM, Zachary Turner <
>>>>>>>>>>>>>>>>>>>>> zturner at google.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 10-15s will be hard without true incremental linking.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> At some point that's going to be the only way to get
>>>>>>>>>>>>>>>>>>>>>> any faster, but incremental linking is hard (putting it lightly), and since
>>>>>>>>>>>>>>>>>>>>>> our full links are already really fast we think we can get reasonably close
>>>>>>>>>>>>>>>>>>>>>> to link.exe incremental speeds with full links. But it's never enough and
>>>>>>>>>>>>>>>>>>>>>> I will always want it to be faster, so you may see incremental linking in
>>>>>>>>>>>>>>>>>>>>>> the future after we hit a performance wall with full link speed :)
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> In any case, I'm definitely interested in seeing what
>>>>>>>>>>>>>>>>>>>>>> kind of numbers you get with /debug:ghash after you get this llvm-objcopy
>>>>>>>>>>>>>>>>>>>>>> feature implemented. So keep me updated :)
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> As an aside, have you tried building with clang
>>>>>>>>>>>>>>>>>>>>>> instead of cl? If you build with clang you wouldn't even have to do this
>>>>>>>>>>>>>>>>>>>>>> llvm-objcopy work, because it would "just work". If you've tried but ran
>>>>>>>>>>>>>>>>>>>>>> into issues I'm interested in hearing about those too. On the other hand,
>>>>>>>>>>>>>>>>>>>>>> it's also reasonable to only switch one thing at a time.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 1:34 PM Leonardo Santagada <
>>>>>>>>>>>>>>>>>>>>>> santagada at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> if we get to < 30s I think most users would prefer
>>>>>>>>>>>>>>>>>>>>>>> it to link.exe, just hopping there is still some more optimizations to get
>>>>>>>>>>>>>>>>>>>>>>> closer to ELF linking times (around 10-15s here).
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 9:50 PM, Zachary Turner <
>>>>>>>>>>>>>>>>>>>>>>> zturner at google.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Generally speaking a good rule of thumb is that
>>>>>>>>>>>>>>>>>>>>>>>> /debug:ghash will be close to or faster than /debug:fastlink, but with none
>>>>>>>>>>>>>>>>>>>>>>>> of the penalties like slow debug time
>>>>>>>>>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 12:44 PM Zachary Turner <
>>>>>>>>>>>>>>>>>>>>>>>> zturner at google.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Chrome is actually one of my exact benchmark
>>>>>>>>>>>>>>>>>>>>>>>>> cases. When building blink_core.dll and browser_tests.exe, i get anywhere
>>>>>>>>>>>>>>>>>>>>>>>>> from a 20-40% reduction in link time. We have some other optimizations in
>>>>>>>>>>>>>>>>>>>>>>>>> the pipeline but not upstream yet.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> My best time so far (including other optimizations
>>>>>>>>>>>>>>>>>>>>>>>>> not yet upstream) is 28s on blink_core.dll, compared to 110s with /debug
>>>>>>>>>>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 12:28 PM Leonardo
>>>>>>>>>>>>>>>>>>>>>>>>> Santagada <santagada at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 9:05 PM, Zachary Turner <
>>>>>>>>>>>>>>>>>>>>>>>>>> zturner at google.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> You probably don't want to go down the same
>>>>>>>>>>>>>>>>>>>>>>>>>>> route that clang goes through to write the object file. If you think
>>>>>>>>>>>>>>>>>>>>>>>>>>> yaml2coff is convoluted, the way clang does it will just give you a
>>>>>>>>>>>>>>>>>>>>>>>>>>> headache. There are multiple abstractions involved to account for
>>>>>>>>>>>>>>>>>>>>>>>>>>> different object file formats (ELF, COFF, MachO) and output formats
>>>>>>>>>>>>>>>>>>>>>>>>>>> (Assembly, binary file). At least with yaml2coff
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I think your phrase got cut there, but yeah I
>>>>>>>>>>>>>>>>>>>>>>>>>> just found AsmPrinter.cpp and it is convoluted.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> It's true that yaml2coff is using the COFFParser
>>>>>>>>>>>>>>>>>>>>>>>>>>> structure, but if you look at the writeCOFF
>>>>>>>>>>>>>>>>>>>>>>>>>>> function in yaml2coff it's pretty bare-metal. The logic you need will be
>>>>>>>>>>>>>>>>>>>>>>>>>>> almost identical, except that instead of checking the COFFParser for the
>>>>>>>>>>>>>>>>>>>>>>>>>>> various fields, you'll check the existing COFFObjectFile, which should have
>>>>>>>>>>>>>>>>>>>>>>>>>>> similar fields.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> The only thing you need to different is when
>>>>>>>>>>>>>>>>>>>>>>>>>>> writing the section table and section contents, to insert a new entry. Since
>>>>>>>>>>>>>>>>>>>>>>>>>>> you're injecting a section into the middle, you'll also probably need to
>>>>>>>>>>>>>>>>>>>>>>>>>>> push back the file pointer of all subsequent sections so that they don't
>>>>>>>>>>>>>>>>>>>>>>>>>>> overlap. (e.g. if the original sections are 1, 2, 3, 4, 5 and you insert
>>>>>>>>>>>>>>>>>>>>>>>>>>> between 2 and 3, then the original sections 3, 4, and 5 would need to have
>>>>>>>>>>>>>>>>>>>>>>>>>>> their FilePointerToRawData offset by the size of the new section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I have the PE/COFF spec open here and I'm happy
>>>>>>>>>>>>>>>>>>>>>>>>>> that I read a bit of it so I actually know what you are talking about...
>>>>>>>>>>>>>>>>>>>>>>>>>> yeah it doesn't seem too complicated.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> If you need to know what values to put for the
>>>>>>>>>>>>>>>>>>>>>>>>>>> other fields in a section header, run `dumpbin /headers foo.obj` on a
>>>>>>>>>>>>>>>>>>>>>>>>>>> clang-generated object file that has a .debug$H section already (e.g. run
>>>>>>>>>>>>>>>>>>>>>>>>>>> clang with -emit-codeview-ghash-section, and look at the properties of the
>>>>>>>>>>>>>>>>>>>>>>>>>>> .debug$H section and use the same values).
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks I will do that and then also look at how
>>>>>>>>>>>>>>>>>>>>>>>>>> the CodeView part of the code does it if I can't understand some of it.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> The only invariant that needs to be maintained
>>>>>>>>>>>>>>>>>>>>>>>>>>> is that Section[N]->FilePointerOfRawData ==
>>>>>>>>>>>>>>>>>>>>>>>>>>> Section[N-1]->FilePointerOfRawData + Section[N-1]->SizeOfRawData
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Well, that and all the sections need to be on the
>>>>>>>>>>>>>>>>>>>>>>>>>> final file... But I'm hopeful.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Anyone has times on linking a big project like
>>>>>>>>>>>>>>>>>>>>>>>>>> chrome with this so that at least I know what kind of performance to expect?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> My numbers are something like:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> 1 pdb per obj file: link.exe takes ~15 minutes
>>>>>>>>>>>>>>>>>>>>>>>>>> and 16GB of ram, lld-link.exe takes 2:30 minutes and ~8GB of ram
>>>>>>>>>>>>>>>>>>>>>>>>>> around 10 pdbs per folder: link.exe takes 1
>>>>>>>>>>>>>>>>>>>>>>>>>> minute and 2-3GB of ram, lld-link.exe takes 1:30 minutes and ~6GB of ram
>>>>>>>>>>>>>>>>>>>>>>>>>> faslink: link.exe takes 40 seconds, but then 20
>>>>>>>>>>>>>>>>>>>>>>>>>> seconds of loading at the first break point in the debugger and we lost DIA
>>>>>>>>>>>>>>>>>>>>>>>>>> support for listing symbols.
>>>>>>>>>>>>>>>>>>>>>>>>>> incremental: link.exe takes 8 seconds, but it
>>>>>>>>>>>>>>>>>>>>>>>>>> only happens when very minor changes happen.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> We have an non negligible number of symbols used
>>>>>>>>>>>>>>>>>>>>>>>>>> on some runtime systems.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 11:52 AM Leonardo
>>>>>>>>>>>>>>>>>>>>>>>>>>> Santagada <santagada at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the tips, I now have something that
>>>>>>>>>>>>>>>>>>>>>>>>>>>> reads the obj file, finds .debug$T sections and global hashes it (proof of
>>>>>>>>>>>>>>>>>>>>>>>>>>>> concept kind of code). What I can't find is: how does clang itself writes
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the coff files with global hashes, as that might help me understand how to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> create the .debug$H section, how to update the file section count and how
>>>>>>>>>>>>>>>>>>>>>>>>>>>> to properly write this back.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> The code on yaml2coff is expecting to be
>>>>>>>>>>>>>>>>>>>>>>>>>>>> working on the yaml COFFParser struct and I'm having quite a bit of a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> headache turning the COFFObjectFile into a COFFParser object or
>>>>>>>>>>>>>>>>>>>>>>>>>>>> compatible... Tomorrow I might try the very non efficient path of coff2yaml
>>>>>>>>>>>>>>>>>>>>>>>>>>>> and then yaml2coff with the hashes header... but it seems way too
>>>>>>>>>>>>>>>>>>>>>>>>>>>> inefficient and convoluted.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jan 19, 2018 at 10:38 PM, Zachary
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Turner <zturner at google.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jan 19, 2018 at 1:02 PM Leonardo
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Santagada <santagada at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jan 19, 2018 at 9:44 PM, Zachary
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Turner <zturner at google.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jan 19, 2018 at 12:29 PM Leonardo
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Santagada <santagada at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> No I didn't, I used cl.exe from the visual
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> studio toolchain. What I'm proposing is a tool for processing .obj files in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> COFF format, reading them and generating the GHASH part.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> To make our build faster we use hundreds of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unity build files (.cpp's with a lot of other .cpp's in them aka munch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> files) but still have a lot of single .cpp's as well (in total something
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like 3.4k .obj files).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ps: sorry for sending to the wrong list, I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> was reading about llvm mailing lists and jumped when I saw what I thought
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> was a lld exclusive list.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> A tool like this would be useful, yes.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We've talked about it internally as well and agreed it would be useful, we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just haven't prioritized it. If you're interested in submitting a patch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> along those lines though, I think it would be a good addition.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm not sure what the best place for it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> would be. llvm-readobj and llvm-objdump seem like obvious choices, but
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> they are intended to be read-only, so perhaps they wouldn't be a good fit.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> llvm-pdbutil is kind of a hodgepodge of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> everything else related to PDBs and symbols, so I wouldn't be opposed to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> making a new subcommand there called "ghash" or something that could
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> process an object file and output a new object file with a .debug$H section.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> A third option would be to make a new tool
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I don't htink it would be that hard to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> write. If you're interested in trying to make a patch for this, I can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> offer some guidance on where to look in the code. Otherwise it's something
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that we'll probably get to, I'm just not sure when.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would love to write it and contribute it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> back, please do tell, I did find some of the code of ghash in lld, but in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fuzzy on the llvm codeview part of it and never seen llvm-readobj/objdump
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> or llvm-pdbutil, but I'm not afraid to look :)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Luckily all of the important code is hidden
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behind library calls, and it should already just do the right thing, so I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspect you won't need to know much about CodeView to do this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think Peter has the right idea about putting
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this in llvm-objcopy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> You can look at one of the existing CopyBinary
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions there, which currently only work for ELF, but you can just make a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new overload that accepts a COFFObjectFile.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would probably start by iterating over each
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the sections (getNumberOfSections / getSectionName) looking for .debug$T
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and .debug$H sections.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If you find a .debug$H section then you can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just skip that object file.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If you find a .debug$T but not a .debug$H,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> then basically do the same thing that LLD does in PDBLinker::mergeDebugT
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (create a CVTypeArray, and pass it to GloballyHashedType::hashTypes. That
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will return an array of hash values. (the format of .debug$H is the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> header, followed by the hash values). Then when you're writing the list of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sections, just add in the .debug$H section right after the .debug$T section.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently llvm-objcopy only writes ELF files,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> so it would need to be taught to write COFF files. We have code to do this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the yaml2obj utility (specifically, in yaml2coff.cpp in the function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> writeCOFF). There may be a way to move this code to somewhere else
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (llvm/Object/COFF.h?) so that it can be re-used by both yaml2coff and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> llvm-objcopy, but in the worst case scenario you could copy the code and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> re-write it to work with these new structures.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lastly, you'll probably want to put all of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this behind an option in llvm-objcopy such as -add-codeview-ghash-section
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Leonardo Santagada
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Leonardo Santagada
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Leonardo Santagada
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Leonardo Santagada
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Leonardo Santagada
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Leonardo Santagada
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>>
>>>>>>>>>>>>> Leonardo Santagada
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>> Leonardo Santagada
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> Leonardo Santagada
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Leonardo Santagada
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Leonardo Santagada
>>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Leonardo Santagada
>>>
>>
>
>
> --
>
> Leonardo Santagada
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180126/474e803b/attachment-0001.html>
More information about the llvm-dev
mailing list