<div dir="ltr">yeah, apparently .bss has a flag of unitialized data that is not being respected on the layout of the coff files (it should skip those sections) but I dunno what to do with .data as it doesn't have a size.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jan 26, 2018 at 7:23 PM, Zachary Turner <span dir="ltr"><<a href="mailto:zturner@google.com" target="_blank">zturner@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">dumpbin has some clues. I ran dumpbin /all on both object files and diffed the results.<div><br></div><div>In the good object file, Section #2 (.data) has File Pointer to Raw Data = 208, but in the bad file Section #2 (.data) has File Pointer to Raw Data = 0.</div><div>Also, Section #3 (.bss) in the good file has Size of Raw Data = 4, but in the bad file Section #3 (.bss) has Size of Raw Data = 0.<br><div><span style="color:rgb(33,33,33)"><br></span></div><div><span style="color:rgb(33,33,33)"><br></span></div></div></div><div class="HOEnZb"><div class="h5"><br><div class="gmail_quote"><div dir="ltr">On Fri, Jan 26, 2018 at 10:06 AM Zachary Turner <<a href="mailto:zturner@google.com" target="_blank">zturner@google.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Interesting. If it is generating yaml files that can't be decoded, then definitely sounds like a bug. If you can provide a reduced test case we can try to fix it, but admittedly it can often take some effort to generate a reduced test case. The best way is to use creduce. Use cl or clang-cl and write the pre-processed output to a file, then run creduce on that file with a test that basically roundtrips from obj2yaml to yaml2obj and return 1 if there's an error. Then let it run for a couple of hours (or days) and you should come back to a minimal repro.<div><br></div><div>Granted, it's understandable if you don't have the time for that :)</div><div><br></div><div><br></div><div>Also, I got rid of my local changes and re-ran the test case and I'm seeing what you see. the 2 yaml files are identical. But the 2 binary files aren't.</div><div><br></div><div><div>00000004: 83 94</div><div>00000050: 00 08</div><div>00000051: 00 02</div><div>00000074: 00 04</div><div>0000077C: 04 0F</div><div>000007A0: 0F 04</div><div>000007C5: 61 62</div><div>000007D0: 62 61</div></div><div><br></div><div>Luckily 00000004 is a pretty easy offset to identify, so we should be able to figure this out. It looks probably some header fields aren't being initialized correctly (not sure why obj2yaml isn't printing this information)</div></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Jan 26, 2018 at 9:59 AM Leonardo Santagada <<a href="mailto:santagada@gmail.com" target="_blank">santagada@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>I'm now thinking that there's a bug in either obj2yaml or yaml2obj, because if I run just those two tools on my codebase it generates yaml files that can't be decoded, will try now to not add any section to the obj file in llvm-objcopy to see if I can link with obj files that I rewrite (but without adding symbols or sections).</div><div><br></div><div>One of the bugs that do annoy me is that the timedatestamp is not carried when obj2yaml writes a file, and that the layout function on yaml2coff does generate different indexes to the sections, none that look wrong, but it seems that it leaves some padding, but I didn't have time to look to closely at why.<br></div></div><div class="gmail_extra"></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jan 26, 2018 at 6:52 PM, Zachary Turner <span dir="ltr"><<a href="mailto:zturner@google.com" target="_blank">zturner@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hmm, ok. In that case let me try again without my local changes. Maybe they are getting in the way :-/<div><div class="m_-6343459584811143114m_2491394413280699048m_-4886203408977771043h5"><br><br><div class="gmail_quote"><div dir="ltr">On Fri, Jan 26, 2018 at 9:51 AM Leonardo Santagada <<a href="mailto:santagada@gmail.com" target="_blank">santagada@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">it is identical to me... wierd.<br></div><div class="gmail_extra"></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jan 26, 2018 at 6:49 PM, Zachary Turner <span dir="ltr"><<a href="mailto:zturner@google.com" target="_blank">zturner@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">(Ignore the fact that my hashes are 8 byte in the "good" file, this is due to some local changes I've been experimenting with)</div><div class="m_-6343459584811143114m_2491394413280699048m_-4886203408977771043m_962751798852437708m_-4300639605599606656HOEnZb"><div class="m_-6343459584811143114m_2491394413280699048m_-4886203408977771043m_962751798852437708m_-4300639605599606656h5"><br><div class="gmail_quote"><div dir="ltr">On Fri, Jan 26, 2018 at 9:48 AM Zachary Turner <<a href="mailto:zturner@google.com" target="_blank">zturner@google.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I did this:<div><br></div><div>// a.cpp</div><div><div>static int x = 0;</div><div>void b(int);<br></div><div>void a(int) {<br></div><div> if (x)</div><div> b(x);</div><div>}</div><div>int main(int argc, char **argv) {<br></div><div> a(argc);</div><div> return x;</div><div>}</div></div><div><br></div><div><br></div><div>clang-cl /Z7 /c a.cpp /Foa.noghash.obj</div><div>clang-cl /Z7 /c a.cpp -mllvm -emit-codeview-ghash-section /Foa.ghash.good.obj</div><div>llvm-objcopy a.noghash.obj a.ghash.bad.obj</div><div>obj2yaml a.ghash.good.obj > a.ghash.good.yaml</div><div>obj2yaml a.ghash.bad.obj > a.ghash.bad.yaml</div><div><br></div><div>Then open these 2 yaml files up in a diff viewer. It looks like the hashes aren't getting emitted at all. For example, in the good yaml file I see this:</div><div><br></div><div><div> - Name: '.debug$H'</div><div> Characteristics: [ IMAGE_SCN_CNT_INITIALIZED_<wbr>DATA, IMAGE_SCN_MEM_DISCARDABLE, IMAGE_SCN_MEM_READ ]</div><div> Alignment: 4</div><div> SectionData: <wbr>C5C93301000001005549419E78044E<wbr>3896D45CD7009428758BE4A1E2B3E0<wbr>22BA267DEE221F5C42B17BCA182AF8<wbr>4584814A8B5E7E3FB17B397A9E3DEA<wbr>75CD5627</div><div> GlobalHashes: </div><div> Version: 0</div><div> HashAlgorithm: 1</div><div> HashValues: </div><div> - 5549419E78044E38</div><div> - 96D45CD700942875</div><div> - 8BE4A1E2B3E022BA</div><div> - 267DEE221F5C42B1</div><div> - 7BCA182AF8458481</div><div> - 4A8B5E7E3FB17B39</div><div> - 7A9E3DEA75CD5627</div></div><div> - Name: .pdata <br></div><div><br></div><div>And in the bad yaml file I see this:</div><div><div> - Name: '.debug$H'</div><div> Characteristics: [ IMAGE_SCN_CNT_INITIALIZED_<wbr>DATA, IMAGE_SCN_MEM_DISCARDABLE, IMAGE_SCN_MEM_READ ]</div><div> Alignment: 4</div><div> SectionData: C5C9330100000000</div><div> GlobalHashes: </div><div> Version: 0</div><div> HashAlgorithm: 0</div><div> - Name: .pdata</div></div><div><br></div><div>Don't focus too much on trying to figure out weird linker errors. Just get the output of obj2yaml to be identical when run under a diff utility, then everything should work fine.</div></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Jan 26, 2018 at 7:27 AM Leonardo Santagada <<a href="mailto:santagada@gmail.com" target="_blank">santagada@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>I'm so close I can almost smell it :)</div><div><br></div><div>I know how bad the code looks, I don't intend to submit this, but if you want to try it out its at: <a href="https://gist.github.com/santagada/544136b1ee143bf31653b1158ac6829e" target="_blank">https://gist.github.com/<wbr>santagada/<wbr>544136b1ee143bf31653b1158ac682<wbr>9e</a><br></div><div><br></div><div>I'm seeing: lld-link.exe: error: duplicate symbol: "<redacted_unmangled>" (<redacted>) in <internal> and in <redacted_filename>.obj, looking at the .yaml dump the symbols are all similar to this:<br></div><div><br></div><div><div><div><span style="color:rgb(0,0,0)"> - </span><span style="color:rgb(128,0,0)">Name</span><span style="color:rgb(0,0,0)">: </span><redacted></div><div><span style="color:rgb(0,0,0)"> </span><span style="color:rgb(128,0,0)">Value</span><span style="color:rgb(0,0,0)">: </span><span style="color:rgb(9,136,90)">0</span></div><div><span style="color:rgb(0,0,0)"> </span><span style="color:rgb(128,0,0)">SectionNumber</span><span style="color:rgb(0,0,0)">: </span><span style="color:rgb(9,136,90)">0</span></div><div><span style="color:rgb(0,0,0)"> </span><span style="color:rgb(128,0,0)">SimpleType</span><span style="color:rgb(0,0,0)">: </span><span style="color:rgb(0,0,255)">IMAGE_SYM_TYPE_NULL</span></div><div><span style="color:rgb(0,0,0)"> </span><span style="color:rgb(128,0,0)">ComplexType</span><span style="color:rgb(0,0,0)">: </span><span style="color:rgb(0,0,255)">IMAGE_SYM_DTYPE_FUNCTION</span></div><div><span style="color:rgb(0,0,0)"> </span><span style="color:rgb(128,0,0)">StorageClass</span><span style="color:rgb(0,0,0)">: </span><span style="color:rgb(0,0,255)">IMAGE_SYM_CLASS_WEAK_EXTERNAL</span></div><div><span style="color:rgb(0,0,0)"> </span><span style="color:rgb(128,0,0)">WeakExternal</span><span style="color:rgb(0,0,0)">: </span></div><div><span style="color:rgb(0,0,0)"> </span><span style="color:rgb(128,0,0)">TagIndex</span><span style="color:rgb(0,0,0)">: </span><span style="color:rgb(9,136,90)">134</span></div><div><span style="color:rgb(0,0,0)"> </span><span style="color:rgb(128,0,0)">Characteristics</span><span style="color:rgb(0,0,0)">: </span><span style="color:rgb(0,0,255)">IMAGE_WEAK_EXTERN_SEARCH_<wbr>LIBRARY</span></div></div></div></div><div class="gmail_extra"></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jan 25, 2018 at 8:01 PM, Zachary Turner <span dir="ltr"><<a href="mailto:zturner@google.com" target="_blank">zturner@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I haven't really dabbled in this part of the COFF format personally, so hopefully I'm not leading you astray :)<div><br></div><div>But I checked the code for coff2yaml, and I see this:</div><div><br></div><div><div> } else if (Symbol.isSectionDefinition()) {</div><div> // This symbol represents a section definition.</div><div> assert(Symbol.<wbr>getNumberOfAuxSymbols() == 1 &&</div><div> "Expected a single aux symbol to describe this section!");</div></div><div><div> const object::coff_aux_section_<wbr>definition *ObjSD =<br></div><div> reinterpret_cast<const object::coff_aux_section_<wbr>definition *>(</div><div> AuxData.data());</div></div><div><br></div><div>So it looks like you need exactly 1 aux symbol for each section symbol.</div><div><br></div><div>I then scrolled up in this function to figure out where AuxData comes from, and it comes from COFFObjectFile::<wbr>getSymbolAuxData. I think that function holds the clue to what you need to do. It looks like you need to set coff::symbol::<wbr>NumberOfAuxSymbols to 1, and then there is a comment in getSymbolAuxData which says:</div><div><br></div><div><div> // AUX data comes immediately after the symbol in COFF</div><div> Aux = reinterpret_cast<const uint8_t *>(Symbol.getRawPtr()) + SymbolSize;</div></div><div><br></div><div>So I think you just need to write the bytes immediately after the coff::symbol. The thing you need to write looks like a coff::coff_aux_section_<wbr>definition structure.</div><div><br></div><div>For the CheckSum, look at WinCOFFObjectWriter::<wbr>writeSection. It looks like its a CRC32 of the actual section contents, which you can generate with a couple of lines of code:</div><div><br></div><div><div> JamCRC JC(/*Init=*/0);</div><div> JC.update(DebugHContents);</div><div> AuxSymbol.CheckSum = JC.getCRC();</div></div><div><br></div><div>Hope this helps</div></div></blockquote></div></div></blockquote></div></blockquote></div></div></div></blockquote></div></div><div class="gmail_extra">
</div></blockquote></div></div></div></div>
</blockquote></div><br><br clear="all"><br></div><div class="gmail_extra">-- <br><div class="m_-6343459584811143114m_2491394413280699048m_-4886203408977771043gmail_signature" data-smartmail="gmail_signature"><br>Leonardo Santagada</div>
</div></blockquote></div></blockquote></div>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><br>Leonardo Santagada</div>
</div>