<div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr">On Fri, Nov 18, 2016 at 8:43 AM Greg Clayton <<a href="mailto:gclayton@apple.com">gclayton@apple.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br class="gmail_msg">
> On Nov 17, 2016, at 5:40 PM, Robinson, Paul <<a href="mailto:paul.robinson@sony.com" class="gmail_msg" target="_blank">paul.robinson@sony.com</a>> wrote:<br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
>> -----Original Message-----<br class="gmail_msg">
>> From: Greg Clayton [mailto:<a href="mailto:gclayton@apple.com" class="gmail_msg" target="_blank">gclayton@apple.com</a>]<br class="gmail_msg">
>> Sent: Thursday, November 17, 2016 5:01 PM<br class="gmail_msg">
>> To: David Blaikie<br class="gmail_msg">
>> Cc: <a href="mailto:llvm-dev@lists.llvm.org" class="gmail_msg" target="_blank">llvm-dev@lists.llvm.org</a>; Robinson, Paul; Eric Christopher; Adrian<br class="gmail_msg">
>> Prantl<br class="gmail_msg">
>> Subject: Re: [llvm-dev] DWARF Generator<br class="gmail_msg">
>><br class="gmail_msg">
>><br class="gmail_msg">
>>> On Nov 17, 2016, at 3:40 PM, David Blaikie <<a href="mailto:dblaikie@gmail.com" class="gmail_msg" target="_blank">dblaikie@gmail.com</a>> wrote:<br class="gmail_msg">
>>><br class="gmail_msg">
>>><br class="gmail_msg">
>>><br class="gmail_msg">
>>> On Thu, Nov 17, 2016 at 3:12 PM Greg Clayton via llvm-dev <llvm-<br class="gmail_msg">
>> <a href="mailto:dev@lists.llvm.org" class="gmail_msg" target="_blank">dev@lists.llvm.org</a>> wrote:<br class="gmail_msg">
>>> I have recently been modifying the DWARF parser and have more patches<br class="gmail_msg">
>> planned and I want to be able to add unit tests that test the internal<br class="gmail_msg">
>> llvm DWARF APIs to ensure they continue to work and also validate the<br class="gmail_msg">
>> changes that I am making. There are not many DWARF unit tests other than<br class="gmail_msg">
>> very simple ones that test DWARF forms currently. I would like to expand<br class="gmail_msg">
>> this to include many more tests.<br class="gmail_msg">
>>><br class="gmail_msg">
>>> I had submitted a patch that I aborted as it was too large. One of the<br class="gmail_msg">
>> issues with the patch was a stand alone DWARF generator that can turn a<br class="gmail_msg">
>> few API calls into the section data required for the DWARFContextInMemory<br class="gmail_msg">
>> class to be able to load DWARF from. The idea is to generate a small blurb<br class="gmail_msg">
>> of DWARF, parse it using our built in DWARF parser and validate that the<br class="gmail_msg">
>> API calls we do when consuming the DWARF match what we expect. The<br class="gmail_msg">
>> original stand along DWARF generator class is in<br class="gmail_msg">
>> unittests/DebugInfo/DWARF/DWARFGenerator2.{h,cpp} in the patch attached.<br class="gmail_msg">
>> The original review suggested that I try to use the AsmPrinter and many of<br class="gmail_msg">
>> its associated classes to generate the DWARF. I attempted to do so and the<br class="gmail_msg">
>> AsmPrinter version is in lib/CodeGen/DwarfGenerator.{h,cpp} in the patch<br class="gmail_msg">
>> attached. This AsmPrinter based code steals code from the DwarfLinker.cpp.<br class="gmail_msg">
>>><br class="gmail_msg">
>>><br class="gmail_msg">
>>><br class="gmail_msg">
>>> I am having trouble getting things to work with the AsmPrinter. I was<br class="gmail_msg">
>> able to get simple DWARF to be emitted with the AsmPrinter version of the<br class="gmail_msg">
>> DWARF generator with code like:<br class="gmail_msg">
>>><br class="gmail_msg">
>>><br class="gmail_msg">
>>> initLLVM();<br class="gmail_msg">
>>> DwarfGen DG;<br class="gmail_msg">
>>> Triple Triple("x86_64--");<br class="gmail_msg">
>>> StringRef Path("/tmp/test.elf");<br class="gmail_msg">
>>> bool DwarfInitSuccess = DG.init(Triple, Path);<br class="gmail_msg">
>>> EXPECT_TRUE(DwarfInitSuccess);<br class="gmail_msg">
>>> uint16_t Version = 4;<br class="gmail_msg">
>>> uint8_t AddrSize = 8;<br class="gmail_msg">
>>> DwarfGenCU &CU = DG.appendCompileUnit(Version, AddrSize);<br class="gmail_msg">
>>> DwarfGenDIE CUDie = CU.getUnitDIE();<br class="gmail_msg">
>>><br class="gmail_msg">
>>> CUDie.addAttribute(DW_AT_name, DW_FORM_strp, "/tmp/main.c");<br class="gmail_msg">
>>> CUDie.addAttribute(DW_AT_language, DW_FORM_data2, DW_LANG_C);<br class="gmail_msg">
>>><br class="gmail_msg">
>>> DwarfGenDIE SubprogramDie = CUDie.addChild(DW_TAG_subprogram);<br class="gmail_msg">
>>> SubprogramDie.addAttribute(DW_AT_name, DW_FORM_strp, "main");<br class="gmail_msg">
>>> SubprogramDie.addAttribute(DW_AT_low_pc, DW_FORM_addr, 0x1000U);<br class="gmail_msg">
>>> SubprogramDie.addAttribute(DW_AT_high_pc, DW_FORM_addr, 0x2000U);<br class="gmail_msg">
>>><br class="gmail_msg">
>>> DwarfGenDIE IntDie = CUDie.addChild(DW_TAG_base_type);<br class="gmail_msg">
>>> IntDie.addAttribute(DW_AT_name, DW_FORM_strp, "int");<br class="gmail_msg">
>>> IntDie.addAttribute(DW_AT_encoding, DW_FORM_data1, DW_ATE_signed);<br class="gmail_msg">
>>> IntDie.addAttribute(DW_AT_byte_size, DW_FORM_data1, 4);<br class="gmail_msg">
>>><br class="gmail_msg">
>>> DwarfGenDIE ArgcDie =<br class="gmail_msg">
>> SubprogramDie.addChild(DW_TAG_formal_parameter);<br class="gmail_msg">
>>> ArgcDie.addAttribute(DW_AT_name, DW_FORM_strp, "argc");<br class="gmail_msg">
>>> //ArgcDie.addAttribute(DW_AT_type, DW_FORM_ref_addr, IntDie); //<br class="gmail_msg">
>> Crashes here...<br class="gmail_msg">
>>><br class="gmail_msg">
>>> DG.generate();<br class="gmail_msg">
>>><br class="gmail_msg">
>>> auto Obj = object::ObjectFile::createObjectFile(Path);<br class="gmail_msg">
>>> if (Obj) {<br class="gmail_msg">
>>> DWARFContextInMemory DwarfContext(*Obj.get().getBinary());<br class="gmail_msg">
>>> uint32_t NumCUs = DwarfContext.getNumCompileUnits();<br class="gmail_msg">
>>> for (uint32_t i=0; i<NumCUs; ++i) {<br class="gmail_msg">
>>> DWARFCompileUnit *U = DwarfContext.getCompileUnitAtIndex(i);<br class="gmail_msg">
>>> if (U)<br class="gmail_msg">
>>> U->getUnitDIE(false)->dump(llvm::outs(), U, -1u);<br class="gmail_msg">
>>> }<br class="gmail_msg">
>>> }<br class="gmail_msg">
>>><br class="gmail_msg">
>>><br class="gmail_msg">
>>> But things fall down if I try to uncomment the DW_FORM_ref_addr line<br class="gmail_msg">
>> above. The problem is that AsmPrinter really expects a full stack of stuff<br class="gmail_msg">
>> to be there and expects people to use the DwarfDebug class and all of its<br class="gmail_msg">
>> associated classes. These associated classes really want to use the "DI"<br class="gmail_msg">
>> objects (DICompileUnit, etc) so to create a compile unit we would need to<br class="gmail_msg">
>> create DICompileUnit object and then make a AsmPrinter/DwarfCompileUnit.<br class="gmail_msg">
>> That stack is pretty heavy and requires the code shown above to create<br class="gmail_msg">
>> many many classes just to represent the simple output we wish to emit.<br class="gmail_msg">
>> Another downside of the AsmPrinter method is we don't know which targets<br class="gmail_msg">
>> people are going to build into their binaries and thus we don't know which<br class="gmail_msg">
>> triples we will be able to use when generating DWARF info. Adrian Prantl<br class="gmail_msg">
>> attempted to help me get things working over here and we kept running into<br class="gmail_msg">
>> roadblocks.<br class="gmail_msg">
>>><br class="gmail_msg">
>>> It'd be great to have more detail about the roadblocks you hit to better<br class="gmail_msg">
>> understand how bad/what the issues are.<br class="gmail_msg">
>><br class="gmail_msg">
>> A few blocks:<br class="gmail_msg">
>><br class="gmail_msg">
>> - DIEString doesn't support DW_FORM_string. DW_FORM_string support might<br class="gmail_msg">
>> have been pulled so that we never emit it from clang, but we would want to<br class="gmail_msg">
>> have a unit test that covers being able to read an inlined C string from a<br class="gmail_msg">
>> DIE. Support won't be that hard to add, but we might not want it so that<br class="gmail_msg">
>> people can't use it by accident and make less efficient DWARF.<br class="gmail_msg">
><br class="gmail_msg">
> Seems to me we originally supported only DW_FORM_string, and then at some<br class="gmail_msg">
> point it was tossed in favor of DW_FORM_strp in order to get space savings<br class="gmail_msg">
> from string pooling. In fact using DW_FORM_string for small strings would<br class="gmail_msg">
> save some more space (admittedly not much) and a bunch of relocations.<br class="gmail_msg">
> (I found data from an old experiment, in a debug build of Clang it saved<br class="gmail_msg">
> ~0.7MB out of a total 340MB of debug-info size, and >360K ELF relocations.)<br class="gmail_msg">
<br class="gmail_msg">
This is true, but it also adversely affects DWARF parsing speed as you will need to manually skip each C string when parsing the DIEs.<br class="gmail_msg">
<br class="gmail_msg">
><br class="gmail_msg">
> I'd favor an API that passed the string down and let the DIE generator<br class="gmail_msg">
> (as opposed to the DWARF generator) pick the form.<br class="gmail_msg">
<br class="gmail_msg">
I have currently added a DIEInlinedString class that can be used for DW_FORM_string attributes.<br class="gmail_msg">
<br class="gmail_msg">
><br class="gmail_msg">
>> - Asserts, asserts, asserts. As we tried to emit DWARF, we got an asserts<br class="gmail_msg">
>> in bool AsmPrinter::doInitialization(Module &M). On the first line:<br class="gmail_msg">
>><br class="gmail_msg">
>> MMI = getAnalysisIfAvailable<MachineModuleInfo>();<br class="gmail_msg">
>><br class="gmail_msg">
>> This asserts if you use the AsmPrinter the way the DwarfLinker and the<br class="gmail_msg">
>> AsmPrinter based DwarfGen does if you call this. You must call this to<br class="gmail_msg">
>> generate the DebugDwarf. If you get past this by installing a Pass then we<br class="gmail_msg">
>> assert at:<br class="gmail_msg">
>><br class="gmail_msg">
>> GCModuleInfo *MI = getAnalysisIfAvailable<GCModuleInfo>();<br class="gmail_msg">
>> assert(MI && "AsmPrinter didn't require GCModuleInfo?");<br class="gmail_msg">
>><br class="gmail_msg">
>> If we don't have this, we don't get a DwarfDebug.<br class="gmail_msg">
>><br class="gmail_msg">
>>><br class="gmail_msg">
>>> Even if we end up adding another set of code to generate DWARF (which<br class="gmail_msg">
>> I'd really like to avoid) we'd want to, at some point, coalesce them back<br class="gmail_msg">
>> together. Given the goal is to try to coalesce the DWARF parsing code in<br class="gmail_msg">
>> LLDB and LLVM, it'd seem unfortunate if that effort just created another<br class="gmail_msg">
>> similar (or larger) amount of work for DWARF generation.<br class="gmail_msg">
>><br class="gmail_msg">
>> This DWARF generator could just live in the unittests/DebugInfo/DWARF<br class="gmail_msg">
>> directory so it wouldn't pollute anything in LLVM it we do choose to use<br class="gmail_msg">
>> it.<br class="gmail_msg">
>>><br class="gmail_msg">
>>> I wanted to pass this patch along in case someone wants to take a look<br class="gmail_msg">
>> at how we can possibly fix the lib/CodeGen/DwarfGenerator.cpp and<br class="gmail_msg">
>> lib/CodeGen/DwarfGenerator.h. The code that sets up all the required<br class="gmail_msg">
>> classes for the AsmPrinter method is in the DwarfGen class from<br class="gmail_msg">
>> lib/CodeGen/DwarfGenerator.cpp in the following function:<br class="gmail_msg">
>>><br class="gmail_msg">
>>> bool DwarfGen::init(Triple TheTriple, StringRef OutputFilename);<br class="gmail_msg">
>>><br class="gmail_msg">
>>> The code in this function was looted from existing DwarfLinker.cpp code.<br class="gmail_msg">
>> This functions requires a valid triple and that triple is used to create a<br class="gmail_msg">
>> lot of the classes required to make the AsmPrinter. I am not sure if any<br class="gmail_msg">
>> other code uses the AsmPrinter like this besides the DwarfLinker.cpp code<br class="gmail_msg">
>> and that code uses its own magic to actually link the DWARF. It does reuse<br class="gmail_msg">
>> some of the functions as I did, but the DwarfLinker doesn't use any of the<br class="gmail_msg">
>> DwarfDebug, DwarfCompileUnit or any of the classes that the<br class="gmail_msg">
>> compiler/assembler uses when making DWARF.<br class="gmail_msg">
>>><br class="gmail_msg">
>>> What's the DwarfLinker code missing that you need? If that code is<br class="gmail_msg">
>> generating essentially arbitrary DWARF, what's blocking using the same<br class="gmail_msg">
>> technique for generating DWARF for parsing tests?<br class="gmail_msg">
>><br class="gmail_msg">
>> They don't use any of the DwarfDebug, DwarfCompileUnit classes. They also<br class="gmail_msg">
>> don't use any of the DI classes when making up the debug info. So both the<br class="gmail_msg">
>> DWARF linker and the generator have similar needs: make DWARF that isn't<br class="gmail_msg">
>> tied too closely to the clang internal classes and DI classes.<br class="gmail_msg">
>>><br class="gmail_msg">
>>> The amount of work required for refactoring the AsmPrinter exceeds the<br class="gmail_msg">
>> time I am going to have, but I would still like to have DWARF API testing<br class="gmail_msg">
>> in the unit tests.<br class="gmail_msg">
>>><br class="gmail_msg">
>>> So my question is if anyone would have objections to using the stand<br class="gmail_msg">
>> along DWARF generator in unittests/DebugInfo/DWARF until we can later get<br class="gmail_msg">
>> the YAML tools to be able to produce DWARF and we can switch to testing<br class="gmail_msg">
>> the DWARF data that way? Chris Bieneman has expressed interest in getting<br class="gmail_msg">
>> a DWARF/YAML layer going.<br class="gmail_msg">
>>><br class="gmail_msg">
>>> Those tools would still want to use pretty similar (conceptually)<br class="gmail_msg">
>> abstractions to LLVM's codegen and llvm-dsymutil. I'd still strongly<br class="gmail_msg">
>> prefer to generalize/keep common APIs here - or better understand why it's<br class="gmail_msg">
>> not practical now (& what it will take/how we make sure we have a plan and<br class="gmail_msg">
>> resources to get there eventually).<br class="gmail_msg">
>>><br class="gmail_msg">
>>> My reasoning is:<br class="gmail_msg">
>>> - I want to be able to test DWARF APIs we have to ensure they work<br class="gmail_msg">
>> correctly as there are no Dwarf API tests right now. I will be adding code<br class="gmail_msg">
>> that changes many things in the DWARF parser and it will be essential to<br class="gmail_msg">
>> verify that there are no regressions in the DWARF APIs.<br class="gmail_msg">
>>> - Not sure which targets would be built into LLVM so it might be hard to<br class="gmail_msg">
>> write tests that cover 32/64 bit addresses and all the variants if we have<br class="gmail_msg">
>> to do things legally via AsmPrinter and valid targets<br class="gmail_msg">
>>><br class="gmail_msg">
>>> Seems like it might be plausible to refactor out whatever features of<br class="gmail_msg">
>> the AsmPrinter these APIs require (so we just harvest that data out of<br class="gmail_msg">
>> AsmPrinter and pass it down in a struct, say - so that other users can<br class="gmail_msg">
>> pass their own struct without needing an AsmPrinter). Though, again,<br class="gmail_msg">
>> interested to know how dsymutil is working in these situations.<br class="gmail_msg">
>><br class="gmail_msg">
>> I can try that method if indeed the only places that use the DwarfDebug<br class="gmail_msg">
>> are the DW_FORM_ref_addr and location lists. I'll let you know how that<br class="gmail_msg">
>> goes.<br class="gmail_msg">
>>><br class="gmail_msg">
>>> - Not enough time to modify AsmPrinter to not require the full DebugInfo<br class="gmail_msg">
>> stack and the classes that it uses (llvm::DwarfCompileUnit which must use<br class="gmail_msg">
>> llvm::DICompileUnit, llvm::DIE class which uses many local classes that<br class="gmail_msg">
>> all depend on the full DwarfDebug stack).<br class="gmail_msg">
>>><br class="gmail_msg">
>>> Will you have time at some later date to come back and revisit this?<br class="gmail_msg">
>> It's understandable that we may choose to incur short term technical debt<br class="gmail_msg">
>> with an understanding that it will be paid off in some timely manner. It'd<br class="gmail_msg">
>> be less desirable if there's no such plan/possibility and we incur a<br class="gmail_msg">
>> fairly clear case of technical debt (redundant DWARF generation libraries<br class="gmail_msg">
>> - especially when this effort is to remove a redundant DWARF parser).<br class="gmail_msg">
>><br class="gmail_msg">
>> Not sure anyone else will need to generate DWARF manually. The two clients<br class="gmail_msg">
>> currently are the DWARF unittests and the DwarfLinker. The DwarfLinker<br class="gmail_msg">
>> worked around these issues. If the AsmPrinter wasn't such an integral part<br class="gmail_msg">
>> of the entire compiler stack, I could take a stab at refactoring it, but I<br class="gmail_msg">
>> don't believe I am the right person to do this at this point as I have no<br class="gmail_msg">
>> experience or knowledge of the various ways that this class is used, or<br class="gmail_msg">
>> how it interacts with other support classes (DwarfDebug, and many many<br class="gmail_msg">
>> other classes).<br class="gmail_msg">
>><br class="gmail_msg">
>> Things that still worry me:<br class="gmail_msg">
>> - not being able to generate DWARF for 32/64 if targets are missing<br class="gmail_msg">
><br class="gmail_msg">
> You mean DWARF-32 and DWARF-64 formats? LLVM doesn't do DWARF-64.<br class="gmail_msg">
> If you mean 64-bit target-machine addresses, I guess I don't understand<br class="gmail_msg">
> the problem. If you have target-dependent tests, then they only work<br class="gmail_msg">
> when the right targets are there. This is extremely common and I'm<br class="gmail_msg">
> not clear why it would be a problem for the DWARF tests.<br class="gmail_msg">
<br class="gmail_msg">
I wasn't aware that there were target-dependent tests. Do you know of one in the unittest directory you can point me to? I did mean 32 bit address target, versus 64 bit address targets. I am not sure how I can test 4 and 8 byte addresses reliably. What triple to I use in the unittest? I can't assume x86_64 as we may have been built on a 32 bit ARM system with only the 32 bit ARM targets.<br class="gmail_msg">
><br class="gmail_msg">
>> - DIEString not supporting DW_FORM_string. I can add support, but I don't<br class="gmail_msg">
>> know if we want it as if we add it people might start using it.<br class="gmail_msg">
><br class="gmail_msg">
> See above. If the API picked the form this would not be a concern.<br class="gmail_msg">
<br class="gmail_msg">
For DWARF parsing speed I still like the DW_FORM_strp.<br class="gmail_msg"></blockquote><div><br></div><div>FWIW I'm with Greg here. I don't find that the "inlined small strings" optimization is really worth it for size, but could be convinced to add it.</div><div><br></div><div>-eric</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br class="gmail_msg">
><br class="gmail_msg">
>> - hacking around asserts by constructing classes and copying code from<br class="gmail_msg">
>> places that properly use the AsmPrinter that way it is supposed to be used<br class="gmail_msg">
>> so that we can use it in a way that it wasn't designed to be used.<br class="gmail_msg">
>><br class="gmail_msg">
>>><br class="gmail_msg">
>>> I made a large effort to try and get things working with the AsmPrinter,<br class="gmail_msg">
>> so I wanted everyone to know that I tried to get that solution working.<br class="gmail_msg">
>> Let me know what you anyone thinks.<br class="gmail_msg">
>>><br class="gmail_msg">
>>> Greg Clayton<br class="gmail_msg">
><br class="gmail_msg">
> --paulr<br class="gmail_msg">
><br class="gmail_msg">
>>><br class="gmail_msg">
>>> _______________________________________________<br class="gmail_msg">
>>> LLVM Developers mailing list<br class="gmail_msg">
>>> <a href="mailto:llvm-dev@lists.llvm.org" class="gmail_msg" target="_blank">llvm-dev@lists.llvm.org</a><br class="gmail_msg">
>>> <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br class="gmail_msg">
<br class="gmail_msg">
</blockquote></div></div>