[llvm-dev] DWARF Generator

[Greg Clayton via llvm-dev]

I have recently been modifying the DWARF parser and have more patches planned and I want to be able to add unit tests that test the internal llvm DWARF APIs to ensure they continue to work and also validate the changes that I am making. There are not many DWARF unit tests other than very simple ones that test DWARF forms currently. I would like to expand this to include many more tests.

I had submitted a patch that I aborted as it was too large. One of the issues with the patch was a stand alone DWARF generator that can turn a few API calls into the section data required for the DWARFContextInMemory class to be able to load DWARF from. The idea is to generate a small blurb of DWARF, parse it using our built in DWARF parser and validate that the API calls we do when consuming the DWARF match what we expect. The original stand along DWARF generator class is in unittests/DebugInfo/DWARF/DWARFGenerator2.{h,cpp} in the patch attached. The original review suggested that I try to use the AsmPrinter and many of its associated classes to generate the DWARF. I attempted to do so and the AsmPrinter version is in lib/CodeGen/DwarfGenerator.{h,cpp} in the patch attached. This AsmPrinter based code steals code from the DwarfLinker.cpp. 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: dwarfgen.patch
Type: application/octet-stream
Size: 90291 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20161117/a50b2c03/attachment.obj>
-------------- next part --------------


I am having trouble getting things to work with the AsmPrinter. I was able to get simple DWARF to be emitted with the AsmPrinter version of the DWARF generator with code like:


   initLLVM();
   DwarfGen DG;
   Triple Triple("x86_64--");
   StringRef Path("/tmp/test.elf");
   bool DwarfInitSuccess = DG.init(Triple, Path);
   EXPECT_TRUE(DwarfInitSuccess);
   uint16_t Version = 4;
   uint8_t AddrSize = 8;
   DwarfGenCU &CU = DG.appendCompileUnit(Version, AddrSize);
   DwarfGenDIE CUDie = CU.getUnitDIE();

   CUDie.addAttribute(DW_AT_name, DW_FORM_strp, "/tmp/main.c");
   CUDie.addAttribute(DW_AT_language, DW_FORM_data2, DW_LANG_C);

   DwarfGenDIE SubprogramDie = CUDie.addChild(DW_TAG_subprogram);
   SubprogramDie.addAttribute(DW_AT_name, DW_FORM_strp, "main");
   SubprogramDie.addAttribute(DW_AT_low_pc, DW_FORM_addr, 0x1000U);
   SubprogramDie.addAttribute(DW_AT_high_pc, DW_FORM_addr, 0x2000U);

   DwarfGenDIE IntDie = CUDie.addChild(DW_TAG_base_type);
   IntDie.addAttribute(DW_AT_name, DW_FORM_strp, "int");
   IntDie.addAttribute(DW_AT_encoding, DW_FORM_data1, DW_ATE_signed);
   IntDie.addAttribute(DW_AT_byte_size, DW_FORM_data1, 4);

   DwarfGenDIE ArgcDie = SubprogramDie.addChild(DW_TAG_formal_parameter);
   ArgcDie.addAttribute(DW_AT_name, DW_FORM_strp, "argc");
   //ArgcDie.addAttribute(DW_AT_type, DW_FORM_ref_addr, IntDie); // Crashes here...

   DG.generate();

   auto Obj = object::ObjectFile::createObjectFile(Path);
   if (Obj) {
     DWARFContextInMemory DwarfContext(*Obj.get().getBinary());
     uint32_t NumCUs = DwarfContext.getNumCompileUnits();
     for (uint32_t i=0; i<NumCUs; ++i) {
       DWARFCompileUnit *U = DwarfContext.getCompileUnitAtIndex(i);
       if (U)
         U->getUnitDIE(false)->dump(llvm::outs(), U, -1u);
     }
   }


But things fall down if I try to uncomment the DW_FORM_ref_addr line above. The problem is that AsmPrinter really expects a full stack of stuff to be there and expects people to use the DwarfDebug class and all of its associated classes. These associated classes really want to use the "DI" objects (DICompileUnit, etc) so to create a compile unit we would need to create DICompileUnit object and then make a AsmPrinter/DwarfCompileUnit. That stack is pretty heavy and requires the code shown above to create many many classes just to represent the simple output we wish to emit. Another downside of the AsmPrinter method is we don't know which targets people are going to build into their binaries and thus we don't know which triples we will be able to use when generating DWARF info. Adrian Prantl attempted to help me get things working over here and we kept running into roadblocks.

I wanted to pass this patch along in case someone wants to take a look at how we can possibly fix the lib/CodeGen/DwarfGenerator.cpp and lib/CodeGen/DwarfGenerator.h. The code that sets up all the required classes for the AsmPrinter method is in the DwarfGen class from lib/CodeGen/DwarfGenerator.cpp in the following function:

bool DwarfGen::init(Triple TheTriple, StringRef OutputFilename);

The code in this function was looted from existing DwarfLinker.cpp code. This functions requires a valid triple and that triple is used to create a lot of the classes required to make the AsmPrinter. I am not sure if any other code uses the AsmPrinter like this besides the DwarfLinker.cpp code and that code uses its own magic to actually link the DWARF. It does reuse some of the functions as I did, but the DwarfLinker doesn't use any of the DwarfDebug, DwarfCompileUnit or any of the classes that the compiler/assembler uses when making DWARF. The amount of work required for refactoring the AsmPrinter exceeds the time I am going to have, but I would still like to have DWARF API testing in the unit tests. 

So my question is if anyone would have objections to using the stand along DWARF generator in unittests/DebugInfo/DWARF until we can later get the YAML tools to be able to produce DWARF and we can switch to testing the DWARF data that way? Chris Bieneman has expressed interest in getting a DWARF/YAML layer going.

My reasoning is:
- I want to be able to test DWARF APIs we have to ensure they work correctly as there are no Dwarf API tests right now. I will be adding code that changes many things in the DWARF parser and it will be essential to verify that there are no regressions in the DWARF APIs.
- Not sure which targets would be built into LLVM so it might be hard to write tests that cover 32/64 bit addresses and all the variants if we have to do things legally via AsmPrinter and valid targets
- Not enough time to modify AsmPrinter to not require the full DebugInfo stack and the classes that it uses (llvm::DwarfCompileUnit which must use llvm::DICompileUnit, llvm::DIE class which uses many local classes that all depend on  the full DwarfDebug stack).

I made a large effort to try and get things working with the AsmPrinter, so I wanted everyone to know that I tried to get that solution working. Let me know what you anyone thinks.

Greg Clayton