[lldb-dev] RFC for DWZ = DW_TAG_imported_unit + DWARF-5 supplementary files

[Greg Clayton via lldb-dev]

> On Aug 24, 2017, at 8:38 AM, Jan Kratochvil <jan.kratochvil at redhat.com> wrote:
> 
> On Wed, 23 Aug 2017 23:55:00 +0200, Greg Clayton wrote:
>>> On Aug 23, 2017, at 2:06 PM, Jan Kratochvil via lldb-dev <lldb-dev at lists.llvm.org> wrote:
>>> Currently it always has to as on non-OSX platforms it is using
>>> DWARFCompileUnit::Index(). But as I plan to implement DWARF-5 .debug_names
>>> index (like __apple_* index) maybe LLDB then no longer needs to populate
>>> m_die_array and so just expanding all DW_TAG_partial_unit into a single
>>> m_die_array for each DW_TAG_compile_unit is fine?
>> 
>> So I glossed over the documentation and I gathered that DWARF type info
>> might be stored in other DWARF files and references from the current file.
> 
> Yes. Arbitrary DIEs, not just type-defining DIEs. DW_TAG_imported_unit can
> happen anywhere according to the DWARF standard but for the DWZ/Fedora case it
> is a bit simpler one can limit the support for only DW_TAG_imported_unit with
> parent of DW_TAG_compile_unit (or DW_TAG_partial_unit for nested imports).
> DWZ ever uses DW_TAG_imported_unit only at the CU/PU top level.
> 
> 
>> SymbolFileDWARFDebugMap is an example of how we do things on MacOS. We have
>> one clang::ASTContext in the SymbolFileDWARFDebugMap, and multiple external
>> .o files (where each ins a SymbolFileDWARF instance) that contain unlinked
>> DWARF. Each SymbolFileDWARF instance will have:
>> 
>>  void SymbolFileDWARF::SetDebugMapModule(const lldb::ModuleSP &module_sp);
>> 
>> called to indicate it is actually part of the SymbolFileDWARFDebugMap.
>> Then there are functions that check the debug map file and return the
>> UniqueDWARFASTTypeMap or the TypeSystem from the SymbolFileDWARFDebugMap if
>> we have one:
> 
> Therefore IIUC you make a single context for all types from a compiled
> program?  Primitive (non-class) types can be different across CUs:
> 	CU1: typedef int  foo_t;
> 	CU2: typedef long foo_t;

We can and we do allow this. We will have two foo_t definitions in the AST context at the same level. Each variable has the correct type as it directly points to the right "foo_t". When these variables are used in expressions, we will only copy 1 of the "foo_t" definitions over into the expression AST context so things normally work out. If you tried to cast something to a "foo_t" you would run into problems though since the type would be ambiguous.
> 
> I have a problem that one DW_TAG_partial_unit can be included by multiple
> DW_TAG_compile_unit.  Therefore DWARFCompileUnit for DW_TAG_partial_unit
> itself cannot map itself by SymbolFileDWARFDebugMap to its parent
> DWARFCompileUnit for DW_TAG_compile_unit (as it has multiple parents).
> 
> I expect you cannot link a single MacOS object file into two diferent
> programs/libraries which are debugged at once by LLDB.

You can. We always open a new instance of the .o file, one for each SymbolFileDWARFDebugMap or unique executable that refers to the .o file.
> 
> 
> 
>> One other idea is to keep all DWARF files separate and stand alone. Your
>> main DWARF file with one or more DW_TAG_imported_unit and all
>> DW_TAG_imported_unit referenced files, each as its own SymbolFileDWARF. Any
>> reference to a DW_FORM_ref_alt would turn into a forward declaration in the
>> current SymbolFileDWARF, so the ASTContext in each SymbolFileDWARF wouldn't
>> know anything about the types,
> 
> Is it applicable even if DW_TAG_imported_unit points to DW_TAG_partial_unit's
> containing DW_TAG_variable, DW_TAG_subprogram and arbitrary other DIEs, not
> just types?

This approach will not work if that is the case. Scratch the above idea...
> 
> 
>> The other approach I might suggest is to write a DWARF linker, maybe using
>> LLVM's DWARF classes (see llvm-dsymutil sources) that takes the top level
>> DWARF and all DW_TAG_imported_unit files and combines them all back into one
>> large DWARF file.
> 
> That would defeat the advantage of DWZ-optimized debug info files.  DWZ
> reduces their size by approx. 30%.  If the relinking was made on-demand then
> it defeats the LLDB performance advantage over GDB - GDB can already read DWZ
> natively (GDB does duplicates the internal representation when reading DWZ
> CUs/PUs).  In such case I could already code such reconstruction of non-DWZ
> debug info when reading m_die_array - but that also seems needlessly slow to
> me.  DWZ should bring even performance advantage of parsing the DWZ common
> file (=imported file) only once.
> 
> 
>> One other idea is to let each DWARF file be separate, and when you need
>> a type from a DW_TAG_imported_unit you log that file as stand alone and copy
>> the type from its clang::ASTContext into the main SymbolFileDWARF's AST
>> context. We copy types all the time in expressions as each on has its own
>> AST context.
> 
> Does this work even for non-type DIEs? 

No it does not. Only types are fine, but anything else will fall down.
> 
> 
>> So there are many solutions. I would vote for linking the DWARF into
>> a single file much like we do with llvm-dsymutil on Mac, but that really
>> depends if the type uniquing is desired within a single DWARF file and not
>> across many shared libraries that all reference common DW_TAG_imported_unit
>> files.
> 
> I agree the DWZ optimization does primarily what -fdebug-types-section does.
> 
> I do not see why to really do relinking files on disk, debugger should not
> need that to read the debug info.
> 
> I do not fully understand why you do the llvm-dsymutil relinking when you
> already have SymbolFileDWARFDebugMap in LLDB.  But I do not know OSX/Mac.


So this is a new and unique case that will require substantial support. Not sure what the best approach is. Symbol files get to pick the IDs of their lldb_private::CompileUnit (which is the DW_TAG_compile_unit DIE offset for normal DWARF), lldb_private::Function (which is the DW_TAG_subprogram DIE offset), lldb_private::Block (which is the DW_TAG_lexical_block and DW_TAG_inlined_subroutine DIE offset), and lldb_private::Type (type DIE offset).

You might need to see what they did for the DWO support where they overload stuff in the DIERef class. See DIERef.cpp. Basically you must ensure that your ID for the things you create in the DWARF parser and hand out to LLDB can easily be used to find the original DIE. DIERef objects must be able to make unique lldb::user_id_t values for each item and given a lldb::user_id_t you must be able to get back to the compile unit, function, block, type etc very efficiently. This space is crowded now that we have normal DWARF, debug map DWARF (SymbolFileDWARFDebugMap) on MacOSX, DWO (https://gcc.gnu.org/wiki/DebugFission) support on other systems, and DWZ on your system. You will need to build in your DWZ support without affecting all of the others that are in place.

The test suite is pretty good for this. I have a patch that attempt to add -fdebug-types-support here:

https://reviews.llvm.org/D32167 <https://reviews.llvm.org/D32167>

The gist of this patch is since we know that .debug_types has no references to any other compile units or type units, we can pretend that the ID for anything in .debug_types is just sizeof(.debug_info) + .debug_types DIE offset. This patch also shows how to have the test suite add -fdebug-types-section to the LLDB test suite to ensure it is tested. Each code test in the LLDB test suite can be duplicated and run with each different kind of debug info flavor. I haven't had time to work out all of the issues in this patch, but it is close.

DWZ is trickier in that it really does want things to be shared between files, including functions, types, and so much more. If you end up trying to have each DWARF file stay separate, you are going to be copying a bunch of stuff from one file's AST context to the other. We have type copying support currently, but I am not sure of the support for variable and function AST copying.

The easiest solution would be to load the main DWARF file, all of its lldb::user_id_t value would just be the normal .debug_info offset. For any files that are mentioned via DW_TAG_imported_unit, their IDs would be the previous .debug_info size + the offset in the file:

dwarf[0] a.out (main DWARF file) lldb::user_id_t is just the actual DIE offset
dwarf[1] /tmp/shared1 (DW_TAG_imported_unit from a.out) lldb::user_id_t is just the DIE offset + sizeof(dwarf[0] .debug_info)
dwarf[2] /tmp/shared2 (DW_TAG_imported_unit from a.out) lldb::user_id_t is just the DIE offset + sizeof(dwarf[0] .debug_info) + sizeof(dwarf[1] .debug_info)

The drawback is this won't allow sharing /tmp/shared1 or /tmp/shared2 between two different top level DWARF files, but it does allow one clang::ASTContext to be used between all of them. SymbolFileDWARFDebugMap makes it lldb::user_id_t contain the CU index in the top 32 bits, and the DIE offset within that .o file's DWARF in the bottom 32 bits. You could do something similar in your case where the top 32 bits is the index of the DWARF file in the "dwarf[]" array that would be maintained in a new SymbolFileDWARFDWZ subclass.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20170824/c7de7f10/attachment.html>