[llvm-dev] [DWARF] De-segregating type units and compile units

[via llvm-dev]

Here's a broader precis of my thinking in taking the library in this direction.

Prior to v4, DWARF kept information of different natures in different object-file sections.  So, .debug_line had the line tables, .debug_macinfo had the macro information, .debug_info had the DIEs, and so on.  In some cases, information in a single section was subdivided according to compilation unit (.debug_info and .debug_line in particular) but the information in each section was all of the same nature.

In v4, DWARF added .debug_types, which was different in two ways.  First, there might be multiple object-file sections with the same name (the COMDATs), and second, the nature of the information was much the same as in .debug_info.  There was a distinction, however, in that .debug_info contained only compile units and .debug_types contained only type units.  There's also the aspect that the LLVM parser is oriented toward the needs of an object-file dumper, which encouraged thinking about the information in object-file terms, i.e. by section.  Hence, type units were supported by using a container of section-oriented classes, each element being able to handle multiple units.

In v5, DWARF corrected the mistake of putting type units in a differently named object-file section.  So, now we can have one or more .debug_info sections, each of which can contain one or more compile and/or type units.  But of course .debug_types still exists if there is some linked object file containing a mix of v4 and 5 compilations.

In parallel, there is increased interest in making the LLVM and LLDB parsers converge, and if the decision goes with using the LLVM parser in LLDB, the LLVM parser needs to better accommodate the needs of a debugger rather than just a dumper.  In that case, the important distinction is not the object-file section, but the nature of the information.  That is, pasting together all the DIE data from .debug_info and .debug_types into a single manageable whole is what LLDB wants, because the source section is irrelevant.  And in fact, within the past year LLDB's parser was modified to understand .debug_types, and models it as being appended to .debug_info for navigation purposes.

Of course the LLVM dumper still thinks about things in object-file section terms, so it's worth being able to look at the DIE information on a per-section basis.

I think I have achieved that with the refactoring.  The DWARF data that is in the nature of DIEs is managed by one container, making it easier for LLDB to look at it as a unified whole.  The container can provide iterators that distinguish between different section names, allowing the dumper to retain a section-oriented view.  The functional patch that enables multiple .debug_info sections for DWARF v5 should be not a whole lot more than changing the parse methods to iterate over possibly multiple instances of object-file sections with that name.

If there are bugs in how intra-section offsets are reported and handled, then it's worth knowing; what that means is that we don't have existing tests that demonstrate the offsets are done correctly.  Happy to improve testing and fix any problems found.

If there is bikeshedding about the names of methods, that's fine too.  Renaming DWARFUnitVector::parse[DWO] to parse[DWO]Section is obviously simple and can be folded into the same patch that renamed the class.

As for merging DWARFTypeUnit and DWARFCompileUnit into DWARFUnit and eliminating the distinction at the class level, I haven't really looked at that.  There are a couple reasons to put that off.  First, Jan Kratochvil has been doing LLDB work in the direction of more subclasses, not fewer, i.e. putting partial and imported units into their own subclasses.  I haven't looked closely at his patches.  Second, I've had interest internally in getting partial units to work reasonably so we can put them into COMDATs and trivially deduplicate the debug info for linkonce functions.  If a separate subclass would help there, merging others would make less sense.

So I'm not fundamentally opposed to merging the unit classes, but I think it can be deferred with little problem while we work out the other aspects.

Other questions/comments always welcome.
--paulr

From: David Blaikie [mailto:dblaikie at gmail.com]
Sent: Wednesday, July 25, 2018 3:36 PM
To: Robinson, Paul
Cc: llvm-dev at lists.llvm.org
Subject: Re: [llvm-dev] [DWARF] De-segregating type units and compile units

A few general design discussions that cross multiple patches, so figure it's worth discussing here:

The DWARFUnitSection is currently (before your changes) representing the contents from a single section - and most/all of the classes in libDebugInfoDWARF use "parse" to initialize their complete state. So it's a bit novel/new/different/worth considering (maybe renaming the function or the like) what it means to make this class now represent data from multiple sections and to be documentend/intended to be used to aggregate the parsing result from multiple sections (both multiple sections with the same name (debug_types currently, or in v5 debug_info comdat sections for type units)). One observable part of the matter of which sections come from is the section relative offsets (you'll see the difference in dumping a file with at least 2 types with type units enabled, with and without split DWARF - without split DWARF comdat sections are used, so each types section offsets are zerod at the start of each type unit because it's a separate section (even though it has the same section name) - whereas in split DWARF, in the .dwo file the offsets are across the whole, singular, debug_types.dwo section)

As for virtuality in the unit hierarchy - reckon it's worth it, or maybe just use a union/conditional for the dumping differences (& collapse/remove the type distinction between type units and compile units - just have DWARFUnit as the only unit class)? (since it's just the type_signature V DWO_id (maybe we could generalize that to "id") and type_offset that's different between them).

On Tue, Jul 24, 2018 at 10:40 AM via llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
Hello DWARF fans,

I've just posted a set of four refactoring patches for DebugInfo/DWARF,
which move in the direction of handling DWARF v4 or v5 type units and
compile units more coherently.

In DWARF v4, type units and compile units are strictly segregated into
the .debug_types and .debug_info sections, respectively.  This division
was pretty ingrained into how DebugInfo/DWARF handled the units.

In DWARF v5, type units and compile units are all in the .debug_info
section, and the .debug_types section is obsolete.  So, we need to have
a container than can handle both kinds of units in a straightforward way.

The refactoring replaces a pair of collections templated on the unit type
with a single collection whose elements are base-class pointers; thus it
can contain elements that are a mix of unit kinds.  Everything that really
mattered was either already virtual or was straightforward to convert to
generic handling.  I think I needed only one type-based conditional, on
top of what already existed.  The code doesn't *quite* support DWARF v5
type units in the .debug_info section, but the new starting point should
make that straightforward.

In a mixed v4/v5 executable, we can pretend the .debug_types sections
are all really .debug_info sections, and tack them onto the end of the
vector of units that already handles a mix of compile and type units.
This is also how the LLDB DWARF parser handles this case, so it's at
least consistent in principle (even if the actual code differs).

The existing distinction between "normal" and split (DWO) units remains;
that has to do with what information exists in which object files, and
is not fundamentally changing in DWARF v5.

Patch 1: De-templatize DWARFUnitSection
         https://reviews.llvm.org/D49741
Patch 2: The TU collection doesn't need to be a deque
         https://reviews.llvm.org/D49742
Patch 3: Rename DWARFUnitSection to DWARFUnitVector
         https://reviews.llvm.org/D49743
Patch 4: Unify handling of type and compile units
         https://reviews.llvm.org/D49744

Thanks,
--paulr

_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180726/c21e49a2/attachment-0001.html>