[llvm-dev] [DWARF] De-segregating type units and compile units

Fri Jul 27 11:33:02 PDT 2018

Thanks for all the context!

As for the section oriented or data oriented view - a section-oriented view
could be maintained and something like STLExtras' concat(R1, R2, ...) could
be used to provide an accessor over all the units across all the sections.
Though I'm not wedded to that direction.

I think it's probably a bit awkward to try to keep them in the same
container and have offsets to remember which parts of that container have
the things from debug_types section and which parts are from debug_info
section. So if the dumping is going to continue working in that way (as
opposed to, say, changing the dumping entirely to not be section-oriented
and have an "all units" section) I'd err on the side of separate containers
at least by section name (if not by section contribution) - and a
concat-based accessor for API users who want to visit all the units
together across all sections.

As for the section abstraction V section name abstraction. Yeah, no big
deal there except some worries about error handling perhaps (the current
API being one object per section contribution, not per section name - so
each object can fail during parsing & the remaining objects remain
unmodified and valid). I'd do the renaming either before or with (2)
(because (2) is where it stops being one-per-section, instead
one-per-section-name). But no big deal either way - end result is the same.

A different name for "parse" that indicates that it's additive would be
helpful (not so much about "parse" V "parseSection" but about the fact that
this parse operation is meant to/able to be run on an already
initialized/populated object and will further populate it - whereas all the
other parse operations take an empty/default initialized object and
populate it - the object goes from an unparsed to a parsed state and is
done, no more parsing/population to happen - whereas that won't be the case
for this data structure). Maybe just "addSection" or "addAndParseSection"
("parseAndAddSection" - though I still worry having "parse" at the start
might make it appear like all the other parse operations here)

It's not that there are bugs in how intra-section offsets work, it's just
that they work/appear differently when dealing with separate sections even
though those sections have the same name. Offsets are section relative.
Looks like the offsets are maintained within the DIEs/Units so it'll
continue to dump fine even if they're all added to the same
DWARFUnitSection/thing.

I still suspect the inheritance was (I implemented the original templated
stuff - and looking at it, seems excessive to me now) & will be overkill
(in part because I'm sort of vaguely hoping that one day all units will be
the same and have a variable header of zero or more DIEs identifiable by
hash with the DIEs respective offset - at which point units generalize/look
more the same) but don't mind for now & we'll just see how it all shakes
out over time.

Long story short, I think it would be best to:

* rename 'parse' for DWARFUnitVector
* keep units from debug_types separate from units from debug_info

Other than that *thumbs up* and commit away.

- Dave

On Thu, Jul 26, 2018 at 8:06 AM <paul.robinson at sony.com> wrote:

> Here's a broader precis of my thinking in taking the library in this
> direction.
>
>
>
> Prior to v4, DWARF kept information of different natures in different
> object-file sections.  So, .debug_line had the line tables, .debug_macinfo
> had the macro information, .debug_info had the DIEs, and so on.  In some
> cases, information in a single section was subdivided according to
> compilation unit (.debug_info and .debug_line in particular) but the
> information in each section was all of the same nature.
>
>
>
> In v4, DWARF added .debug_types, which was different in two ways.  First,
> there might be multiple object-file sections with the same name (the
> COMDATs), and second, the nature of the information was much the same as in
> .debug_info.  There was a distinction, however, in that .debug_info
> contained only compile units and .debug_types contained only type units.
> There's also the aspect that the LLVM parser is oriented toward the needs
> of an object-file dumper, which encouraged thinking about the information
> in object-file terms, i.e. by section.  Hence, type units were supported by
> using a container of section-oriented classes, each element being able to
> handle multiple units.
>
>
>
> In v5, DWARF corrected the mistake of putting type units in a differently
> named object-file section.  So, now we can have one or more .debug_info
> sections, each of which can contain one or more compile and/or type units.
> But of course .debug_types still exists if there is some linked object file
> containing a mix of v4 and 5 compilations.
>
>
>
> In parallel, there is increased interest in making the LLVM and LLDB
> parsers converge, and if the decision goes with using the LLVM parser in
> LLDB, the LLVM parser needs to better accommodate the needs of a debugger
> rather than just a dumper.  In that case, the important distinction is not
> the object-file section, but the nature of the information.  That is,
> pasting together all the DIE data from .debug_info and .debug_types into a
> single manageable whole is what LLDB wants, because the source section is
> irrelevant.  And in fact, within the past year LLDB's parser was modified
> to understand .debug_types, and models it as being appended to .debug_info
> for navigation purposes.
>
>
>
> Of course the LLVM dumper still thinks about things in object-file section
> terms, so it's worth being able to look at the DIE information on a
> per-section basis.
>
>
>
> I think I have achieved that with the refactoring.  The DWARF data that is
> in the nature of DIEs is managed by one container, making it easier for
> LLDB to look at it as a unified whole.  The container can provide iterators
> that distinguish between different section names, allowing the dumper to
> retain a section-oriented view.  The functional patch that enables multiple
> .debug_info sections for DWARF v5 should be not a whole lot more than
> changing the parse methods to iterate over possibly multiple instances of
> object-file sections with that name.
>
>
>
> If there are bugs in how intra-section offsets are reported and handled,
> then it's worth knowing; what that means is that we don't have existing
> tests that demonstrate the offsets are done correctly.  Happy to improve
> testing and fix any problems found.
>
>
>
> If there is bikeshedding about the names of methods, that's fine too.
> Renaming DWARFUnitVector::parse[DWO] to parse[DWO]Section is obviously
> simple and can be folded into the same patch that renamed the class.
>
>
>
> As for merging DWARFTypeUnit and DWARFCompileUnit into DWARFUnit and
> eliminating the distinction at the class level, I haven't really looked at
> that.  There are a couple reasons to put that off.  First, Jan Kratochvil
> has been doing LLDB work in the direction of more subclasses, not fewer,
> i.e. putting partial and imported units into their own subclasses.  I
> haven't looked closely at his patches.  Second, I've had interest
> internally in getting partial units to work reasonably so we can put them
> into COMDATs and trivially deduplicate the debug info for linkonce
> functions.  If a separate subclass would help there, merging others would
> make less sense.
>
>
>
> So I'm not fundamentally opposed to merging the unit classes, but I think
> it can be deferred with little problem while we work out the other aspects.
>
>
>
> Other questions/comments always welcome.
>
> --paulr
>
>
>
> *From:* David Blaikie [mailto:dblaikie at gmail.com]
> *Sent:* Wednesday, July 25, 2018 3:36 PM
> *To:* Robinson, Paul
> *Cc:* llvm-dev at lists.llvm.org
> *Subject:* Re: [llvm-dev] [DWARF] De-segregating type units and compile
> units
>
>
>
> A few general design discussions that cross multiple patches, so figure
> it's worth discussing here:
>
> The DWARFUnitSection is currently (before your changes) representing the
> contents from a single section - and most/all of the classes in
> libDebugInfoDWARF use "parse" to initialize their complete state. So it's a
> bit novel/new/different/worth considering (maybe renaming the function or
> the like) what it means to make this class now represent data from multiple
> sections and to be documentend/intended to be used to aggregate the parsing
> result from multiple sections (both multiple sections with the same name
> (debug_types currently, or in v5 debug_info comdat sections for type
> units)). One observable part of the matter of which sections come from is
> the section relative offsets (you'll see the difference in dumping a file
> with at least 2 types with type units enabled, with and without split DWARF
> - without split DWARF comdat sections are used, so each types section
> offsets are zerod at the start of each type unit because it's a separate
> section (even though it has the same section name) - whereas in split
> DWARF, in the .dwo file the offsets are across the whole, singular,
> debug_types.dwo section)
>
> As for virtuality in the unit hierarchy - reckon it's worth it, or maybe
> just use a union/conditional for the dumping differences (& collapse/remove
> the type distinction between type units and compile units - just have
> DWARFUnit as the only unit class)? (since it's just the type_signature V
> DWO_id (maybe we could generalize that to "id") and type_offset that's
> different between them).
>
>
>
> On Tue, Jul 24, 2018 at 10:40 AM via llvm-dev <llvm-dev at lists.llvm.org>
> wrote:
>
> Hello DWARF fans,
>
> I've just posted a set of four refactoring patches for DebugInfo/DWARF,
> which move in the direction of handling DWARF v4 or v5 type units and
> compile units more coherently.
>
> In DWARF v4, type units and compile units are strictly segregated into
> the .debug_types and .debug_info sections, respectively.  This division
> was pretty ingrained into how DebugInfo/DWARF handled the units.
>
> In DWARF v5, type units and compile units are all in the .debug_info
> section, and the .debug_types section is obsolete.  So, we need to have
> a container than can handle both kinds of units in a straightforward way.
>
> The refactoring replaces a pair of collections templated on the unit type
> with a single collection whose elements are base-class pointers; thus it
> can contain elements that are a mix of unit kinds.  Everything that really
> mattered was either already virtual or was straightforward to convert to
> generic handling.  I think I needed only one type-based conditional, on
> top of what already existed.  The code doesn't *quite* support DWARF v5
> type units in the .debug_info section, but the new starting point should
> make that straightforward.
>
> In a mixed v4/v5 executable, we can pretend the .debug_types sections
> are all really .debug_info sections, and tack them onto the end of the
> vector of units that already handles a mix of compile and type units.
> This is also how the LLDB DWARF parser handles this case, so it's at
> least consistent in principle (even if the actual code differs).
>
> The existing distinction between "normal" and split (DWO) units remains;
> that has to do with what information exists in which object files, and
> is not fundamentally changing in DWARF v5.
>
> Patch 1: De-templatize DWARFUnitSection
>          https://reviews.llvm.org/D49741
> Patch 2: The TU collection doesn't need to be a deque
>          https://reviews.llvm.org/D49742
> Patch 3: Rename DWARFUnitSection to DWARFUnitVector
>          https://reviews.llvm.org/D49743
> Patch 4: Unify handling of type and compile units
>          https://reviews.llvm.org/D49744
>
> Thanks,
> --paulr
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180727/538b18d7/attachment-0001.html>