<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 29, 2016 at 11:50 PM, Eric Christopher via cfe-dev <span dir="ltr"><<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><br><div class="gmail_quote"><span><div dir="ltr">On Tue, Mar 29, 2016 at 11:20 PM Robinson, Paul <<a href="mailto:Paul_Robinson@playstation.sony.com" target="_blank">Paul_Robinson@playstation.sony.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Skipping a serialization and doing something clever about LTO uniquing sounds awesome. I'm guessing you achieve this by extracting types out of DI metadata
and packaging them as lumps-o-DWARF that the back-end can then paste together? Reading between the lines a bit here.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"></span></p></div></div></blockquote><div><br></div></span><div>Pretty much, yes.</div><span><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Can you share data about how much "pure" types dominate the size of debug info? Or at least the current metadata scheme? (Channeling Sean Silva here: show
me the data!) Does this hold for C as well as C++?</span></p></div></div></blockquote></span><div>They're huge. It's ridiculous. Take a look at the size of the metadata and then the size of the stuff we put in there versus dwarf.</div></div></div></blockquote><div><br></div><div>Because numbers are nice to have, I modified Clang to generate every type as 'int' (patch attached - I may've screwed some things up) & then compiled llvm-tblgen's object files with -flto (I would've used all of clang, but I don't have the lto plugin setup, so I couldn't get past tblgen)<br><br>Without debug info: 77 MB of bitcode files<br>With debug info: 24 MB<br>With debug info, but no types: 46 MB<br><br>so... 59% is pure type descriptions (these are the pure ones, the same things we put in type units - I didn't even remove the injected declarations (so if you compile example programs with this - you'll find that the DW_TAG_base_type for "int" has a child for every member function declaration that's defined (even used inline functions) in this translation unit) for this particular test, at least. Clang would be a larger/more representative sample.<br></div><div><br></div><div>I confirmed that both with and without types, there were the same number (48542) of subprogram definitions and without types there were no instances of DICompositeType (both of these were confirmed with xargs/llvm-dis/grep, nothing fancy)<br><br><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div><br></div><div>And yes, it also trivially holds for C.</div><span><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Not much discussion of data objects and code objects (other than concrete subprograms), is that because they basically aren't changing? Still defined in the
metadata and still managed/emitted by the back-end?</span></p></div></div></blockquote><div><br></div></span><div>Yep. A way of looking at it is more that it is related to things in the IR and so needs IR to represent it.</div><span><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Please say something about types (which you're thinking of as a front-end thing) defined within scopes (which it looks like you're thinking of as a back-end
thing). Not seeing how to get the scoping right.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> </span></p></div></div></blockquote><div><br></div></span><div>Basic idea is non-defining declarations holding types and be the abstract origin for the concrete function? Honestly, I wish they were type unitable at the moment, but that might be something to look into. The current plan at least. This will make some debug info a little bit larger, but only for things like nested types where we need to throw an extra declaration (i.e. the same sorts of places that type units make things larger).</div><div><br></div><div>At any rate, the first thing is to get the APIs split anyhow.</div><div><br></div><div>-eric</div><div><div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Thanks!<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">--paulr<u></u><u></u></span></p>
<p class="MsoNormal"><a name="m_-3608509686125761661_m_4039372186788237136_m_7647402183533570110_m_-4897511106553300520__MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></a></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> cfe-dev [mailto:<a href="mailto:cfe-dev-bounces@lists.llvm.org" target="_blank">cfe-dev-bounces@lists.llvm.org</a>]
<b>On Behalf Of </b>Eric Christopher via cfe-dev<br>
<b>Sent:</b> Tuesday, March 29, 2016 6:01 PM<br>
<b>To:</b> Clang Dev; llvm-dev<br>
<b>Subject:</b> [cfe-dev] RFC: Up front type information generation in clang and llvm<u></u><u></u></span></p>
</div>
</div></div></div></div><div lang="EN-US" link="blue" vlink="purple"><div><div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">Hi All,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">This is something that's been talked about for some time and it's probably time to propose it.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">The "We" in this document is everyone on the cc line plus me.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Please go ahead and take a look.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks!<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">-eric<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Objective (and TL;DR)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">=================<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Migrate debug type information generation from the backends to the front end.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">This will enable:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">1. Separation of concerns and maintainability: LLVM shouldn’t have to know about C preprocessor macros, Obj-C properties, or extensive details about debug information binary formats.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2. Performance: Skipping a serialization should speed up normal compilations.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">3. Memory usage: The DI metadata structures are smaller than they were, but are still fairly large and pointer heavy.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Motivation<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">========<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Currently, types in LLVM debug info are described by the DIType class hierarchy. This hierarchy evolved organically from a more flexible sea-of-nodes representation into what it is today - a large, only somewhat format neutral representation
of debug types. Making this more format neutral will only increase the memory use - and for no reason as type information is static (or nearly so). Debug formats already have a memory efficient serialization, their own binary format so we should support a
front end emitting type information with sufficient representation to allow the backend to emit debug information based on the more normal IR features: functions, scopes, variables, etc.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Scope/Impact<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">===========<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">This is going to involve large scale changes across both LLVM and clang. This will also affect any out-of-tree front ends, however, we expect the impact to be on the order of a large API change rather than needing massive infrastructure
changes.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Related work<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">==========<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">This is related to the efforts to support CodeView in LLVM and clang as well as efforts to reduce overall memory consumption when compiling with debug information enabled; in particular efforts to prune LTO memory usage.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Concerns<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">========<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">We need a good story for transitioning all the debug info testcases in the backend without giving up coverage and/or readability. David believes he has a plan here.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Proposal<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">=======<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Short version<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">-----------------<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">1. Split the DIBuilder API into Types (+Macros, Imports, …) and Line Table.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2. Split the clang CGDebugInfo API into Types and Line Table to match.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">3. Add a LLVM DWARF emission library similar to the existing CodeView one.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">4. Migrate the Types API into a clang internal API taking clang AST structures and use the LLVM binary emission libraries to produce type information.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">5. Remove the old binary emission out of LLVM.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Questions/Thoughts/Elaboration<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">-------------------------------------------<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Splitting the DIBuilder API<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">~~~~~~~~~~~~~~~~~~~~<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Will DISubprogram be part of both?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * We should split it in two: Full declarations with type and a slimmed down version with an abstract origin.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">How will we reference types in the DWARF blob?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * ODR types can be referenced by name<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * Non-odr types by full DWARF hash<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * Each type can be a pair(tuple) of identifier (DITypeRef today) and blob.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * For < DWARF4 we can emit each type as a unit, but not a DWARF Type Unit and use references and module relocations for the offsets. (See below)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">How will we handle references in DWARF2 or global relocations for non-type template parameters?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * We can use a “relocation” metadata as part of the format.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * Representable as a tuple that has the DIType and the offset within the DIBlob as where to write the final relocation/offset for the reference at emission time.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Why break up the types at all?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * To enable non-debug format aware linking and type uniquing for LTO that won’t be huge in size. We break up the types so we don’t need to parse debug information to link two modules together efficiently.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Any other concerns there?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * Debug information without type units might be slightly larger in this scheme due to parents being duplicated (declarations and abstract origin, not full parents). It may be possible to extend dsymutil/etc to merge all siblings into
a common parent. Open question for better ways to solve this.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">How should we handle DWARF5/Apple Accelerator Tables?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * Thoughts:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * We can parse the dwarf in the back end and generate them.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * We can emit in the front end for the base case of non-LTO (with help from the backend for relocation aspects).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * We can use dsymutil on LTO debug information to generate them.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Why isn’t this a more detailed spec?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * Mostly because we’ve thought about the issues, but we can’t plan for everything during implementation.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Future work<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">----------------<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Not contained as part of this, but an obvious future direction is that the Module linker could grow support for debug aware linking. Then we can have all of the type information for a single translation unit in a single blob and use the
debug aware linking to handle merging types.<u></u><u></u></p>
</div>
</div>
</div></div></div></blockquote></div></div></div></div>
<br>_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br>
<br></blockquote></div><br></div></div>