<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Mar 29, 2016, at 11:35 PM, mats petersson via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" class="">cfe-dev@lists.llvm.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class="">How will this affect other languages that generate debug info - not that you should care about those, I'm just curious - my Pascal compiler does not generate clang-style AST, and does not use clang at all. I currently have code that in uses DIBuilder directly...<br class=""></div></div></div></blockquote><div><br class=""></div><div>I don’t think that the code for generating DWARF types should move into Clang, but rather in a separate library that can be shared by multiple frontends. It can even keep most of the existing DIBuilder interface (but we may need to split DIBuilder in a types vs. everything else part).</div><div><br class=""></div><div>-- adrian</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><br class="">--<br class=""></div>Mats<br class=""></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On 30 March 2016 at 04:15, Eric Christopher via cfe-dev <span dir="ltr" class=""><<a href="mailto:cfe-dev@lists.llvm.org" target="_blank" class="">cfe-dev@lists.llvm.org</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class=""><br class=""><br class=""><div class="gmail_quote"><div class=""><div class="h5"><div dir="ltr" class="">On Tue, Mar 29, 2016 at 8:11 PM Peter Collingbourne <<a href="mailto:peter@pcc.me.uk" target="_blank" class="">peter@pcc.me.uk</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote">On Tue, Mar 29, 2016 at 7:43 PM, Eric Christopher <span dir="ltr" class=""><<a href="mailto:echristo@gmail.com" target="_blank" class="">echristo@gmail.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr" class=""><br class=""><br class=""><div class="gmail_quote"><span class=""><div dir="ltr" class="">On Tue, Mar 29, 2016 at 7:31 PM Peter Collingbourne <<a href="mailto:peter@pcc.me.uk" target="_blank" class="">peter@pcc.me.uk</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote">Thanks for sharing this. Mostly seems like a reasonable plan to me. A few comments below.</div><div class="gmail_quote"><br class=""></div></div></div></blockquote><div class=""><br class=""></div></span><div class="">Thanks Peter!</div><div class=""><div class=""><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"></div><div class="gmail_quote"></div></div></div><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote">On Tue, Mar 29, 2016 at 6:00 PM, Eric Christopher via cfe-dev <span dir="ltr" class=""><<a href="mailto:cfe-dev@lists.llvm.org" target="_blank" class="">cfe-dev@lists.llvm.org</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr" class=""><div class="">Hi All,</div><div class=""><br class=""></div><div class="">This is something that's been talked about for some time and it's probably time to propose it.</div><div class=""><br class=""></div><div class="">The "We" in this document is everyone on the cc line plus me.</div><div class=""><br class=""></div><div class="">Please go ahead and take a look.</div><div class=""><br class=""></div><div class="">Thanks!</div><div class=""><br class=""></div><div class="">-eric</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Objective (and TL;DR)<br class=""></div><div class="">=================</div><div class=""><br class=""></div><div class="">Migrate debug type information generation from the backends to the front end.</div><div class=""><br class=""></div><div class="">This will enable:</div><div class="">1. Separation of concerns and maintainability: LLVM shouldn’t have to know about C preprocessor macros, Obj-C properties, or extensive details about debug information binary formats.</div><div class="">2. Performance: Skipping a serialization should speed up normal compilations.</div><div class="">3. Memory usage: The DI metadata structures are smaller than they were, but are still fairly large and pointer heavy.</div><div class=""><br class=""></div><div class="">Motivation</div><div class="">========</div><div class=""><br class=""></div><div class="">Currently, types in LLVM debug info are described by the DIType class hierarchy. This hierarchy evolved organically from a more flexible sea-of-nodes representation into what it is today - a large, only somewhat format neutral representation of debug types. Making this more format neutral will only increase the memory use - and for no reason as type information is static (or nearly so). Debug formats already have a memory efficient serialization, their own binary format so we should support a front end emitting type information with sufficient representation to allow the backend to emit debug information based on the more normal IR features: functions, scopes, variables, etc.</div><div class=""><br class=""></div><div class="">Scope/Impact</div><div class="">===========</div><div class=""><br class=""></div><div class="">This is going to involve large scale changes across both LLVM and clang. This will also affect any out-of-tree front ends, however, we expect the impact to be on the order of a large API change rather than needing massive infrastructure changes.</div><div class=""><br class=""></div><div class="">Related work</div><div class="">==========</div><div class=""><br class=""></div><div class="">This is related to the efforts to support CodeView in LLVM and clang as well as efforts to reduce overall memory consumption when compiling with debug information enabled; in particular efforts to prune LTO memory usage.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Concerns</div><div class="">========</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">We need a good story for transitioning all the debug info testcases in the backend without giving up coverage and/or readability. David believes he has a plan here.</div><div class=""><br class=""></div><div class="">Proposal</div><div class="">=======</div><div class=""><br class=""></div><div class="">Short version</div><div class="">-----------------</div><div class=""><br class=""></div><div class="">1. Split the DIBuilder API into Types (+Macros, Imports, …) and Line Table.</div><div class="">2. Split the clang CGDebugInfo API into Types and Line Table to match.</div><div class="">3. Add a LLVM DWARF emission library similar to the existing CodeView one.</div><div class="">4. Migrate the Types API into a clang internal API taking clang AST structures and use the LLVM binary emission libraries to produce type information.</div><div class="">5. Remove the old binary emission out of LLVM.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Questions/Thoughts/Elaboration</div><div class="">-------------------------------------------</div><div class=""><br class=""></div><div class="">Splitting the DIBuilder API</div><div class="">~~~~~~~~~~~~~~~~~~~~</div><div class="">Will DISubprogram be part of both?</div><div class=""> * We should split it in two: Full declarations with type and a slimmed down version with an abstract origin.</div><div class=""><br class=""></div><div class="">How will we reference types in the DWARF blob?</div><div class=""> * ODR types can be referenced by name</div><div class=""> * Non-odr types by full DWARF hash</div><div class=""> * Each type can be a pair(tuple) of identifier (DITypeRef today) and blob.</div><div class=""> * For < DWARF4 we can emit each type as a unit, but not a DWARF Type Unit and use references and module relocations for the offsets. (See below)</div><div class=""><br class=""></div><div class="">How will we handle references in DWARF2 or global relocations for non-type template parameters?</div><div class=""> * We can use a “relocation” metadata as part of the format.</div><div class=""> * Representable as a tuple that has the DIType and the offset within the DIBlob as where to write the final relocation/offset for the reference at emission time.</div><div class=""><br class=""></div><div class="">Why break up the types at all?</div><div class=""> * To enable non-debug format aware linking and type uniquing for LTO that won’t be huge in size. We break up the types so we don’t need to parse debug information to link two modules together efficiently.</div></div></blockquote><div class=""><br class=""></div></div></div></div><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class="">How do you plan to handle abbreviations? You wouldn't necessarily be able to embed them directly in the blob, as when doing LTO each compilation unit would have its own set of abbreviations. I suppose you could do something like treat them as a special sort of reference to an abbreviation table entry, or maybe pre-allocate in the frontend (but would complicate cross-frontend LTO) but curious what you have in mind.</div></div></div></div></blockquote><div class=""><br class=""></div></div></div><div class="">Thanks for reminding me, I knew I was forgetting something I'd talked about when writing all of this down. :)</div><div class=""><br class=""></div><div class="">Basically to handle abbreviations you can do them the similarly to types by creating a blob with an index/hash/etc and then reference that as part of the type tuple, e.g.:</div><div class=""><br class=""></div><div class="">$1 = { DIAbbrev: 0x1234, DIBlob: <blah> }</div><div class="">$2 = { DIType: <ID>, DIAbbrev: $1, DIBlob: <blah> }</div><div class=""><br class=""></div><div class="">and keep them uniqued during emission and remember to merge these as well during module merge time.</div></div></div></blockquote><div class=""><br class=""></div></div></div></div><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class="">Makes sense, but wouldn't you need multiple abbreviations for each DIType, in order to represent DITypes formed of multiple DIEs (e.g. enums, records)?</div><div class=""><br class=""></div><div class="">Maybe something like this would work:</div><div class=""><div class=""><br class=""></div><div class="">$1 = { DIAbbrev: 0x1234, DIBlob: DW_TAG_enumeration_type<blah> }</div><div class="">$2 = { DIAbbrev: 0x5678, DIBlob: DW_TAG_enumerator<blah> }</div></div><div class=""><div class="">$3 = { DIType: <ID>, DIAbbrev: [(0, $1), (8, $2), (16, $2)], DIBlob: <8 bytes of DW_TAG_enumeration_type attrs><8 bytes of DW_TAG_enumerator attrs><8 bytes of DW_TAG_enumerator attrs><0> }</div></div><div class=""><br class=""></div><div class="">?</div></div></div></div></blockquote><div class=""><br class=""></div></div></div><div class="">*nod* That (or something similar) will work.</div><span class="HOEnZb"><font color="#888888" class=""><div class=""><br class=""></div><div class="">-eric</div></font></span><div class=""><div class="h5"><div class=""><br class=""></div><div class=""> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""><br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr" class=""><div class="gmail_quote"><span class=""><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""><br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr" class=""><div class="">Any other concerns there?</div><div class=""> * Debug information without type units might be slightly larger in this scheme due to parents being duplicated (declarations and abstract origin, not full parents). It may be possible to extend dsymutil/etc to merge all siblings into a common parent. Open question for better ways to solve this.</div></div></blockquote><div class=""><br class=""></div></div></div></div><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class="">When we were thinking about teaching the backend to produce blobs from IR metadata we were thinking about cases where the debug info emitter would discover special member functions during IR traversal. I guess since we're moving all of that to the frontend we can just ask the frontend directly which special members are needed on the class. That solves the problem for a single translation unit. But what do you plan to do in the multiple translation unit case where two TUs declare different special members on a class? Would it be fine to just emit the two definitions and let the debugger sort it out? I guess this is the type of thing that debuggers normally deal with in the non-LTO case, so I suppose so?</div></div></div></div></blockquote><div class=""><br class=""></div></span><div class="">Pretty much. This is one area where I have... disagreements with the DWARF committee and I don't think there's anything else we can do here. TBH right now I think we'd have issues with type units and special member functions since we're using ODR-ness to unique.</div><span class=""><font color="#888888" class=""><div class=""><br class=""></div><div class="">-eric</div></font></span><span class=""><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"></blockquote></div></div></div><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr" class=""><div class="">How should we handle DWARF5/Apple Accelerator Tables?</div><div class=""> * Thoughts:</div><div class=""> * We can parse the dwarf in the back end and generate them.</div><div class=""> * We can emit in the front end for the base case of non-LTO (with help from the backend for relocation aspects).</div><div class=""> * We can use dsymutil on LTO debug information to generate them.</div><div class=""><br class=""></div><div class="">Why isn’t this a more detailed spec?</div><div class=""> * Mostly because we’ve thought about the issues, but we can’t plan for everything during implementation.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Future work</div><div class="">----------------</div><div class=""><br class=""></div><div class="">Not contained as part of this, but an obvious future direction is that the Module linker could grow support for debug aware linking. Then we can have all of the type information for a single translation unit in a single blob and use the debug aware linking to handle merging types.</div></div>
<br class=""></blockquote></div></div></div><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">_______________________________________________<br class="">
cfe-dev mailing list<br class="">
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank" class="">cfe-dev@lists.llvm.org</a><br class="">
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" target="_blank" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br class="">
<br class=""></blockquote></div></div></div><div dir="ltr" class=""><div class="gmail_extra"><br class=""><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div class=""><div dir="ltr" class="">-- <div class="">Peter</div></div></div>
</div></div></blockquote></span></div></div>
</blockquote></div></div></div><div dir="ltr" class=""><div class="gmail_extra"><br class=""><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div class=""><div dir="ltr" class="">-- <div class="">Peter</div></div></div>
</div></div></blockquote></div></div></div></div>
<br class="">_______________________________________________<br class="">
cfe-dev mailing list<br class="">
<a href="mailto:cfe-dev@lists.llvm.org" class="">cfe-dev@lists.llvm.org</a><br class="">
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" target="_blank" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br class="">
<br class=""></blockquote></div><br class=""></div>
_______________________________________________<br class="">cfe-dev mailing list<br class=""><a href="mailto:cfe-dev@lists.llvm.org" class="">cfe-dev@lists.llvm.org</a><br class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev<br class=""></div></blockquote></div><br class=""></body></html>