<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi,<div class=""><br class=""></div><div class="">I tend to agree with Eric, but since I'm too busy to compute data or to sign-up for doing the work at that time, I won't weight strongly on this.<div class=""><br class=""></div><div class="">Especially Eric's point b) is worrying to me: unless the work to "correctly" design is unbearable, having work performed on CodeView that would make things harder to do for Dwarf later is a red flag IMO.</div><div class=""><div class=""><br class=""><div class="">I'd like to answer especially:</div><div class=""> "I don't think it's an imminent priority for anyone to do this for DWARF, so I worry that if we start building infrastructure for it, it will end up over engineered."</div><div class=""><br class=""></div><div class="">LTO is impacted a lot by debug info memory size and CPU time. ThinLTO is impacted even on a larger scale. So it should be an "imminent priority" to address that, and we (well Adrian and Duncan) worked a lot on the "non-type" part of it recently, improving the current state significantly. However debug info is still a bottleneck and we aim at doing better. The plan about moving type emission in the front-end is definitively appealing to me because of that.</div><div class=""><br class=""></div><div class="">Another aspect is that when building LLVM, we are linking 56 separate binaries (I'm not talking about archive) from largely overlapping sets of object files. </div><div class="">It means that any work that is performed during LTO/Codegen on these files but could be moved during the compile phase is already almost a x56 speedup win (and lower peak memory during LTO).</div><div class="">Knowing that the peak memory is reached during CodeGen and that the Dwarf emission is a large part of it, this is a major candidate to be moved in the compile phase.</div><div class=""><br class=""></div><div class="">-- </div><div class="">Mehdi</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Apr 27, 2016, at 4:53 PM, Eric Christopher via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="">I don't agree in general here because of:</div><div class=""><br class=""></div><div class="">a) maintainability - there isn't a one true path through things and now is scattering more windows knowledge through debug info and lto</div><div class=""><br class=""></div><div class="">b) higher bar for implementing similar dwarf functionality - there's nothing here that makes it at any point better for our general debug info support. Incrementally updating to an intermediate step is much easier and a lower bar than needing to implement everything up to and including a format aware linker and support that through ThinLTO, the JIT, and full LTO.</div><div class=""><br class=""></div><div class="">c) if there's no reason to do this for dwarf there's no reason to do it for windows. The existing proposal was a way to get you type emission in the front end so that you'd have to do less work. Ultimately though I don't see a reason to do this if all of the platforms don't look the same.</div><div class=""><br class=""></div><div class="">d) ThinLTO/ORC won't support the debug info you have in your proposal right now without patches</div><div class=""><br class=""></div><div class="">e) You're regressing LTO linking performance hugely for windows with debug until you write the patches that enable format aware linking of code view information</div><div class=""><br class=""></div><div class="">I'm open to arguments on any of these points from anyone.</div><div class=""><br class=""></div><div class="">-eric</div></div><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_quote" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div dir="ltr" class="">On Wed, Apr 27, 2016 at 4:41 PM Reid Kleckner <<a href="mailto:rnk@google.com" class="">rnk@google.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div dir="ltr" class="">My general feeling is that this design represents a mid-point between our current metadata design, and a future design where frontends just emit type information and LTO links it in a format-aware way.<div class=""><br class=""></div><div class="">I don't think it's an imminent priority for anyone to do this for DWARF, so I worry that if we start building infrastructure for it, it will end up overengineered.</div><div class=""><br class=""></div><div class="">Also, people seem to agree that in the long term, we really need a format-aware linker, and maybe LTO should just use one. Supposedly Frédéric has patches to llvm-dsymutil to make one for DWARF, but he hasn't found the time to upstream them.</div><div class=""><br class=""></div><div class="">Together, these reasons make me feel that we should limit the short-term scope to just CodeView, and add utilities to lib/Linker for performing basic tasks like type stream merging or type extraction, possibly with forward declaration of composite types.</div><div class=""><br class=""></div><div class="">In the future, when we do this work for DWARF, we can add a new DIType* stand-in similar to what you are describing.</div><div class=""><br class=""></div><div class="">The working patch that I have for just CodeView, all types as a single blob, is up here:<span class="Apple-converted-space"> </span><a href="http://reviews.llvm.org/D19236" target="_blank" class="">http://reviews.llvm.org/D19236</a><span class="Apple-converted-space"> </span>While it doesn't deal with type blobs or LTO type merging yet, I think it shows that there is surprisingly little need to bifurcate other parts of LLVM.</div><div class=""><br class=""></div><div class="">Thoughts?</div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Tue, Mar 29, 2016 at 6:00 PM, Eric Christopher<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:echristo@gmail.com" target="_blank" class="">echristo@gmail.com</a>></span><span class="Apple-converted-space"> </span>wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div dir="ltr" class=""><div class="">Hi All,</div><div class=""><br class=""></div><div class="">This is something that's been talked about for some time and it's probably time to propose it.</div><div class=""><br class=""></div><div class="">The "We" in this document is everyone on the cc line plus me.</div><div class=""><br class=""></div><div class="">Please go ahead and take a look.</div><div class=""><br class=""></div><div class="">Thanks!</div><div class=""><br class=""></div><div class="">-eric</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Objective (and TL;DR)<br class=""></div><div class="">=================</div><div class=""><br class=""></div><div class="">Migrate debug type information generation from the backends to the front end.</div><div class=""><br class=""></div><div class="">This will enable:</div><div class="">1. Separation of concerns and maintainability: LLVM shouldn’t have to know about C preprocessor macros, Obj-C properties, or extensive details about debug information binary formats.</div><div class="">2. Performance: Skipping a serialization should speed up normal compilations.</div><div class="">3. Memory usage: The DI metadata structures are smaller than they were, but are still fairly large and pointer heavy.</div><div class=""><br class=""></div><div class="">Motivation</div><div class="">========</div><div class=""><br class=""></div><div class="">Currently, types in LLVM debug info are described by the DIType class hierarchy. This hierarchy evolved organically from a more flexible sea-of-nodes representation into what it is today - a large, only somewhat format neutral representation of debug types. Making this more format neutral will only increase the memory use - and for no reason as type information is static (or nearly so). Debug formats already have a memory efficient serialization, their own binary format so we should support a front end emitting type information with sufficient representation to allow the backend to emit debug information based on the more normal IR features: functions, scopes, variables, etc.</div><div class=""><br class=""></div><div class="">Scope/Impact</div><div class="">===========</div><div class=""><br class=""></div><div class="">This is going to involve large scale changes across both LLVM and clang. This will also affect any out-of-tree front ends, however, we expect the impact to be on the order of a large API change rather than needing massive infrastructure changes.</div><div class=""><br class=""></div><div class="">Related work</div><div class="">==========</div><div class=""><br class=""></div><div class="">This is related to the efforts to support CodeView in LLVM and clang as well as efforts to reduce overall memory consumption when compiling with debug information enabled; in particular efforts to prune LTO memory usage.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Concerns</div><div class="">========</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">We need a good story for transitioning all the debug info testcases in the backend without giving up coverage and/or readability. David believes he has a plan here.</div><div class=""><br class=""></div><div class="">Proposal</div><div class="">=======</div><div class=""><br class=""></div><div class="">Short version</div><div class="">-----------------</div><div class=""><br class=""></div><div class="">1. Split the DIBuilder API into Types (+Macros, Imports, …) and Line Table.</div><div class="">2. Split the clang CGDebugInfo API into Types and Line Table to match.</div><div class="">3. Add a LLVM DWARF emission library similar to the existing CodeView one.</div><div class="">4. Migrate the Types API into a clang internal API taking clang AST structures and use the LLVM binary emission libraries to produce type information.</div><div class="">5. Remove the old binary emission out of LLVM.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Questions/Thoughts/Elaboration</div><div class="">-------------------------------------------</div><div class=""><br class=""></div><div class="">Splitting the DIBuilder API</div><div class="">~~~~~~~~~~~~~~~~~~~~</div><div class="">Will DISubprogram be part of both?</div><div class=""> * We should split it in two: Full declarations with type and a slimmed down version with an abstract origin.</div><div class=""><br class=""></div><div class="">How will we reference types in the DWARF blob?</div><div class=""> * ODR types can be referenced by name</div><div class=""> * Non-odr types by full DWARF hash</div><div class=""> * Each type can be a pair(tuple) of identifier (DITypeRef today) and blob.</div><div class=""> * For < DWARF4 we can emit each type as a unit, but not a DWARF Type Unit and use references and module relocations for the offsets. (See below)</div><div class=""><br class=""></div><div class="">How will we handle references in DWARF2 or global relocations for non-type template parameters?</div><div class=""> * We can use a “relocation” metadata as part of the format.</div><div class=""> * Representable as a tuple that has the DIType and the offset within the DIBlob as where to write the final relocation/offset for the reference at emission time.</div><div class=""><br class=""></div><div class="">Why break up the types at all?</div><div class=""> * To enable non-debug format aware linking and type uniquing for LTO that won’t be huge in size. We break up the types so we don’t need to parse debug information to link two modules together efficiently.</div><div class=""><br class=""></div><div class="">Any other concerns there?</div><div class=""> * Debug information without type units might be slightly larger in this scheme due to parents being duplicated (declarations and abstract origin, not full parents). It may be possible to extend dsymutil/etc to merge all siblings into a common parent. Open question for better ways to solve this.</div><div class=""><br class=""></div><div class="">How should we handle DWARF5/Apple Accelerator Tables?</div><div class=""> * Thoughts:</div><div class=""> * We can parse the dwarf in the back end and generate them.</div><div class=""> * We can emit in the front end for the base case of non-LTO (with help from the backend for relocation aspects).</div><div class=""> * We can use dsymutil on LTO debug information to generate them.</div><div class=""><br class=""></div><div class="">Why isn’t this a more detailed spec?</div><div class=""> * Mostly because we’ve thought about the issues, but we can’t plan for everything during implementation.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Future work</div><div class="">----------------</div><div class=""><br class=""></div><div class="">Not contained as part of this, but an obvious future direction is that the Module linker could grow support for debug aware linking. Then we can have all of the type information for a single translation unit in a single blob and use the debug aware linking to handle merging types.</div></div></blockquote></div><br class=""></div></blockquote></div><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">LLVM Developers mailing list</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="mailto:llvm-dev@lists.llvm.org" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">llvm-dev@lists.llvm.org</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a></div></blockquote></div><br class=""></div></div></div></div></body></html>