<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Mar 22, 2016, at 9:56 PM, David Blaikie <<a href="mailto:dblaikie@gmail.com" class="">dblaikie@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><br class=""><div class="gmail_extra"><br class=""><div class="gmail_quote">On Tue, Mar 22, 2016 at 9:48 PM, Duncan Exon Smith <span dir="ltr" class=""><<a href="mailto:dexonsmith@apple.com" target="_blank" class="">dexonsmith@apple.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto" class=""><span class=""><div class=""><br class=""></div><div class="">On Mar 22, 2016, at 8:04 PM, David Blaikie <<a href="mailto:dblaikie@gmail.com" target="_blank" class="">dblaikie@gmail.com</a>> wrote:<br class=""><br class=""></div><blockquote type="cite" class=""><div class=""><div dir="ltr" class="">+pcc, who had some other ideas/patch out for improving memory usage of debug info<br class="">+Reid, who's responsible for the windows/CodeView/PDB debug info which is motivating some of the ideas about changes to type emission<br class=""><br class="">So how does this relate, or not, to Peter's (pcc) work trying to reduce the DIE overhead during code gen? Are you folks chasing different memory bottlenecks? Are they both relevant (perhaps in different scenarios)?<br class=""></div></div></blockquote><div class=""><br class=""></div></span>I think this is orthogonal to Peter's work, although (2) may help DIE overhead, I'm not sure. The primary goal is actually CPU speedup when lazy-loading only a very small number of functions from a module (e.g, a module that gets imported into 1000 others needs to be lazy loaded 1000 times), but I'm using the memory stats as rough guidance for which metadata exists and takes time to parse. The memory is also a problem, but the peak memory scales linearly so it's not as bad. </div></blockquote><div class=""><br class=""></div><div class="">I was mostly curious why you & Peter ended up with different bottlenecks if you're both going after the same scenario. But I see you're specifically targeting /Thin/LTO whereas he's more interested in traditional LTO. So not surprising you're hitting the memory bottlenecks in modules linking, etc, but less so in CodeGen.<br class=""></div></div></div></div></div></blockquote><div><br class=""></div><div>We have the same issue using ThinLTO that exist with FullLTO during CodeGen, maybe even more difficult to scale than FullLTO (more redundant work because of all the imported debug info). However by design ThinLTO partitioning helps to handle the scalability somehow, provided that we apply a few fixes.</div><div>However the IR loading/linking itself is on another level worse than FullLTO here (the number of IR lazy load is ~quadratic in the number of modules instead of linear for FullLTO). So this something that needs to be addressed on top of what Peter does, and with higher priority considering the impact.</div><div>Ultimately it'll need some bitcode format redesign to be really efficient, but that's not gonna happen in the short term (we aiming at a couple of weeks milestone right now).</div><div><br class=""></div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class="">Did you investigate much further the straight LTO memory issues before prioritizing ThinLTO? (just curious if you got LTO to something acceptable usable for your needs, if you knew what the next biggest problem was, etc - not too important though, just curious about any institutional knowledge lying around on these different axes)</div></div></div></div></div></blockquote><div><br class=""></div><div>We'd like to have ThinLTO "good enough" to obsolete FullLTO for most use cases. So our effort are focused on ThinLTO right now.</div><div><br class=""></div><div>-- </div><div>Mehdi</div><div><br class=""></div><div><br class=""></div><div><br class=""></div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto" class=""><div class=""><div class="h5"><div class=""><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class="">Baking into the IR more about types as units has pretty direct overlap with Reid/CodeView/etc - so, yeah, that'll takes ome discussion (but, as you say, it's not in your immediate plan anyway, so we can come back to that - but would be good for whoever gets there first to discuss it with the others)</div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Tue, Mar 22, 2016 at 7:28 PM, Duncan P. N. Exon Smith <span dir="ltr" class=""><<a href="mailto:dexonsmith@apple.com" target="_blank" class="">dexonsmith@apple.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I have some ideas to allow the BitcodeReader to lazy-load debug info<br class="">
metadata, and wanted to air this on llvm-dev before getting too deep<br class="">
into the code.<br class="">
<br class="">
Motivation<br class="">
==========<br class="">
<br class="">
Based on some analysis Mehdi ran (ping him for details), there are three<br class="">
(related) compile-time bottlenecks we're seeing with `-flto=thin -g`:<br class="">
<br class="">
a) Reading the large number of Metadata bitcode records in the global<br class="">
metadata block. I'm talking about raw `BitStreamer` calls here.<br class="">
<br class="">
b) Creating unnecessary `DI*` instances (that aren't relevant to code).<br class="">
<br class="">
c) Emitting unnecessary `DI*` instances (that aren't relevant to code).<br class="">
<br class="">
Here is my recollection of some peak memory stats on a small testcase<br class="">
during thin-LTO, which should be a decent indicator of (b):<br class="">
<br class="">
- ~150MB: DILocation<br class="">
- ~100MB: DISubprogram<br class="">
- ~70MB: DILocalVariable<br class="">
- ~50MB: (cumulative) DIType descendents<br class="">
<br class="">
It looks, suprisingly, like types are not the primary bottleneck.<br class="">
<br class="">
There are caveats:<br class="">
<br class="">
- `DISubprogram` declarations -- member function descriptors -- are<br class="">
part of the type hierarchy.<br class="">
- Most of the type hierarchy gets uniqued at parse time.<br class="">
- As a result, these data are a poor indicator for (a).<br class="">
<br class="">
Even so, non-types are substantial.<br class="">
<br class="">
Related work<br class="">
============<br class="">
<br class="">
Teresa has some post-processing in-place/in-review to avoid importing<br class="">
metadata unnecessarily, but IIUC: it won't address (a) and (b), only<br class="">
(c) (maybe I'm wrong?); and it only helps -flto=thin, not other<br class="">
lazy-loaders.<br class="">
<br class="">
I heard a rumour that Eric has a grand plan to factor away the type<br class="">
hierarchy -- awesome if true -- but I think most of this is worthwhile<br class="">
regardless.<br class="">
<br class="">
Proposal<br class="">
========<br class="">
<br class="">
Short version<br class="">
-------------<br class="">
<br class="">
1. Serialize metadata in Function blocks where possible.<br class="">
2. Reverse the `DISubprogram`/`DICompileUnit` link.<br class="">
3. Create a `METADATA_SUBPROGRAM_BLOCK`.<br class="">
<br class="">
Type-related work Eric will make unnecessary if he's fast:<br class="">
<br class="">
4. Remove `DICompositeType`s from `retainedTypes:`, similar to (2).<br class="">
5. Create a `METADATA_COMPOSITE_TYPE_BLOCK`, similar to (3).<br class="">
<br class="">
Long version<br class="">
------------<br class="">
<br class="">
1. If a piece of metadata is referenced from only a single `Function`,<br class="">
serialize that metadata in the function's metadata block instead of<br class="">
the global metadata block.<br class="">
<br class="">
This addresses problems (a) and (b), primarily targeting<br class="">
`DILocation`s. It should pick up lots of other stuff, depending on<br class="">
how much inlining has happened.<br class="">
<br class="">
(I have a draft of the writer side, still working on the reader.)<br class="">
<br class="">
2. Reverse the `DISubprogram`/`DICompileUnit` link (David and I have<br class="">
talked about this in the past in barely-related threads). The<br class="">
direct effect is that subprograms that are not pointed at by any<br class="">
code (!dbg attachments or @llvm.dbg.value intrinsics) get dropped.<br class="">
<br class="">
This addresses problem (c). If a consumer is only linking/loading a<br class="">
subset of a module's functions, this naturally filters subprograms<br class="">
to the relevant ones. Also, with limited inlining (and assuming<br class="">
(1)), it addresses problems (a) and (b), too.<br class="">
<br class="">
Adrian volunteered to implement this and is apparently almost ready<br class="">
to post a patch (still working on testcase update script logic I<br class="">
believe (probably other details, don't let me oversell it)).<br class="">
<br class="">
3. Create a special `METADATA_SUBPROGRAM_BLOCK` for each `DISubprogram`<br class="">
in the global metadata block. Store the relevant `DISubprogram` and<br class="">
all of the subprogram's `DILexicalBlock`s and `DILocalVariable`s.<br class="">
The block can be lazy-loaded on an all-or-nothing basis.<br class="">
<br class="">
In combination with (2), this addresses (a) and (b) in cases that<br class="">
(1) doesn't catch. A lazy-loading module will only load the<br class="">
subprogram blocks that get referenced.<br class="">
<br class="">
(I have a basic design for this that accounts for references into<br class="">
the middle of block; I'll see what happens when I flesh it out.)<br class="">
<br class="">
I think this will solve the non-type bottlenecks.<br class="">
<br class="">
If Eric hasn't solved types by then, we can do similar things to the IR<br class="">
for the debug info type hierarchy.<br class="">
<br class="">
4. Implement my proposal to remove the `DICompositeType` name map from<br class="">
`retainedTypes:`.<br class="">
<br class="">
<a href="http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20160125/327936.html" rel="noreferrer" target="_blank" class="">http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20160125/327936.html</a><br class="">
<br class="">
Similar to (2) above, this will naturally filter the types that get<br class="">
linked in to the ones actually used by the code being linked.<br class="">
<br class="">
It should also allow the reader to skip records for types that have<br class="">
already been loaded in the main module.<br class="">
<br class="">
5. Create a special `METADATA_COMPOSITE_TYPE_BLOCK`, similar to (3) but<br class="">
for composite types and their members. This avoids the raw bitcode<br class="">
reading overhead. (This is totally undesigned at this point.)<br class="">
<br class="">
<br class="">
</blockquote></div><br class=""></div>
</div></blockquote></div></div></div></div></blockquote></div><br class=""></div></div>
</div></blockquote></div><br class=""></body></html>