<div dir="ltr">I have no objections to any of this FWIW :)<div><br></div><div>-eric</div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Mar 29, 2016 at 6:46 PM Duncan P. N. Exon Smith <<a href="mailto:dexonsmith@apple.com">dexonsmith@apple.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
> On 2016-Mar-22, at 20:11, Eric Christopher <<a href="mailto:echristo@gmail.com" target="_blank">echristo@gmail.com</a>> wrote:<br>
><br>
>> On Tue, Mar 22, 2016 at 8:04 PM David Blaikie <<a href="mailto:dblaikie@gmail.com" target="_blank">dblaikie@gmail.com</a>> wrote:<br>
>> +pcc, who had some other ideas/patch out for improving memory usage of debug info<br>
>> +Reid, who's responsible for the windows/CodeView/PDB debug info which is motivating some of the ideas about changes to type emission<br>
><br>
> So I discussed this with Adrian and Mehdi at the social last Thursday and I'm getting set to finish the write up. I think it'll have some bearing on this proposal as I think it'll change how we want to take a look at the format of DISubprogram metadata a bit more.<br>
<br>
(The interesting bit here is to make a clearer split between<br>
DISubprogram declarations (part of the type hierarchY) and<br>
DISubprogram definitions (part of the code/line table/variable<br>
locations). I think that'll end up being mostly orthogonal to what<br>
I'm trying to do.)<br>
<br>
> That said, most of it is orthogonal to the changes Duncan is talking about here. Just puts the pressure on to get the other proposal written up.<br>
<br>
Which is now here:<br>
<a href="http://lists.llvm.org/pipermail/llvm-dev/2016-March/097773.html" rel="noreferrer" target="_blank">http://lists.llvm.org/pipermail/llvm-dev/2016-March/097773.html</a><br>
<br>
>> Baking into the IR more about types as units has pretty direct overlap with Reid/CodeView/etc - so, yeah, that'll takes ome discussion (but, as you say, it's not in your immediate plan anyway, so we can come back to that - but would be good for whoever gets there first to discuss it with the others)<br>
<br>
After thinking for a few days, I don't think this will bake anything<br>
new into the IR. If anything it removes a few special cases.<br>
<br>
More at the bottom.<br>
<br>
>>> Motivation<br>
>>> ==========<br>
>>><br>
>>> Based on some analysis Mehdi ran (ping him for details), there are three<br>
>>> (related) compile-time bottlenecks we're seeing with `-flto=thin -g`:<br>
>>><br>
>>> a) Reading the large number of Metadata bitcode records in the global<br>
>>> metadata block. I'm talking about raw `BitStreamer` calls here.<br>
>>><br>
>>> b) Creating unnecessary `DI*` instances (that aren't relevant to code).<br>
>>><br>
>>> c) Emitting unnecessary `DI*` instances (that aren't relevant to code).<br>
>>><br>
>>> Here is my recollection of some peak memory stats on a small testcase<br>
>>> during thin-LTO, which should be a decent indicator of (b):<br>
>>><br>
>>> - ~150MB: DILocation<br>
>>> - ~100MB: DISubprogram<br>
>>> - ~70MB: DILocalVariable<br>
>>> - ~50MB: (cumulative) DIType descendents<br>
>>><br>
>>> It looks, suprisingly, like types are not the primary bottleneck.<br>
<br>
(Probably wrong for (a), BTW. Caveats matter.)<br>
<br>
>>> There are caveats:<br>
>>><br>
>>> - `DISubprogram` declarations -- member function descriptors -- are<br>
>>> part of the type hierarchy.<br>
>>> - Most of the type hierarchy gets uniqued at parse time.<br>
>>> - As a result, these data are a poor indicator for (a).<br>
<br>
((a) is the main bottleneck for compile-time of -flto=thin (since it's<br>
quadratic in the number of files). (b) only affects memory. Also<br>
important, but at least it scales linearly.)<br>
<br>
>>> Even so, non-types are substantial.<br>
>>><br>
>>> Proposal<br>
>>> ========<br>
>>><br>
>>> Short version<br>
>>> -------------<br>
>>><br>
>>> 4. Remove `DICompositeType`s from `retainedTypes:`, similar to (2).<br>
<br>
This is the part that's relevant to the new RFC Eric just posted.<br>
<br>
>>> Long version<br>
>>> -------------<br>
>>><br>
>>> 4. Implement my proposal to remove the `DICompositeType` name map from<br>
>>> `retainedTypes:`.<br>
>>><br>
>>> <a href="http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20160125/327936.html" rel="noreferrer" target="_blank">http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20160125/327936.html</a><br>
>>><br>
>>> Similar to (2) above, this will naturally filter the types that get<br>
>>> linked in to the ones actually used by the code being linked.<br>
>>><br>
>>> It should also allow the reader to skip records for types that have<br>
>>> already been loaded in the main module.<br>
<br>
The essential things I want to accomplish with this part:<br>
<br>
- Make `type:` operands less special: instead of referencing types<br>
indirectly through MDString, point directly at the type node.<br>
<br>
- Stop using `retainedTypes:` by default (only for -gfull, etc.).<br>
<br>
- Avoid blowing up memory in -flto=full (which converting MDString<br>
references back to pointers would do naively, through<br>
re-introducing cycles). Note that this needs to be handled<br>
somehow at BitcodeReader time.<br>
<br>
After chatting with Eric, I don't think this conflicts at all with the<br>
other RFC. Unifying the `type:` operands might actually help both.<br>
<br>
One good point David mentioned last week was that we don't want to<br>
teach the IR any more about types. Rather than inventing some new<br>
context (as I suggested originally), I figure LTO plugins can just<br>
pass a (StringRef -> DIType*) map to the BitcodeReader, and the Module<br>
itself doesn't need to know anything about it.</blockquote></div>