[llvm-dev] Reducing DWARF emitter memory consumption

[David Blaikie via llvm-dev]

Will look more closely soon - but I'd really try just writing out type
units to MC as soon as they're done. It should be relatively non-intrusive
(we build type units once, there's no ambiguity about when they're done) -
for non-fission+type units it might be a bit tricky, because the type units
still need a relocation for the stmt_list* (I'm trying to find where that's
added now... I seem to have lost it), but fission+type units should produce
entirely static type units that are knowable the moment the type is being
emitted so far as I can tell (including the type hash and everything - you
can write the bytes out to the AsmStreamer, etc and forget about them
entirely except to keep the hash to know that you don't need to emit it
again.

I imagine this would provide all the memory savings we would need for much
of anything (since types are most of the debug info), and, if not, would be
a good start.

*I think we might know what the stmt_list relocation is up-front, though -
if that's the case we'd be able to be as aggressive as I described is the
case for fission

On Fri, Feb 5, 2016 at 3:17 PM, Peter Collingbourne <peter at pcc.me.uk> wrote:

> Hi all,
>
> We have profiled [1] the memory usage in LLVM when LTO'ing Chromium, and
> we've found that one of the top consumers of memory is the DWARF emitter in
> lib/CodeGen/AsmPrinter/Dwarf*. I've been reading the DWARF emitter code and
> I have a few ideas in mind for how to reduce its memory consumption. One
> idea I've had is to restructure the emitter so that (for the most part) it
> directly produces the bytes and relocations that need to go into the DWARF
> sections without going through other data structures such as DIE and
> DIEValue.
>
> I understand that the DWARF emitter needs to accommodate incomplete
> entities
> that may be completed elsewhere during tree construction (e.g. abstract
> origins
> for inlined functions, special members for types), so here's a quick
> high-level
> sketch of the data structures that I believe could support this design:
>
> struct DIEBlock {
>   SmallVector<char, 1> Data;
>   std::vector<InternalReloc> IntRelocs;
>   std::vector<ExternalReloc> ExtRelocs;
>   DIEBlock *Next;
> };
>
> // This would be used to represent things like DW_AT_type references to
> types
> struct InternalReloc {
>   size_t Offset; // offset within DIEBlock::Data
>   DIEBlock *Target; // the offset within Target is at
> Data[Offset...Offset+Size]
> };
>
> // This would be used to represent things like pointers to
> .debug_loc/.debug_str or to functions/globals
> struct ExternalReloc {
>   size_t Offset; // offset within DIEBlock::Data
>   MCSymbol *Target; // the offset within Target is at
> Data[Offset...Offset+Size]
> };
>
> struct DwarfBuilder {
>   DIEBlock *First;
>   DIEBlock *Cur;
>   DenseMap<DISubprogram *, DIEBlock *> Subprograms;
>   DenseMap<DIType *, DIEBlock *> Types;
>   DwarfBuilder() : First(new DIEBlock), Cur(First) {}
>   // builder implementation goes here...
> };
>
> Normally, the DwarfBuilder will just emit bytes to Cur->Data (with possibly
> internal or external relocations to IntRelocs/ExtRelocs), but if it ever
> needs to create a "gap" for an incomplete data structure (e.g. at the end
> of a
> subprogram or a struct type), it will create a new DIEBlock New, store it
> to
> Cur->Next, store Cur in a DenseMap associated with the subprogram/type/etc
> and store New to Cur. To fill a gap later, the DwarfBuilder can pull the
> DIEBlock out of the DenseMap and start appending there. Once the IR is
> fully
> visited, the debug info writer will walk the linked list starting at First,
> calculate a byte offset for each DIEBlock, apply any internal relocations
> and write Data using the AsmPrinter (e.g. using EmitBytes, or maybe some
> other new interface that also supports relocations and avoids copying).
>
> Does that sound reasonable? Is there anything I haven't accounted for?
>
> Thanks,
> --
> Peter
>
> [1] https://code.google.com/p/chromium/issues/detail?id=583551#c15
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160205/4d6c85f5/attachment.html>