[llvm-dev] Reducing DWARF emitter memory consumption

[Eric Christopher via llvm-dev]

On Wed, Feb 10, 2016 at 2:42 PM Peter Collingbourne <peter at pcc.me.uk> wrote:

> On Fri, Feb 05, 2016 at 05:35:14PM -0800, David Blaikie wrote:
> > On Fri, Feb 5, 2016 at 5:04 PM, Peter Collingbourne <peter at pcc.me.uk>
> wrote:
> >
> > > Thanks, I'll look into that. (Though earlier you told me that debug
> info
> > > for types could be extended while walking the IR, so I wouldn't have
> > > thought
> > > that would have worked.)
> > >
> > >
> > Yeah, had to think about it more - and as I think about it - I'm
> moderately
> > sure type units (which don't include these latent extensions) will be
> > pretty close to static. With just the stmt_list relocation in non-fission
> > type units which /should/ still be knowable up-front.
>
> I've implemented a change which does this, and looked at impact on memory
> consumption and binary size running "llc" on Chromium's 50 largest (by
> bitcode size) translation units. Bottom line: *huge* savings in total
> memory
> consumption, median 17% when compared to before the change, median 7% when
> compared to type units disabled.
>
> (I'm not yet confident that my patch is correct (some of the section sizes
> are different and I'll need to double check what's going on there) but I'll
> send it out once I'm confident in it.)
>
> I think we can do better, though. With type units enabled, the size of
> .debug_info as a fraction of (.debug_info + .debug_types) is median ~40%,
> so I think there's another ~12% that can be saved by avoiding DIE/DIEValue
> retention for debug_info, bringing the total to ~30%. I expect numbers with
> type units disabled to be in the same ballpark (with type units enabled,
> we consume ~25% more space in the object file on .debug_info +
> .debug_types,
> so the proportional savings may be less, but the absolute memory
> consumption
> should be lower).  This also roughly lines up with the heap profiler
> figures
> from before.
>
> My conclusion from all this: I think we should do it, and I think it would
> especially help in LTO mode with type units disabled: the type units
> feature
> is redundant with LTO deduplication and would therefore add unnecessary
> bloat
> to object files, which would mean increased memory usage (I measured a ~10%
> median increase in memory usage comparing the current type units
> implementation
> against type units disabled -- not an entirely fair comparison, but
> probably
> good enough).
>
> I have a plan in mind for doing this incrementally: we will start using the
> more efficient data structure at the leaves of the DIE tree, and gradually
> expand out to the root. You'll see what that looks like once I have my
> first
> patch ready.
>
>
This sounds reasonable and I like the plan as well.

Thanks for looking into this Peter!

-eric


> Thanks,
> --
> Peter
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160210/422a1217/attachment.html>