[llvm-dev] Possible Memory Savings for tools emitting large amounts of existing data through MC

[David Blaikie via llvm-dev]

On Mon, Feb 29, 2016 at 3:36 PM, Peter Collingbourne <peter at pcc.me.uk>
wrote:

> Hi David,
>
> The way I imagined that we might want to extend the MCStreamer API (this
> was motivated by DIEData) is by allowing clients to move bytes and fixups
> into the MC layer.
>
> This is the sort of API that I was imagining:
>
> void MoveAndEmitFragment(SmallVectorImpl<char> &&Data,
>                          SmallVectorImpl<MCFixup> &&Fixups);
>
> Note that this mirrors the fields
> MCEncodedFragmentWithContents::Contents and
> MCEncodedFragmentWithFixups::Fixups
> and the arguments could be directly moved into the fields of a newly
> created
> MCDataFragment.
>
> Would that work for your use case?
>

Not quite, unfortunately - the issue is that we're doing a task that is
essentially "linking + a bit" - so imagine linking a bunch of files
together, the final, say, debug_info.dwo section is made up of the
concatenation of all the debug_info.dwo sections of the inputs. So it's
fragmented and it's already available, memory mapped, never in a
SmallVector, etc.


>
> Peter
>
> On Mon, Feb 29, 2016 at 03:18:22PM -0800, David Blaikie via llvm-dev wrote:
> > Just in case it interests anyone else, I'm playing around with trying to
> > broaden the MCStreamer API to allow for emission of bytes without copying
> > the contents into a local buffer first (either because you already have a
> > buffer, or the bytes are already present in another file, etc) in
> > http://reviews.llvm.org/D17694 . In theory there's some overlap with lld
> > here (no doubt it already does this sort of thing, but not in a way, I
> > assume, we could reuse from other tools at the moment) and my motivation,
> > llvm-dwp, looks very much like "linking with a few extra steps".
> >
> > But to check that these changes might be more generally applicable, I
> > thought I'd solicit data from anyone building tools that might be memory
> > constrained as well.
> >
> > First that comes to mind (Eric suggested/mentioned) is llvm-dsymutil.
> >
> > Adrian/Fred - do you guys ever have trouble with memory usage of
> > llvm-dsymutil? Do you have an example you could provide that has high
> > memory usage, so I could see if any simple changes based on my prototype
> MC
> > changes would help.
> >
> > A quick glance at dsymutil's code indicates it might benefit slightly, at
> > least - in the string table emission, for example (it looks very similar
> to
> > string table emission in dwp - just being able to reference the strings
> in
> > the StringMap rather than copying them into MCStreamer could help (also I
> > found using a DenseMap<StringRef to the memory mapped input helped as
> well
> > - but that's a change you can make locally without any MCStreamer
> > improvements) - other parts might be trickier, and consist of parts of
> > referencable data (like the line table header)  and parts that are not
> > referencable (like their contents) - my prototype could be extended to
> > handle that)
>
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
> --
> Peter
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160229/b9689273/attachment.html>