[llvm-dev] Possible Memory Savings for tools emitting large amounts of existing data through MC

[Peter Collingbourne via llvm-dev]

Hi David,

The way I imagined that we might want to extend the MCStreamer API (this
was motivated by DIEData) is by allowing clients to move bytes and fixups
into the MC layer.

This is the sort of API that I was imagining:

void MoveAndEmitFragment(SmallVectorImpl<char> &&Data,
                         SmallVectorImpl<MCFixup> &&Fixups);

Note that this mirrors the fields
MCEncodedFragmentWithContents::Contents and MCEncodedFragmentWithFixups::Fixups
and the arguments could be directly moved into the fields of a newly created
MCDataFragment.

Would that work for your use case?

Peter

On Mon, Feb 29, 2016 at 03:18:22PM -0800, David Blaikie via llvm-dev wrote:
> Just in case it interests anyone else, I'm playing around with trying to
> broaden the MCStreamer API to allow for emission of bytes without copying
> the contents into a local buffer first (either because you already have a
> buffer, or the bytes are already present in another file, etc) in
> http://reviews.llvm.org/D17694 . In theory there's some overlap with lld
> here (no doubt it already does this sort of thing, but not in a way, I
> assume, we could reuse from other tools at the moment) and my motivation,
> llvm-dwp, looks very much like "linking with a few extra steps".
> 
> But to check that these changes might be more generally applicable, I
> thought I'd solicit data from anyone building tools that might be memory
> constrained as well.
> 
> First that comes to mind (Eric suggested/mentioned) is llvm-dsymutil.
> 
> Adrian/Fred - do you guys ever have trouble with memory usage of
> llvm-dsymutil? Do you have an example you could provide that has high
> memory usage, so I could see if any simple changes based on my prototype MC
> changes would help.
> 
> A quick glance at dsymutil's code indicates it might benefit slightly, at
> least - in the string table emission, for example (it looks very similar to
> string table emission in dwp - just being able to reference the strings in
> the StringMap rather than copying them into MCStreamer could help (also I
> found using a DenseMap<StringRef to the memory mapped input helped as well
> - but that's a change you can make locally without any MCStreamer
> improvements) - other parts might be trickier, and consist of parts of
> referencable data (like the line table header)  and parts that are not
> referencable (like their contents) - my prototype could be extended to
> handle that)

> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev


-- 
Peter