[llvm-dev] Possible Memory Savings for tools emitting large amounts of existing data through MC

[Frédéric Riss via llvm-dev]

> On Feb 29, 2016, at 3:18 PM, David Blaikie <dblaikie at gmail.com> wrote:
> 
> Just in case it interests anyone else, I'm playing around with trying to broaden the MCStreamer API to allow for emission of bytes without copying the contents into a local buffer first (either because you already have a buffer, or the bytes are already present in another file, etc) in http://reviews.llvm.org/D17694 <http://reviews.llvm.org/D17694> . In theory there's some overlap with lld here (no doubt it already does this sort of thing, but not in a way, I assume, we could reuse from other tools at the moment) and my motivation, llvm-dwp, looks very much like "linking with a few extra steps".
> 
> But to check that these changes might be more generally applicable, I thought I'd solicit data from anyone building tools that might be memory constrained as well.
> 
> First that comes to mind (Eric suggested/mentioned) is llvm-dsymutil.
> 
> Adrian/Fred - do you guys ever have trouble with memory usage of llvm-dsymutil? Do you have an example you could provide that has high memory usage, so I could see if any simple changes based on my prototype MC changes would help. 
> 
> A quick glance at dsymutil's code indicates it might benefit slightly, at least - in the string table emission, for example (it looks very similar to string table emission in dwp - just being able to reference the strings in the StringMap rather than copying them into MCStreamer could help (also I found using a DenseMap<StringRef to the memory mapped input helped as well - but that's a change you can make locally without any MCStreamer improvements) - other parts might be trickier, and consist of parts of referencable data (like the line table header)  and parts that are not referencable (like their contents) - my prototype could be extended to handle that)

Yes, I agree dsymutil could see slight gains from the parts you mention. It wouldn’t be groundbreaking though, the biggest contributor would always be the DIE tree that we recreate completely.

Fred
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160229/638c8626/attachment.html>