[llvm-dev] Possible Memory Savings for tools emitting large amounts of existing data through MC

[Adrian Prantl via llvm-dev]

> On Feb 29, 2016, at 3:18 PM, David Blaikie <dblaikie at gmail.com> wrote:
> 
> Just in case it interests anyone else, I'm playing around with trying to broaden the MCStreamer API to allow for emission of bytes without copying the contents into a local buffer first (either because you already have a buffer, or the bytes are already present in another file, etc) in http://reviews.llvm.org/D17694 <http://reviews.llvm.org/D17694> . In theory there's some overlap with lld here (no doubt it already does this sort of thing, but not in a way, I assume, we could reuse from other tools at the moment) and my motivation, llvm-dwp, looks very much like "linking with a few extra steps".
> 
> But to check that these changes might be more generally applicable, I thought I'd solicit data from anyone building tools that might be memory constrained as well.
> 
> First that comes to mind (Eric suggested/mentioned) is llvm-dsymutil.
> 
> Adrian/Fred - do you guys ever have trouble with memory usage of llvm-dsymutil? Do you have an example you could provide that has high memory usage, so I could see if any simple changes based on my prototype MC changes would help. 

Since dsymutil processes object files one after another, memory usage wasn’t really a problem so far, but you could try running llvm-dsymutil on bin/clang for a larger example (takes about a minute to finish).
> 
> A quick glance at dsymutil's code indicates it might benefit slightly, at least - in the string table emission, for example (it looks very similar to string table emission in dwp - just being able to reference the strings in the StringMap rather than copying them into MCStreamer could help (also I found using a DenseMap<StringRef to the memory mapped input helped as well - but that's a change you can make locally without any MCStreamer improvements) - other parts might be trickier, and consist of parts of referencable data (like the line table header)  and parts that are not referencable (like their contents) - my prototype could be extended to handle that)

-- adrian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160229/b7d688ed/attachment.html>