[llvm-dev] Possible Memory Savings for tools emitting large amounts of existing data through MC

[David Blaikie via llvm-dev]

Just in case it interests anyone else, I'm playing around with trying to
broaden the MCStreamer API to allow for emission of bytes without copying
the contents into a local buffer first (either because you already have a
buffer, or the bytes are already present in another file, etc) in
http://reviews.llvm.org/D17694 . In theory there's some overlap with lld
here (no doubt it already does this sort of thing, but not in a way, I
assume, we could reuse from other tools at the moment) and my motivation,
llvm-dwp, looks very much like "linking with a few extra steps".

But to check that these changes might be more generally applicable, I
thought I'd solicit data from anyone building tools that might be memory
constrained as well.

First that comes to mind (Eric suggested/mentioned) is llvm-dsymutil.

Adrian/Fred - do you guys ever have trouble with memory usage of
llvm-dsymutil? Do you have an example you could provide that has high
memory usage, so I could see if any simple changes based on my prototype MC
changes would help.

A quick glance at dsymutil's code indicates it might benefit slightly, at
least - in the string table emission, for example (it looks very similar to
string table emission in dwp - just being able to reference the strings in
the StringMap rather than copying them into MCStreamer could help (also I
found using a DenseMap<StringRef to the memory mapped input helped as well
- but that's a change you can make locally without any MCStreamer
improvements) - other parts might be trickier, and consist of parts of
referencable data (like the line table header)  and parts that are not
referencable (like their contents) - my prototype could be extended to
handle that)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160229/91164401/attachment.html>