[llvm-dev] Possible Memory Savings for tools emitting large amounts of existing data through MC

[David Blaikie via llvm-dev]

On Mon, Feb 29, 2016 at 3:36 PM, Adrian Prantl <aprantl at apple.com> wrote:

>
> On Feb 29, 2016, at 3:18 PM, David Blaikie <dblaikie at gmail.com> wrote:
>
> Just in case it interests anyone else, I'm playing around with trying to
> broaden the MCStreamer API to allow for emission of bytes without copying
> the contents into a local buffer first (either because you already have a
> buffer, or the bytes are already present in another file, etc) in
> http://reviews.llvm.org/D17694 . In theory there's some overlap with lld
> here (no doubt it already does this sort of thing, but not in a way, I
> assume, we could reuse from other tools at the moment) and my motivation,
> llvm-dwp, looks very much like "linking with a few extra steps".
>
> But to check that these changes might be more generally applicable, I
> thought I'd solicit data from anyone building tools that might be memory
> constrained as well.
>
> First that comes to mind (Eric suggested/mentioned) is llvm-dsymutil.
>
> Adrian/Fred - do you guys ever have trouble with memory usage of
> llvm-dsymutil? Do you have an example you could provide that has high
> memory usage, so I could see if any simple changes based on my prototype MC
> changes would help.
>
>
> Since dsymutil processes object files one after another,
>

As does llvm-dwp. Think of llvm-dwp more like a linker with a few extra
bits. But the MCStreamer API means any bytes you write to the streamer stay
in memory until you "Finish" - so if you're dwp/linking large enough
inputs, you have them all in memory when you really don't need them. For
example, the dwp file I was generating is 7GB, but the tool with the memory
improvements only has a high water mark of 2.3GB.


> memory usage wasn’t really a problem so far, but you could try running
> llvm-dsymutil on bin/clang for a larger example (takes about a minute to
> finish).
>

Was thinking of something more accessible to me, on a non-Darwin platform.
Is there a way I can generate the dsym inputs across Clang on a non-Darwin
platform? (what happens if I run dsymutil on my ELF object files?)

>
> A quick glance at dsymutil's code indicates it might benefit slightly, at
> least - in the string table emission, for example (it looks very similar to
> string table emission in dwp - just being able to reference the strings in
> the StringMap rather than copying them into MCStreamer could help (also I
> found using a DenseMap<StringRef to the memory mapped input helped as well
> - but that's a change you can make locally without any MCStreamer
> improvements) - other parts might be trickier, and consist of parts of
> referencable data (like the line table header)  and parts that are not
> referencable (like their contents) - my prototype could be extended to
> handle that)
>
>
> -- adrian
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160229/f51d10a6/attachment.html>