[llvm-dev] Possible Memory Savings for tools emitting large amounts of existing data through MC

[David Blaikie via llvm-dev]

On Mon, Feb 29, 2016 at 3:51 PM, Adrian Prantl <aprantl at apple.com> wrote:

>
> On Feb 29, 2016, at 3:46 PM, David Blaikie <dblaikie at gmail.com> wrote:
>
>
>
> On Mon, Feb 29, 2016 at 3:36 PM, Adrian Prantl <aprantl at apple.com> wrote:
>
>>
>> On Feb 29, 2016, at 3:18 PM, David Blaikie <dblaikie at gmail.com> wrote:
>>
>> Just in case it interests anyone else, I'm playing around with trying to
>> broaden the MCStreamer API to allow for emission of bytes without copying
>> the contents into a local buffer first (either because you already have a
>> buffer, or the bytes are already present in another file, etc) in
>> http://reviews.llvm.org/D17694 . In theory there's some overlap with lld
>> here (no doubt it already does this sort of thing, but not in a way, I
>> assume, we could reuse from other tools at the moment) and my motivation,
>> llvm-dwp, looks very much like "linking with a few extra steps".
>>
>> But to check that these changes might be more generally applicable, I
>> thought I'd solicit data from anyone building tools that might be memory
>> constrained as well.
>>
>> First that comes to mind (Eric suggested/mentioned) is llvm-dsymutil.
>>
>> Adrian/Fred - do you guys ever have trouble with memory usage of
>> llvm-dsymutil? Do you have an example you could provide that has high
>> memory usage, so I could see if any simple changes based on my prototype MC
>> changes would help.
>>
>>
>> Since dsymutil processes object files one after another,
>>
>
> As does llvm-dwp. Think of llvm-dwp more like a linker with a few extra
> bits. But the MCStreamer API means any bytes you write to the streamer stay
> in memory until you "Finish" - so if you're dwp/linking large enough
> inputs, you have them all in memory when you really don't need them. For
> example, the dwp file I was generating is 7GB, but the tool with the memory
> improvements only has a high water mark of 2.3GB.
>
>
>> memory usage wasn’t really a problem so far, but you could try running
>> llvm-dsymutil on bin/clang for a larger example (takes about a minute to
>> finish).
>>
>
> Was thinking of something more accessible to me, on a non-Darwin platform.
> Is there a way I can generate the dsym inputs across Clang on a non-Darwin
> platform? (what happens if I run dsymutil on my ELF object files?)
>
>
> At this point probably nothing. Dsymutil acts on STABS symbol table
> entries that are (I guess) not present in a typical ELF binary. Dsymutil
> also only implements MachO relocations and has lots of other things where
> the ELF implementation is missing. It’s probably not too much work to wire
> all this up, but so far nobody did it.
>

& no easy way for me to get a representative (or pathalogically large,
even) set of machO files to play with, I take it? It's no worries - just
figured I'd give it a go if it was convenient.


>
> -- adrian
>
>
>> A quick glance at dsymutil's code indicates it might benefit slightly, at
>> least - in the string table emission, for example (it looks very similar to
>> string table emission in dwp - just being able to reference the strings in
>> the StringMap rather than copying them into MCStreamer could help (also I
>> found using a DenseMap<StringRef to the memory mapped input helped as well
>> - but that's a change you can make locally without any MCStreamer
>> improvements) - other parts might be trickier, and consist of parts of
>> referencable data (like the line table header)  and parts that are not
>> referencable (like their contents) - my prototype could be extended to
>> handle that)
>>
>>
>> -- adrian
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160229/38484a1c/attachment.html>