[llvm-dev] Possible Memory Savings for tools emitting large amounts of existing data through MC

[David Blaikie via llvm-dev]

On Mon, Feb 29, 2016 at 8:41 PM, Frédéric Riss <friss at apple.com> wrote:

>
> On Feb 29, 2016, at 3:46 PM, David Blaikie <dblaikie at gmail.com> wrote:
>
>
>
> On Mon, Feb 29, 2016 at 3:36 PM, Adrian Prantl <aprantl at apple.com> wrote:
>
>>
>> On Feb 29, 2016, at 3:18 PM, David Blaikie <dblaikie at gmail.com> wrote:
>>
>> Just in case it interests anyone else, I'm playing around with trying to
>> broaden the MCStreamer API to allow for emission of bytes without copying
>> the contents into a local buffer first (either because you already have a
>> buffer, or the bytes are already present in another file, etc) in
>> http://reviews.llvm.org/D17694 . In theory there's some overlap with lld
>> here (no doubt it already does this sort of thing, but not in a way, I
>> assume, we could reuse from other tools at the moment) and my motivation,
>> llvm-dwp, looks very much like "linking with a few extra steps".
>>
>> But to check that these changes might be more generally applicable, I
>> thought I'd solicit data from anyone building tools that might be memory
>> constrained as well.
>>
>> First that comes to mind (Eric suggested/mentioned) is llvm-dsymutil.
>>
>> Adrian/Fred - do you guys ever have trouble with memory usage of
>> llvm-dsymutil? Do you have an example you could provide that has high
>> memory usage, so I could see if any simple changes based on my prototype MC
>> changes would help.
>>
>>
>> Since dsymutil processes object files one after another,
>>
>
> As does llvm-dwp. Think of llvm-dwp more like a linker with a few extra
> bits. But the MCStreamer API means any bytes you write to the streamer stay
> in memory until you "Finish" - so if you're dwp/linking large enough
> inputs, you have them all in memory when you really don't need them. For
> example, the dwp file I was generating is 7GB, but the tool with the memory
> improvements only has a high water mark of 2.3GB.
>
>
> I’m  a bit surprised by those numbers. If the output is 7GB, don’t you
> need to have a high watermark of 7GB at emission time even with your scheme?
>

Nope, which is the great thing - the input files are memory mapped (reading
with libObject) and by delaying the output a bit more, we can literally be
reading bytes from the memory mapped input and writing them out to the
output file - at no point do we then need to have the entire contents in
memory.


> Also, in D17694 you mention that the memory peak goes from 9.6GB to 2.3GB.
> Is this dirty memory or allocated memory?
>

Allocated - I used valgrind's --tool=massif to analyze the memory usage.


> When investigating the memory use of dsymutil, I found out that the
> exponential growth of the MC vectors would hide the real memory usage (eg
> showing 2GB when the code actually used just a bit over 1GB).
>

True, there could be some allocated but undirtied pages. Not sure if
Valgrind accounts for that.


> Just curious, I think your approach makes a lot of sense.
>
> Fred
>
> memory usage wasn’t really a problem so far, but you could try running
>> llvm-dsymutil on bin/clang for a larger example (takes about a minute to
>> finish).
>>
>
> Was thinking of something more accessible to me, on a non-Darwin platform.
> Is there a way I can generate the dsym inputs across Clang on a non-Darwin
> platform? (what happens if I run dsymutil on my ELF object files?)
>
>>
>> A quick glance at dsymutil's code indicates it might benefit slightly, at
>> least - in the string table emission, for example (it looks very similar to
>> string table emission in dwp - just being able to reference the strings in
>> the StringMap rather than copying them into MCStreamer could help (also I
>> found using a DenseMap<StringRef to the memory mapped input helped as well
>> - but that's a change you can make locally without any MCStreamer
>> improvements) - other parts might be trickier, and consist of parts of
>> referencable data (like the line table header)  and parts that are not
>> referencable (like their contents) - my prototype could be extended to
>> handle that)
>>
>>
>> -- adrian
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160229/51ec58bb/attachment-0001.html>