[llvm-dev] Possible Memory Savings for tools emitting large amounts of existing data through MC
David Blaikie via llvm-dev
llvm-dev at lists.llvm.org
Mon Feb 29 20:59:11 PST 2016
On Mon, Feb 29, 2016 at 8:41 PM, Frédéric Riss <friss at apple.com> wrote:
> On Feb 29, 2016, at 3:46 PM, David Blaikie <dblaikie at gmail.com> wrote:
> On Mon, Feb 29, 2016 at 3:36 PM, Adrian Prantl <aprantl at apple.com> wrote:
>> On Feb 29, 2016, at 3:18 PM, David Blaikie <dblaikie at gmail.com> wrote:
>> Just in case it interests anyone else, I'm playing around with trying to
>> broaden the MCStreamer API to allow for emission of bytes without copying
>> the contents into a local buffer first (either because you already have a
>> buffer, or the bytes are already present in another file, etc) in
>> http://reviews.llvm.org/D17694 . In theory there's some overlap with lld
>> here (no doubt it already does this sort of thing, but not in a way, I
>> assume, we could reuse from other tools at the moment) and my motivation,
>> llvm-dwp, looks very much like "linking with a few extra steps".
>> But to check that these changes might be more generally applicable, I
>> thought I'd solicit data from anyone building tools that might be memory
>> constrained as well.
>> First that comes to mind (Eric suggested/mentioned) is llvm-dsymutil.
>> Adrian/Fred - do you guys ever have trouble with memory usage of
>> llvm-dsymutil? Do you have an example you could provide that has high
>> memory usage, so I could see if any simple changes based on my prototype MC
>> changes would help.
>> Since dsymutil processes object files one after another,
> As does llvm-dwp. Think of llvm-dwp more like a linker with a few extra
> bits. But the MCStreamer API means any bytes you write to the streamer stay
> in memory until you "Finish" - so if you're dwp/linking large enough
> inputs, you have them all in memory when you really don't need them. For
> example, the dwp file I was generating is 7GB, but the tool with the memory
> improvements only has a high water mark of 2.3GB.
> I’m a bit surprised by those numbers. If the output is 7GB, don’t you
> need to have a high watermark of 7GB at emission time even with your scheme?
Nope, which is the great thing - the input files are memory mapped (reading
with libObject) and by delaying the output a bit more, we can literally be
reading bytes from the memory mapped input and writing them out to the
output file - at no point do we then need to have the entire contents in
> Also, in D17694 you mention that the memory peak goes from 9.6GB to 2.3GB.
> Is this dirty memory or allocated memory?
Allocated - I used valgrind's --tool=massif to analyze the memory usage.
> When investigating the memory use of dsymutil, I found out that the
> exponential growth of the MC vectors would hide the real memory usage (eg
> showing 2GB when the code actually used just a bit over 1GB).
True, there could be some allocated but undirtied pages. Not sure if
Valgrind accounts for that.
> Just curious, I think your approach makes a lot of sense.
> memory usage wasn’t really a problem so far, but you could try running
>> llvm-dsymutil on bin/clang for a larger example (takes about a minute to
> Was thinking of something more accessible to me, on a non-Darwin platform.
> Is there a way I can generate the dsym inputs across Clang on a non-Darwin
> platform? (what happens if I run dsymutil on my ELF object files?)
>> A quick glance at dsymutil's code indicates it might benefit slightly, at
>> least - in the string table emission, for example (it looks very similar to
>> string table emission in dwp - just being able to reference the strings in
>> the StringMap rather than copying them into MCStreamer could help (also I
>> found using a DenseMap<StringRef to the memory mapped input helped as well
>> - but that's a change you can make locally without any MCStreamer
>> improvements) - other parts might be trickier, and consist of parts of
>> referencable data (like the line table header) and parts that are not
>> referencable (like their contents) - my prototype could be extended to
>> handle that)
>> -- adrian
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the llvm-dev