[llvm-dev] Possible Memory Savings for tools emitting large amounts of existing data through MC

[Adrian Prantl via llvm-dev]

> On Feb 29, 2016, at 4:10 PM, David Blaikie <dblaikie at gmail.com> wrote:
> 
> 
> 
> On Mon, Feb 29, 2016 at 3:51 PM, Adrian Prantl <aprantl at apple.com <mailto:aprantl at apple.com>> wrote:
> 
>> On Feb 29, 2016, at 3:46 PM, David Blaikie <dblaikie at gmail.com <mailto:dblaikie at gmail.com>> wrote:
>> 
>> 
>> 
>> On Mon, Feb 29, 2016 at 3:36 PM, Adrian Prantl <aprantl at apple.com <mailto:aprantl at apple.com>> wrote:
>> 
>>> On Feb 29, 2016, at 3:18 PM, David Blaikie <dblaikie at gmail.com <mailto:dblaikie at gmail.com>> wrote:
>>> 
>>> Just in case it interests anyone else, I'm playing around with trying to broaden the MCStreamer API to allow for emission of bytes without copying the contents into a local buffer first (either because you already have a buffer, or the bytes are already present in another file, etc) in http://reviews.llvm.org/D17694 <http://reviews.llvm.org/D17694> . In theory there's some overlap with lld here (no doubt it already does this sort of thing, but not in a way, I assume, we could reuse from other tools at the moment) and my motivation, llvm-dwp, looks very much like "linking with a few extra steps".
>>> 
>>> But to check that these changes might be more generally applicable, I thought I'd solicit data from anyone building tools that might be memory constrained as well.
>>> 
>>> First that comes to mind (Eric suggested/mentioned) is llvm-dsymutil.
>>> 
>>> Adrian/Fred - do you guys ever have trouble with memory usage of llvm-dsymutil? Do you have an example you could provide that has high memory usage, so I could see if any simple changes based on my prototype MC changes would help. 
>> 
>> Since dsymutil processes object files one after another,
>> 
>> As does llvm-dwp. Think of llvm-dwp more like a linker with a few extra bits. But the MCStreamer API means any bytes you write to the streamer stay in memory until you "Finish" - so if you're dwp/linking large enough inputs, you have them all in memory when you really don't need them. For example, the dwp file I was generating is 7GB, but the tool with the memory improvements only has a high water mark of 2.3GB.
>>  
>> memory usage wasn’t really a problem so far, but you could try running llvm-dsymutil on bin/clang for a larger example (takes about a minute to finish).
>> 
>> Was thinking of something more accessible to me, on a non-Darwin platform. Is there a way I can generate the dsym inputs across Clang on a non-Darwin platform? (what happens if I run dsymutil on my ELF object files?) 
> 
> At this point probably nothing. Dsymutil acts on STABS symbol table entries that are (I guess) not present in a typical ELF binary. Dsymutil also only implements MachO relocations and has lots of other things where the ELF implementation is missing. It’s probably not too much work to wire all this up, but so far nobody did it.
> 
> & no easy way for me to get a representative (or pathalogically large, even) set of machO files to play with, I take it? It's no worries - just figured I'd give it a go if it was convenient.

I can definitely go and grab you a clang build directory from one of the green dragon bots for example; but all the paths are hardcoded so you’d have to install them in the exact same location. In theory everything doing file access should be handled by the LLVM low-level libraries, so this *could* work.

-- adrian
>  
> 
> -- adrian
> 
>>> 
>>> A quick glance at dsymutil's code indicates it might benefit slightly, at least - in the string table emission, for example (it looks very similar to string table emission in dwp - just being able to reference the strings in the StringMap rather than copying them into MCStreamer could help (also I found using a DenseMap<StringRef to the memory mapped input helped as well - but that's a change you can make locally without any MCStreamer improvements) - other parts might be trickier, and consist of parts of referencable data (like the line table header)  and parts that are not referencable (like their contents) - my prototype could be extended to handle that)
>> 
>> -- adrian

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160229/49981cb2/attachment.html>