<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Feb 29, 2016, at 4:10 PM, David Blaikie <<a href="mailto:dblaikie@gmail.com" class="">dblaikie@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><br class="Apple-interchange-newline"><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_quote" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">On Mon, Feb 29, 2016 at 3:51 PM, Adrian Prantl<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:aprantl@apple.com" target="_blank" class="">aprantl@apple.com</a>></span><span class="Apple-converted-space"> </span>wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><br class=""><div class=""><span class=""><blockquote type="cite" class=""><div class="">On Feb 29, 2016, at 3:46 PM, David Blaikie <<a href="mailto:dblaikie@gmail.com" target="_blank" class="">dblaikie@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><br class=""><div class="gmail_extra"><br class=""><div class="gmail_quote">On Mon, Feb 29, 2016 at 3:36 PM, Adrian Prantl<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:aprantl@apple.com" target="_blank" class="">aprantl@apple.com</a>></span><span class="Apple-converted-space"> </span>wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><br class=""><div class=""><span class=""><blockquote type="cite" class=""><div class="">On Feb 29, 2016, at 3:18 PM, David Blaikie <<a href="mailto:dblaikie@gmail.com" target="_blank" class="">dblaikie@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class="">Just in case it interests anyone else, I'm playing around with trying to broaden the MCStreamer API to allow for emission of bytes without copying the contents into a local buffer first (either because you already have a buffer, or the bytes are already present in another file, etc) in<span class="Apple-converted-space"> </span><a href="http://reviews.llvm.org/D17694" target="_blank" class="">http://reviews.llvm.org/D17694</a><span class="Apple-converted-space"> </span>. In theory there's some overlap with lld here (no doubt it already does this sort of thing, but not in a way, I assume, we could reuse from other tools at the moment) and my motivation, llvm-dwp, looks very much like "linking with a few extra steps".<br class=""><br class="">But to check that these changes might be more generally applicable, I thought I'd solicit data from anyone building tools that might be memory constrained as well.<br class=""><br class="">First that comes to mind (Eric suggested/mentioned) is llvm-dsymutil.<br class=""><br class="">Adrian/Fred - do you guys ever have trouble with memory usage of llvm-dsymutil? Do you have an example you could provide that has high memory usage, so I could see if any simple changes based on my prototype MC changes would help. <br class=""></div></div></blockquote><div class=""><br class=""></div></span><div class="">Since dsymutil processes object files one after another,</div></div></div></blockquote><div class=""><br class=""></div><div class="">As does llvm-dwp. Think of llvm-dwp more like a linker with a few extra bits. But the MCStreamer API means any bytes you write to the streamer stay in memory until you "Finish" - so if you're dwp/linking large enough inputs, you have them all in memory when you really don't need them. For example, the dwp file I was generating is 7GB, but the tool with the memory improvements only has a high water mark of 2.3GB.</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><div class="">memory usage wasn’t really a problem so far, but you could try running llvm-dsymutil on bin/clang for a larger example (takes about a minute to finish).</div></div></div></blockquote><div class=""><br class=""></div><div class="">Was thinking of something more accessible to me, on a non-Darwin platform. Is there a way I can generate the dsym inputs across Clang on a non-Darwin platform? (what happens if I run dsymutil on my ELF object files?) <br class=""></div></div></div></div></div></blockquote><div class=""><br class=""></div></span><div class="">At this point probably nothing. Dsymutil acts on STABS symbol table entries that are (I guess) not present in a typical ELF binary. Dsymutil also only implements MachO relocations and has lots of other things where the ELF implementation is missing. It’s probably not too much work to wire all this up, but so far nobody did it.</div></div></div></blockquote><div class=""><br class="">& no easy way for me to get a representative (or pathalogically large, even) set of machO files to play with, I take it? It's no worries - just figured I'd give it a go if it was convenient.<br class=""></div></div></div></blockquote><div><br class=""></div><div>I can definitely go and grab you a clang build directory from one of the green dragon bots for example; but all the paths are hardcoded so you’d have to install them in the exact same location. In theory everything doing file access should be handled by the LLVM low-level libraries, so this *could* work.</div><div><br class=""></div>-- adrian<br class=""><blockquote type="cite" class=""><div class=""><div class="gmail_quote" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><span class="HOEnZb"><font color="#888888" class=""><div class=""><br class=""></div><div class="">-- adrian</div></font></span><span class=""><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><span class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><br class="">A quick glance at dsymutil's code indicates it might benefit slightly, at least - in the string table emission, for example (it looks very similar to string table emission in dwp - just being able to reference the strings in the StringMap rather than copying them into MCStreamer could help (also I found using a DenseMap<StringRef to the memory mapped input helped as well - but that's a change you can make locally without any MCStreamer improvements) - other parts might be trickier, and consist of parts of referencable data (like the line table header) and parts that are not referencable (like their contents) - my prototype could be extended to handle that)<br class=""></div></div></blockquote></span></div><span class=""><font color="#888888" class=""><br class=""><div class="">-- adrian</div></font></span></div></blockquote></div></div></div></div></blockquote></span></div></div></blockquote></div></div></blockquote></div><br class=""></body></html>