<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi Angelina,<div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Jun 19, 2016, at 7:51 AM, Angelina Lee <<a href="mailto:angelee@seas.wustl.edu" class="">angelee@seas.wustl.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class=""><div class="">There seems to be two main concerns with the use of LTO. </div></div></div></div></blockquote><div><br class=""></div><div>To be clear: for the purpose of CSI, the concern is not about making is friendly with LTO, it is about *requiring* LTO to make it efficient. This is design flaw that has no justification.</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><div class=""> The first one is the hit on build time, and the second one is the argument of simplicity --- that is, it may not actually be simpler to use LTO.<br class=""><br class="">Regarding the performance hit on build time, we would like to emphasize that, CSI is not tied to the use of LTO --- it does not rely on LTO to work correctly; LTO is simply a mean to optimize the final executable. Having that being said, as Kostya pointed it out, and based what our understanding of thinLTO, we believe that thinLTO can do most of the optimizations that CSI will benefit from. CSI uses LTO only for optimization, but it also seems that later down the road, such an optimization will cost less.<br class=""></div></div></div></div></blockquote><div><br class=""></div><div>I understand this, and I still don’t see any reason to force the use of LTO to optimize the CSI use-case. We should offer a mechanism that makes is no different for the tool writer, but does not require LTO. I also believe such mechanism can be implemented in the compiler easily.</div><div>And I haven’t seen any reasons raised by anyone not to do it in this thread.</div><div><br class=""></div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><div class="">Regarding simplicity, Mehdi, you raise a good point, but I think Reid's point that, linking is more incremental still stands. In particular, we can separate out the application developer and the tool user --- the tool user does not have to be the one that build the application to generate the CSI-enabled bitcode files. These days, there are versions of libc that you can get w/ all the debug symbols compiled in versus the version that doesn't. Imagine if by default, the applications / libraries also come with just a collection of bitcode files that are CSI enabled, then the tool user doesn't even need to have the source directly nor mess with the application's build system --- he / she can simply link the CSI-compatible</div></div></div></div></blockquote><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><div class="">bitcodes with any tool that he / she intends to use.<br class=""></div></div></div></div></blockquote><div><br class=""></div><div>All of what you describe is interesting, and a good justification for making CSI *friendly* with LTO (which I never contested by the way). However it does not seem to invalidate my point that *requiring* LTO is fundamentally unnecessary here to support the “mainstream” flow that already exists with ASAN and other sanitizer for instance.</div><div><br class=""></div><div>— </div><div>Mehdi</div><div><br class=""></div><div><br class=""></div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><div class=""><br class=""></div>Cheers,<br class=""></div>Angelina<br class=""></div><div class="gmail_extra"><br clear="all" class=""><div class=""><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr" class=""><div class=""><span style="font-size:12.8000001907349px" class="">----</span></div><span style="font-size:12.8000001907349px" class="">I-Ting Angelina Lee</span><div style="font-size:12.8000001907349px" class="">Assistant Professor, Washington University in St. Louis</div><div style="font-size:12.8000001907349px" class="">Department of Computer Science and Engineering </div><div class=""><span style="font-size:12.8000001907349px" class="">Homepage: </span><font color="#1155cc" class=""><u class=""><a href="http://www.cse.wustl.edu/~angelee/" target="_blank" class="">http://www.cse.wustl.edu/~angelee/</a></u></font></div></div></div></div>
<br class=""><div class="gmail_quote">On Fri, Jun 17, 2016 at 3:22 PM, Mehdi Amini <span dir="ltr" class=""><<a href="mailto:mehdi.amini@apple.com" target="_blank" class="">mehdi.amini@apple.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word" class="">
<br class="">
<div class=""><span class="">
<blockquote type="cite" class="">
<div class="">On Jun 17, 2016, at 11:56 AM, Reid Kleckner <<a href="mailto:rnk@google.com" target="_blank" class="">rnk@google.com</a>> wrote:</div>
<br class="">
<div class="">
<div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class="">
<div class="gmail_extra">
<div class="gmail_quote">Mehdi, I think TB is saying that LTO is actually *more* incremental than standard compilation for people writing instrumentation tools.</div>
</div>
</div>
</div>
</blockquote>
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class="">
<div class="gmail_extra">
<div class="gmail_quote"><br class="">
</div>
<div class="gmail_quote">It's far simpler to rebuild your CSI instrumentation hooks and re-run LTO on Apache than it is to re-run someone else's complicated build system.
</div>
</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
</span><div class="">I *highly* doubt about that. I’d even say that I’m heavily convinced of the opposite: any build system has a facility to add a compiler flag. What could be easier than adding `-csi=path_to_my_tool` to the CFLAGS?</div>
<div class=""><br class="">
</div>
<div class="">On the other hand, LTO is a whole other beast. First you also need to modify the CFLAGS to enable LTO, so you don’t avoid the “complicated build system” interaction.</div>
<div class="">But then you’ll need to make sure the rest of the toolchain is compatible (Never had an issue with static archives? I did a lot of time!). Also what if the projects involves subprojects with dylib? You need to make sure every link is correctly performed
with the instrumented runtime.</div>
<div class=""><br class="">
</div>
<div class="">Nothing “impossible” here, but I don’t buy at all the “it is simpler” argument.</div><span class="">
<div class=""><br class="">
</div>
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class="">
<div class="gmail_extra">
<div class="gmail_quote">In fact, you save all the time that clang would spend parsing C and C++ source code, since you're starting with several blobs of semi-optimized LLVM IR.</div>
</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
</span><div class="">LTO alone on the clang binary is longer than a full clean build (and if you build all of the llvm binaries, we one or two order of magnitudes away), so it won't buy you anything. And even if it would (apparently on Apache httpd it would), I doubt it would
be significant enough to make it an argument to *tie* the design to LTO.</div><span class="">
<br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class="">
<div class="gmail_extra">
<div class="gmail_quote">It's worth exploring what you mentioned up-thread, where you inline the instrumentation into the code during normal compliation instead of waiting until link time, but I think it makes more sense to make it easy to build tools before
we try to make those new tools compile faster. </div>
</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
</span><div class="">So a few line earlier your argument for relying on LTO was to save “parsing C and C++” (so basically compile time), and now you’re saying that “making those new tools compile faster” should not drive the design?</div>
<div class=""><br class="">
</div>
<div class="">I agree with “making it easy to build tools” part, I just don’t see any reason why forcing to use LTO supports this goal.</div><span class="">
<br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class="">
<div class="gmail_extra">
<div class="gmail_quote">It seems like a nice-to-have feature for instrumentation tools that manage to graduate from research idea to production tool.</div>
</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
</span><div class="">I believe this last sentence isn’t related to the LTO aspect, CSI looks like a great idea, let’s just do it in a way that it does not require LTO.</div>
<div class=""><br class="">
</div>
<div class="">— </div>
<div class="">Mehdi</div>
<div class=""><br class="">
</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class="">
<div class="gmail_extra">
<div class="gmail_quote"><br class="">
</div>
<div class="gmail_quote"><span class="">On Fri, Jun 17, 2016 at 11:42 AM, Mehdi Amini via llvm-dev<span class=""> </span><span dir="ltr" class=""><<a href="mailto:llvm-dev@lists.llvm.org" target="_blank" class="">llvm-dev@lists.llvm.org</a>></span><span class=""> </span>wrote:<br class="">
</span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word" class="">
<div class=""><span class=""><span class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class="gmail_extra">
<div class="gmail_quote">
<div class="">This reduction in the number of compile operations needed, and in the number intermediate object/bitcode files produced, is indeed an advantage of the CSI approach. </div>
</div>
</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
</span>
<div class="">It is a very artificial advantage, what are you saving? Temporary Disk Space?</div>
<span class="">
<div class=""><br class="">
</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class="gmail_extra">
<div class="gmail_quote">
<div class="">As an aside, we've been experimenting with linking CSI-instrumented bitcodes against the "null tool," which implements every instrumentation hook as a nop, and comparing the performance of those binaries against production binaries. Our preliminary
tests have shown some promising results. For generating main executables, using LTO to link CSI-instrumented bitcodes with the null tool produces executables that are as fast as the production executables. For generating dynamic libraries, however, using
LTO to link the CSI-instrumented bitcode of a dynamic library with the null tool seems to produce a binary that is slower than production. (The Apache HTTP server benchmark we've tried runs roughly 30% slower when using such null-tool-instrumented dynamic
libraries.) These results suggest that using LTO to link CSI-instrumented bitcodes with the null tool is almost, but not quite, able to produce binaries with production performance</div>
</div>
</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
</span>
<div class="">These results suggests that “adding instrumentation has a cost” nothing more, and is unrelated to LTO at all.</div>
<div class="">You would provide the runtime to the compiler directly during the compile phase and you would get the same results.</div>
<span class="">
<div class=""><br class="">
</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class="gmail_extra">
<div class="gmail_quote">
<div class="">, which would allow tool users to only compile their sources once.</div>
</div>
</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
</span>
<div class="">LTO means basically “compiles during the link”. You won’t save much.</div>
<div class=""><br class="">
</div>
<div class="">I haven’t seen a single compelling argument to *tie* CSI to LTO in this thread until now.</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">— </div>
<div class="">Mehdi</div>
</span><span class="">
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class="gmail_extra">
<div class="gmail_quote">
<div class=""> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><span class="">
<div bgcolor="#FFFFFF" text="#000000" class=""><br class="">
It's possible that the math doesn't really work out in practice if the cost of the LTO-link dwarfs the compile times.<span class=""><font color="#888888" class=""><br class="">
<pre cols="72" class="">--
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project
</pre>
</font></span></div>
<br class=""></span><span class="">
_______________________________________________<br class="">
LLVM Developers mailing list<br class="">
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank" class="">llvm-dev@lists.llvm.org</a><br class="">
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br class="">
<br class="">
</span></blockquote>
</div>
<br class="">
</div>
</div><span class="">
_______________________________________________<br class="">
LLVM Developers mailing list<br class="">
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank" class="">llvm-dev@lists.llvm.org</a><br class="">
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" target="_blank" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br class="">
</span></div>
</blockquote>
</span></div>
<br class="">
</div><span class="">
<br class="">
_______________________________________________<br class="">
LLVM Developers mailing list<br class="">
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank" class="">llvm-dev@lists.llvm.org</a><br class="">
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a></span></blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</blockquote></div><br class=""></div>
</div></blockquote></div><br class=""></div></body></html>