<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Sep 17, 2014 at 6:46 AM, Daniel Stewart <span dir="ltr"><<a href="mailto:stewartd@codeaurora.org" target="_blank">stewartd@codeaurora.org</a>></span> wrote:<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div lang="EN-US" link="#0563C1" vlink="#954F72"><div><p class="MsoNormal">Looking at the existing flow of passes for LTO, it appears that most all passes are run on a per file basis, before the call to the gold linker. I’m looking to get people’s feedback on whether there would be an advantage to waiting to run a number of these passes until the linking stage. For example, I believe I saw a post a little while back about postponing vectorization until the linking stage. It seems to me that there could be an advantage to postponing (some) passes until the linking stage, where the entire graph is available. In general, what do people think about the idea of a different flow of LTO where more passes are postponed until the linking stage?</p></div></div></blockquote><div><br></div><div>AFAIK, we still mostly obey the per-TU optimization flags. E.g. if you pass -O3 for each TU, we will run -O3 passes without really keeping in mind that we are doing LTO (or if we do, then it is fairly minimal). The per-TU optimization flags can have an enormous impact on the final binary size. Here are some data points I have recently collected on one of our first-party games:</div><div><br></div><div>noLTO O3perTU:</div><div>71.1MiB (control)</div><div><br></div><div>LTO O3perTU:</div><div>71.8MiB (1% larger)</div><div><br></div><div>LTO O0perTU:</div><div>67.4MiB (5% smaller)</div><div><br></div><div>LTO O1perTU:</div><div>68.5MiB (4% smaller)</div><div><br></div><div>LTO OsperTU:</div><div>65.3MiB (8% smaller)</div><div><br></div><div>This is with a 3.4-based compiler btw, but is in keeping with what I observed last Summer, so I assume that the significant effect on binary size is still present today.</div><div>FYI, these elf sizes are also with no debug info.</div><div><br></div><div>Here is a visualization of those same binary files, but broken down by text and data sections (as given by llvm-size; bss was not significantly affected so it was omitted):</div><div><img src="http://i.imgur.com/Ie5Plgx.png" style="margin-right: 0px;" width="562" height="379"><br><br></div><div><a href="http://i.imgur.com/Ie5Plgx.png">http://i.imgur.com/Ie5Plgx.png</a></div><div><br></div><div>As you can see (and would expect), LTO does a good job of reducing the data size, since it can use whole-program analysis to eliminate it. This benefit does not depend on the per-TU optimization level, also as you would expect.<br></div><div>The text section however has a different behavior. I'm still investigating, but I suspect any size regression is largely due to excessive inlining (as I think most people would expect). It is interesting to note that between the -Os LTO case and -O3 LTO case, there is a text size difference of (20.7/14.3 - 1) ~ 45%. Also, looking at this again, I don't understand why I didn't do anything with -O2 (I'll eventually need to re-obtain these datasets with a ToT compiler anyway, and I will be sure to grab -O2 data); my experience is that Clang's -O3 is sufficiently similar to -O2 that I'm fairly confident that this missing data is not going to significantly alter the findings of my preliminary analysis in the upcoming days.</div><div><br></div><div>For starters, here is a plot showing how much of the total text size is attributable to functions of each size, comparing -O3 noLTO with -O3 LTO:</div><div><img src="http://i.imgur.com/pfIo0sy.png" style="margin-right: 0px;" width="562" height="393"><br><br></div><div><a href="http://i.imgur.com/pfIo0sy.png">http://i.imgur.com/pfIo0sy.png</a> [*]<br></div><div>To understand this plot, imagine if you were to take all the functions in the binary, and group them into a small number of buckets of similarly-sized functions. Each bar represents one bucket, and the height of the bar represents the total size of all the functions in that bucket. The width and position of the bucket indicate which range of sizes it corresponds to.</div><div>Although the general behavior is a shift in the distribution to the right (functions are becoming larger with LTO), there is also an increase in total area under the bars, which is perhaps best visualized by looking at the same plot, but with each bar indicating the cumulative total (imagine that you were to call std::accumulate on the list of bar heights from the previous plot):</div><div><img src="http://i.imgur.com/q7Iq7AH.png" style="margin-right: 0px;" width="562" height="393"><br><br></div><div><a href="http://i.imgur.com/q7Iq7AH.png">http://i.imgur.com/q7Iq7AH.png</a><br></div><div>The overall text size regression adds up to nearly 25%.</div><div><br></div><div><br></div><div>[*] The two outliers in the non-LTO case are:</div><div>- the global initializers (_GLOBAL__I_a), whose size is significantly reduced by LTO from about 400k to 100k (this single function corresponds to the entire furthest-right bar). Note: the right-most bar for the LTO dataset (>100kB functions) is this function (slimmed down to about 100k) and one other that was subjected to an unusually large amount of inlining and grew from 2k to about 125k.</div><div>- an unusually large dead function that LTO was able to remove but was not being removed before (this single function corresponds to the entire second-to-furthest-right bar).</div><div><br></div><div>-- Sean Silva</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div lang="EN-US" link="#0563C1" vlink="#954F72"><div><p class="MsoNormal"> <u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">Daniel Stewart<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">--<u></u><u></u></p><p class="MsoNormal">Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p></div></div><br>_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:LLVMdev@cs.uiuc.edu">LLVMdev@cs.uiuc.edu</a> <a href="http://llvm.cs.uiuc.edu" target="_blank">http://llvm.cs.uiuc.edu</a><br>
<a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev</a><br>
<br></blockquote></div></div></div>