<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi,<div class=""><br class=""></div><div class="">TLDR: I totally support considering compile time regression as bug.</div><div class=""><br class=""></div><div class="">I'm glad you bring this topic. Also it is worth pointing at this recent thread: <a href="http://lists.llvm.org/pipermail/llvm-dev/2016-March/096488.html" class="">http://lists.llvm.org/pipermail/llvm-dev/2016-March/096488.html</a></div><div class="">And also this blog post comparing the evolution of clang and gcc on this aspect: <a href="http://hubicka.blogspot.nl/2016/03/building-libreoffice-with-gcc-6-and-lto.html" class="">http://hubicka.blogspot.nl/2016/03/building-libreoffice-with-gcc-6-and-lto.html</a></div><div class=""><br class=""></div><div class="">I will repeat myself here, since we also noticed internally that compile time was slowly degrading with time. Bruno and Chris are working on some infrastructure and tooling to help tracking closely compile time regressions.</div><div class=""><br class="">We had this conversation internally about the tradeoff between compile-time and runtime performance, and I planned to bring-up the topic on the list in the coming months, but was waiting for more tooling to be ready.</div><div class="">Apparently in the past (years/decade ago?) the project was very conservative on adding any optimizations that would impact compile time, however there is no explicit policy (that I know of) to address this tradeoff. <br class="">The closest I could find would be what Chandler wrote in: <a href="http://reviews.llvm.org/D12826" class="">http://reviews.llvm.org/D12826</a> ; for instance for O2 he stated that "if an optimization increases compile time by 5% or increases code size by 5% for a particular benchmark, that benchmark should also be one which sees a 5% runtime improvement".<br class=""><br class="">My hope is that with better tooling for tracking compile time in the future, we'll reach a state where we'll be able to consider "breaking" the compile-time regression test as important as breaking any test: i.e. the offending commit should be reverted unless it has been shown to significantly (hand wavy...) improve the runtime performance.</div><div class=""><br class=""></div><div class="">Since you raise the discussion now, I take the opportunity to push on the "more aggressive" side: I think the policy should be a balance between the improvement the commit brings compared to the compile time slow down. Something along the line as what you wrote in my quote above.</div><div class="">You are referring to "large" compile time regressions (aside: what is "large"?), while Bruno has graphs that shows that the compile time regressions are mostly a lot of 1-3% regressions in general, spread over tens of commits. </div><div class="">Also (and this where we need better tooling) *unexpected* compile-time slow down are what makes me worried: i.e. the author of the commit adds something but didn't expect the compile time to be "significantly" impacted. This is motivated by Bruno/Chris data.</div><div class="">Tracking this more closely may also help to triage thing between O2 and O3 when a commit introduces a compile time slow-down but also brings significant enough runtime improvements.</div><div class=""><br class=""></div><div class="">-- </div><div class="">Mehdi</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div><br class=""></div><div><br class=""></div><div><br class=""><blockquote type="cite" class=""><div class="">On Mar 31, 2016, at 1:28 PM, Chandler Carruth via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">LLVM has a wonderful policy regarding broken commits: we revert to green. We ask that a test case be available within a reasonable time frame (preferably before, but some exceptions can be made), but otherwise we revert the offending patch, even if it contains nice features that people want, and keep the tree green. This is an awesome policy.<div class=""><br class=""></div><div class="">I would like to suggest we adopt and follow the same policy for compile time regressions that are large, and especially for ones that are super-linear. As an example from the previous thread:<br class=""><div class=""><br class=""><div class="gmail_quote"><div dir="ltr" class="">On Mon, Mar 14, 2016 at 12:14 PM Bruno Cardoso Lopes via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">> There is a possibility that r259673 could play a role here.<br class="">
><br class="">
> For the buildSchedGraph() method, there is the -dag-maps-huge-region that<br class="">
> has the default value of 1000. When I commited the patch, I was expecting<br class="">
> people to lower this value as needed and also suggested this, but this has<br class="">
> not happened. 1000 is very high, basically "unlimited".<br class="">
><br class="">
> It would be interesting to see what results you get with e.g. -mllvm<br class="">
> -dag-maps-huge-region=50. Of course, since this is a trade-off between<br class="">
> compile time and scheduler freedom, some care should be taken before<br class="">
> lowering this in trunk.<br class="">
<br class="">
Indeed we hit this internally, filed a PR:<br class="">
<a href="https://llvm.org/bugs/show_bug.cgi?id=26940" rel="noreferrer" target="_blank" class="">https://llvm.org/bugs/show_bug.cgi?id=26940</a></blockquote><div class=""><br class=""></div><div class="">I think we should have rolled back r259673 as soon as the test case was available.</div><div class=""><br class=""></div><div class="">Thoughts?</div></div></div></div></div>
_______________________________________________<br class="">LLVM Developers mailing list<br class=""><a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a><br class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev<br class=""></div></blockquote></div><br class=""></div></body></html>