<html><head><style>body{font-family:Helvetica,Arial;font-size:13px}</style></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">We have measured null/empty tests before. On a well stabilized machine there was no problem detecting regressions.</div> <br> <div id="bloop_sign_1480367672761116928" class="bloop_sign"></div> <br><p class="airmail_on">On November 17, 2016 at 3:00:23 PM, Matthias Braun (<a href="mailto:mbraun@apple.com">mbraun@apple.com</a>) wrote:</p> <blockquote type="cite" class="clean_bq"><span><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div></div><div>
<title></title>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Nov 17, 2016, at 2:55 PM, Mehdi Amini <<a href="mailto:mehdi.amini@apple.com" class="">mehdi.amini@apple.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi Gerolf,
<div class=""><br class=""></div>
<div class="">This is really cool!</div>
<div class="">I’m very excited about this initiative and I hope
we’ll be able to get to a stage where compile time regression are
handled like other regression: if they are not expected / justified
by the commit author promptly, the commit should be reverted in the
meantime!</div>
<div class=""><br class=""></div>
<div class="">I’d like to suggest adding to CTMark the “empty”
compile test (and maybe “empty + one empty function”), unless it is
too noisy to measure.</div>
<div class="">It is an interesting test to complete the existing
ones because it measures the general overhead of setting up all the
“infrastructure” (static initializers, creating a pass pipeline,
etc.)</div>
</div>
</div>
</blockquote>
<div>That would indeed be a very interesting test, however this
will be way too short to measure predictably on its
own. </div>
<div>I could see it working if we had a flag that artifically runs
the compilation pipeline hundreds of times or alternatively puts
the whole compiler into something like googlebenchmark.</div>
<div><br class=""></div>
<div>- Matthias</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div class=""><br class=""></div>
<div class="">— </div>
<div class="">Mehdi</div>
<div class=""><br class=""></div>
<div class=""><br class=""></div>
<div class=""><br class=""></div>
<div class=""><br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Nov 15, 2016, at 1:27 PM, Gerolf Hoflehner via
llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div class=""><br class=""></div>
<div class="">
<div class="">Hi,</div>
<div class=""><br class=""></div>
<div class="">this is about kicking-off regular compile-time
tracking for LLVM and CLANG on the green dragon: <a href="http://lab.llvm.org:8080/green/view/Compile%20Time/" class="">http://lab.llvm.org:8080/green/view/Compile%20Time/</a>. The
goal is to stay on top of compile-time issues immediately when they
occur so they can be assessed rather than creeping in unnoticed.
The methodology is simple: form a CTMark suite out of 10 “long”
compiling tests of the LLVM test suite and track it closely for
ARM64 O0g and Os. When there is a jump in compile-time of more than
2.5% in one of the tests an email notification will be sent to the
committer and a bug will be filed in bugzilla. The 2.5% threshold
is large enough to be above the noise level and should be
motivating enough to root cause the issue. Also watch for spikes of
>10% in O3 LTO. </div>
<div class=""><br class=""></div>
<div class="">There will be process and effort. Chris Matthews put
together CTMark together with Michael and Matthias, and keeps the
servers running and constantly improves tooling. Michael stepped up
helping with the ongoing process of watching for issues and filing
PRs until that part is automated. It will require the effort of
everyone to control and possibly improve compile-time over time.
And everybody is invited to root cause past regressions.</div>
<div class=""><br class=""></div>
<div class="">
<div class="">The need for this work has been recognized by the
communities and also externally. It has been observed more than
once that llvm and clang are getting slower over time, for example
in this thread: <a href="http://lists.llvm.org/pipermail/llvm-dev/2016-March/096491.html" class="">http://lists.llvm.org/pipermail/llvm-dev/2016-March/096491.html</a>.
Phoronix reported recently compile-time slow downs in clang 3.9 vs
3.8: <a href="http://www.phoronix.com/scan.php?page=article&item=llvm-clang-39&num=4" class="">http://www.phoronix.com/scan.php?page=article&item=llvm-clang-39&num=4</a>.
At the LLVM Developer Meeting Michael showed double digit
compile-time increases in clang 3.7 and 3.8 in Os/O3 and O0 in his
Loop Passes presentation (<a href="https://llvmdevelopersmeetingbay2016.sched.org/event/8Z0B/loop-passes-adding-new-features-while-reducing-technical-debt" class="">https://llvmdevelopersmeetingbay2016.sched.org/event/8Z0B/loop-passes-adding-new-features-while-reducing-technical-debt</a>):</div>
<div class=""><span id="cid:0114352C-550B-4B19-B70A-C56A42E85BE7@apple.com" class=""><PastedGraphic-3.png></span></div>
</div>
<div class=""><br class=""></div>
<div class=""><br class=""></div>
<div class="">The selection of 10 tests out of the LLVM tests
invites criticism. The motivation here is to pick something
“reasonable” and adjust as needed moving forward. Specifically this
means running a wider set of tests from time to time and adjust the
tests in CTMark either by adding more tests or by removing
in-effective tests. Internally we also track a set of benchmarks
and check for correlations of compile-time regressions to CTMark.
And most importantly I think the shared interest and commitment of
the community to compile-time will carry this forward.</div>
<div class=""><br class=""></div>
<div class="">Special thanks to Chris and Michael to get this
started!</div>
<div class=""><br class=""></div>
<div class="">-Gerolf</div>
</div>
</div>
_______________________________________________<br class="">
LLVM Developers mailing list<br class="">
<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a><br class="">
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br class="">
</div>
</blockquote>
</div>
<br class=""></div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div></div></span></blockquote></body></html>