[www] r176209 - Add LNT statistics project

[David Tweed]

Hi Renato, sounds like a useful project to have. However, since unlike
correctness regressions there's less time pressure, there are other
statistical measures with greater resolving power you could use if you don't
need to only consider the latest sample, eg, you can use change point
detection tests between 5 samples and the next five samples. (Of course this
will still only produce estimates of when there was a big perf change in one
go, moving averages may be the way to deal with the "0.5% here, 0.5%, ...,
soon it adds up to real slowdowns".

Cheers,
Dave

Modified: www/trunk/OpenProjects.html
URL:
http://llvm.org/viewvc/llvm-project/www/trunk/OpenProjects.html?rev=176209&r
1=176208&r2=176209&view=diff
============================================================================
==
--- www/trunk/OpenProjects.html (original)
+++ www/trunk/OpenProjects.html Wed Feb 27 15:09:57 2013
@@ -11,6 +11,7 @@
     <li><a href="#programs">Compile programs with the LLVM
Compiler</a></li>
     <li><a href="#llvmtest">Add programs to the llvm-test suite</a></li>
     <li><a href="#benchmark">Benchmark the LLVM compiler</a></li>
+    <li><a href="#statistics">Benchmark Statistics and Warning
System</a></li>
     <li><a href="#misc_imp">Miscellaneous Improvements</a></li>
   </ol></li>
 
@@ -201,6 +202,44 @@ have the resources to go finding out why
 
 </div>
 
+
+<!--
======================================================================= -->
+<div class="www_subsubsection">
+  <a name="statistics">Benchmark Statistics and Warning System</a>
+</div>
+
+<div class="www_text">
+
+<p>The <a href='http://llvm.org/perf/db_default/v4/nts/recent_activity'>
+LNT perf database</a> has some nice features like detect moving average,
+standard deviations, variations, etc. But the report page give too much
emphasis
+on the individual variation (where noise can be higher than signal), eg.
+<a
href='http://llvm.org/perf/db_default/v4/nts/graph?plot.0=10.341.3&highlight
_run=8943'>
+this case</a>.</p>
+
+<p>The first part of the project would be to create an analysis tool that
would
+track moving averages and report:
+<ul>
+ <li>If the current result is higher/lower than the previous moving average
by
+     more than (configurable) S standard deviations</li>
+ <li>If the current moving average is more than S standard deviations of
the
+     Base run</li>
+ <li>If the last A moving averages are in constant increase/decrease of
more
+     than P percent</li>
+</ul>
+
+<p>The second part would be to create a web page which would show all
related
+benchmarks (possibly configurable, like a dashboard) and show the basic
statistics
+with red/yellow/green colour codes to show status and links to more
detailed
+analysis of each benchmark.</p>
+
+<p>A possible third part would be to be able to automatically cross
reference
+different builds, so that if you group them by architecture/compiler/number
+of CPUs, this automated tool would understand that the changes are more
common
+to one particular group.</p>
+
+</div>
+
 <!--
======================================================================= -->
 <div class="www_subsubsection">
   <a name="misc_imp">Miscellaneous Improvements</a>


_______________________________________________
llvm-commits mailing list
llvm-commits at cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits