<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">No I don’t think so, it makes the regression algorithm less vulnerable to both spikes and drops. The old approach with mins would ignore a spike, but trigger on a single point drop. Now a single sample that is largely different is completely ignored. Large improvements are also detected, except in the case where the data is bimodal, and the lesser mode is near the upper mode’s improvement.</div><div class=""><br class=""></div><div class=""><br class=""></div><br class=""><div><blockquote type="cite" class=""><div class="">On Sep 2, 2015, at 10:33 AM, James Molloy <<a href="mailto:james@jamesmolloy.co.uk" class="">james@jamesmolloy.co.uk</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Also, doesn't this make the regression detection algorithm very vulnerable to transient spikes (perhaps due to wonky data) or large compiler improvements? A one-off difference shouldn't affect the difference computed on aggregate.</div><br class=""><div class="gmail_quote"><div dir="ltr" class="">On Wed, 2 Sep 2015 at 17:28 Daniel Dunbar via llvm-commits <<a href="mailto:llvm-commits@lists.llvm.org" class="">llvm-commits@lists.llvm.org</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class=""><div class="">Hi Chris,</div><div class=""><br class=""></div><div class="">This change has some questionable behavior and works pretty poorly when dealing with instances that have reasonable sample counts per run (e.g., 10).</div><div class=""><br class=""></div><div class="">Given this ComparisonResult:</div><div class=""> ComparisonResult(safe_min, False, False, [0.873164, 0.822148, 0.877115, 0.830048, 0.830452, 0.897407, 0.817558, 0.842067, 0.836447, 0.875904], [0.695299, 0.798883, 0.676503, 0.671114, 0.648465, 0.653936, 0.673405, 0.646322, 0.654594, 0.667564], 0.05, False, )</div><div class="">where all the previous samples are from the immediately preceding run (not from multiple runs), then this ends up computing:</div><div class="">--</div><div class="">cr.current == 0.8176</div><div class="">cr.previous == 0.6463</div><div class="">cr.delta == 0.0187</div><div class="">cr.pct_delta == 2.34%</div><div class="">--</div><div class="">which doesn't make any sense to the user who is just seeing those values and wondering where the strange delta value came from.</div><div class=""><br class=""></div><div class="">If we are going to keep this behavior, then I think that:</div><div class="">a. The cr.previous value should be updated to be the selected value, so that all the values are at least consistent in the UI.</div><div class="">b. This behavior should only be enabled when using prior samples across runs.</div><div class=""><br class=""></div><div class="">I would also prefer this was modeled as a different "aggregation function" that shows in the UI, and maybe make that an per-instance default.</div><div class=""><br class=""></div><div class=""> - Daniel</div><div class=""><br class=""></div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Wed, Jun 3, 2015 at 2:01 PM, Chris Matthews <span dir="ltr" class=""><<a href="mailto:cmatthews5@apple.com" target="_blank" class="">cmatthews5@apple.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Author: cmatthews<br class="">
Date: Wed Jun 3 16:01:23 2015<br class="">
New Revision: 238965<br class="">
<br class="">
URL: <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__llvm.org_viewvc_llvm-2Dproject-3Frev-3D238965-26view-3Drev&d=BQMFaQ&c=eEvniauFctOgLOKGJOplqw&r=sCYjGfqxzUYZGf3nWUM6D263xFvlkMzkQU_bdBSE1DY&m=wsyRVVznnryv8J12BgkYv1yGstMzJAc2yYTE6zHmSk0&s=rqEVThyTpkHLe5uAH7n1zuRy9iFYSbtA61VMWihwlrU&e=" rel="noreferrer" target="_blank" class="">http://llvm.org/viewvc/llvm-project?rev=238965&view=rev</a><br class="">
Log:<br class="">
Improved regression detection with min of diffs<br class="">
<br class="">
This change reduces false positives in LNT's regression detection<br class="">
algorithm by having it use more past data. Instead of calculating the<br class="">
delta as the difference of min(current)-min(prev), we use the smallest<br class="">
difference between min(current) and all prevs. When the data has noise<br class="">
or is multimodal this reduces our chances of flagging a false positive<br class="">
regression.<br class="">
<br class="">
Modified:<br class="">
lnt/trunk/lnt/server/reporting/analysis.py<br class="">
lnt/trunk/tests/server/reporting/analysis.py<br class="">
<br class="">
Modified: lnt/trunk/lnt/server/reporting/analysis.py<br class="">
URL: <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__llvm.org_viewvc_llvm-2Dproject_lnt_trunk_lnt_server_reporting_analysis.py-3Frev-3D238965-26r1-3D238964-26r2-3D238965-26view-3Ddiff&d=BQMFaQ&c=eEvniauFctOgLOKGJOplqw&r=sCYjGfqxzUYZGf3nWUM6D263xFvlkMzkQU_bdBSE1DY&m=wsyRVVznnryv8J12BgkYv1yGstMzJAc2yYTE6zHmSk0&s=_6z2OunP4brgSkGgYGq-qd0mVVW2v1aMLysorzU_QBg&e=" rel="noreferrer" target="_blank" class="">http://llvm.org/viewvc/llvm-project/lnt/trunk/lnt/server/reporting/analysis.py?rev=238965&r1=238964&r2=238965&view=diff</a><br class="">
==============================================================================<br class="">
--- lnt/trunk/lnt/server/reporting/analysis.py (original)<br class="">
+++ lnt/trunk/lnt/server/reporting/analysis.py Wed Jun 3 16:01:23 2015<br class="">
@@ -15,6 +15,21 @@ UNCHANGED_FAIL = 'UNCHANGED_FAIL'<br class="">
MIN_VALUE_PRECISION = 0.0001<br class="">
<br class="">
<br class="">
+def absmin_diff(current, prevs):<br class="">
+ """Min of differences between current sample and all previous samples.<br class="">
+ Given more than one min, use the last one detected which is probably a<br class="">
+ newer value. Returns (difference, prev used)<br class="">
+ """<br class="">
+ diffs = [abs(current-prev) for prev in prevs]<br class="">
+ smallest_pos = 0<br class="">
+ smallest = diffs[0]<br class="">
+ for i, diff in enumerate(diffs):<br class="">
+ if diff <= smallest:<br class="">
+ smallest = diff<br class="">
+ smallest_pos = i<br class="">
+ return current-prevs[smallest_pos], prevs[smallest_pos]<br class="">
+<br class="">
+<br class="">
def calc_geomean(run_values):<br class="">
# NOTE Geometric mean applied only to positive values, so fix it by<br class="">
# adding MIN_VALUE to each value and substract it from the result.<br class="">
@@ -48,8 +63,8 @@ class ComparisonResult:<br class="">
<br class="">
# Compute the comparison status for the test value.<br class="">
if self.current and self.previous and self.previous != 0:<br class="">
- self.delta = self.current - self.previous<br class="">
- self.pct_delta = self.delta / self.previous<br class="">
+ self.delta, value = absmin_diff(self.current, prev_samples)<br class="">
+ self.pct_delta = self.delta / value<br class="">
else:<br class="">
self.delta = 0<br class="">
self.pct_delta = 0.0<br class="">
<br class="">
Modified: lnt/trunk/tests/server/reporting/analysis.py<br class="">
URL: <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__llvm.org_viewvc_llvm-2Dproject_lnt_trunk_tests_server_reporting_analysis.py-3Frev-3D238965-26r1-3D238964-26r2-3D238965-26view-3Ddiff&d=BQMFaQ&c=eEvniauFctOgLOKGJOplqw&r=sCYjGfqxzUYZGf3nWUM6D263xFvlkMzkQU_bdBSE1DY&m=wsyRVVznnryv8J12BgkYv1yGstMzJAc2yYTE6zHmSk0&s=f4bpeqVu2SpwaN3ZuMAiORuuUkL9oa7aH4pZqLoDCy8&e=" rel="noreferrer" target="_blank" class="">http://llvm.org/viewvc/llvm-project/lnt/trunk/tests/server/reporting/analysis.py?rev=238965&r1=238964&r2=238965&view=diff</a><br class="">
==============================================================================<br class="">
--- lnt/trunk/tests/server/reporting/analysis.py (original)<br class="">
+++ lnt/trunk/tests/server/reporting/analysis.py Wed Jun 3 16:01:23 2015<br class="">
@@ -2,8 +2,10 @@<br class="">
#<br class="">
# RUN: python %s<br class="">
import unittest<br class="">
+<br class="">
from lnt.server.reporting.analysis import ComparisonResult, REGRESSED, IMPROVED<br class="">
from lnt.server.reporting.analysis import UNCHANGED_PASS, UNCHANGED_FAIL<br class="">
+from lnt.server.reporting.analysis import absmin_diff<br class="">
<br class="">
FLAT_LINE = [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0,<br class="">
1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]<br class="">
@@ -309,5 +311,18 @@ class ComparisonResultTest(unittest.Test<br class="">
# Fixme<br class="">
# self.assertEquals(slow.get_value_status(), IMPROVED)<br class="">
<br class="">
+<br class="">
+class AbsMinTester(unittest.TestCase):<br class="">
+<br class="">
+ def test_absmin(self):<br class="">
+ """Test finding smallest difference."""<br class="">
+ self.assertEqual(absmin_diff(1, [2, 2, 3]), (-1, 2))<br class="">
+ self.assertEqual(absmin_diff(1, [1, 2, 3]), (0, 1))<br class="">
+ self.assertEqual(absmin_diff(1, [2]), (-1, 2))<br class="">
+ self.assertEqual(absmin_diff(1.5, [1, 4, 4]), (0.5, 1))<br class="">
+ self.assertEqual(absmin_diff(5, [1, 2, 1]), (3, 2))<br class="">
+ self.assertEqual(absmin_diff(1, [2, 0, 3]), (1, 0))<br class="">
+<br class="">
+<br class="">
if __name__ == '__main__':<br class="">
unittest.main()<br class="">
<br class="">
<br class="">
_______________________________________________<br class="">
llvm-commits mailing list<br class="">
<a href="mailto:llvm-commits@cs.uiuc.edu" target="_blank" class="">llvm-commits@cs.uiuc.edu</a><br class="">
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.cs.uiuc.edu_mailman_listinfo_llvm-2Dcommits&d=BQMFaQ&c=eEvniauFctOgLOKGJOplqw&r=sCYjGfqxzUYZGf3nWUM6D263xFvlkMzkQU_bdBSE1DY&m=wsyRVVznnryv8J12BgkYv1yGstMzJAc2yYTE6zHmSk0&s=9pJrOn6nX0Xi_wSfTb722WQfcw1euEyRZFzG-AG8MCQ&e=" rel="noreferrer" target="_blank" class="">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a><br class="">
</blockquote></div><br class=""></div>
_______________________________________________<br class="">
llvm-commits mailing list<br class="">
<a href="mailto:llvm-commits@lists.llvm.org" target="_blank" class="">llvm-commits@lists.llvm.org</a><br class="">
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.llvm.org_cgi-2Dbin_mailman_listinfo_llvm-2Dcommits&d=BQMFaQ&c=eEvniauFctOgLOKGJOplqw&r=sCYjGfqxzUYZGf3nWUM6D263xFvlkMzkQU_bdBSE1DY&m=wsyRVVznnryv8J12BgkYv1yGstMzJAc2yYTE6zHmSk0&s=wQRSiZEjIWuBqIAOhNk8wcFE9H6GsHnydq6CLXqLQc8&e=" rel="noreferrer" target="_blank" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits</a><br class="">
</blockquote></div>
</div></blockquote></div><br class=""></body></html>