<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Eww, we don’t perform well there at all! The noise in current set jumps up right into the previous range. <div class=""><br class=""></div><div class="">I will add this test case to the example regression list.</div><div class=""><br class=""></div>I agree with a and b. C is a little more work, I will do that sometimes soon. <div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Sep 2, 2015, at 9:27 AM, Daniel Dunbar via llvm-commits <<a href="mailto:llvm-commits@lists.llvm.org" class="">llvm-commits@lists.llvm.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class="">Hi Chris,</div><div class=""><br class=""></div><div class="">This change has some questionable behavior and works pretty poorly when dealing with instances that have reasonable sample counts per run (e.g., 10).</div><div class=""><br class=""></div><div class="">Given this ComparisonResult:</div><div class=""> ComparisonResult(safe_min, False, False, [0.873164, 0.822148, 0.877115, 0.830048, 0.830452, 0.897407, 0.817558, 0.842067, 0.836447, 0.875904], [0.695299, 0.798883, 0.676503, 0.671114, 0.648465, 0.653936, 0.673405, 0.646322, 0.654594, 0.667564], 0.05, False, )</div><div class="">where all the previous samples are from the immediately preceding run (not from multiple runs), then this ends up computing:</div><div class="">--</div><div class="">cr.current == 0.8176</div><div class="">cr.previous == 0.6463</div><div class="">cr.delta == 0.0187</div><div class="">cr.pct_delta == 2.34%</div><div class="">--</div><div class="">which doesn't make any sense to the user who is just seeing those values and wondering where the strange delta value came from.</div><div class=""><br class=""></div><div class="">If we are going to keep this behavior, then I think that:</div><div class="">a. The cr.previous value should be updated to be the selected value, so that all the values are at least consistent in the UI.</div><div class="">b. This behavior should only be enabled when using prior samples across runs.</div><div class=""><br class=""></div><div class="">I would also prefer this was modeled as a different "aggregation function" that shows in the UI, and maybe make that an per-instance default.</div><div class=""><br class=""></div><div class=""> - Daniel</div><div class=""><br class=""></div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Wed, Jun 3, 2015 at 2:01 PM, Chris Matthews <span dir="ltr" class=""><<a href="mailto:cmatthews5@apple.com" target="_blank" class="">cmatthews5@apple.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Author: cmatthews<br class="">
Date: Wed Jun 3 16:01:23 2015<br class="">
New Revision: 238965<br class="">
<br class="">
URL: <a href="http://llvm.org/viewvc/llvm-project?rev=238965&view=rev" rel="noreferrer" target="_blank" class="">http://llvm.org/viewvc/llvm-project?rev=238965&view=rev</a><br class="">
Log:<br class="">
Improved regression detection with min of diffs<br class="">
<br class="">
This change reduces false positives in LNT's regression detection<br class="">
algorithm by having it use more past data. Instead of calculating the<br class="">
delta as the difference of min(current)-min(prev), we use the smallest<br class="">
difference between min(current) and all prevs. When the data has noise<br class="">
or is multimodal this reduces our chances of flagging a false positive<br class="">
regression.<br class="">
<br class="">
Modified:<br class="">
lnt/trunk/lnt/server/reporting/analysis.py<br class="">
lnt/trunk/tests/server/reporting/analysis.py<br class="">
<br class="">
Modified: lnt/trunk/lnt/server/reporting/analysis.py<br class="">
URL: <a href="http://llvm.org/viewvc/llvm-project/lnt/trunk/lnt/server/reporting/analysis.py?rev=238965&r1=238964&r2=238965&view=diff" rel="noreferrer" target="_blank" class="">http://llvm.org/viewvc/llvm-project/lnt/trunk/lnt/server/reporting/analysis.py?rev=238965&r1=238964&r2=238965&view=diff</a><br class="">
==============================================================================<br class="">
--- lnt/trunk/lnt/server/reporting/analysis.py (original)<br class="">
+++ lnt/trunk/lnt/server/reporting/analysis.py Wed Jun 3 16:01:23 2015<br class="">
@@ -15,6 +15,21 @@ UNCHANGED_FAIL = 'UNCHANGED_FAIL'<br class="">
MIN_VALUE_PRECISION = 0.0001<br class="">
<br class="">
<br class="">
+def absmin_diff(current, prevs):<br class="">
+ """Min of differences between current sample and all previous samples.<br class="">
+ Given more than one min, use the last one detected which is probably a<br class="">
+ newer value. Returns (difference, prev used)<br class="">
+ """<br class="">
+ diffs = [abs(current-prev) for prev in prevs]<br class="">
+ smallest_pos = 0<br class="">
+ smallest = diffs[0]<br class="">
+ for i, diff in enumerate(diffs):<br class="">
+ if diff <= smallest:<br class="">
+ smallest = diff<br class="">
+ smallest_pos = i<br class="">
+ return current-prevs[smallest_pos], prevs[smallest_pos]<br class="">
+<br class="">
+<br class="">
def calc_geomean(run_values):<br class="">
# NOTE Geometric mean applied only to positive values, so fix it by<br class="">
# adding MIN_VALUE to each value and substract it from the result.<br class="">
@@ -48,8 +63,8 @@ class ComparisonResult:<br class="">
<br class="">
# Compute the comparison status for the test value.<br class="">
if self.current and self.previous and self.previous != 0:<br class="">
- self.delta = self.current - self.previous<br class="">
- self.pct_delta = self.delta / self.previous<br class="">
+ self.delta, value = absmin_diff(self.current, prev_samples)<br class="">
+ self.pct_delta = self.delta / value<br class="">
else:<br class="">
self.delta = 0<br class="">
self.pct_delta = 0.0<br class="">
<br class="">
Modified: lnt/trunk/tests/server/reporting/analysis.py<br class="">
URL: <a href="http://llvm.org/viewvc/llvm-project/lnt/trunk/tests/server/reporting/analysis.py?rev=238965&r1=238964&r2=238965&view=diff" rel="noreferrer" target="_blank" class="">http://llvm.org/viewvc/llvm-project/lnt/trunk/tests/server/reporting/analysis.py?rev=238965&r1=238964&r2=238965&view=diff</a><br class="">
==============================================================================<br class="">
--- lnt/trunk/tests/server/reporting/analysis.py (original)<br class="">
+++ lnt/trunk/tests/server/reporting/analysis.py Wed Jun 3 16:01:23 2015<br class="">
@@ -2,8 +2,10 @@<br class="">
#<br class="">
# RUN: python %s<br class="">
import unittest<br class="">
+<br class="">
from lnt.server.reporting.analysis import ComparisonResult, REGRESSED, IMPROVED<br class="">
from lnt.server.reporting.analysis import UNCHANGED_PASS, UNCHANGED_FAIL<br class="">
+from lnt.server.reporting.analysis import absmin_diff<br class="">
<br class="">
FLAT_LINE = [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0,<br class="">
1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]<br class="">
@@ -309,5 +311,18 @@ class ComparisonResultTest(unittest.Test<br class="">
# Fixme<br class="">
# self.assertEquals(slow.get_value_status(), IMPROVED)<br class="">
<br class="">
+<br class="">
+class AbsMinTester(unittest.TestCase):<br class="">
+<br class="">
+ def test_absmin(self):<br class="">
+ """Test finding smallest difference."""<br class="">
+ self.assertEqual(absmin_diff(1, [2, 2, 3]), (-1, 2))<br class="">
+ self.assertEqual(absmin_diff(1, [1, 2, 3]), (0, 1))<br class="">
+ self.assertEqual(absmin_diff(1, [2]), (-1, 2))<br class="">
+ self.assertEqual(absmin_diff(1.5, [1, 4, 4]), (0.5, 1))<br class="">
+ self.assertEqual(absmin_diff(5, [1, 2, 1]), (3, 2))<br class="">
+ self.assertEqual(absmin_diff(1, [2, 0, 3]), (1, 0))<br class="">
+<br class="">
+<br class="">
if __name__ == '__main__':<br class="">
unittest.main()<br class="">
<br class="">
<br class="">
_______________________________________________<br class="">
llvm-commits mailing list<br class="">
<a href="mailto:llvm-commits@cs.uiuc.edu" class="">llvm-commits@cs.uiuc.edu</a><br class="">
<a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits" rel="noreferrer" target="_blank" class="">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a><br class="">
</blockquote></div><br class=""></div>
_______________________________________________<br class="">llvm-commits mailing list<br class=""><a href="mailto:llvm-commits@lists.llvm.org" class="">llvm-commits@lists.llvm.org</a><br class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits<br class=""></div></blockquote></div><br class=""></div></body></html>