[LNT] r238965 - Improved regression detection with min of diffs
Chris Matthews via llvm-commits
llvm-commits at lists.llvm.org
Wed Sep 2 11:36:56 PDT 2015
Eww, we don’t perform well there at all! The noise in current set jumps up right into the previous range.
I will add this test case to the example regression list.
I agree with a and b. C is a little more work, I will do that sometimes soon.
> On Sep 2, 2015, at 9:27 AM, Daniel Dunbar via llvm-commits <llvm-commits at lists.llvm.org> wrote:
>
> Hi Chris,
>
> This change has some questionable behavior and works pretty poorly when dealing with instances that have reasonable sample counts per run (e.g., 10).
>
> Given this ComparisonResult:
> ComparisonResult(safe_min, False, False, [0.873164, 0.822148, 0.877115, 0.830048, 0.830452, 0.897407, 0.817558, 0.842067, 0.836447, 0.875904], [0.695299, 0.798883, 0.676503, 0.671114, 0.648465, 0.653936, 0.673405, 0.646322, 0.654594, 0.667564], 0.05, False, )
> where all the previous samples are from the immediately preceding run (not from multiple runs), then this ends up computing:
> --
> cr.current == 0.8176
> cr.previous == 0.6463
> cr.delta == 0.0187
> cr.pct_delta == 2.34%
> --
> which doesn't make any sense to the user who is just seeing those values and wondering where the strange delta value came from.
>
> If we are going to keep this behavior, then I think that:
> a. The cr.previous value should be updated to be the selected value, so that all the values are at least consistent in the UI.
> b. This behavior should only be enabled when using prior samples across runs.
>
> I would also prefer this was modeled as a different "aggregation function" that shows in the UI, and maybe make that an per-instance default.
>
> - Daniel
>
>
> On Wed, Jun 3, 2015 at 2:01 PM, Chris Matthews <cmatthews5 at apple.com <mailto:cmatthews5 at apple.com>> wrote:
> Author: cmatthews
> Date: Wed Jun 3 16:01:23 2015
> New Revision: 238965
>
> URL: http://llvm.org/viewvc/llvm-project?rev=238965&view=rev <http://llvm.org/viewvc/llvm-project?rev=238965&view=rev>
> Log:
> Improved regression detection with min of diffs
>
> This change reduces false positives in LNT's regression detection
> algorithm by having it use more past data. Instead of calculating the
> delta as the difference of min(current)-min(prev), we use the smallest
> difference between min(current) and all prevs. When the data has noise
> or is multimodal this reduces our chances of flagging a false positive
> regression.
>
> Modified:
> lnt/trunk/lnt/server/reporting/analysis.py
> lnt/trunk/tests/server/reporting/analysis.py
>
> Modified: lnt/trunk/lnt/server/reporting/analysis.py
> URL: http://llvm.org/viewvc/llvm-project/lnt/trunk/lnt/server/reporting/analysis.py?rev=238965&r1=238964&r2=238965&view=diff <http://llvm.org/viewvc/llvm-project/lnt/trunk/lnt/server/reporting/analysis.py?rev=238965&r1=238964&r2=238965&view=diff>
> ==============================================================================
> --- lnt/trunk/lnt/server/reporting/analysis.py (original)
> +++ lnt/trunk/lnt/server/reporting/analysis.py Wed Jun 3 16:01:23 2015
> @@ -15,6 +15,21 @@ UNCHANGED_FAIL = 'UNCHANGED_FAIL'
> MIN_VALUE_PRECISION = 0.0001
>
>
> +def absmin_diff(current, prevs):
> + """Min of differences between current sample and all previous samples.
> + Given more than one min, use the last one detected which is probably a
> + newer value. Returns (difference, prev used)
> + """
> + diffs = [abs(current-prev) for prev in prevs]
> + smallest_pos = 0
> + smallest = diffs[0]
> + for i, diff in enumerate(diffs):
> + if diff <= smallest:
> + smallest = diff
> + smallest_pos = i
> + return current-prevs[smallest_pos], prevs[smallest_pos]
> +
> +
> def calc_geomean(run_values):
> # NOTE Geometric mean applied only to positive values, so fix it by
> # adding MIN_VALUE to each value and substract it from the result.
> @@ -48,8 +63,8 @@ class ComparisonResult:
>
> # Compute the comparison status for the test value.
> if self.current and self.previous and self.previous != 0:
> - self.delta = self.current - self.previous
> - self.pct_delta = self.delta / self.previous
> + self.delta, value = absmin_diff(self.current, prev_samples)
> + self.pct_delta = self.delta / value
> else:
> self.delta = 0
> self.pct_delta = 0.0
>
> Modified: lnt/trunk/tests/server/reporting/analysis.py
> URL: http://llvm.org/viewvc/llvm-project/lnt/trunk/tests/server/reporting/analysis.py?rev=238965&r1=238964&r2=238965&view=diff <http://llvm.org/viewvc/llvm-project/lnt/trunk/tests/server/reporting/analysis.py?rev=238965&r1=238964&r2=238965&view=diff>
> ==============================================================================
> --- lnt/trunk/tests/server/reporting/analysis.py (original)
> +++ lnt/trunk/tests/server/reporting/analysis.py Wed Jun 3 16:01:23 2015
> @@ -2,8 +2,10 @@
> #
> # RUN: python %s
> import unittest
> +
> from lnt.server.reporting.analysis import ComparisonResult, REGRESSED, IMPROVED
> from lnt.server.reporting.analysis import UNCHANGED_PASS, UNCHANGED_FAIL
> +from lnt.server.reporting.analysis import absmin_diff
>
> FLAT_LINE = [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0,
> 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
> @@ -309,5 +311,18 @@ class ComparisonResultTest(unittest.Test
> # Fixme
> # self.assertEquals(slow.get_value_status(), IMPROVED)
>
> +
> +class AbsMinTester(unittest.TestCase):
> +
> + def test_absmin(self):
> + """Test finding smallest difference."""
> + self.assertEqual(absmin_diff(1, [2, 2, 3]), (-1, 2))
> + self.assertEqual(absmin_diff(1, [1, 2, 3]), (0, 1))
> + self.assertEqual(absmin_diff(1, [2]), (-1, 2))
> + self.assertEqual(absmin_diff(1.5, [1, 4, 4]), (0.5, 1))
> + self.assertEqual(absmin_diff(5, [1, 2, 1]), (3, 2))
> + self.assertEqual(absmin_diff(1, [2, 0, 3]), (1, 0))
> +
> +
> if __name__ == '__main__':
> unittest.main()
>
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at cs.uiuc.edu <mailto:llvm-commits at cs.uiuc.edu>
> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits <http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits>
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20150902/5499786a/attachment.html>
More information about the llvm-commits
mailing list