[LNT] r238965 - Improved regression detection with min of diffs

Wed Sep 2 09:27:54 PDT 2015

Hi Chris,

This change has some questionable behavior and works pretty poorly when
dealing with instances that have reasonable sample counts per run (e.g.,
10).

Given this ComparisonResult:
  ComparisonResult(safe_min, False, False, [0.873164, 0.822148, 0.877115,
0.830048, 0.830452, 0.897407, 0.817558, 0.842067, 0.836447, 0.875904],
[0.695299, 0.798883, 0.676503, 0.671114, 0.648465, 0.653936, 0.673405,
0.646322, 0.654594, 0.667564], 0.05, False, )
where all the previous samples are from the immediately preceding run (not
from multiple runs), then this ends up computing:
--
cr.current == 0.8176
cr.previous == 0.6463
cr.delta == 0.0187
cr.pct_delta == 2.34%
--
which doesn't make any sense to the user who is just seeing those values
and wondering where the strange delta value came from.

If we are going to keep this behavior, then I think that:
a. The cr.previous value should be updated to be the selected value, so
that all the values are at least consistent in the UI.
b. This behavior should only be enabled when using prior samples across
runs.

I would also prefer this was modeled as a different "aggregation function"
that shows in the UI, and maybe make that an per-instance default.

 - Daniel

On Wed, Jun 3, 2015 at 2:01 PM, Chris Matthews <cmatthews5 at apple.com> wrote:

> Author: cmatthews
> Date: Wed Jun  3 16:01:23 2015
> New Revision: 238965
>
> URL: http://llvm.org/viewvc/llvm-project?rev=238965&view=rev
> Log:
> Improved regression detection with min of diffs
>
> This change reduces false positives in LNT's regression detection
> algorithm by having it use more past data.  Instead of calculating the
> delta as the difference of min(current)-min(prev), we use the smallest
> difference between min(current) and all prevs.  When the data has noise
> or is multimodal this reduces our chances of flagging a false positive
> regression.
>
> Modified:
>     lnt/trunk/lnt/server/reporting/analysis.py
>     lnt/trunk/tests/server/reporting/analysis.py
>
> Modified: lnt/trunk/lnt/server/reporting/analysis.py
> URL:
> http://llvm.org/viewvc/llvm-project/lnt/trunk/lnt/server/reporting/analysis.py?rev=238965&r1=238964&r2=238965&view=diff
>
> ==============================================================================
> --- lnt/trunk/lnt/server/reporting/analysis.py (original)
> +++ lnt/trunk/lnt/server/reporting/analysis.py Wed Jun  3 16:01:23 2015
> @@ -15,6 +15,21 @@ UNCHANGED_FAIL = 'UNCHANGED_FAIL'
>  MIN_VALUE_PRECISION = 0.0001
>
>
> +def absmin_diff(current, prevs):
> +    """Min of differences between current sample and all previous samples.
> +    Given more than one min, use the last one detected which is probably a
> +    newer value. Returns (difference, prev used)
> +    """
> +    diffs = [abs(current-prev) for prev in prevs]
> +    smallest_pos = 0
> +    smallest = diffs[0]
> +    for i, diff in enumerate(diffs):
> +        if diff <= smallest:
> +            smallest = diff
> +            smallest_pos = i
> +    return current-prevs[smallest_pos], prevs[smallest_pos]
> +
> +
>  def calc_geomean(run_values):
>      # NOTE Geometric mean applied only to positive values, so fix it by
>      # adding MIN_VALUE to each value and substract it from the result.
> @@ -48,8 +63,8 @@ class ComparisonResult:
>
>          # Compute the comparison status for the test value.
>          if self.current and self.previous and self.previous != 0:
> -            self.delta = self.current - self.previous
> -            self.pct_delta = self.delta / self.previous
> +            self.delta, value = absmin_diff(self.current, prev_samples)
> +            self.pct_delta = self.delta / value
>          else:
>              self.delta = 0
>              self.pct_delta = 0.0
>
> Modified: lnt/trunk/tests/server/reporting/analysis.py
> URL:
> http://llvm.org/viewvc/llvm-project/lnt/trunk/tests/server/reporting/analysis.py?rev=238965&r1=238964&r2=238965&view=diff
>
> ==============================================================================
> --- lnt/trunk/tests/server/reporting/analysis.py (original)
> +++ lnt/trunk/tests/server/reporting/analysis.py Wed Jun  3 16:01:23 2015
> @@ -2,8 +2,10 @@
>  #
>  # RUN: python %s
>  import unittest
> +
>  from lnt.server.reporting.analysis import ComparisonResult, REGRESSED,
> IMPROVED
>  from lnt.server.reporting.analysis import UNCHANGED_PASS, UNCHANGED_FAIL
> +from lnt.server.reporting.analysis import absmin_diff
>
>  FLAT_LINE = [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0,
> 1.0,
>               1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
> @@ -309,5 +311,18 @@ class ComparisonResultTest(unittest.Test
>          # Fixme
>          # self.assertEquals(slow.get_value_status(), IMPROVED)
>
> +
> +class AbsMinTester(unittest.TestCase):
> +
> +    def test_absmin(self):
> +        """Test finding smallest difference."""
> +        self.assertEqual(absmin_diff(1, [2, 2, 3]), (-1, 2))
> +        self.assertEqual(absmin_diff(1, [1, 2, 3]), (0, 1))
> +        self.assertEqual(absmin_diff(1, [2]), (-1, 2))
> +        self.assertEqual(absmin_diff(1.5, [1, 4, 4]), (0.5, 1))
> +        self.assertEqual(absmin_diff(5, [1, 2, 1]), (3, 2))
> +        self.assertEqual(absmin_diff(1, [2, 0, 3]), (1, 0))
> +
> +
>  if __name__ == '__main__':
>      unittest.main()
>
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20150902/e0c68451/attachment.html>