[LNT] r238965 - Improved regression detection with min of diffs
James Molloy via llvm-commits
llvm-commits at lists.llvm.org
Wed Sep 2 10:33:35 PDT 2015
Also, doesn't this make the regression detection algorithm very vulnerable
to transient spikes (perhaps due to wonky data) or large compiler
improvements? A one-off difference shouldn't affect the difference computed
on aggregate.
On Wed, 2 Sep 2015 at 17:28 Daniel Dunbar via llvm-commits <
llvm-commits at lists.llvm.org> wrote:
> Hi Chris,
>
> This change has some questionable behavior and works pretty poorly when
> dealing with instances that have reasonable sample counts per run (e.g.,
> 10).
>
> Given this ComparisonResult:
> ComparisonResult(safe_min, False, False, [0.873164, 0.822148, 0.877115,
> 0.830048, 0.830452, 0.897407, 0.817558, 0.842067, 0.836447, 0.875904],
> [0.695299, 0.798883, 0.676503, 0.671114, 0.648465, 0.653936, 0.673405,
> 0.646322, 0.654594, 0.667564], 0.05, False, )
> where all the previous samples are from the immediately preceding run (not
> from multiple runs), then this ends up computing:
> --
> cr.current == 0.8176
> cr.previous == 0.6463
> cr.delta == 0.0187
> cr.pct_delta == 2.34%
> --
> which doesn't make any sense to the user who is just seeing those values
> and wondering where the strange delta value came from.
>
> If we are going to keep this behavior, then I think that:
> a. The cr.previous value should be updated to be the selected value, so
> that all the values are at least consistent in the UI.
> b. This behavior should only be enabled when using prior samples across
> runs.
>
> I would also prefer this was modeled as a different "aggregation function"
> that shows in the UI, and maybe make that an per-instance default.
>
> - Daniel
>
>
> On Wed, Jun 3, 2015 at 2:01 PM, Chris Matthews <cmatthews5 at apple.com>
> wrote:
>
>> Author: cmatthews
>> Date: Wed Jun 3 16:01:23 2015
>> New Revision: 238965
>>
>> URL: http://llvm.org/viewvc/llvm-project?rev=238965&view=rev
>> Log:
>> Improved regression detection with min of diffs
>>
>> This change reduces false positives in LNT's regression detection
>> algorithm by having it use more past data. Instead of calculating the
>> delta as the difference of min(current)-min(prev), we use the smallest
>> difference between min(current) and all prevs. When the data has noise
>> or is multimodal this reduces our chances of flagging a false positive
>> regression.
>>
>> Modified:
>> lnt/trunk/lnt/server/reporting/analysis.py
>> lnt/trunk/tests/server/reporting/analysis.py
>>
>> Modified: lnt/trunk/lnt/server/reporting/analysis.py
>> URL:
>> http://llvm.org/viewvc/llvm-project/lnt/trunk/lnt/server/reporting/analysis.py?rev=238965&r1=238964&r2=238965&view=diff
>>
>> ==============================================================================
>> --- lnt/trunk/lnt/server/reporting/analysis.py (original)
>> +++ lnt/trunk/lnt/server/reporting/analysis.py Wed Jun 3 16:01:23 2015
>> @@ -15,6 +15,21 @@ UNCHANGED_FAIL = 'UNCHANGED_FAIL'
>> MIN_VALUE_PRECISION = 0.0001
>>
>>
>> +def absmin_diff(current, prevs):
>> + """Min of differences between current sample and all previous
>> samples.
>> + Given more than one min, use the last one detected which is probably
>> a
>> + newer value. Returns (difference, prev used)
>> + """
>> + diffs = [abs(current-prev) for prev in prevs]
>> + smallest_pos = 0
>> + smallest = diffs[0]
>> + for i, diff in enumerate(diffs):
>> + if diff <= smallest:
>> + smallest = diff
>> + smallest_pos = i
>> + return current-prevs[smallest_pos], prevs[smallest_pos]
>> +
>> +
>> def calc_geomean(run_values):
>> # NOTE Geometric mean applied only to positive values, so fix it by
>> # adding MIN_VALUE to each value and substract it from the result.
>> @@ -48,8 +63,8 @@ class ComparisonResult:
>>
>> # Compute the comparison status for the test value.
>> if self.current and self.previous and self.previous != 0:
>> - self.delta = self.current - self.previous
>> - self.pct_delta = self.delta / self.previous
>> + self.delta, value = absmin_diff(self.current, prev_samples)
>> + self.pct_delta = self.delta / value
>> else:
>> self.delta = 0
>> self.pct_delta = 0.0
>>
>> Modified: lnt/trunk/tests/server/reporting/analysis.py
>> URL:
>> http://llvm.org/viewvc/llvm-project/lnt/trunk/tests/server/reporting/analysis.py?rev=238965&r1=238964&r2=238965&view=diff
>>
>> ==============================================================================
>> --- lnt/trunk/tests/server/reporting/analysis.py (original)
>> +++ lnt/trunk/tests/server/reporting/analysis.py Wed Jun 3 16:01:23 2015
>> @@ -2,8 +2,10 @@
>> #
>> # RUN: python %s
>> import unittest
>> +
>> from lnt.server.reporting.analysis import ComparisonResult, REGRESSED,
>> IMPROVED
>> from lnt.server.reporting.analysis import UNCHANGED_PASS, UNCHANGED_FAIL
>> +from lnt.server.reporting.analysis import absmin_diff
>>
>> FLAT_LINE = [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0,
>> 1.0,
>> 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
>> @@ -309,5 +311,18 @@ class ComparisonResultTest(unittest.Test
>> # Fixme
>> # self.assertEquals(slow.get_value_status(), IMPROVED)
>>
>> +
>> +class AbsMinTester(unittest.TestCase):
>> +
>> + def test_absmin(self):
>> + """Test finding smallest difference."""
>> + self.assertEqual(absmin_diff(1, [2, 2, 3]), (-1, 2))
>> + self.assertEqual(absmin_diff(1, [1, 2, 3]), (0, 1))
>> + self.assertEqual(absmin_diff(1, [2]), (-1, 2))
>> + self.assertEqual(absmin_diff(1.5, [1, 4, 4]), (0.5, 1))
>> + self.assertEqual(absmin_diff(5, [1, 2, 1]), (3, 2))
>> + self.assertEqual(absmin_diff(1, [2, 0, 3]), (1, 0))
>> +
>> +
>> if __name__ == '__main__':
>> unittest.main()
>>
>>
>> _______________________________________________
>> llvm-commits mailing list
>> llvm-commits at cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>>
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20150902/7befe352/attachment.html>
More information about the llvm-commits
mailing list