<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hopefully fixed in r238974, or maybe now detectable by r238973 which checks we are being sane. If it continues, can you grab me a “print locals()” inside the start of ComparisonResult.__init__. I know the geomean calculation likes to pass Nones around and lists with None inside, but I don’t know what cases that is expected. No tests in that part of the code. <div class=""><br class=""></div><div class="">I might refactor it a little more if I can get a handle on all the cases where Nones are getting passed through.<br class=""><div class=""><br class=""></div><div class=""><div><blockquote type="cite" class=""><div class="">On Jun 3, 2015, at 7:20 AM, Chris Matthews <<a href="mailto:cmatthews5@apple.com" class="">cmatthews5@apple.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="content-type" content="text/html; charset=utf-8" class=""><div dir="auto" class=""><div class="">This looks familiar. I wonder if I fixed only locally. Will fix today!<br class=""><br class="">Sent from my iPhone</div><div class=""><br class="">On Jun 3, 2015, at 7:16 AM, James Molloy <<a href="mailto:james@jamesmolloy.co.uk" class="">james@jamesmolloy.co.uk</a>> wrote:<br class=""><br class=""></div><blockquote type="cite" class=""><div class=""><div dir="ltr" class="">Hi Chris,<br class=""><br class="">I just updated our LNT instance and this commit causes the following internal server error:<br class=""><div class=""><br class=""></div><div class="">... snip ...</div><div class=""><div class=""> File "/home/llvm-test/sandbox/lnt/lnt/server/ui/templates/v4_run.html", line 334, in block "body"</div><div class=""> {% set cr = request_info.sri.get_geomean_comparison_result(</div><div class=""> File "/home/llvm-test/sandbox/lnt/lnt/server/reporting/analysis.py", line 286, in get_geomean_comparison_result</div><div class=""> bigger_is_better=field.bigger_is_better)</div><div class=""> File "/home/llvm-test/sandbox/lnt/lnt/server/reporting/analysis.py", line 60, in __init__</div><div class=""> self.stddev = stats.standard_deviation(samples)</div><div class=""> File "/home/llvm-test/sandbox/lnt/lnt/util/stats.py", line 36, in standard_deviation</div><div class=""> m = mean(l)</div><div class=""> File "/home/llvm-test/sandbox/lnt/lnt/util/stats.py", line 15, in mean</div><div class=""> return sum(l)/len(l)</div><div class="">TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'</div></div><div class=""><br class=""></div><div class="">Could you please take a look?</div><div class=""><br class=""></div><div class="">Cheers,</div><div class=""><br class=""></div><div class="">James</div><div class=""><br class=""></div><div class="">[P.S: if you reply to <a href="mailto:james.molloy@arm.com" class="">james.molloy@arm.com</a> perhaps I might get your reply - I still lay the blame at gmail's forwarding eating your messages]</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="">On Tue, 19 May 2015 at 03:11 Chris Matthews <<a href="mailto:cmatthews5@apple.com" class="">cmatthews5@apple.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Author: cmatthews<br class="">
Date: Mon May 18 20:59:20 2015<br class="">
New Revision: 237658<br class="">
<br class="">
URL: <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__llvm.org_viewvc_llvm-2Dproject-3Frev-3D237658-26view-3Drev&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=mQ4LZ2PUj9hpadE3cDHZnIdEwhEBrbAstXeMaFoB9tg&m=hlSrqTcmRBON9OCba2v2Zn0fIsNvy5cwSdLlBAkG2wo&s=dp8UqojU4x-vOe92vDtq0jOvYKDLqUFHYfOPEOw1V_0&e=" target="_blank" class="">http://llvm.org/viewvc/llvm-project?rev=237658&view=rev</a><br class="">
Log:<br class="">
Refactor ComparisonResult to be more self contained<br class="">
<br class="">
Modified:<br class="">
lnt/trunk/lnt/server/reporting/analysis.py<br class="">
lnt/trunk/lnt/server/ui/templates/v4_run.html<br class="">
lnt/trunk/tests/server/reporting/analysis.py<br class="">
<br class="">
Modified: lnt/trunk/lnt/server/reporting/analysis.py<br class="">
URL: <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__llvm.org_viewvc_llvm-2Dproject_lnt_trunk_lnt_server_reporting_analysis.py-3Frev-3D237658-26r1-3D237657-26r2-3D237658-26view-3Ddiff&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=mQ4LZ2PUj9hpadE3cDHZnIdEwhEBrbAstXeMaFoB9tg&m=hlSrqTcmRBON9OCba2v2Zn0fIsNvy5cwSdLlBAkG2wo&s=a94GveESXxCe1huABIe9cbIWAHXj7YQp3fMDT79zatk&e=" target="_blank" class="">http://llvm.org/viewvc/llvm-project/lnt/trunk/lnt/server/reporting/analysis.py?rev=237658&r1=237657&r2=237658&view=diff</a><br class="">
==============================================================================<br class="">
--- lnt/trunk/lnt/server/reporting/analysis.py (original)<br class="">
+++ lnt/trunk/lnt/server/reporting/analysis.py Mon May 18 20:59:20 2015<br class="">
@@ -31,43 +31,67 @@ def calc_geomean(run_values):<br class="">
class ComparisonResult:<br class="">
"""A ComparisonResult is ultimatly responsible for determining if a test<br class="">
improves, regresses or does not change, given some new and old data."""<br class="">
-<br class="">
- def __init__(self,cur_value, prev_value, delta, pct_delta, stddev, MAD,<br class="">
- cur_failed, prev_failed, samples, prev_samples, stddev_mean = None,<br class="">
- confidence_lv = .05, bigger_is_better = False):<br class="">
- self.current = cur_value<br class="">
- self.previous = prev_value<br class="">
- self.delta = delta<br class="">
- self.pct_delta = pct_delta<br class="">
- self.stddev = stddev<br class="">
- self.MAD = MAD<br class="">
+<br class="">
+ def __init__(self, aggregation_fn,<br class="">
+ cur_failed, prev_failed, samples, prev_samples,<br class="">
+ confidence_lv=0.05, bigger_is_better=False):<br class="">
+ self.aggregation_fn = aggregation_fn<br class="">
+ if samples:<br class="">
+ self.current = aggregation_fn(samples)<br class="">
+ else:<br class="">
+ self.current = None<br class="">
+ if prev_samples:<br class="">
+ self.previous = aggregation_fn(prev_samples)<br class="">
+ else:<br class="">
+ self.previous = None<br class="">
+<br class="">
+ # Compute the comparison status for the test value.<br class="">
+ if self.current and self.previous and self.previous != 0:<br class="">
+ self.delta = self.current - self.previous<br class="">
+ self.pct_delta = self.delta / self.previous<br class="">
+ else:<br class="">
+ self.delta = 0<br class="">
+ self.pct_delta = 0.0<br class="">
+<br class="">
+ # If we have multiple values for this run, use that to estimate the<br class="">
+ # distribution.<br class="">
+ if samples and len(samples) > 1:<br class="">
+ self.stddev = stats.standard_deviation(samples)<br class="">
+ self.MAD = stats.median_absolute_deviation(samples)<br class="">
+ else:<br class="">
+ self.stddev = None<br class="">
+ self.MAD = None<br class="">
+<br class="">
+ self.stddev_mean = None # Only calculate this if needed.<br class="">
self.failed = cur_failed<br class="">
self.prev_failed = prev_failed<br class="">
self.samples = samples<br class="">
self.prev_samples = prev_samples<br class="">
- self.stddev_mean = stddev_mean<br class="">
+<br class="">
self.confidence_lv = confidence_lv<br class="">
self.bigger_is_better = bigger_is_better<br class="">
<br class="">
+ @property<br class="">
+ def stddev_mean(self):<br class="">
+ """The mean around stddev for current sampples. Cached after first call.<br class="">
+ """<br class="">
+ if not self.stddev_mean:<br class="">
+ self.stddev_mean = stats.mean(self.samples)<br class="">
+ return self.stddev_mean<br class="">
+<br class="">
def __repr__(self):<br class="">
"""Print this ComparisonResult's constructor.<br class="">
-<br class="">
+<br class="">
Handy for generating test cases for comparisons doing odd things."""<br class="">
- frmt = "{}(" + "{}, " * 11 + ")"<br class="">
- return frmt.format("ComparisonResult",<br class="">
- self.current,<br class="">
- self.previous,<br class="">
- self.delta,<br class="">
- self.pct_delta,<br class="">
- self.stddev,<br class="">
- self.MAD,<br class="">
- self.failed,<br class="">
- self.prev_failed,<br class="">
- self.samples,<br class="">
- self.prev_samples,<br class="">
- self.stddev_mean,<br class="">
- self.confidence_lv,<br class="">
- self.bigger_is_better)<br class="">
+ fmt = "{}(" + "{}, " * 7 + ")"<br class="">
+ return fmt.format(self.__class__.__name__,<br class="">
+ self.aggregation_fn.__name__,<br class="">
+ self.failed,<br class="">
+ self.prev_failed,<br class="">
+ self.samples,<br class="">
+ self.prev_samples,<br class="">
+ self.confidence_lv,<br class="">
+ bool(self.bigger_is_better))<br class="">
<br class="">
def is_result_interesting(self):<br class="">
"""is_result_interesting() -> bool<br class="">
@@ -237,77 +261,27 @@ class RunInfo(object):<br class="">
if s[field.index] is not None]<br class="">
prev_values = [s[field.index] for s in prev_samples<br class="">
if s[field.index] is not None]<br class="">
- if run_values:<br class="">
- run_value = self.aggregation_fn(run_values)<br class="">
- else:<br class="">
- run_value = None<br class="">
- if prev_values:<br class="">
- prev_value = self.aggregation_fn(prev_values)<br class="">
- else:<br class="">
- prev_value = None<br class="">
-<br class="">
- # If we have multiple values for this run, use that to estimate the<br class="">
- # distribution.<br class="">
- if run_values and len(run_values) > 1:<br class="">
- stddev = stats.standard_deviation(run_values)<br class="">
- MAD = stats.median_absolute_deviation(run_values)<br class="">
- stddev_mean = stats.mean(run_values)<br class="">
- else:<br class="">
- stddev = None<br class="">
- MAD = None<br class="">
- stddev_mean = None<br class="">
-<br class="">
- # If we are missing current or comparison values we are done.<br class="">
- if run_value is None or prev_value is None:<br class="">
- return ComparisonResult(<br class="">
- run_value, prev_value, delta=None,<br class="">
- pct_delta = None, stddev = stddev, MAD = MAD,<br class="">
- cur_failed = run_failed, prev_failed = prev_failed,<br class="">
- samples = run_values, prev_samples = prev_values,<br class="">
- confidence_lv = self.confidence_lv,<br class="">
- bigger_is_better = field.bigger_is_better)<br class="">
-<br class="">
- # Compute the comparison status for the test value.<br class="">
- delta = run_value - prev_value<br class="">
- if prev_value != 0:<br class="">
- pct_delta = delta / prev_value<br class="">
- else:<br class="">
- pct_delta = 0.0<br class="">
-<br class="">
- return ComparisonResult(run_value, prev_value, delta,<br class="">
- pct_delta, stddev, MAD,<br class="">
- run_failed, prev_failed, run_values,<br class="">
- prev_values, stddev_mean, self.confidence_lv,<br class="">
- bigger_is_better = field.bigger_is_better)<br class="">
-<br class="">
+<br class="">
+ r = ComparisonResult(self.aggregation_fn,<br class="">
+ run_failed, prev_failed, run_values,<br class="">
+ prev_values, self.confidence_lv,<br class="">
+ bigger_is_better=field.bigger_is_better)<br class="">
+ print repr(r)<br class="">
+ return r<br class="">
<br class="">
- def get_geomean_comparison_result(self, run, compare_to, field, tests,<br class="">
- comparison_window=[]):<br class="">
+ def get_geomean_comparison_result(self, run, compare_to, field, tests):<br class="">
if tests:<br class="">
prev_values,run_values = zip(*[(cr.previous,cr.current) for _,_,cr in tests])<br class="">
else:<br class="">
prev_values,run_values = [], []<br class="">
<br class="">
- run_geomean = calc_geomean(run_values)<br class="">
- prev_geomean = calc_geomean(prev_values)<br class="">
-<br class="">
- if run_geomean and prev_geomean:<br class="">
- delta = run_geomean - prev_geomean<br class="">
- if prev_geomean != 0:<br class="">
- pct_delta = delta / prev_geomean<br class="">
- else:<br class="">
- pct_delta = 0.0<br class="">
- else:<br class="">
- delta = pct_delta = 0<br class="">
-<br class="">
- return ComparisonResult(run_geomean, prev_geomean, delta,<br class="">
- pct_delta, stddev=None, MAD=None,<br class="">
- cur_failed=run_values and not run_geomean,<br class="">
- prev_failed=prev_values and not prev_geomean,<br class="">
- samples=[run_geomean] if run_geomean else [],<br class="">
- prev_samples=[prev_geomean] if prev_geomean else [],<br class="">
+ return ComparisonResult(calc_geomean,<br class="">
+ cur_failed=bool(run_values),<br class="">
+ prev_failed=bool(prev_values),<br class="">
+ samples=run_values,<br class="">
+ prev_samples=prev_values,<br class="">
confidence_lv=0,<br class="">
- bigger_is_better = field.bigger_is_better)<br class="">
+ bigger_is_better=field.bigger_is_better)<br class="">
<br class="">
def _load_samples_for_runs(self, run_ids):<br class="">
# Find the set of new runs to load.<br class="">
<br class="">
Modified: lnt/trunk/lnt/server/ui/templates/v4_run.html<br class="">
URL: <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__llvm.org_viewvc_llvm-2Dproject_lnt_trunk_lnt_server_ui_templates_v4-5Frun.html-3Frev-3D237658-26r1-3D237657-26r2-3D237658-26view-3Ddiff&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=mQ4LZ2PUj9hpadE3cDHZnIdEwhEBrbAstXeMaFoB9tg&m=hlSrqTcmRBON9OCba2v2Zn0fIsNvy5cwSdLlBAkG2wo&s=QXpbzkMZse99YZyDX5HVN2Zx9PkBRLGpeRbNT6msGpo&e=" target="_blank" class="">http://llvm.org/viewvc/llvm-project/lnt/trunk/lnt/server/ui/templates/v4_run.html?rev=237658&r1=237657&r2=237658&view=diff</a><br class="">
==============================================================================<br class="">
--- lnt/trunk/lnt/server/ui/templates/v4_run.html (original)<br class="">
+++ lnt/trunk/lnt/server/ui/templates/v4_run.html Mon May 18 20:59:20 2015<br class="">
@@ -332,7 +332,7 @@<br class="">
</tbody><br class="">
<tfoot><br class="">
{% set cr = request_info.sri.get_geomean_comparison_result(<br class="">
- run, compare_to, field, tests, request_info.comparison_window) %}<br class="">
+ run, compare_to, field, tests) %}<br class="">
<td><input type="checkbox" name="mean" value="{{<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__machine.id&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=mQ4LZ2PUj9hpadE3cDHZnIdEwhEBrbAstXeMaFoB9tg&m=hlSrqTcmRBON9OCba2v2Zn0fIsNvy5cwSdLlBAkG2wo&s=gAmmnPAoTUCrKVDFvfJ8x_Aa2oJr3HSn_xkYyghkybw&e=" target="_blank" class="">machine.id</a>}}.{{field.index}}"></td><br class="">
<td><a href="{{graph_base}}&mean={{<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__machine.id&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=mQ4LZ2PUj9hpadE3cDHZnIdEwhEBrbAstXeMaFoB9tg&m=hlSrqTcmRBON9OCba2v2Zn0fIsNvy5cwSdLlBAkG2wo&s=gAmmnPAoTUCrKVDFvfJ8x_Aa2oJr3HSn_xkYyghkybw&e=" target="_blank" class="">machine.id</a>}}.{{field.index}}">Geometric Mean</a></td><br class="">
{{ get_cell_value(cr) }}<br class="">
<br class="">
Modified: lnt/trunk/tests/server/reporting/analysis.py<br class="">
URL: <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__llvm.org_viewvc_llvm-2Dproject_lnt_trunk_tests_server_reporting_analysis.py-3Frev-3D237658-26r1-3D237657-26r2-3D237658-26view-3Ddiff&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=mQ4LZ2PUj9hpadE3cDHZnIdEwhEBrbAstXeMaFoB9tg&m=hlSrqTcmRBON9OCba2v2Zn0fIsNvy5cwSdLlBAkG2wo&s=X3KNE8s_qHD2EVtyXDGw3v6aMKY0Ju48Rkw-hpHTjqI&e=" target="_blank" class="">http://llvm.org/viewvc/llvm-project/lnt/trunk/tests/server/reporting/analysis.py?rev=237658&r1=237657&r2=237658&view=diff</a><br class="">
==============================================================================<br class="">
--- lnt/trunk/tests/server/reporting/analysis.py (original)<br class="">
+++ lnt/trunk/tests/server/reporting/analysis.py Mon May 18 20:59:20 2015<br class="">
@@ -4,7 +4,7 @@<br class="">
import unittest<br class="">
import lnt.util.stats as stats<br class="">
from lnt.server.reporting.analysis import ComparisonResult, REGRESSED, IMPROVED<br class="">
-from lnt.server.reporting.analysis import UNCHANGED_PASS<br class="">
+from lnt.server.reporting.analysis import UNCHANGED_PASS, UNCHANGED_FAIL<br class="">
<br class="">
<br class="">
class ComparisonResultTest(unittest.TestCase):<br class="">
@@ -13,15 +13,8 @@ class ComparisonResultTest(unittest.Test<br class="">
def test_comp(self):<br class="">
"""Test a real example."""<br class="">
curr_samples = [0.0887, 0.0919, 0.0903]<br class="">
- prev = 0.0858<br class="">
- cur = min(curr_samples)<br class="">
- stddev = stats.standard_deviation(curr_samples)<br class="">
- MAD = stats.median_absolute_deviation(curr_samples)<br class="">
- stddev_mean = stats.mean(curr_samples)<br class="">
- uninteresting = ComparisonResult(cur, prev, cur-prev,<br class="">
- (cur-prev)/prev, stddev, MAD,<br class="">
- False, False, curr_samples, [prev],<br class="">
- stddev_mean)<br class="">
+ prev = [0.0858]<br class="">
+ uninteresting = ComparisonResult(min, False, False, curr_samples, prev)<br class="">
<br class="">
self.assertFalse(<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__uninteresting.is&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=mQ4LZ2PUj9hpadE3cDHZnIdEwhEBrbAstXeMaFoB9tg&m=hlSrqTcmRBON9OCba2v2Zn0fIsNvy5cwSdLlBAkG2wo&s=LNXkZ03tcVIIBrsTu6UL5jOhzxYt1FE16ln5zGl35Es&e=" class="">uninteresting.is</a>_result_interesting())<br class="">
self.assertEquals(uninteresting.get_test_status(), UNCHANGED_PASS)<br class="">
@@ -29,31 +22,36 @@ class ComparisonResultTest(unittest.Test<br class="">
<br class="">
def test_slower(self):<br class="">
"""Test getting a simple regression."""<br class="">
- slower = ComparisonResult(10, 5, 5, 0.5, None, None,<br class="">
- False, False, [10], [5], None)<br class="">
+ slower = ComparisonResult(min,<br class="">
+ False, False, [10], [5])<br class="">
self.assertEquals(slower.get_value_status(), REGRESSED)<br class="">
self.assertTrue(<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__slower.is&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=mQ4LZ2PUj9hpadE3cDHZnIdEwhEBrbAstXeMaFoB9tg&m=hlSrqTcmRBON9OCba2v2Zn0fIsNvy5cwSdLlBAkG2wo&s=Z5kxq32Ae5b15BStaNJTr9T0hhnF6FXiedX6JhP_eq8&e=" class="">slower.is</a>_result_interesting())<br class="">
<br class="">
def test_faster(self):<br class="">
"""Test getting a simple improvement."""<br class="">
<br class="">
- faster = ComparisonResult(5, 10, -5, -0.5, None, None,<br class="">
- False, False, [5], [10], None)<br class="">
+ faster = ComparisonResult(min,<br class="">
+ False, False, [5], [10])<br class="">
self.assertEquals(faster.get_value_status(), IMPROVED)<br class="">
self.assertTrue(<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__faster.is&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=mQ4LZ2PUj9hpadE3cDHZnIdEwhEBrbAstXeMaFoB9tg&m=hlSrqTcmRBON9OCba2v2Zn0fIsNvy5cwSdLlBAkG2wo&s=T7kqvyItOJV5W2QAHss-VlNdSKjL4YpLZomO4eECkD0&e=" class="">faster.is</a>_result_interesting())<br class="">
<br class="">
def test_improved_status(self):<br class="">
"""Test getting a test status improvement."""<br class="">
- improved = ComparisonResult(None, None, None, None, None, None,<br class="">
- False, True, [5], [10], None)<br class="">
+ improved = ComparisonResult(min,<br class="">
+ False, True, [1], None)<br class="">
self.assertEquals(improved.get_test_status(), IMPROVED)<br class="">
<br class="">
def test_regressed_status(self):<br class="">
"""Test getting a test status improvement."""<br class="">
- improved = ComparisonResult(None, None, None, None, None, None,<br class="">
- True, False, [5], [10], None)<br class="">
+ improved = ComparisonResult(min,<br class="">
+ True, False, None, [10])<br class="">
self.assertEquals(improved.get_test_status(), REGRESSED)<br class="">
<br class="">
+ def test_keep_on_failing_status(self):<br class="">
+ """Test getting a repeated fail."""<br class="">
+ improved = ComparisonResult(min,<br class="">
+ True, True, None, None)<br class="">
+ self.assertEquals(improved.get_test_status(), UNCHANGED_FAIL)<br class="">
<br class="">
if __name__ == '__main__':<br class="">
unittest.main()<br class="">
<br class="">
<br class="">
_______________________________________________<br class="">
llvm-commits mailing list<br class="">
<a href="mailto:llvm-commits@cs.uiuc.edu" target="_blank" class="">llvm-commits@cs.uiuc.edu</a><br class="">
<a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits" target="_blank" class="">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a><br class="">
</blockquote></div>
</div></blockquote></div>_______________________________________________<br class="">llvm-commits mailing list<br class=""><a href="mailto:llvm-commits@cs.uiuc.edu" class="">llvm-commits@cs.uiuc.edu</a><br class="">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits<br class=""></div></blockquote></div><br class=""></div></div></body></html>