<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p><span style="color:#1F497D">Just some general observations:<o:p></o:p></span></p>
<p><span style="color:#1F497D">Firstly, just to note that when I talk about looking at what statisticians have developed I’m not being snobbish, it’s that pretty much any methodology will show up big effects it’s getting the best “power” on small effects when
you’ve got marginal sample sizes that’s tricky and where a lot of people have already spent a long time thinking about these things.<o:p></o:p></span></p>
<p>On Jun 30, 2013 8:12 PM, "Anton Korobeynikov" <<a href="mailto:anton@korobeynikov.info">anton@korobeynikov.info</a>> wrote:<br>
><br>
> > Getting 10 samples at different commits will give you similar accuracy if<br>
> > behaviour doesn't change, and you can rely on 10-point blocks before and > after each change to have the same result.<br>
> Right. But this way you will have 10-commits delay. So, you will need<br>
> 3-4 additional test runs to pinpoint the offending commit in the worst<br>
> case.<br>
><br>
> > This is why I proposed something like moving averages.<br>
> Moving average will "smooth" the result. So, only really big changes<br>
> will be caught by it.<br>
><o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Just to state the obvious, statistics is best able to detect small effects the fewer extraneous things you try to estimate precisely. So I don’t quite see why an appropriately
robust change-point estimator isn’t what we’d like to use here. (Someone earlier in the thread suggested it wasn’t, but I didn’t follow why.) In such a case you can use the 2-3 results from several consecutive commits in the “before” region and 2-3 results
from several consecutive results in the after region, which seems a reasonable fit for the experimental situation. (My objection to smoothing is just that it’s summarising data before using a statistical test for no good reason, not that tracking samples over
a window seems problematic.)<o:p></o:p></span></p>
<p><span style="color:#1F497D">| </span>Like any result in statistics, the result should be quoted together with a +/- figure derived from the statistical method used. Generally, low sample size means<span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">
</span>high +/-.<span style="color:#1F497D"><o:p></o:p></span></p>
<p><span style="color:#1F497D">“Yes, but...” </span><span style="font-family:Wingdings;color:#1F497D">J</span><span style="color:#1F497D"> That’s absolutely true, but even +/- figures can be overly optimistic/overly pessimistic depending how well the actual
distributions in practice match the assumptions about the distributions implicit in the statistical test. (As you can probably tell, I’m heavily Bayesian and regard statistics as ways of coherently assigning numbers to your beliefs and assumptions, along with
new data, so making assumptions – that are going to be re-examined as things progress -- is fine; objective, assumption-free statistics doesn’t really exist for me.)<o:p></o:p></span></p>
<p><span style="color:#1F497D">Cheers,<o:p></o:p></span></p>
<p><span style="color:#1F497D">Dave<o:p></o:p></span></p>
</div>
<br>
<font face="Arial" color="Black" size="2">-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents
to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.<br>
</font>
</body>
</html>