<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:413281351;
mso-list-type:hybrid;
mso-list-template-ids:-1535626386 621427996 134807555 134807557 134807553 134807555 134807557 134807553 134807555 134807557;}
@list l0:level1
{mso-level-start-at:2;
mso-level-number-format:bullet;
mso-level-text:\F0D8;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;
mso-fareast-font-family:Calibri;
mso-bidi-font-family:"Times New Roman";
color:#1F497D;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi Renato,<o:p></o:p></span></p>
<div>
<div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">(noise can make you slower, it can't make you faster)</span><o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#1F497D">| </span>I disagree.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#1F497D">| </span>Noise that makes you slower is context-switch, I/O stalls, device drivers, hypervisors. But there's also noise that makes you much faster, like accidentally well placed cache lines, OS cache for lookups,
memory maps, etc.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#1F497D">| </span>What you're calling "program speed" is the fastest the program can go, assuming there is no OS, but when the OS interferes, you can go much faster (open a browser, close and open again) or much slower
(as we all know).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I think the complication is that we're using terms slightly differently: I think of things in terms of the "inlier" distribution and the contaminating "outlier"
(aka noise) distributions. Things like cache-line placement, memory maps, etc, are part of the variability of the inlier distribution that you're looking to get a picture of. The outliers/noise is what is obscuring what you're interested in, and in an ideal
world you'd just be able to "magically" remove its effects. In the same sense, the "program speed" is not one number, it is fundamentally a distribution (varying with all the factors that are inextricably part of runnig the code), although one might end up
summarising it with a median or mean in further tests. With this view then noise can only ever slow you down. But I can see that there'd be a view where any variation is viewed as noise one could have noise that makes things faster.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#1F497D">| </span>Bare metal noise's standard deviation up (slow) is definitely higher than down (faster), but when OSs are in play, especially with out-of-order CPUs like the A9, I wouldn't jump to conclusions.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Again, I think it's a question of perspective: is a CPU reordering instructions an intrinsic part of the variability in how a given piece of compiled code runs,
or is it noise? (Obviously I go for the former view.)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">> But to be honest I'm sure someone must have figured out the pros and cons of various stats for benchmarking<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#1F497D">| </span>Benchmarking is art, not science. Each with its own school(s), ideas, beliefs. We could devote our entire lifetimes discussing this and we'd still be arguing. ;)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I'm afraid I've got to disagree with you on the opening sentence: benchmarking is a science in the sense that once a user has decided what goals/properties
they consider important, then one set of statistical procedure can be determined to be better at achieving those goals than another. It's perfectly fine for different people to disagree about what goals/properties are important, but for a given choice there
should be a best set. Unfortunately I wish I could be certain a bit more about what properties the various statistical estimators have on the kind of data you'd get in benchmarking. (Reason that I think the statistics is important is precisely because once
it's finalised it's going to be left to run without looking at the actual raw data -- at least until it throws up something that's deemed worthy of investigation. As such, you wouldn't spot if it was generating non-informative results.)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">However, in a bigger sense all this appears to be a second order issue. Talking to one of the guys here at ARM, it looks like issues such as code & data alignment
tend to be random but historically correlated, so that just running your benchmark multiple times won't tend to change the initial ones chosen very much; as such the "inlier" distribution you get one day can differ from the "inlier" distribution you get another
day, so unless you explicitly try to control it that can swamp your results anyway.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Cheers,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Dave<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
<br>
<font face="Arial" color="Black" size="2">-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents
to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.<br>
</font>
</body>
</html>