[www] r176209 - Add LNT statistics project

[David Tweed]

Hi Renato,
(noise can make you slower, it can't make you faster)

| I disagree.

| Noise that makes you slower is context-switch, I/O stalls, device drivers, hypervisors. But there's also noise that makes you much faster, like accidentally well placed cache lines, OS cache for lookups, memory maps, etc.

| What you're calling "program speed" is the fastest the program can go, assuming there is no OS, but when the OS interferes, you can go much faster (open a browser, close and open again) or much slower (as we all know).

I think the complication is that we're using terms slightly differently: I think of things in terms of the "inlier" distribution and the contaminating "outlier" (aka noise) distributions. Things like cache-line placement, memory maps, etc, are part of the variability of the inlier distribution that you're looking to get a picture of. The outliers/noise is what is obscuring what you're interested in, and in an ideal world you'd just be able to "magically" remove its effects. In the same sense, the "program speed" is not one number, it is fundamentally a distribution (varying with all the factors that are inextricably part of runnig the code), although one might end up summarising it with a median or mean in further tests. With this view then noise can only ever slow you down. But I can see that there'd be a view where any variation is viewed as noise one could have noise that makes things faster.

| Bare metal noise's standard deviation up (slow) is definitely higher than down (faster), but when OSs are in play, especially with out-of-order CPUs like the A9, I wouldn't jump to conclusions.

Again, I think it's a question of perspective: is a CPU reordering instructions an intrinsic part of the variability in how a given piece of compiled code runs, or is it noise? (Obviously I go for the former view.)

> But to be honest I'm sure someone must have figured out the pros and cons of various stats for benchmarking

| Benchmarking is art, not science. Each with its own school(s), ideas, beliefs. We could devote our entire lifetimes discussing this and we'd still be arguing. ;)

I'm afraid I've got to disagree with you on the opening sentence: benchmarking is a science in the sense that once a user has decided what goals/properties they consider important, then one set of statistical procedure can be determined to be better at achieving those goals than another. It's perfectly fine for different people to disagree about what goals/properties are important, but for a given choice there should be a best set. Unfortunately I wish I could be certain a bit more about what properties the various statistical estimators have on the kind of data you'd get in benchmarking. (Reason that I think the statistics is important is precisely because once it's finalised it's going to be left to run without looking at the actual raw data -- at least until it throws up something that's deemed worthy of investigation. As such, you wouldn't spot if it was generating non-informative results.)

However, in a bigger sense all this appears to be a second order issue. Talking to one of the guys here at ARM, it looks like issues such as code & data alignment tend to be random but historically correlated, so that just running your benchmark multiple times won't tend to change the initial ones chosen very much; as such the "inlier" distribution you get one day can differ from the "inlier" distribution you get another day, so unless you explicitly try to control it that can swamp your results anyway.

Cheers,
Dave

-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20130301/73506c4a/attachment.html>