[www] r176209 - Add LNT statistics project

[Renato Golin]

On 28 February 2013 19:28, David Blaikie <dblaikie at gmail.com> wrote:

> I'm still confused as to which things you're talking about. My
> suggestion is that we can get higher confidence on the performance of
> tests by running them multiple times.


Ok, now I know why I was confused... and ended up confusing you...

My idea was not to run the whole test multiple times to achieve a nice
mean+sigma, but to increase the number of iterations inside one single run
of the test.

The big problems I find in benchmarks is that the prologue/epilogue
normally is more non-deterministic than the test itself, because they're
based on I/O, setting up the memory (thus populating caches, etc). If you
run multiple times, you'll end up with a small-enough sigma over the whole
test, but not small enough over the prologue/epilogue.

So, going back all the way to the beginning: I propose to spot the core of
each benchmark and force all of them to print times of just that core.

If you need to run a few times to normalize memory/cache, do it before
start timing. If the benchmark is a set of many calls, wrap the working
part of main into a separate function and call it many times. In essence,
everything to make the preparation and shutdown phases nor relevant.

It's not the same as you're proposing at all, and I see value on both
propositions. Yours is a bit easier to implement, and may have good enough
results for the time being. But I think that we should think of a library
to measure time on all platforms (easier said than done), so we can have
multiple time tests on the same benchmark, and possibly simplify the whole
infrastructure in the process (by making the benchmarks more complex).

Now that I got what you mean, I think it's a simple idea that should be
done before my proposition, possibly together with my GSoC project.

Sorry for the confusion, I really got it upside-down. ;)

cheers,
--renato
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20130228/aed53820/attachment.html>