[www] r176209 - Add LNT statistics project

[David Blaikie]

On Thu, Feb 28, 2013 at 9:00 AM, Renato Golin <renato.golin at linaro.org> wrote:
> On 28 February 2013 16:29, David Blaikie <dblaikie at gmail.com> wrote:
>>
>> If I were doing this that's probably where I would want to head.
>> Essentially gut the existing test suite running infrastructure (this
>> needs to be done sooner or later anyway because the whole thing is
>> terribly crufty) & make it smart enough to choose the number of
>> executions dynamically to get compensate for the noise to some level
>> of confidence (& presumably have some cutoff threshold where the test
>> would be considered just too noisy to be worthwhile).
>
>
> While the current system has many problems, it has the minimum set of
> features I'd expect from continuous integration + benchmarks. I can do the
> analysis off-line for now, that's not a big problem. (though, having a json
> interface would be great).

To be clear the intention is not to rewrite LNT - but the test-suite
beneath it. It's a complex hodge-podge of shell, Make, C, awk, etc...
difficult to maintain/add new features to. LNT was built with the
intention that the test-suite execution could be rewritten beneath it.

> Re-writing the whole system would create other inefficiencies, since we'd be
> focused on the things that the current system fails, but essentially, we'll
> have the same time we had before, so the things that the current system does
> well will be missing. I have seen many test infrastructures been re-written
> and not a single time the version 2.0 completely replaced 1.0.
>
> Also, dynamic benchmarks are not good for regression analysis, as they can
> potentially change number of cycles between runs

Hardly my forte, though I don't immediately see why changing the
number of cycles would make regression analysis invalid.

> (OS being busy and all that).

If the OS is differently loaded each time you run the tests you're
going to have a hard time doing regression analysis anyway, aren't
you?

> So, all in all, the current system is not so bad, and if it could be
> extensible with a json interface (is it?) like the buildbot, we could do
> much more with it as it is.
>
> But yes, increasing (statically) the run-time for most applications would be
> an easy and good move.
>
> cheers,
> --renato
>
>