[LLVMdev] RFC:LNT Improvements

[Tobias Grosser]

Hi Renato,

On 30/04/2014 11:05, Renato Golin wrote:
> On 30 April 2014 07:50, Tobias Grosser <tobias at grosser.es> wrote:
>> In general, I see such changes as a second step. First, we want to have a
>> system in place that allows us to reliably detect if a benchmark is noisy or
>> not, second we want to increase the number of benchmarks that are not noisy
>> and where we can use the results.
>
> I personally use the test-suite for correctness, not performance and
> would not like to have its run time increased by any means.

I agree, we should not complicate the current use as a correctness 
test-suite. Though I don't think any of the changes proposed this.

I also believe we should as a first step not touch the test suite at 
all, but just improve how LNT reports the results it gets.

> As discussed in the BoF last year, if we could separate test run from
> benchmark run before we do any change, I'd appreciate.

To my understanding, the first patches should just improve LNT to report 
how reliable the results are it reports. So there is no way that this 
can effect the test suite runs, which means I do not see why we would 
want to delay such changes.

In fact, if we have a good idea which kernels are reliable and which 
ones are not, we can probably use this information to actually mark 
benchmarks that are known to be noisy.

> I want to have a separate benchmark bot on the subset that makes sense
> to work as benchmark, but I don't want the noise of the rest.

Right. There are two steps to get here:

1) Measure and show if a benchmark result is reliable

2) Avoid running the know to be noisy/unreliable benchmarks

>> 1. Get 5-10 samples per run
>> 2. Do the Wilcoxon/Mann-Whitney test
>
> 5-10 samples on an ARM board is not feasible. Currently it takes 1
> hour to run the whole set. Making it run for 5-10 hours will reduce
> its value to zero.

Reporting numbers that are not 100% reliable makes the results useless 
as well. As ARM boards are cheap, you could just put 5 boxes in place 
and we get the samples we need. Even if this is not yet feasible, I 
would rather run 5 samples of the benchmarks you really care, then 
running everything once and getting unreliable number.

>> I am a little sceptical on this. Machines should generally not be noisy.

Let me rephrase. "Machines on which you would like to run benchmarks 
should have a consistent and low enough level of noise"

> ARM machines work at a much lower power level than Intel ones. The
> scheduler is a lot more aggressive and the quality of the peripherals
> is *a lot* worse.
>
> Even if you set up the board for benchmarks (fix the scheduler, put
> everything up to 11), the quality of the external hardware (USB, SD,
> eMMC, etc) and their drivers do a lot of damage to any meaningful
> number you may extract if the moon is full and Jupiter is in
> Sagittarius.
>
> So...
 >
 >
>
>> However, if for some reason there is noise on the machine, the noise is as
>> likely to appear during this pre-noise-test than during the actual benchmark
>> runs, maybe during both, but maybe also only during the benchmark. So I am
>> afraid we might often run in the situation where this test says OK but the
>> later test is still suffering noise.
>
> ...this is not entirely true, on ARM.

So do you think the benchmark.sh script proposed by Yi Kong is useful 
for ARM?

Cheers,
Tobias