[PATCH] Add benchmarking-only mode to the test suite

[Tobias Grosser]

On 12/08/2014 01:19, Yi Kong wrote:
> On 11 August 2014 22:54, Tobias Grosser wrote:
>> On 11/08/2014 07:55, Tobias Grosser wrote:
>>> Hi,
>>>
>>> thanks for the investigation.
>>>
>>> As it seems purely beneficial in terms of reducing another variable, I
>>> disabled it on the public builders. Let's check in a couple of days if
>>> the reported results become more stable.
>>
>>
>> It seems the latest build (e.g. 3mm) still have this surprising performance
>> behaviour. Let's give it some time, but I am afraid at least on x86
>> something surprising is still going on.
>>
>> Tobias
>>
>
> Yes, for polybench it seems to make no difference at all. However it
> does slightly reduce variance for other tests(maximum MAD of compile
> time tests reduced from ~0.08 to ~0.02), so it's still a useful
> change.
>
> For builders that takes only a few samples per run, polybench results
> are too unstable. Do you still believe that we should keep them in
> benchmarking only mode?

They don't show up in my runs as problematic, so I prefer to keep them. 
I currently did not yet enable the benchmark only mode, but would like 
to do so. On the other side, if that means I loose the polybench kernels
I probably don't switch.

Anyhow. If they show up as problematic in your runs, please feel free to 
drop them from the benchmark runs. We can readd them after someone 
understood their performance behaviour better.

Cheers,
Tobias