[PATCH] Add benchmarking-only mode to the test suite

[Yi Kong]

On 11 August 2014 22:54, Tobias Grosser wrote:
> On 11/08/2014 07:55, Tobias Grosser wrote:
>> Hi,
>>
>> thanks for the investigation.
>>
>> As it seems purely beneficial in terms of reducing another variable, I
>> disabled it on the public builders. Let's check in a couple of days if
>> the reported results become more stable.
>
>
> It seems the latest build (e.g. 3mm) still have this surprising performance
> behaviour. Let's give it some time, but I am afraid at least on x86
> something surprising is still going on.
>
> Tobias
>

Yes, for polybench it seems to make no difference at all. However it
does slightly reduce variance for other tests(maximum MAD of compile
time tests reduced from ~0.08 to ~0.02), so it's still a useful
change.

For builders that takes only a few samples per run, polybench results
are too unstable. Do you still believe that we should keep them in
benchmarking only mode?

-Yi