[PATCH] Add benchmarking-only mode to the test suite

[Yi Kong]

On 12 August 2014 00:19, Yi Kong <kongy.dev at gmail.com> wrote:
> On 11 August 2014 22:54, Tobias Grosser wrote:
>> On 11/08/2014 07:55, Tobias Grosser wrote:
>>> Hi,
>>>
>>> thanks for the investigation.
>>>
>>> As it seems purely beneficial in terms of reducing another variable, I
>>> disabled it on the public builders. Let's check in a couple of days if
>>> the reported results become more stable.
>>
>>
>> It seems the latest build (e.g. 3mm) still have this surprising performance
>> behaviour. Let's give it some time, but I am afraid at least on x86
>> something surprising is still going on.
>>
>> Tobias
>>
>
> Yes, for polybench it seems to make no difference at all. However it
> does slightly reduce variance for other tests(maximum MAD of compile
> time tests reduced from ~0.08 to ~0.02), so it's still a useful
> change.

I inspected the results from recent runs and it turns out although
ASLR reduces variance for the same run, it increases the noise when
comparing different revisions.

For example, http://llvm.org/perf/db_perf/v4/nts/8225, it reports 7
false positives(the only thing changed in this revision was ARM
specific) while we used to get less than 3 per run.

I will look into why that causes the regression and in the meanwhile,
I think you should re-enable ASLR on the builders.

-Yi