[llvm-dev] Benchmarks for LLVM-generated Binaries

[Dean Michael Berris via llvm-dev]

> On 15 Sep 2016, at 13:04, Mehdi Amini <mehdi.amini at apple.com> wrote:
> 
>> 
>> On Sep 14, 2016, at 6:57 PM, Dean Michael Berris <dean.berris at gmail.com> wrote:
>> 
>> Thanks everyone, I'll go with the "libs/" as a top-level directory in test-suite.
>> 
>>> On 15 Sep 2016, at 03:08, Matthias Braun <matze at braunis.de> wrote:
>>> 
>>> Have you seen the prototype for googlebenchmark integration I did in the past:
>>> 
>>> https://reviews.llvm.org/D18428 (though probably out of date for todays test-suite)
>>> 
>> 
>> Not yet, but thanks for the pointer Matthias!
>> 
>>> +1 for copying the googlebechmark into the test-suite.
>>> 
>>> However I do not think this should simply go into MultiSource: We currently have a number of additional plugins in the lit test runner such as measuring the runtime of the benchmark executable, determining code size, we still plan to add a mode to run benchmarks multiple times, we run the bechmark under perf (or iOS specific tools) to collect performance counters… Many of those are questionable measurements for a googlebenchmark executable which has varying runtime because it runs the test more/less often.
>>> We really should introduce a new benchmarking mode for this.
>>> 
>> 
>> Sounds good to me, but probably something for later down the road.
> 
> I think these benchmarks should not run by default as long as there is no proper integration to report the “correct” timing in lit. Otherwise it’ll pollute the reports / database.
> 

That sounds reasonable too. The "easy" way to do this would be to gate the building/running of the benchmarks under a CMake option, and defaulting that to OFF and let specific runners turn those on if they'd like to.

Cheers

-- Dean