[PATCH] D36717: [test-suite] Add SPEC CPU 2017

[Matthias Braun via Phabricator via llvm-commits]

MatzeB added a comment.

In https://reviews.llvm.org/D36717#845089, @kristof.beyls wrote:

> In https://reviews.llvm.org/D36717#844930, @Meinersbur wrote:
>
> > One more remark: The 'ref' dataset of 638.imagick_s on all the computers I tried on took between 2 and 3 hours. Submitted results to SPEC <https://www.spec.org/cpu2017/results/res2017q2/cpu2017-20161026-00017.html> are in a similar range. Unfortunately `timeit.py` has a 7200 seconds (2 hours) limit hardcoded (for `--limit-cpu` __and__ `--timeout`). I had to remove that limit to be able to have a successful run.
> >
> > Can we make the limit configurable or set higher?
>
>
> Let's make it configurable. There isn't any hard-coded value that will work well in all situations.


Yes. I think the easiest way to implement this would be reading an option from lit.cfg (`config.xxx` inside the litsupport code) so that benchmarks with special needs can place a lit.local.cfg in their directory to change this setting.
I think it would also be best to not make this setting a fixed value but a multiplier. So you would specify 638.imagick_s gets 3x the amount of time as other benchmarks. The nice thing about specifying this in relative terms is that someone benchmarking on really slow (or fast) devices can change the default/base timeout without the need to adapt all the per-benchmark timeouts as well.


https://reviews.llvm.org/D36717