[llvm-dev] Using lnt to run SPEC tests: possible or not?

[Bill Seurer via llvm-dev]

Thanks Kristof.

I am hitting some odd cmake issues (it isn't handing pthread stuff 
properly for some reason) that I haven't seen before but this should be 
me going.  I've used lnt for running the llvm test suite many times 
using "runtest nt" without problem.

Hmmm.  powerpc64 isn't listed in the known architectures in 
test_suite.py so I may need to update some of these files.

On 06/20/18 02:02, Kristof Beyls wrote:
> Hi Bill,
> 
> Yes, it is possible to run SPEC tests through LNT/test-suite.
> You won’t get official reportable SPEC numbers as you don’t use the 
> official build-and-run framework from SPEC, but OTOH you get the extra 
> features in LNT/test-suite working out of the box, e.g. the ones 
> documented at 
> http://blog.llvm.org/2016/06/using-lnt-to-track-performance.html.
> 
> To make use of this, I think you need to use the cmake driver in the 
> test-suite.
> Which, if you use the LNT wrapper to drive runs, is invoked using “lnt 
> runtest test-suite”.
> 
> A few example of the important options on the command line that I hope 
> will get you started:
> 
>   * Running SPEC2K6int reference inputs:lnt runtest test-suite
>     --only-test=External/SPEC/CINT2006 --cmake-define
>     TEST_SUITE_SPEC2006_ROOT=/work/Benchmark/spec2006 --cmake-define
>     TEST_SUITE_RUN_TYPE=ref
> 
>   * Running SPEC2K6int train inputs:lnt runtest test-suite
>     --only-test=External/SPEC/CINT2006 --cmake-define
>     TEST_SUITE_SPEC2006_ROOT=/work/Benchmark/spec2006 --cmake-define
>     TEST_SUITE_RUN_TYPE=train
>   * Running SPEC2Kfp reference inputs: lnt runtest test-suite
>     --only-test=External/SPEC/CFP2000 --cmake-define
>     TEST_SUITE_SPEC2000_ROOT=/work/Benchmark/spec2000 --cmake-define
>     TEST_SUITE_RUN_TYPE=ref
> 
> Thanks,
> 
> Kristof
> 
> 
>> On 19 Jun 2018, at 23:53, Bill Seurer via llvm-dev 
>> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>>
>> It appears that lnt has the capability to run the 
>> SPEC2000/2006/2017(maybe) tests from looking at the options and from 
>> poking around in the mailing list archives.  Or does it?  I 
>> experimented a bit and cannot get it to work right.  However, it isn't 
>> really documented anywhere so maybe I didn't set it up properly.  Plus 
>> there's all the mention of llvmgcc...
>>
>> So is anyone doing this?  Is there some setup information that I 
>> missed?  Thanks for any tips on how to do this!
>> -- 
>>
>> -Bill Seurer
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> 


-- 

-Bill Seurer