[llvm-dev] Using lnt to run SPEC tests: possible or not?

[Matthias Braun via llvm-dev]

If you don't want to submit the results to an LNT server you can also just run the test-suite directly without going through LNT.

You should be able to do something like:
cmake -DCMAKE_C_COMPILER=/path/to/my/clang -DTEST_SUITE_SPECXXX_ROOT -DTEST_SUITE_SUBDIRS=External/SPECXXX -GNinja /path/to/test-suite
# NOte: use ccmake or cmake -L to see a list of variables you can tweak.
ninja
llvm-lit

to build and run SPEC via the test-suite infrastructure.

- Matthias

> On Jun 20, 2018, at 12:20 PM, Bill Seurer via llvm-dev <llvm-dev at lists.llvm.org> wrote:
> 
> Thanks Kristof.
> 
> I am hitting some odd cmake issues (it isn't handing pthread stuff properly for some reason) that I haven't seen before but this should be me going.  I've used lnt for running the llvm test suite many times using "runtest nt" without problem.
> 
> Hmmm.  powerpc64 isn't listed in the known architectures in test_suite.py so I may need to update some of these files.
> 
> On 06/20/18 02:02, Kristof Beyls wrote:
>> Hi Bill,
>> Yes, it is possible to run SPEC tests through LNT/test-suite.
>> You won’t get official reportable SPEC numbers as you don’t use the official build-and-run framework from SPEC, but OTOH you get the extra features in LNT/test-suite working out of the box, e.g. the ones documented at http://blog.llvm.org/2016/06/using-lnt-to-track-performance.html.
>> To make use of this, I think you need to use the cmake driver in the test-suite.
>> Which, if you use the LNT wrapper to drive runs, is invoked using “lnt runtest test-suite”.
>> A few example of the important options on the command line that I hope will get you started:
>>  * Running SPEC2K6int reference inputs:lnt runtest test-suite
>>    --only-test=External/SPEC/CINT2006 --cmake-define
>>    TEST_SUITE_SPEC2006_ROOT=/work/Benchmark/spec2006 --cmake-define
>>    TEST_SUITE_RUN_TYPE=ref
>>  * Running SPEC2K6int train inputs:lnt runtest test-suite
>>    --only-test=External/SPEC/CINT2006 --cmake-define
>>    TEST_SUITE_SPEC2006_ROOT=/work/Benchmark/spec2006 --cmake-define
>>    TEST_SUITE_RUN_TYPE=train
>>  * Running SPEC2Kfp reference inputs: lnt runtest test-suite
>>    --only-test=External/SPEC/CFP2000 --cmake-define
>>    TEST_SUITE_SPEC2000_ROOT=/work/Benchmark/spec2000 --cmake-define
>>    TEST_SUITE_RUN_TYPE=ref
>> Thanks,
>> Kristof
>>> On 19 Jun 2018, at 23:53, Bill Seurer via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> <mailto:llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>> wrote:
>>> 
>>> It appears that lnt has the capability to run the SPEC2000/2006/2017(maybe) tests from looking at the options and from poking around in the mailing list archives. Or does it?  I experimented a bit and cannot get it to work right.  However, it isn't really documented anywhere so maybe I didn't set it up properly.  Plus there's all the mention of llvmgcc...
>>> 
>>> So is anyone doing this?  Is there some setup information that I missed?  Thanks for any tips on how to do this!
>>> -- 
>>> 
>>> -Bill Seurer
>>> 
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> <mailto:llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> 
> 
> -- 
> 
> -Bill Seurer
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180620/50a90d6e/attachment.html>