[LLVMdev] [Polly] Set up performance tester for GSOC2013 FastPolly project

[Tobias Grosser]

On 06/06/2013 11:17 AM, Star Tan wrote:
> Hi Tobias,
>
>
> I am recently trying to set up the performance tester for FastPolly project. According to your suggestion, I plan to use the LNT infrastructure to set up the performance tester. For this purpose, I think I should do this job in three steps:
>
>
> First, I will add PolyBench to LLVM test-suite since PolyBench is the critical benchmarks for FastPolly. I have adjust the PolyBench-c-3.2 so we can put it into LLVM test-suite/SingleSource/Benchmarks/.  Please find the attached PolyBench_for_LLVM_test_suite.tgz.

Polybench should already be part of the LLVM test suite. Please have a 
source at SingleSource/Benchmarks/Polybench/

> Second, I think I should select some benchmarks for FastPolly testing. Running all LLVM test-suite benchmarks is very time-consuming and unnecessary. My idea is still to run the mediebench  and PolyBench using LNT infrastructure. Please find the attached Prliminary PolyBench evaluation results (LNT_PolyBench.html) using LNT infrastructure.

I believe in the end we should run all benchmarks. It is important to
see the performance over a wider range of input programs. To get started 
you can obviously focus on specific programs or problems.

> Third, I think I should evaluate the compile-time and running time in both Polly-enabling case and Polly-disabling case. Currently, I am not clear how to configurate LNT so it can test the same benchmark using different parameters in the same runtime test, but I will investigate it.

The LNT infrastructure will provide you with compile-time and run-time
statistics. We should set up the tester with 'clang -O3' and 
'polly-clang -O3' and compare the compile-time of both runs against each 
other.

> Do you think it is a good way to set up a performance tester for the FastPolly project? If you agree with me, I will try to commit the PolyBench to llvm test-suite in the next step.

You can skip the step with committing the test suite.

A the moment I do not have dedicated machines available to set up public 
performance testers, so we need to rely on Sebastian's internal testers 
or your own experiments. I may be able to get some machines in the not 
so distance future. Until we reach this point, I propose you make 
yourself familiar with LNT, the LLVM test suite and possibly also the 
buildbot system we have. If you have a machine where you can run this 
locally, this would already be a good start.

Cheers,
Tobias