[LLVMdev] execution time of bytecode and native

[Chris Lattner]

On Thu, 30 Jun 2005, Tanu Sharma wrote:
> I am compiling SPEC 2000 benchmarks with llvm .Got stuck with 
> calculating "execution time" of all the .bc and native files.
>
> The log for nightly test itself gives execution times but I am passing 
> the bytecode files to my pass which gives another bytecode file.I have 
> to calculate execution time of such bytecode and native files as well.If 
> i simply do this:

You should look into the llvm-test framework.  All of the tests in it (as 
well as 'external tests' which we don't provide source for, like SPEC) 
allow you to do things like this.  If you look at llvm-test/TEST.*Makefile 
and llvm-test/TEST.*report files, you'll see ways to automatically 
generate custom reports.

Unfortunately there is no documentation for how to build a custom report, 
but there are many examples.  The 'jit' report (TEST.jit.Makefile and 
TEST.jit.report) are a simple example to demonstrate this.  For example, 
go into llvm-test/MultiSource/Benchmarks/Olden and run:

$ make TEST=jit report

This should spit out something like this (on X86):

Name:               | Total CodeGen InstSel LiveVar RA     FPStack Peep   | MCSize #MCInsts #Glob | #store #load #fp #fxch |
treeadd/treeadd     | 6.38  0.0300  0.0133  0.0023  0.0054 0.0001  0.0002 | 1000   245      115   | *      10    *   *     |
power/power         | 4.87  0.0899  0.0368  0.0044  0.0366 0.0022  0.0003 | 5720   1336     702   | 105    216   790 58    |
tsp/tsp             | 4.72  0.1400  0.0439  0.0073  0.0591 0.0023  0.0005 | 4463   1128     106   | 62     149   469 28    |
bh/bh               | 3.09  0.2700  0.1231  0.0107  0.1184 0.0030  0.0006 | 11983  2483     137   | 223    408   968 27    |
bisort/bisort       | 2.86  0.0800  0.0332  0.0046  0.0163 0.0002  0.0002 | 3825   914      148   | 17     64    *   *     |
em3d/em3d           | 2.84  0.0600  0.0569  0.0034  0.0195 0.0003  0.0002 | 3005   727      406   | 32     79    58  1     |
voronoi/voronoi     | 1.11  0.1900  0.0601  0.0081  0.0755 0.0024  0.0009 | 8676   2205     322   | 141    275   761 38    |
health/health       | 0.97  0.0800  0.0298  0.0036  0.0171 0.0004  0.0002 | 3438   823      268   | 35     49    101 6     |
perimeter/perimeter | 0.64  0.0800  0.0307  0.0040  0.0149 0.0002  0.0002 | 4266   987      79    | 21     57    *   *     |
mst/mst             | 0.21  0.0500  0.0167  0.0025  0.0117 0.0001  0.0001 | 2094   491      223   | 15     33    *   *     |

This recursively traverses llvm-test building all of the programs in the 
subdirectory from where you start.  The report stuff can emit a text file 
like this, HTML files, CSV, and latex output.

There is a lot of stuff there to play with, I would suggest giving it a 
try.  To use SPEC 95/2K, set up the External/SPEC/* stuff and it should 
just work (including automatic management of input/output files etc).

BTW, the nightly tester is just one client of this framework.

-Chris

> time lli spec_benchmark.bc OR time ./spec_benchmark.native
>
> I need to know the command line options and also how to run for "test" or "ref" etc and like.Is
>
> I run experiments (run my pass) some 30 times so is there any way to achieve it globally ?
>
> I appreciate your time,
>
> Thanks
>
>
>
>
>
> ---------------------------------
> Yahoo! Sports
> Rekindle the Rivalries. Sign up for Fantasy Football

-Chris

-- 
http://nondot.org/sabre/
http://llvm.cs.uiuc.edu/