[LLVMdev] execution time of bytecode and native

[Chris Lattner]

On Wed, 20 Jul 2005, Tanu Sharma wrote:

> Hello All,
>
> Thanks for the reply.I can generate the reports by compiling Spec 
> through llvm, but that couldn't resolve my problem.

ok

> I m trying to determine execution time for the bytecode and native files 
> , which are obtained as a result of running my pass over the original 
> bytecode .I am running these experiments on spec benchmark.

ok

> In SPEC we have command line tools such as runspec where we can specify 
> the parameters such as ref or test etc, but if I am taking the llvm-test 
> approach, I have to make all the changes in Makefile.spec.Is that 
> correct?

No, you should be able to make changes to the local makefile itself in the 
program's directory (if you want to change the args passed to it).  If you 
want to change how it is compiled, there are several options: the easiest 
of which is to just change the 'gccas' or 'gccld' tools to include you 
pass.

> Even for this purpose I am including my pass in 
> Makefile.spec.Is there any simpler or command line option possible?

That works.  Another option is to (ab)use the "STATS" variable.  If you 
set STATS to something on the make command line, the contents are passed 
to gccas and gccld.  Through this mechanism, you could modify gccas/ld to 
take your option and do what you want with it.

It was originally designed to be used like this:

$ make STATS='-stats -time-passes'

... but you can pass whatever you want.

> Because of this particular problem, i am not able to use simple commands 
> like "time " to determine time as I can't work on individual benchmarks.

When the llvm-test harness runs a program, it takes care of diffing the 
output (to make sure the program is working) and timing it.  The time 
info for each run is output to the Output/*.time files.

> If I sound confusing please let me know.Please help me with this.Thanks 
> for taking out time !!

Keep asking questions, and we'll keep trying to help!

-Chris


>
> Chris Lattner <sabre at nondot.org> wrote:
> On Thu, 30 Jun 2005, Tanu Sharma wrote:
>> I am compiling SPEC 2000 benchmarks with llvm .Got stuck with
>> calculating "execution time" of all the .bc and native files.
>>
>> The log for nightly test itself gives execution times but I am passing
>> the bytecode files to my pass which gives another bytecode file.I have
>> to calculate execution time of such bytecode and native files as well.If
>> i simply do this:
>
> You should look into the llvm-test framework. All of the tests in it (as
> well as 'external tests' which we don't provide source for, like SPEC)
> allow you to do things like this. If you look at llvm-test/TEST.*Makefile
> and llvm-test/TEST.*report files, you'll see ways to automatically
> generate custom reports.
>
> Unfortunately there is no documentation for how to build a custom report,
> but there are many examples. The 'jit' report (TEST.jit.Makefile and
> TEST.jit.report) are a simple example to demonstrate this. For example,
> go into llvm-test/MultiSource/Benchmarks/Olden and run:
>
> $ make TEST=jit report
>
> This should spit out something like this (on X86):
>
> Name: | Total CodeGen InstSel LiveVar RA FPStack Peep | MCSize #MCInsts #Glob | #store #load #fp #fxch |
> treeadd/treeadd | 6.38 0.0300 0.0133 0.0023 0.0054 0.0001 0.0002 | 1000 245 115 | * 10 * * |
> power/power | 4.87 0.0899 0.0368 0.0044 0.0366 0.0022 0.0003 | 5720 1336 702 | 105 216 790 58 |
> tsp/tsp | 4.72 0.1400 0.0439 0.0073 0.0591 0.0023 0.0005 | 4463 1128 106 | 62 149 469 28 |
> bh/bh | 3.09 0.2700 0.1231 0.0107 0.1184 0.0030 0.0006 | 11983 2483 137 | 223 408 968 27 |
> bisort/bisort | 2.86 0.0800 0.0332 0.0046 0.0163 0.0002 0.0002 | 3825 914 148 | 17 64 * * |
> em3d/em3d | 2.84 0.0600 0.0569 0.0034 0.0195 0.0003 0.0002 | 3005 727 406 | 32 79 58 1 |
> voronoi/voronoi | 1.11 0.1900 0.0601 0.0081 0.0755 0.0024 0.0009 | 8676 2205 322 | 141 275 761 38 |
> health/health | 0.97 0.0800 0.0298 0.0036 0.0171 0.0004 0.0002 | 3438 823 268 | 35 49 101 6 |
> perimeter/perimeter | 0.64 0.0800 0.0307 0.0040 0.0149 0.0002 0.0002 | 4266 987 79 | 21 57 * * |
> mst/mst | 0.21 0.0500 0.0167 0.0025 0.0117 0.0001 0.0001 | 2094 491 223 | 15 33 * * |
>
> This recursively traverses llvm-test building all of the programs in the
> subdirectory from where you start. The report stuff can emit a text file
> like this, HTML files, CSV, and latex output.
>
> There is a lot of stuff there to play with, I would suggest giving it a
> try. To use SPEC 95/2K, set up the External/SPEC/* stuff and it should
> just work (including automatic management of input/output files etc).
>
> BTW, the nightly tester is just one client of this framework.
>
> -Chris
>
>> time lli spec_benchmark.bc OR time ./spec_benchmark.native
>>
>> I need to know the command line options and also how to run for "test" or "ref" etc and like.Is
>>
>> I run experiments (run my pass) some 30 times so is there any way to achieve it globally ?
>>
>> I appreciate your time,
>>
>> Thanks
>>
>>
>>
>>
>>
>> ---------------------------------
>> Yahoo! Sports
>> Rekindle the Rivalries. Sign up for Fantasy Football
>
> -Chris
>
>

-Chris

-- 
http://nondot.org/sabre/
http://llvm.org/