[LLVMdev] LLVM benchmarks against GCC

["?=Valery A.Khamenya=?koi8-r?Q?"]

> Be careful comparing these numbers.  I see that we have a 23x speedup
> today over GCC on the "burg" test, but we go from 0.093 -> 0.004s.  :)
> The shorter the test runs get, the more noisy they get, so unfortunately
> we're not getting a realistic 23x speedup here. ;-)
> [...]
> These are even more dubious.  In particular, only the first 6 rows contain
> programs with reasonable runtimes.  This means that the 7x speedups for
> going from 0.021 -> 0.003 don't really count.  :)
> [...]
> This one is just noise, if you look today it's 1.0's straight across the
> board.  Also note that the test runs for 0.003 seconds, which is the
> resolution of the time command on the system the program is being run on:
> this is not a good test for checking performance.  :)

yes, i saw, that the numbers are quite discrete, it is just another 
feature telling the same. Well, but who really needs insecure numbers?..
maybe it would be a good idea to change the test a bit or make the 
same test but in a loop? otherwize it makes no sense to include them
into benchmarking...

> Hrm, you're not happy with the 2.25x speedup we get now?  

opss, sorry, my mistake.

> With GCC it
> takes 9.639s to run the test, with LLVM-CBE it takes 4.3s, and with
> LLVM-LLC-LS it takes 4.963s.  I think these are pretty good numbers.  :)

oh, you are right, i like those numbers :)

> Actually we spend *VERY* little time tuning and tweaking the optimizer for
> performance.  Something that would be *INCREDIBLY* useful would be for
> someone to pick some benchmark or other program we do poorly on (e.g.
> Ptrdist-ks), and find out *WHAT* we could be doing to improve it.  A good
> way to do this is to take the program, run it in a profiler (llvm-prof or
> gprof) find the hot spots, and see what we're code generating for them,
> and suggest ways that it could be improved.  If something performs well
> with the CBE but not with LLC-LS, then compare the native machine code
> generated, if it performs poorly with both, then it's probably an LLVM
> optimization.
> 
> At this point there are a huge number of possibilities for improvement.
> We have very little in the way of loop optimizations, and we don't
> actually use an interprocedural pointer analysis (I'm hope to rectify this
> for 1.3, it should make a huge difference).  Even if you're not into
> hacking on LLVM optimizations, identifying code that we could improve
> (and reducing them down to small examples of code we compile poorly) is
> incredibly useful.
> 
> Consider this a small plea for help.  :)  Once we know what to fix, it's
> usually pretty easy to do so, but identifying the problems takes time, and
> we have plenty of other things we need to be doing as well.

well, 90+ % of my spare time is dedicated to SuSE 9 and win32/cygwin.
Neither of two is good for LLVM today. But me and others always 
have a web client! Would it be a big deal to add small functionality 
to the http://llvm.cs.uiuc.edu/demo/ page? I mean, why not to add
output in assembler there? Then everyone could try to help you from 
anywhere. If i remember right, Misha was the one who prepared 
this nice page, could we ask Misha for that great add-on?

with best regards,
Valery.