[cfe-dev] Compilation benchmark: bzip2

[Chris Lattner]

On Dec 22, 2007, at 8:49 PM, Sanghyeon Seo wrote:

> I decided to measure clang's performance by compiling bzip2.
> bzip2 is available from http://www.bzip.org/
> Makefile used for benchmark is here:
> http://sparcs.kaist.ac.kr/~tinuviel/devel/llvm/Makefile.bzip2

Cool

> Result (Minimum of 3):
>
> tinuviel at debian:~/clang$ tar zxf src/bzip2-1.0.4.tar.gz
> tinuviel at debian:~/clang$ cp make/Makefile.bzip2 bzip2-1.0.4/Makefile
> tinuviel at debian:~/clang$ cd bzip2-1.0.4
>
> tinuviel at debian:~/clang/bzip2-1.0.4$ make -s
>
> GCC: real 0.613s user 0.556s sys 0.052s
> tcc: real 0.046s user 0.028s sys 0.012s
> GCC -S: real 0.555s user 0.488s sys 0.052s
> clang: real 0.298s user 0.248s sys 0.040s
> clang+llvm-as: real 0.636s user 0.576s sys 0.048s

Just so I understand what is going on here:
   "GCC" -> "gcc -O0 -c"
   "GCC -S" -> "gcc -O0 -S"
   "tcc" -> "tcc -c"
   "clang" -> clang -emit-llvm
   "clang+llvm-as" -> clang -emit-llvm | llvm-as

These are interesting numbers, but not very relevant.  GCC -S is doing  
a *lot* more than clang -emit-llvm.  To get a useful comparison  
between gcc vs clang codegen, you'd need to link the llvm code  
generator into clang to get it to emit a native .s file.  Likewise, if  
you want "clang emission of llvm bytecode", you should link the  
bytecode writer into clang, instead of using llvm-as (which is  
obviously not very fast).  A really rough functional approximation  
would be "clang -emit-llvm | llvm-as | llc -fast -regalloc=local", but  
this will obviously be much slower than linking llc components into  
clang.

To me, the one interesting thing out of this is that the difference  
between gcc -c and gcc -S is ~14%.  That's a pretty big cost just for  
an assembler.  Maybe someone should work on finishing the llvm .o file  
emitter at some point ;-).

> clang+llvm-as spent about half the time in assembler.
> gcc spent less than 10% time in assembler.

Right, but these assemblers are not the same thing at all :)

At this point, llvm -O0 code generation is not nearly as fast as it  
should be.  That said, our -O2 or -O3 codegen is a lot faster than GCC  
in most cases.  In the future, we'll put more effort into making -O0  
codegen fast.

-Chris