[LLVMdev] FYI: Phoronix GCC vs. LLVM-GCC benchmarks

Wed Sep 16 02:22:54 PDT 2009

On 2009-09-16 11:42, Olivier Meurant wrote:
> On Tue, Sep 15, 2009 at 5:35 PM, David Greene <dag at cray.com
> <mailto:dag at cray.com>> wrote:
>
>
>     Remember, the goal here isn't to show how great LLVM is.  It's to
>     get an
>     honest assessment of where we are at.  Phoronix did us a big
>     favor.  Getting
>     more details about his tests would help us even more.
>
>
> It's exactly what I'm trying to do. I'm not even a LLVM contributor,
> so I don't feel responsible for a good or bad performances of LLVM. :)
> Anyway, thanks for pointing this target differences. After some hours
> at recompiling gcc, here we are :
>
> Some information on the test platform :
>
> cat /etc/lsb-release
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=9.04
> DISTRIB_CODENAME=jaunty
> DISTRIB_DESCRIPTION="Ubuntu 9.04"
>
> uname -a
> Linux zaraki 2.6.28-15-generic #49-Ubuntu SMP Tue Aug 18 18:40:08 UTC
> 2009 i686 GNU/Linux
>
> /proc/cpuinfo
> dual core Intel(R) Core(TM)2 CPU X6800  @ 2.93GHz (cache size 4096)
>
> llvm-gcc -v
> Using built-in specs.
> Target: i686-pc-linux-gnu
> Configured with: ../llvm-gcc4.2-2.5.source/configure
> --prefix=/mounts/zion/disks/0/localhome/tbrethou/2.5/prerelease2/llvm-gcc/obj/../install
> --program-prefix=llvm-
> --enable-llvm=/localhome/tbrethou/2.5/prerelease2/llvm-2.5/
> --disable-bootstrap --enable-languages=c,c++,fortran
> Thread model: posix
> gcc version 4.2.1 (Based on Apple Inc. build 5636) (LLVM build 2.5)
> (Binary from official x86 2.5 release)
>
> gcc -v
> Using built-in specs.
> Target: i686-pc-linux-gnu
> Configured with: ./configure --prefix=/usr/local
> Thread model: posix
> gcc version 4.4.1 (GCC)

Phoronix tested with gcc 4.2.1 (Apple's).

>
> The compilation line are :
> gcc -o bin src.c -O3
> llvm-gcc -o bin src.c -O3

The Makefile uses -O by default, perhaps that is what Phoronix used too.

>
>
> Results :
> LLVM average : 13553161.95
> gcc average : 14624441.45
> Average : gcc is better with 8%
>
> The full report :
>         "LLVM"  "GCC"   "Difference"    "Difference %"
>
> "Run 1" 13842616        14693115.8      -850499.8       -6.14
> "Run 2" 13773202.9      14785734.9      -1012532        -7.35
> "Run 3" 13671309.3      14985114        -1313804.7      -9.61
> "Run 4" 12916939        14640392.9      -1723453.9      -13.34
> "Run 5" 13572133.3      14425575        -853441.7       -6.29
> "Run 6" 13810096.5      14694240.6      -884144.1       -6.4
> "Run 7" 13427435.8      14695094.6      -1267658.8      -9.44
> "Run 8" 14056659.9      14468968.4      -412308.5       -2.93
> "Run 9" 13699125.3      14373360.9      -674235.6       -4.92
> "Run 10 12762101.5      14482817.4      -1720715.9      -13.48
>
> "Average"       13553161.95     14624441.45     -1071279.5      -7.99
> "Standard deviation"    412810.77       187898.68      
> 430895.31       3.46
>
> Is there again something wrong on this test ?

No, it seems fine.

I get different results on x86-64 though (with -O), the difference is
very small (4% with 3% stddev, or 3% with 1.5% stddev).

Maybe you should try to reproduce one of the other benchmarks that show
a higher difference, according to phoronix JOhn the Ripper shown 40%
performance drop, there was also a huge drop for blowfish.

	GCC 4.4.1 	GCC 4.2.4 	LLVM-GCC 2.5 	LLVM-GCC 2.6 	GCC-4.4/LLVM2.5
GCC-4.4/LLVM2.6 	GCC-4.4/GCC-4.2

	144663277 	139072037 	154747143 	151467440 	0.935 	0.955 	1.040

	144413249 	139042455 	150949846 	150322055 	0.957 	0.961 	1.039

	144915166 	138131283 	150630261 	150650540 	0.962 	0.962 	1.049

	145602711 	138587982 	154993781 	147281539 	0.939 	0.989 	1.051

	145547868 	139224734 	150650813 	147998859 	0.966 	0.983 	1.045

	145719970 	139224734 	144055081 	152474829 	1.012 	0.956 	1.047
average 	145143706.83 	138880537.5 	151004487.5 	150032543.67 	0.962
0.968 	1.045
stdev 	551855.54 	435155.79 	3961926.31 	2009642.82 	0.027 	0.015 	0.005

Using built-in specs.
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 4.4.1-4'
--with-bugurl=file:///usr/share/doc/gcc-4.4/README.Bugs
--enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr
--enable-shared --enable-multiarch --enable-linker-build-id
--with-system-zlib --libexecdir=/usr/lib --without-included-gettext
--enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.4
--program-suffix=-4.4 --enable-nls --enable-clocale=gnu
--enable-libstdcxx-debug --enable-mpfr --enable-objc-gc
--with-arch-32=i486 --with-tune=generic --enable-checking=release
--build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.4.1 (Debian 4.4.1-4)

llvm2.5:
Using built-in specs.
Target: x86_64-unknown-linux-gnu
Configured with: ../llvm-gcc-4.2/configure --program-prefix=llvm-
--enable-llvm=/home/edwin/llvm2.5/obj --enable-languages=c,c++,fortran
Thread model: posix
gcc version 4.2.1 (Based on Apple Inc. build 5636) (LLVM build)

llvm2.6pre1:
Using built-in specs.
Target: x86_64-unknown-linux-gnu
Configured with: ../llvm-gcc4.2-2.6.source/configure
--program-prefix=llvm- --enable-languages=c,c++,fortran
--enable-llvm=/home/edwin/llvm2.6/obj/
Thread model: posix
gcc version 4.2.1 (Based on Apple Inc. build 5649) (LLVM build)

Using built-in specs.
Target: x86_64-linux-gnu
Configured with: ../src/configure -v
--enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr
--enable-shared --with-system-zlib --libexecdir=/usr/lib
--without-included-gettext --enable-threads=posix --enable-nls
--with-gxx-include-dir=/usr/include/c++/4.2 --program-suffix=-4.2
--enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc
--enable-mpfr --with-tune=generic --enable-checking=release
--build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.2.4 (Debian 4.2.4-6)

Best regards,
--Edwin