[LLVMdev] FYI: Phoronix GCC vs. LLVM-GCC benchmarks
Olivier Meurant
meurant.olivier at gmail.com
Wed Sep 16 06:05:19 PDT 2009
I have run the john the ripper test.
I have used the official archive (same version as phoronix) from
http://www.openwall.com/john/g/john-1.7.3.1.tar.bz2
To build with llvm-gcc, replace the line CC = gcc with CC = llvm-gcc.
I have used the following command to build : make clean linux-x86-sse2
(seems to be the best on x86-32)
The makefile invocation is "gcc -c -Wall -O2 -fomit-frame-pointer
-funroll-loops src.c" and "llvm-gcc -c -Wall -O2 -fomit-frame-pointer
-funroll-loops src.c".
I have runned 10 tests : "./john --test" in the run directory. It does 6
benchmarking on various algorithms :
Benchmarking: Traditional DES [128/128 BS SSE2]... DONE
LLVM GCC Difference Difference %
Run 1 2371 2358 13 0.55
Run 2 2499 2497 2 0.08
Run 3 2489 2487 2 0.08
Run 4 2305 2504 -199 -8.63
Run 5 2499 2445 54 2.16
Run 6 2404 2503 -99 -4.12
Run 7 2482 2502 -20 -0.81
Run 8 2479 2475 4 0.16
Run 9 2463 2489 -26 -1.06
Run 10 2484 2483 1 0.04
Average 2447.5 2474.3 -26.8 -1.15
Std dev 65.69 44.5 71.81 3.07
==> Similar results
Benchmarking: BSDI DES (x725) [128/128 BS SSE2]... DONE
LLVM GCC Difference Difference %
Run 1 72584 81280 -8696 -11.98
Run 2 76620 79795 -3175 -4.14
Run 3 79820 75264 4556 5.71
Run 4 76339 81484 -5145 -6.74
Run 5 81484 76441 5043 6.19
Run 6 80742 81433 -691 -0.86
Run 7 81510 79104 2406 2.95
Run 8 81049 79872 1177 1.45
Run 9 80409 81100 -691 -0.86
Run 10 80204 80921 -717 -0.89
Average 79076.1 79669.4 -593.3 -0.92
Std dev 2937.56 2181.15 4262.03 5.59
==> Similar results
Benchmarking: FreeBSD MD5 [32/32]... DONE
LLVM GCC Difference Difference %
Run 1 7552 8009 -457 -6.05
Run 2 7739 7724 15 0.19
Run 3 7997 7696 301 3.76
Run 4 8038 8041 -3 -0.04
Run 5 7474 7938 -464 -6.21
Run 6 7871 8078 -207 -2.63
Run 7 7884 7980 -96 -1.22
Run 8 7870 8025 -155 -1.97
Run 9 7989 8046 -57 -0.71
Run 10 7986 7989 -3 -0.04
Average 7840 7952.6 -112.6 -1.49
Std dev 193.87 133.78 227.92 2.98
==> Similar results
Benchmarking: OpenBSD Blowfish (x32) [32/32]... DONE
LLVM GCC Difference Difference %
Run 1 494 495 -1 -0.2
Run 2 457 485 -28 -6.13
Run 3 492 474 18 3.66
Run 4 494 492 2 0.4
Run 5 486 469 17 3.5
Run 6 491 495 -4 -0.81
Run 7 495 493 2 0.4
Run 8 490 490 0 0
Run 9 493 494 -1 -0.2
Run 10 493 492 1 0.2
Average 488.5 487.9 0.6 0.08
Std dev 11.37 9.19 12.56 2.67
==> Similar results
Benchmarking: Kerberos AFS DES [48/64 4K MMX]... DONE
LLVM GCC Difference Difference %
Run 1 399001 403712 -4711 -1.18
Run 2 396697 377292 19405 4.89
Run 3 395520 401971 -6451 -1.63
Run 4 392396 404172 -11776 -3
Run 5 392294 376217 16077 4.1
Run 6 395571 404172 -8601 -2.17
Run 7 400128 402995 -2867 -0.72
Run 8 397516 395110 2406 0.61
Run 9 396748 403507 -6759 -1.7
Run 10 396263 403712 -7449 -1.88
Average 396213.4 397286 -1072.6 -0.27
Std dev 2497.59 11150.79 10620.52 2.69
==> Similar results
Benchmarking: LM DES [128/128 BS SSE2]... DONE
LLVM GCC Difference Difference %
Run 1 6879 11433 -4554 -66.2
Run 2 8984 12252 -3268 -36.38
Run 3 9142 12182 -3040 -33.25
Run 4 8802 12205 -3403 -38.66
Run 5 8756 11971 -3215 -36.72
Run 6 9227 12224 -2997 -32.48
Run 7 8667 12191 -3524 -40.66
Run 8 9163 11942 -2779 -30.33
Run 9 9117 12254 -3137 -34.41
Run 10 9076 12166 -3090 -34.05
Average 8781.3 12082 -3300.7 -38.31
Std dev 695.06 252.95 487.97 10.26
==> This one is interesting as gcc is better with near 40%
I have no idea why but anyone interested could take a look at LM_fmt.c which
seems to define test and source for this algorithm.
Olivier.
On Wed, Sep 16, 2009 at 2:40 PM, Renato Golin <rengolin at systemcall.org>wrote:
> 2009/9/16 Olivier Meurant <meurant.olivier at gmail.com>:
> > Average 13836499.46 12447973.17 1388526.29 10.03
> > Standard deviation 53189.13 515638.56
> > 522400.98 3.77
>
>
> That was pretty much what I was expecting from the article... numbers.
> It doesn't matter who's best, you can't be best in all areas, but
> profiling must be done right. The standard deviation is, at least, one
> order of magnitude lower than the difference between the two averages,
> which means the measurements have a meaning and express some reality.
>
> There is little value in the results on the article. People had to
> re-do it properly again, anyway...
>
> Shouldn't it be part of the standard release? I mean, add profiling as
> an automated task before every big release, comparing to previous
> versions of LLVM and other important compilers. Not to be waving the
> results about, but to know the weakness and work on them on the next
> release. It might take a while to build such infrastructure, but it's
> a good thing to do, I guess.
>
> cheers,
> --renato
>
> Reclaim your digital rights, eliminate DRM, learn more at
> http://www.defectivebydesign.org/what_is_drm
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20090916/90b427bd/attachment.html>
More information about the llvm-dev
mailing list