[LLVMdev] -march and -mtune options on x86

[Chandler Carruth]

On Mon, Jan 16, 2012 at 12:29 AM, Ghassan Shobaki <ghassan_shobaki at yahoo.com
> wrote:

> Let me describe more precisely what I am doing and why the results I got
> may help improve LLVM's performance on modern x86-64 processors regardless
> of the front end (GCC, Clang or DragonEgg).
>
> I am running ALL my tests on an Intel Xeon E5540 processor, which is an
> x86-64 Nehalem processor. The OS is a 64-bit version of Ubuntu. So, I am
> running all my tests on the same x86-64 machine and am only experimenting
> with compiler options. What I meant by running in x86-32 mode is simply
> using the -m32 option (passed through the llvm-gcc front end, but I believe
> the front end is irrelevant with regard to this matter, any confirmation?).
>

That is not quite accurate. This option is inherently a frontend option.
The frontend lowers that into various different options to the backend, and
what those are varies from frontend to frontend. Please try using Clang or
DragonEgg. That said, I think I've got a pretty good understanding of what
went on with these options... read my detailed responses below.


The original reason for using this option was forcing the back end to work
> with a smaller number of physical registers to study register pressure
> reduction under more stringent constraints (as part of an academic research
> project). However, the observation that I am trying to communicate in this
> posting may not be related to register pressure.
>
> The observation is that using the -march=core2 and -mtune=core2 options
> makes a significant positive difference in x86-32 mode (that is, with the
> -m32 option), while it does not make any significant difference in the
> x86-64 mode (without the -m32 option).
>

This is exactly the phenomenon we were describing to you.

With no '-march' options, the default architecture for a 32-bit build is
i386, which is very old and has quite limited instructions available.

With no '-march' options, the default architecture for a 64-bit build is
x86-64, which is newer than i686 and has a broad and very useful
instruction set available to it.

The delta of architectures from i386 -> core2 is *massive*. The delta from
x86-64 -> core2 is quite small.

My hypothesis is that the LLVM back end is making some good target-specific
> optimizations when the -m32 option is used but those good optimizations are
> disabled when this option is not used.
>

That is not how the backend works. The target specific optimizations are
being *turned off* by specifying '-m32' in order to hit the very old target
of i386.

 Here are the SPEC CPU2006 geometric-mean scores that I am measuring:
>
> Native x86-64 mode (without the -m32 option):
> Using -O3 only:                                               INT score:
> 19.24     FP score: 15.64
>  Using -O3 -march=core2 -mtune=core2:     INT score: 19.16     FP score:
> 15.57
>  So, there is no significant difference in this case. The small
> difference may just be random variation.
>
>  x86-32 mode (adding the -m32 option):
> Using -O3 -m32 only:                                               INT
> score: 16.86     FP score: 14.09
>  Using -O3 -m32 -march=core2 -mtune=core2:     INT score: 17.02     FP
> score: 15.24
>

Note that these scores are still lower than those x86-64 got in both modes.
I'm not sure why you think some optimization was enabled here rather than
disabled...


>  So, there is a significant difference in this case. In fact, the 8%
> geometric-mean improvement on FP2006 is a huge improvement that comes out
> of many double-digit percentage improvements on some individual benchmarks.
> The biggest improvement is 48% on gromacs, which is a CPU-intensive FP
> benchmark.
>

I haven't tested it, but I'd be willing to bet this is a difference of
using x87 FPU versus the SSE FP operations... I'd have to look into the
compiled result much more to tell, but genuinely it isn't interesting. The
behavior you describe is exactly what I would expect from these options.


>  The above geometric means are consistent with the logical expectation
> that LLVM's performance in the native x86-64 mode is better, because more
> spill code is generated when the -m32 option is used. However, these
> aggregate numbers hide the fact that LLVM generates much faster code for
> some benchmarks when the -m32 option is used. An extreme example is the
> bwaves benchmark, which has a score of 15.5 in the native mode and a score
> of 23 (a 48% speedup) when the -m32 option is used. If LLVM is capable of
> achieving the higher score in the -m32 mode, it should be able to achieve
> at least the same score in the native mode. So, the question is: are there
> any good optimizations that we are losing in the native x86-64 mode? If
> yes, can we enable them to get better performance on x86-64?
>

Have you looked at the generated code for optimizations made in 32-bits
that are missed in 64-bits? That's the hard part. I know for a fact (having
done such an investigation) that there are some, but I strongly doubt they
are responsible for these huge swings in performance you're observing.

There is a much bigger difference than any backend optimizations between
32-bit binaries and 64-bit binaries: pointers are half the size in 32-bits.
This can cause significant shrink to in-memory data structures which can
cause significantly denser cache lines and significantly better
performance. That would seem like a much better explanation for a few
benchmarks being significantly faster in 32-bit mode.

It might still be worth investigating to see if there are missed
optimizations here, but all of the ones I have found have been subtle
results of different code patterns due to differently sized and structure
data. I think you're going to need to do a lot of very detailed analysis to
draw any conclusions about missed optimizations.


> It seems to me that the back end is somehow assuming that modern x86-64
> machines magically solve all the scheduling and tuning problems and do not
> need any help from the compiler. Any truth to this?
>

None at all. Please go look at the x86-64 backend before making wild
guesses. For reference, the 64-bit and 32-bit x86 backend *share the same
code*.


> Can anyone who is interested in performance on x86-64 try rerunning
> his/her tests using the -m32 mode to see if that gives any speedup?
>

I do this all the time. If you want to help, please actually investigate
the performance differences you find, isolate specific missed optimizations
with clear testcases demonstrating the transform that should be applied and
isn't and file bugs or submit patches.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20120116/305aa457/attachment.html>