[LLVMdev] Problems with custom calling convention on Mac OS X

[David Terei]

Charles Davis wrote:
> Just today I added support for a new 'alignstack' function attribute.
> With it, you can force the stack to be 16-byte aligned (or n-byte
> aligned, if you so desire) in your functions. This way, you can make
> calls to dylibs on Mac OS X without triggering the misaligned stack error.

I finally got around to properly playing around with 'alignstack' today 
and encountered a problem. It works as specified, indeed aligning the 
stack properly but interacts badly with the GHC calling convention. The 
problem is the GHC calling convention uses unconventional registers for 
argument passing. On x86-32 this is the four registers, ebx, ebp, esi, 
edi. The 'alignstack' attribute causes the ebp register to be clobbered.

e.g

_s1eJ_ret:
## BB#0:
    pushl %ebp
    movl  %esp, %ebp
    andl  $-16, %esp
    subl  $32, %esp
    movl  %ebp, 16(%esp)      ## 4-byte Spill
    [...]
    movl  16(%esp), %eax      ## 4-byte Reload
    movl  %edi, -4(%eax)

I'm not really sure what to do about this at the moment, will keep 
investigating, perhaps you have an idea though?

Cheers,
David