[LLVMdev] Problems with custom calling convention on Mac OS X

[Charles Davis]

David Terei wrote:
> Hi all,
> 
> I'm working on using LLVM as a back-end for the Haskell GHC compiler. As 
> part of that work I have modified LLVM to include a new custom calling 
> convention for performance reasons as outlined previously in a 
> conversation on this mailing list:
> 
> http://nondot.org/sabre/LLVMNotes/GlobalRegisterVariables.txt
> 
> This custom calling convention on x86-32 needs to handle just 4 
> parameters, passing them in %ebx, %ebp, %esi, %edi. These are all callee 
> saved registers. To implement the custom calling convention I change 
> llvm in two places:
> 
> 1) X86CallingConv.td : Add new calling convention.
> 2) X86RegisterInfo.cpp : Modify 'getCalleeSavedRegs' to remove the above 
> registers from the callee saved registers.
> 
> This works fine mostly. On Linux, the generated code passes the GHC 
> testsuite. On Mac however the GHC testsuite fails on any code which uses 
> the ffi (which is implemented by libffi 
> [http://sourceware.org/libffi/]). Programs which fail segfault with the 
> error '__dyld_misaligned_stack_error'. The issue seems to be from my 
> investigations that the ffi call should be 16-byte aligned as per Mac OS 
> X's ABI.
That's correct. Therefore, when generating code, you must ensure that
the stack is 16-byte aligned, one way or another, if you want to make a
dylib call on Mac OS X.
> 
> I'm hoping someone is able to confirm that my changes would have 
> introduced this bug and how to go about fixing it.
Something I'm doing right now may be of interest to you.

Just today I added support for a new 'alignstack' function attribute.
With it, you can force the stack to be 16-byte aligned (or n-byte
aligned, if you so desire) in your functions. This way, you can make
calls to dylibs on Mac OS X without triggering the misaligned stack error.

Of course, it's a no-op right now (I still have to do the backend work).
But if you add this attribute to the emitted LLVM functions, when I do
finish this, the stack will be properly aligned and you won't get this
error anymore. It will cost you, though, in terms of code size and
performance.
> 
> Another minor issue is that the generated code has a strong tendency to 
> manipulate the stack pointer when its not required. For a large amount 
> of functions, the generated code will start and finish with sp 
> manipulation to give the function some space despite the function not 
> otherwise using the stack.
> 
> e.g
> 
> Utils_doStatefulOp1_entry:
>    subl $4, %esp
>    movl 4(%ebp), %eax
>    movl 8(%ebp), %ecx
>    movl (%ebp), %esi
>    movl %eax, 8(%ebp)
>    movl %ecx, 4(%ebp)
>    addl $4, %ebp
>    addl $4, %esp
>    jmp  stg_ap_pp_fast
> 
> It would be nice to fix this up as well.
That's normal. But it would be nice to fix this.

Chip