[LLVMdev] Problems with custom calling convention on Mac OS X

[David Terei]

Hi all,

I'm working on using LLVM as a back-end for the Haskell GHC compiler. As 
part of that work I have modified LLVM to include a new custom calling 
convention for performance reasons as outlined previously in a 
conversation on this mailing list:

http://nondot.org/sabre/LLVMNotes/GlobalRegisterVariables.txt

This custom calling convention on x86-32 needs to handle just 4 
parameters, passing them in %ebx, %ebp, %esi, %edi. These are all callee 
saved registers. To implement the custom calling convention I change 
llvm in two places:

1) X86CallingConv.td : Add new calling convention.
2) X86RegisterInfo.cpp : Modify 'getCalleeSavedRegs' to remove the above 
registers from the callee saved registers.

This works fine mostly. On Linux, the generated code passes the GHC 
testsuite. On Mac however the GHC testsuite fails on any code which uses 
the ffi (which is implemented by libffi 
[http://sourceware.org/libffi/]). Programs which fail segfault with the 
error '__dyld_misaligned_stack_error'. The issue seems to be from my 
investigations that the ffi call should be 16-byte aligned as per Mac OS 
X's ABI.

I'm hoping someone is able to confirm that my changes would have 
introduced this bug and how to go about fixing it.

Another minor issue is that the generated code has a strong tendency to 
manipulate the stack pointer when its not required. For a large amount 
of functions, the generated code will start and finish with sp 
manipulation to give the function some space despite the function not 
otherwise using the stack.

e.g

Utils_doStatefulOp1_entry:
   subl $4, %esp
   movl 4(%ebp), %eax
   movl 8(%ebp), %ecx
   movl (%ebp), %esi
   movl %eax, 8(%ebp)
   movl %ecx, 4(%ebp)
   addl $4, %ebp
   addl $4, %esp
   jmp  stg_ap_pp_fast

It would be nice to fix this up as well.

Cheers,
David