[LLVMdev] How to represent __attribute__((fastcall)) functions in the IL

[Rafael Espíndola]

Functions with __attribute__((fastcall)) pop their arguments and take
up to two arguments in ecx and edx. Currently we represent them by
just setting the x86_fastcallcc calling convention. The problem is
that the ABI has some strange conventions on when a register is used
or not. For example:

void __attribute__((fastcall)) foo1(int y);

will take 'y' in ecx, but

struct S1 {
  int x;
};
void __attribute__((fastcall)) foo2(struct S1 y);

will use the stack. Even more surprising is that

void __attribute__((fastcall)) foo8(struct S1 a, int b);

will take 'a' in the stack but 'b' in *edx*. That is, the first
argument consumed ecx but didn't use it. The implement this the IL
needs to be able to represent that
* an argument will go on registers
* a register was consumed but unused

A way to do this is to make clang produce byval for anything that goes
on the stack, but that produces worse code as the caller now has to
use an alloca instead of a simple value.

Another option (which I implemented in a patch just sent to
llvm-commits) is use the inreg attribute which we already use for the
regparm C attribute. The above requirements are handled by putting
inreg in the arguments that should go in register and creating a inreg
padding argument when a register should be consumed but not used.

The above examples would be irgened to:

declare x86_fastcallcc void @foo1(i32 inreg

declare x86_fastcallcc void @foo2(i32 inreg  /*dummy*/ , i32)

declare x86_fastcallcc void @foo8(i32 inreg /*dummy*/, i32, i32 inreg)

As Eli pointed out on llvm-commits, this would change the meaning of
existing  x86_fastcallcc IL files, but given that they are only
produced for fastcall C functions and those are fairly broken without
this fix, I assume that is OK.

Cheers,
Rafael