[LLVMdev] Tail call optimization thoughts

[Anton Korobeynikov]

Hello, Arnold.

Only quick comments, I'll try to make a full review a little bit later.

> 0.)a fast calling convention (maybe use the current
>     CallingConv::Fast, or create a CallingConv::TailCall)
> 1.) lowering of formal arguments
>     like for example x86_LowerCCCArguments in  stdcall mode
>     we need to make sure that later mentioned CALL_CLOBBERED_REG is
>     not used (remove it from availableregisters in callingconv
>     for argument passing?)
This can be acceptable, only if function has internal linkage.
Otherwise, we cannot change calling convention of the function. So
extension if this problem:
Let us have the following sequence of calls: a->b->c
Call b->c is tail call.

We need to formulate conditions of the call to be "suitable for tail
call". On x86 it's the following:
1. The calling conventions of b and c are "compatible". There are two
scenarios: caller cleans the stack (C calling convention), callee cleans
the stack (stdcall calling convention). The call is not suitable for
tail call, if b and c cleans the stack differently. For example, let b
has C CC and c - stdcall CC. In this case making tail call results to
double cleaning (on in a and one in c), which is incorrect.
2. The same situtation is for struct return functions: the caller pushes
extra pointer to return value, but callee is required to clean this
pointer out of the stack. So, call b->c is "suitable for tail call" iff
both b and c are struct return.
3. Various tricky cases with FP. I don't remeber correctly, but there
can be some problems with functions returning FP values.
4. PIC. %ebx should be live in order to make a call via PLT.
Unfortunately, %ebx is callee-saved, that's why we can assemble tail
calls only to functions within the same module (PLT is not required).
5. %ecx is livein for regparm(3) functions.
... (maybe something I forget)

--
WBR, Anton Korobeynikov