[LLVMdev] Using sret AND thiscall calling convention

[Eli Friedman]

On Fri, Aug 12, 2011 at 1:11 AM, Damien Gleizes <gleizesd at gmail.com> wrote:
>
> Hi all,
>
> I am trying to generate LLVM IR that calls an external C++ function
> returning a structure by copy:
> vec3 vec3::Cross(const vec3& iV) const;
>
> Here is the LLVM IR that I am generating for win32 ABI, which says that the
> first parameter should be a pointer to the return value:
>
> %vec3 = type <{ float, float, float }>
>
> define void @CPP_Return_Struct(%vec3* %v1, %vec3* %v2) inlinehint
> alignstack(16) {
> ENTRY:
>  %"@0" = alloca %vec3, align 4
>  call x86_thiscallcc void @vec3_Cross(%vec3* sret %"@0", %vec3* %v1, %vec3*
> %v2)
>  ret void
> }
>
> declare x86_thiscallcc void @vec3_Cross(%vec3* sret, %vec3*, %vec3*) align 2
>
> It seems that the code generated by the JIT takes the first parameter for
> the this value because of the X86_thiscall calling convention. However, in
> this particular case, the first parameter is the return value which is
> tagged with the sret attribute.
>
> I have manually modified the X86GenCallingConv.inc to treat the second
> parameter as this value when the first has been tagged with sret attribute,
> and it is working fine.
>
> But I am not sure if it is a (known?) bug or if I do not generate the
> correct LLVM IR.
>
> I found nothing about this on the mailing list, I apologize if I missed any
> important information.

I would suggest avoiding any complicated calls from LLVM-generated
code into Visual Studio-generated code.  For example,
http://llvm.org/bugs/show_bug.cgi?id=5058 ,
http://llvm.org/bugs/show_bug.cgi?id=5064 .  Not sure about the
particular issue you're running into; looks like a bug at first
glance.  The file that mostly controls this sort of thing is
lib/Target/X86/X86CallingConv.td .

-Eli