[llvm-dev] Calling virtual "ELF" functions - BC code

[David Chisnall via llvm-dev]

On 5 Feb 2018, at 21:19, via llvm-dev <llvm-dev at lists.llvm.org> wrote:
> 
> Hello everyone,
> 
> I encountered a strange behaviour which I can't explain.
> 
> I'm developing an application under Windows 7 64bit. This application is using the LLVM library to load and resolve BC files generated with clang. Then the program picks some functions from the module and executes them. That's the theory - but now it's getting strange.
> 
> The BC files are compiled for 64bit windows but have the ELF abi (the application has PECOFF). When I now pick a normal function and call them, everything is fine. But then I decided to call a function which is virtual - the program and the ELF file are both sharing the same interface for this with pure virtual functions. When I now call a virtual function, then I will reach the correct function - but the return value will be faulty, which then corruptes the stack and it crashes. But the this pointer - when the function was called - is correct.
> What's more stranger: I casted the address of the virtual function "down" to a normal function, but with the same return value. When I call this construct, then the return value is correct!

This doesn’t sound like an ELF issue.  On Windows, virtual functions use the ‘thiscall’ calling convention, where the *callee* cleans up the stack.  In the standard Windows x86 calling convention, ecx is the first integer or pointer argument.  In the thiscall calling convention, it is the `this` pointer, so if you are creating a function with the normal calling convention and calling it with the thiscall convention then you will expect to see something like this: the argument is in the right place, but the caller expects the callee to clean up the stack and the callee expects the caller to do it.

In your IR, are the functions that you intend to use as virtual functions declared to use the X86_ThisCall calling convention?

David