[LLVMdev] Replacing IR call instructions by Intrinsics

[Jan Teske]

Hello,

I want to add some instrumentation to the call instructions of
compiled X86 binaries. Specifically, the return address shall be
encrypted before it is pushed on the stack. This is something that
cannot be done in IR. So I thought of using an Intrinsic that can
replace IR call instructions. That way I could still write IR passes
and would not have to get down to the machine layer (except for
implementing the Intrinsic).

I would like to know, if what I'm trying to do is even possible with
Intrinsics. Basically, the Intrinsic would have to do everything call
does, and more. It would:
- get the address of the called functions and the arguments of the
call as parameters
- push the arguments on the stack
- encrypt the return address and push the encrypted value on the stack
- jump to the address of the called function

I'm aware that I will also have to change the ret instruction. But I
already figured out that this will not work on the IR layer. My
question is only about this call Intrinsic.

Problem 1: Would I be able to get the return address (i.e. the address
of the instruction after the call instruction) somehow from within the
intrinsic? I guess, memory addresses are not known before the machine
code is generated. Is there some mechanism in LLVM, that would help me?

Problem 2: What about the argument and return types of the intrinsics?
I guess, I could use an i64 for the (64-bit) function address and a
vararg type for the arguments? But I have no idea how to specify the
return value. Since the return value depends on the specific function
that is called, I simply cannot know it. It could be void, a pointer
or a struct, for example.

These are only the problems I can see until now. More might arise when
I try to dig into how intrinsics are implemented. But before doing
this, I would like to know if it's worth the effort or if I would be
better of looking at lower levels (MachineInstr, MC layer, ...).

Thanks for your help!
Jan