[LLVMdev] LLVM backend: Treat some function calls specially

[Jonas Gefele]

Hello,

I am writing an LLVM backend that generates byte code for a custom virtual
machine. Standard function calls are lowered to a simple
   CALL $offsetInByteCode
This works fine so far using the standard machinery of LLVM. But some
functions are not implemented in byte code, but delegated to native
implementations within the VM. Calls to these functions use a non-standard
calling convention and should be lowered to another opcode:
   CALL_NATIVE $functionIndex
where $functionIndex is a magic byte telling the VM which native function
should be called. As the set of native functions and function indices will
be extended over time, it should not be hardcoded into the backend but
given as input at compile time.


I have some vague ideas on how I could implement this, but would like to
ask for your ideas in order to keep me from going too far in a
wrong/cumbersome direction.


The first option that came into my mind is adding a new intrinsic
"llvm.myVM.call_native(i32 %funcIdx, ...)" and then linking at compile
time a module that looks like
   define void @foo(i32 %a) {
      call @llvm.myVM.call_native(i32 1, i32 %a)
   }
   define void @bar(i32 %a, i32 %b) {
      call @llvm.myVM.call_native(i32 2, i32 %a, i32 %b)
   }
   ...
But here I am unsure how to handle the return type of this intrinsic (can
it be overloaded to accept any return type?) and whether I can reuse the
TargetLowering::LowerCall and CCState::AnalyzeFormalArguments stuff.



Maybe another option is to add some kind of metadata to these function
declarations and then check in "LowerCall" whether these metadata flags
are present. If so, then extract the funcIdx from metadata and lower this
call in a special way. If this is doable, are there some example of how I
can do that (or similar things)?


Maybe, you have an even better idea?


Many thanks in advance,
Jonas