[cfe-dev] How to add a new calling convention?

[David Chisnall via cfe-dev]

Hi,

On 13/07/2021 07:47, Gaier, Bjoern via cfe-dev wrote:
> Thank you for the reply guys :D
> 
> Seems like adding the calling convention might bring more pain then I thought...
> Is it a better idea to tackle this issue on IR level then? Making Clang to not compile to object files, but IR instead, patching the effected calls and then passing it to static compiler? I haven't done such a thing so I honestly have no idea how easy it is to locate the calls that need patching (not every function call would need that) and then actually applying the patch.

I think you need to do both parts anyway: clang doesn't generate object 
code, clang generates LLVM IR and LLVM generates object code.  If you 
want a new calling convention exposed to C/C++ then you first need to 
expose a new clang calling convention, make clang lower it to a new IR 
calling convention, and then modify the back ends to handle the new 
calling convention.

The good news is that if you add the calling conventions to the two 
enums (for C in clang, for IR in LLVM) and run a build with something 
like `ninja -k100` then it will tell you all of the places outside of 
the back end where you need to add handling for your new calling 
convention.  You can probably find a similar example near all of these 
to copy.  I've done this a bunch of times and never felt the need to 
note down all of the places because the incomplete enum coverage in 
switch warning finds them all for me.

You need to modify a *lot* of places, but each individual change is 
typically tiny.

David