[cfe-dev] How to add a new calling convention?

[Keane, Erich via cfe-dev]

Adding calling conventions is actually quite awful, you can find my old patches implementing RegCall on Phab somewhere if you'd like some examples.

A big part of the problem is that we store calling conventions directly as a part of the FunctionType, so every one you need to add increases the size of FunctionType (and therefore harms the whole compiler, even if not used), which is quite unfortunate to have to do.[0].  You DO need it to be in the type system (even in your example) because pointers to these functions need to correctly copy/have the correct semantics.

Then, once you get it into the type system correctly, you need to handle it in clang-codegen, in the lib/CodeGen/TargetInfo.cpp versions of EACH target that you want to support it on.  This code itself is actually quite terrible in itself, as it is quite disorganized and unmaintained.  Additionally, X86/X86-64 is actually implemented across FOUR different types there, since there are those plus bifurcated for Linux and Windows.  This is less of a concern for YOUR type, since the new 'calling convention' is actually not affecting the normal calling mechanism (that is, it doesn't change how parameter types are passed), but you'll still want to make sure we label the function types correctly in the IR.

THEN, you'd need to get it to be in the IR, and do what Nathan says below.  I'd STRONGLY discourage adding a new calling convention if at all possible...  

[0] I have on my backlog to design/implement a mechanism to get some of these bits 'back' for other platforms by making the bits target-specific (so we're not using bits in x86 mode to represent arm calling conventions that are impossible to get, and also to free up the use of these bits for calling conventions of other targets), but I haven't had a chance to get to it yet.

-----Original Message-----
From: cfe-dev <cfe-dev-bounces at lists.llvm.org> On Behalf Of Nathan Sidwell via cfe-dev
Sent: Monday, July 12, 2021 9:37 AM
To: Gaier, Bjoern <Bjoern.Gaier at horiba.com>; clang developer list <cfe-dev at lists.llvm.org>
Subject: Re: [cfe-dev] How to add a new calling convention?

On 7/12/21 3:22 AM, Gaier, Bjoern via cfe-dev wrote:
> Hello Clang-Devs,
> 
> this question originally came from the LLVM Discord server:
> 
> Currently we are using an inhouse library which a different department 
> developed for us. I'm not aware about what they are doing in there 
> implementation, however whenever we use functions from said library we 
> are instructed to save the GS register. Currently we manage this by 
> using inline assembly, we push the register before the call, do the 
> call and pop it back. However this is very error prone because you can 
> easily forget to do so.

Ew, that's horrible.  (I recall having to do it once when a library clobbered SI & DI, which are supposed to be callee saved.)

> 
> My hope was to teach Clang an additional calling convention which we 
> could explicitly declare with "__specialcall" or something. That 
> convention would push the GS register for us and then do the normal
> x86_64 calling convention. In Discord I was told that this is possible 
> but that there are no instructions for this.
> 
> Can someone guide me in the right direction for doing such a thing? 
> Hopefully in a way a beginner can pick it up >o<

You're going to have to modify the X86 code generation directly.  Mark the functions of interest with some kind of new attribute, and then look for that in the call generation code.

llvm/lib/Target/X86/X86ISelLowering.cpp is the file of interest there, make GS conditionally a caller-saved register in some way.  More than that I don't know.

(Just in case you're unaware, GS is used to access TLS data in the x86 ABI.  Not sure if that's true on windows though.)

> 
> Thank you in advance and kind greetings
> 
> Björn
> 
> Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816, 
> USt.ID-Nr. DE 114 165 789 Geschäftsführer: Dr. Hiroshi Nakamura, Dr.
> Robert Plank, Markus Bode, Takashi Nagano, Junichi Tajika, Ergin Cansiz.
> 
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
> 


--
Nathan Sidwell
_______________________________________________
cfe-dev mailing list
cfe-dev at lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev