[cfe-dev] How to add a new calling convention?

[Nathan Sidwell via cfe-dev]

On 7/12/21 3:22 AM, Gaier, Bjoern via cfe-dev wrote:
> Hello Clang-Devs,
> 
> this question originally came from the LLVM Discord server:
> 
> Currently we are using an inhouse library which a different department 
> developed for us. I'm not aware about what they are doing in there 
> implementation, however whenever we use functions from said library we 
> are instructed to save the GS register. Currently we manage this by 
> using inline assembly, we push the register before the call, do the call 
> and pop it back. However this is very error prone because you can easily 
> forget to do so.

Ew, that's horrible.  (I recall having to do it once when a library 
clobbered SI & DI, which are supposed to be callee saved.)

> 
> My hope was to teach Clang an additional calling convention which we 
> could explicitly declare with "__specialcall" or something. That 
> convention would push the GS register for us and then do the normal 
> x86_64 calling convention. In Discord I was told that this is possible 
> but that there are no instructions for this.
> 
> Can someone guide me in the right direction for doing such a thing? 
> Hopefully in a way a beginner can pick it up >o<

You're going to have to modify the X86 code generation directly.  Mark 
the functions of interest with some kind of new attribute, and then look 
for that in the call generation code.

llvm/lib/Target/X86/X86ISelLowering.cpp is the file of interest there, 
make GS conditionally a caller-saved register in some way.  More than 
that I don't know.

(Just in case you're unaware, GS is used to access TLS data in the x86 
ABI.  Not sure if that's true on windows though.)

> 
> Thank you in advance and kind greetings
> 
> Björn
> 
> Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816, 
> USt.ID-Nr. DE 114 165 789 Geschäftsführer: Dr. Hiroshi Nakamura, Dr. 
> Robert Plank, Markus Bode, Takashi Nagano, Junichi Tajika, Ergin Cansiz.
> 
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
> 


-- 
Nathan Sidwell