[cfe-dev] Handling ignored calling conventions

[Douglas Gregor]

On Sep 10, 2012, at 12:26 PM, Aaron Ballman <aaron at aaronballman.com> wrote:

> I'm trying to figure out a rational design to solve a pretty big
> problem in clang.  Right now, clang happily accepts calling
> conventions even for architectures that don't support them.  For
> instance, stdcall on x64.  Generally speaking, they get lowered from
> clang into LLVM IR during codegen and LLVM simply ignores the problem.
> However, this still causes problems such as with name mangling.  The
> real kicker is: the Win32 APIs have this mismatch *everywhere* and
> MSVC will silently ignore nonsense calling conventions in the most
> literal sense of the term.  Eg)
> 
> int __stdcall foo();
> int __fastcall foo { return 0; }
> 
> On x86, this will err because of the mismatch.  On x64, IPF and ARM,
> this compiles fine.  This is documented behavior on MSDN, FWIW.  In
> any case, it's causing us fits on Win64 because we can't link against
> any of the Win32 APIs thanks to mangling.  We mangle as though it's
> really stdcall since that's what the attribute says but the OS
> libraries mangle as though it were cdecl for x64.
> 
> There's been a fair amount of discussion on IRC about the topic, and a
> whirlwind of patches from me trying to solve the problem.  What I
> believe the consensus has been is:
> 
> 0) We will ignore calling conventions that do not make sense for the
> target architecture
> 1) We want to warn when calling conventions are ignored because of
> target differences
>  a) But only if the calling convention wouldn't otherwise cause an
> error (eg, __stdcall int foo = 12 should still err regardless of
> architecture and not warn).
> 2) When we ignore a calling convention, we modify it to be CC_Default
> at the sema level
> a) This does cause problems with rewriters since it now has the wrong
> spelling, but this is acceptable because we have the same problem with
> other attributes
> 
> This gives us the desired behavior both in terms of semantics as well
> as name mangling, and we generate more sensible code for LLVM.
> 
> I am looking for a bit of design help as to how to model this.  In my
> current incarnation, I am handling all of this in sema by looking at
> the target triple and comparing it to the calling convention.  But
> this feels messy to me.  I am thinking that pushing this information
> into TargetInfo and subclasses is more sensible, but am looking for
> confirmation.  Something along the lines of:
> 
> virtual bool TargetInfo::AcceptsCallingConvention( CallingConv CC ) const;
> 
> So that Sema can just do:
> 
> if (!Context.getTargetInfo().AcceptsCallingConvention(CC))
>  Ignore = true;

Pushing this information into TargetInfo makes sense, but I don't think this solves the whole problem. If __stdcall and __fastcall are the same on MSVC's x64, the AcceptsCallingConversion function doesn't give us enough information to tell that. I think what you need is for the target to provide:

  - A mapping from a given calling convention to its canonical calling convention, which establishes identity in the type system
  - The default calling conversion if none is supplied
  - Whether a particular calling conversion is supported or not

> Do you agree with what I think the consensus is?  Does this seem like
> a reasonable solution to you?


Seems like a good direction.

	- Doug