[cfe-dev] Handling ignored calling conventions

[Aaron Ballman]

I'm trying to figure out a rational design to solve a pretty big
problem in clang.  Right now, clang happily accepts calling
conventions even for architectures that don't support them.  For
instance, stdcall on x64.  Generally speaking, they get lowered from
clang into LLVM IR during codegen and LLVM simply ignores the problem.
 However, this still causes problems such as with name mangling.  The
real kicker is: the Win32 APIs have this mismatch *everywhere* and
MSVC will silently ignore nonsense calling conventions in the most
literal sense of the term.  Eg)

int __stdcall foo();
int __fastcall foo { return 0; }

On x86, this will err because of the mismatch.  On x64, IPF and ARM,
this compiles fine.  This is documented behavior on MSDN, FWIW.  In
any case, it's causing us fits on Win64 because we can't link against
any of the Win32 APIs thanks to mangling.  We mangle as though it's
really stdcall since that's what the attribute says but the OS
libraries mangle as though it were cdecl for x64.

There's been a fair amount of discussion on IRC about the topic, and a
whirlwind of patches from me trying to solve the problem.  What I
believe the consensus has been is:

0) We will ignore calling conventions that do not make sense for the
target architecture
1) We want to warn when calling conventions are ignored because of
target differences
  a) But only if the calling convention wouldn't otherwise cause an
error (eg, __stdcall int foo = 12 should still err regardless of
architecture and not warn).
2) When we ignore a calling convention, we modify it to be CC_Default
at the sema level
 a) This does cause problems with rewriters since it now has the wrong
spelling, but this is acceptable because we have the same problem with
other attributes

This gives us the desired behavior both in terms of semantics as well
as name mangling, and we generate more sensible code for LLVM.

I am looking for a bit of design help as to how to model this.  In my
current incarnation, I am handling all of this in sema by looking at
the target triple and comparing it to the calling convention.  But
this feels messy to me.  I am thinking that pushing this information
into TargetInfo and subclasses is more sensible, but am looking for
confirmation.  Something along the lines of:

virtual bool TargetInfo::AcceptsCallingConvention( CallingConv CC ) const;

So that Sema can just do:

if (!Context.getTargetInfo().AcceptsCallingConvention(CC))
  Ignore = true;

Do you agree with what I think the consensus is?  Does this seem like
a reasonable solution to you?

Thanks!

~Aaron