[llvm-dev] __attribute__((apple_abi)): targeting Apple/ARM64 ABI from Linux (and others)

[Adrien Guinet via llvm-dev]

Hello everyone,

I made a quick patch to clang/llvm to introduce an "apple_abi" function attribute
(https://github.com/aguinet/llvm-project/commit/c4905ded3afb3182435df30e527955031cb0d098),
to be able to compile functions for the Apple ARM64 ABI when targeting other ARM64 OSes
(e.g. Linux). This can be seen as the Apple version of the already existing "ms_abi"
attribute.

In this mail, I will describe why we would want to do such a thing, the current
implementation and some remaining questions I have about this (like "isn't this a terrible
idea").

Motivation
==========

The motivation comes a bit from far away, I'll try to make it quick. We have various
libraries that targets what I call the "infernal combo", that is
Android/iOS/OSX/Windows/Linux, for every major architectures that are supported by these
OS. For iOS, we thus need to support armv7/arm64 and all their flavors. As we like to test
things, we have to run at some points binaries under iOS/{armv7,arm64}, which is not
something really easy to do provided the official Apple hardware.

In one of other attempts to make all this mess easier to handle, we adapted the
https://github.com/shinh/maloader project (that will be open source if all of this works)
to load ARM64 MachO under Linux and run the final binary using qemu-user. This can be seen
as a very light version of wine [1] for iOS.

Where troubles come in
----------------------

All of this could have "just worked", but Apple has a different ABI than the "official"
ARM64 one. All of this is explained here:
https://developer.apple.com/documentation/xcode/writing_arm64_code_for_apple_platforms .

In an attempt to fix this problem, the idea is to follow the "wine [1] spirit" and write
"ABI wrappers" against the libSystem functions that cause troubles, and have the custom
MachO loader use these functions instead of the linux/libc ones. The next problem to solve
is how to write these wrappers.

One idea was thus to implement the counter-part of "ms_abi" for Apple, hence "apple_abi".

The current implementation & questions
======================================

The current implementation introduces the CC_AArch64_Apple calling convention, to enforce
the usage of Apple's CC when necessary. This has mainly been inspired by how CC_Win64 works.

There are I think at least these limitations:

* this supposes that the original targeted CC is Apple ARM64 AAPCS. In its current form,
there is no way to support for instance vector calls (see for instance
https://github.com/aguinet/llvm-project/commit/c4905ded3afb3182435df30e527955031cb0d098#diff-f124368bac3e5d7be20450aa83b166daR218)
* I haven't tested for a binary that targets Windows/ARM64, so chances are there are bugs
in this configuration
* obviously the commit misses proper tests

My questions would be:
* the fact that we can't target Apple's vector calls ABI shows that having one
CC_AArch64Apple (as CC_Win64 exists) calling convention might not be the right
implementation of this "apple_abi" attribute. Has someone better suggestions?

* For variadic functions (which are among the functions that have different ABIs), GCC and
Clang have __builtin_ms_va_list. My understanding is that we should have the Apple
equivalent, but I'm not sure to completely understand what's at stake here. Said
differently, is this builtin used to make sure we use the va_list type of the Apple ABI,
should the need arise to forward it to another function that uses the Apple ABI?

Example with printf
===================

For now, we manage to compile this simple example for iOS/arm64:

#include <stdio.h>

int main(int argc, char** argv)
{
  printf("number of args: %d, argv: %s, %s, %s\n", argc, argv[0], argv[1], argv[2]);
  return 0;
}

and run it under the combo maloader/qemu-user under Linux/x64, using this wrapper for printf:

__attribute__((apple_abi)) int darwin_aarch64_printf(const char* format, ...)
{
  va_list args;
  va_start(args, format);
  const int ret = vprintf(format, args);
  va_end(args);
  return ret;
}

The fact that va_start/va_end works by using the Linux ABI from a function whose arguments
use the Apple ABI seems completely magical to me, so if someone knows why this work I
would also be interested!


Is this a terrible idea?
========================

Building these "ABI wrappers" using an "apple_abi" attribute seemed a good idea at the
beginning, but this already raises some concerns (see above), and I'd be willing to hear
any arguments that show that this is actually a bad idea.

Thanks everyone!


P.S.: about the original motivation, the darlinghq project [2] can be seen as the real
wine [1] for OSX [3]. Unfortunately, as far as I know, it still doesn't have official
support for iOS/ARM64 binaries (and I'm not sure they will aim at full emulation, only
support on native arm64 hardware).

[1] https://www.winehq.org/
[2] https://www.darlinghq.org/
[3] What I say here isn't entirely true, as darlinghq moved away from this "wine" model
(which can be seen very basically as make a loader for the targeted architecture, create
wrappers for system libraries and run all of this in userland). For those interested in
more information, I recommend reading the article in
http://blog.darlinghq.org/2017/02/the-mach-o-transition-darling-in-past-5.html