[clang] [clang] Always pass `fp128` arguments indirectly on Windows (PR #115052)

[Trevor Gross via cfe-commits]

https://github.com/tgross35 updated https://github.com/llvm/llvm-project/pull/115052

>From 60b07161e8668c2bc3ee5d7a4c470a90a7673178 Mon Sep 17 00:00:00 2001
From: Trevor Gross <tmgross at umich.edu>
Date: Tue, 5 Nov 2024 07:00:35 -0500
Subject: [PATCH 1/2] [clang] Add fp128 ABI tests for MinGW (NFC)

Duplicate `win64-i128.c` to `win64-fp128.c` and update with the current
behavior of `__float128`.
---
 clang/test/CodeGen/win64-fp128.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
 create mode 100644 clang/test/CodeGen/win64-fp128.c

diff --git a/clang/test/CodeGen/win64-fp128.c b/clang/test/CodeGen/win64-fp128.c
new file mode 100644
index 0000000000000..33e2441ddf314
--- /dev/null
+++ b/clang/test/CodeGen/win64-fp128.c
@@ -0,0 +1,20 @@
+// RUN: %clang_cc1 -triple x86_64-windows-gnu -emit-llvm -o - %s \
+// RUN:    | FileCheck %s --check-prefix=GNU64
+// __float128 is unsupported on MSVC
+
+__float128 fp128_ret(void) { return 0; }
+// GNU64: define dso_local fp128 @fp128_ret()
+
+__float128 fp128_args(__float128 a, __float128 b) { return a * b; }
+// GNU64: define dso_local fp128 @fp128_args(fp128 noundef %a, fp128 noundef %b)
+
+void fp128_vararg(int a, ...) {
+  // GNU64-LABEL: define dso_local void @fp128_vararg
+  __builtin_va_list ap;
+  __builtin_va_start(ap, a);
+  __float128 i = __builtin_va_arg(ap, __float128);
+  // movaps  xmm0, xmmword ptr [rax]
+  // GNU64: load ptr, ptr
+  // GNU64: load fp128, ptr
+  __builtin_va_end(ap);
+}

>From afaf2cb8b8a99a064a8daa5e11c45c1e7c3ee770 Mon Sep 17 00:00:00 2001
From: Trevor Gross <tmgross at umich.edu>
Date: Tue, 5 Nov 2024 05:53:10 -0500
Subject: [PATCH 2/2] [clang] Pass `fp128` indirectly and return in xmm0 on
 Windows

Clang currently passes and returns `__float128` in vector registers on
MinGW targets, which is LLVM's default ABI for `fp128`. However, the
Windows x86-64 calling convention [1] states the following:

    __m128 types, arrays, and strings are never passed by immediate
    value. Instead, a pointer is passed to memory allocated by the
    caller. Structs and unions of size 8, 16, 32, or 64 bits, and __m64
    types, are passed as if they were integers of the same size. Structs
    or unions of other sizes are passed as a pointer to memory allocated
    by the caller. For these aggregate types passed as a pointer,
    including __m128, the caller-allocated temporary memory must be
    16-byte aligned.

Based on the above it sounds like `__float128` should be passed
indirectly; this is what MinGW GCC already does, so change Clang to
match. Passing by value also causes problems with varargs. E.g. the
below completes successfully when built with GCC but has a runtime crash
when built with Clang:

    void va_f128(int count, ...) {
        va_list args;
        va_start(args, count);
        __float128 val = va_arg(args, __float128);
        va_end(args);
    }

    int main() {
        va_f128(0, 0.0);
    }

Change `f128` passing to use the stack and make the return in xmm0
explicit. This is the identical to `i128`, and passing is the same as
GCC.

Regarding return values, the documentation states:

    A scalar return value that can fit into 64 bits, including the __m64
    type, is returned through RAX. Non-scalar types including floats,
    doubles, and vector types such as __m128, __m128i, __m128d are
    returned in XMM0.

This makes it sound like it should be acceptable to return `__float128`
in xmm0; however, GCC returns `__float128` on the stack. That above ABI
statement as well as consistency with `i128` (which is returned in xmm0)
mean that it would likely be better for GCC to change its return ABI to
match Clang rather than the other way around, so that portion is left
as-is.

Clang's MSVC targets do not support `__float128` or `_Float128`, but
these changes would also apply there if it is eventually enabled.

[1]: https://learn.microsoft.com/en-us/cpp/build/x64-calling-convention?view=msvc-170
---
 clang/lib/CodeGen/Targets/X86.cpp | 5 +++++
 clang/test/CodeGen/win64-fp128.c  | 5 ++---
 2 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/clang/lib/CodeGen/Targets/X86.cpp b/clang/lib/CodeGen/Targets/X86.cpp
index b7a1374d5b399..145c60d89d79e 100644
--- a/clang/lib/CodeGen/Targets/X86.cpp
+++ b/clang/lib/CodeGen/Targets/X86.cpp
@@ -3390,6 +3390,9 @@ ABIArgInfo WinX86_64ABIInfo::classify(QualType Ty, unsigned &FreeSSERegs,
 
     case BuiltinType::Int128:
     case BuiltinType::UInt128:
+    case BuiltinType::Float128:
+      // 128-bit float and integer types share the same ABI.
+      
       // If it's a parameter type, the normal ABI rule is that arguments larger
       // than 8 bytes are passed indirectly. GCC follows it. We follow it too,
       // even though it isn't particularly efficient.
@@ -3400,6 +3403,8 @@ ABIArgInfo WinX86_64ABIInfo::classify(QualType Ty, unsigned &FreeSSERegs,
 
       // Mingw64 GCC returns i128 in XMM0. Coerce to v2i64 to handle that.
       // Clang matches them for compatibility.
+      // NOTE: GCC actually returns f128 indirectly but will hopefully change.
+      // See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=115054#c8.
       return ABIArgInfo::getDirect(llvm::FixedVectorType::get(
           llvm::Type::getInt64Ty(getVMContext()), 2));
 
diff --git a/clang/test/CodeGen/win64-fp128.c b/clang/test/CodeGen/win64-fp128.c
index 33e2441ddf314..943325eb373f7 100644
--- a/clang/test/CodeGen/win64-fp128.c
+++ b/clang/test/CodeGen/win64-fp128.c
@@ -3,17 +3,16 @@
 // __float128 is unsupported on MSVC
 
 __float128 fp128_ret(void) { return 0; }
-// GNU64: define dso_local fp128 @fp128_ret()
+// GNU64: define dso_local <2 x i64>  @fp128_ret()
 
 __float128 fp128_args(__float128 a, __float128 b) { return a * b; }
-// GNU64: define dso_local fp128 @fp128_args(fp128 noundef %a, fp128 noundef %b)
+// GNU64: define dso_local <2 x i64> @fp128_args(ptr noundef %0, ptr noundef %1)
 
 void fp128_vararg(int a, ...) {
   // GNU64-LABEL: define dso_local void @fp128_vararg
   __builtin_va_list ap;
   __builtin_va_start(ap, a);
   __float128 i = __builtin_va_arg(ap, __float128);
-  // movaps  xmm0, xmmword ptr [rax]
   // GNU64: load ptr, ptr
   // GNU64: load fp128, ptr
   __builtin_va_end(ap);