[llvm] [Hexagon] Bugfix in VarArg lowering: Special case for musl (PR #157564)

[Benedikt Lukas Huber via llvm-commits]

https://github.com/benedikt-l-huber updated https://github.com/llvm/llvm-project/pull/157564

>From a0804ee60f9fba8f5acb94d335948ca26a121b38 Mon Sep 17 00:00:00 2001
From: Benedikt Huber <benedikt.huber at tuwien.ac.at>
Date: Mon, 8 Sep 2025 21:21:20 +0200
Subject: [PATCH 1/2] [Hexagon] Bugfix in VarArg lowering: Special case for
 musl

When compiling for musl varargs are treated like non varargs.
Thus the check must also include the State.isVarArg(),
since this is determined by the TreatAsVarArg argument,
that depends on the musl environment.

Possibly, this fixes #145206.
---
 llvm/lib/Target/Hexagon/HexagonCallingConv.td |  8 ++++++--
 llvm/test/CodeGen/Hexagon/vararg-musl.ll      | 15 +++++++++++++++
 2 files changed, 21 insertions(+), 2 deletions(-)
 create mode 100644 llvm/test/CodeGen/Hexagon/vararg-musl.ll

diff --git a/llvm/lib/Target/Hexagon/HexagonCallingConv.td b/llvm/lib/Target/Hexagon/HexagonCallingConv.td
index fd6d873dd4188..96c41cdb2b7ee 100644
--- a/llvm/lib/Target/Hexagon/HexagonCallingConv.td
+++ b/llvm/lib/Target/Hexagon/HexagonCallingConv.td
@@ -6,6 +6,10 @@
 //
 //===----------------------------------------------------------------------===//
 
+
+class CCIfBothVarArgAndArgVarArg<CCAction A>
+  : CCIf<"State.isVarArg() && ArgFlags.isVarArg()", A>;
+
 def CC_HexagonStack: CallingConv<[
   CCIfType<[i32,v2i16,v4i8],
     CCAssignToStack<4,4>>,
@@ -23,7 +27,7 @@ def CC_Hexagon_Legacy: CallingConv<[
 
   CCIfByVal<
     CCPassByVal<8,8>>,
-  CCIfArgVarArg<
+  CCIfBothVarArgAndArgVarArg<
     CCDelegateTo<CC_HexagonStack>>,
 
   // Pass split values in pairs, allocate odd register if necessary.
@@ -53,7 +57,7 @@ def CC_Hexagon: CallingConv<[
 
   CCIfByVal<
     CCPassByVal<8,1>>,
-  CCIfArgVarArg<
+  CCIfBothVarArgAndArgVarArg<
     CCDelegateTo<CC_HexagonStack>>,
 
   // Pass split values in pairs, allocate odd register if necessary.
diff --git a/llvm/test/CodeGen/Hexagon/vararg-musl.ll b/llvm/test/CodeGen/Hexagon/vararg-musl.ll
new file mode 100644
index 0000000000000..b902dded32153
--- /dev/null
+++ b/llvm/test/CodeGen/Hexagon/vararg-musl.ll
@@ -0,0 +1,15 @@
+; RUN: llc -mtriple=hexagon-unknown-linux-musl < %s | FileCheck %s -check-prefix=MUSL
+; RUN: llc -mtriple=hexagon-unknown-none-elf   < %s | FileCheck %s -check-prefix=NONMUSL
+
+; MUSL-NOT: memw
+; NONMUSL: memw
+
+declare i32 @f0(i32 %a0, ...)
+
+define i32 @f1(i32 %a0, i32 %a1) #0 {
+b1:
+  %v7 = call i32 (i32, ...) @f0(i32 %a0, i32 %a1)
+  ret i32 %v7
+}
+
+attributes #0 = { nounwind }

>From e376fc7d7f8ebd6bea47b1e0b665e8cacb8cf298 Mon Sep 17 00:00:00 2001
From: Benedikt Huber <benedikt.huber at tuwien.ac.at>
Date: Wed, 17 Sep 2025 13:47:14 +0200
Subject: [PATCH 2/2] [Hexagon] CallingConv rename class and explain in comment

---
 llvm/lib/Target/Hexagon/HexagonCallingConv.td | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/llvm/lib/Target/Hexagon/HexagonCallingConv.td b/llvm/lib/Target/Hexagon/HexagonCallingConv.td
index 96c41cdb2b7ee..dceb70c8abbf2 100644
--- a/llvm/lib/Target/Hexagon/HexagonCallingConv.td
+++ b/llvm/lib/Target/Hexagon/HexagonCallingConv.td
@@ -6,8 +6,13 @@
 //
 //===----------------------------------------------------------------------===//
 
-
-class CCIfBothVarArgAndArgVarArg<CCAction A>
+// We cannot use the standard CCIfArgVarArg class here since in Hexagon there
+// exists a special case for a musl environment.  In a musl environment VarArgs
+// are treated like non VarArgs.  I.e., in a musl enviroment unnamed arguments
+// can also be passed in registers.  The CCIfArgVarArg class only checks each
+// individual argument, but not whether State.isVarArg() is true.  We also have
+// to check State.isVarArg() which is determined by the TreatAsVarArg argument.
+class CCIfStateVarArgAndArgVarArg<CCAction A>
   : CCIf<"State.isVarArg() && ArgFlags.isVarArg()", A>;
 
 def CC_HexagonStack: CallingConv<[
@@ -27,7 +32,7 @@ def CC_Hexagon_Legacy: CallingConv<[
 
   CCIfByVal<
     CCPassByVal<8,8>>,
-  CCIfBothVarArgAndArgVarArg<
+  CCIfStateVarArgAndArgVarArg<
     CCDelegateTo<CC_HexagonStack>>,
 
   // Pass split values in pairs, allocate odd register if necessary.
@@ -57,7 +62,7 @@ def CC_Hexagon: CallingConv<[
 
   CCIfByVal<
     CCPassByVal<8,1>>,
-  CCIfBothVarArgAndArgVarArg<
+  CCIfStateVarArgAndArgVarArg<
     CCDelegateTo<CC_HexagonStack>>,
 
   // Pass split values in pairs, allocate odd register if necessary.