[llvm] fix preserve_none in X86 backend (PR #192300)

[via llvm-commits]

llvmbot wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-backend-x86

Author: Jerry Dang (kuroyukiasuna)

<details>
<summary>Changes</summary>

This is a issue that's specific to the X86 backend: preserve_none with -O1, IR looks like:
```
tail call preserve_nonecc void @<!-- -->_Z14func(ptr noundef nonnull byval(%struct.S) align 8 %1)
```
Where we have tailcall on a func with preserve_none cc, and a byval in the parameter.
This slips through the `isEligibleForSiblingCallOpt` check and thinks this is a sibling call, because `byval` for `preserve_none` is passed as a register, not memory. As a result, the lowering logic thinks this is a sibling call and does not reserve stack space for it. But preserve_none with byval would try copy outgoing args in the stack, corrupting the frame. This leads to a runtime crash.

The propose fix is to check for non-standard CC like `preserve_none` and disqualify it for sibling call.

Fixes #<!-- -->188930 

---
Full diff: https://github.com/llvm/llvm-project/pull/192300.diff


2 Files Affected:

- (modified) llvm/lib/Target/X86/X86ISelLoweringCall.cpp (+9) 
- (modified) llvm/test/CodeGen/X86/preserve_nonecc_call.ll (+18) 


``````````diff
diff --git a/llvm/lib/Target/X86/X86ISelLoweringCall.cpp b/llvm/lib/Target/X86/X86ISelLoweringCall.cpp
index 37c80e27f4bd2..f1a0a0047a4fd 100644
--- a/llvm/lib/Target/X86/X86ISelLoweringCall.cpp
+++ b/llvm/lib/Target/X86/X86ISelLoweringCall.cpp
@@ -2953,6 +2953,15 @@ bool X86TargetLowering::isEligibleForSiblingCallOpt(
   if (!mayTailCallThisCC(CalleeCC))
     return false;
 
+  // preserve_nonecc uses a distinct caller/callee-saved register convention.
+  // Be conservative and avoid sibling tail calls when byval arguments are
+  // present, as they require stack argument copying with stricter layout
+  // constraints.
+  if (CalleeCC == CallingConv::PreserveNone)
+    for (const auto &Out : Outs)
+      if (Out.Flags.isByVal())
+        return false;
+
   // If -tailcallopt is specified, make fastcc functions tail-callable.
   MachineFunction &MF = DAG.getMachineFunction();
   X86MachineFunctionInfo *FuncInfo = MF.getInfo<X86MachineFunctionInfo>();
diff --git a/llvm/test/CodeGen/X86/preserve_nonecc_call.ll b/llvm/test/CodeGen/X86/preserve_nonecc_call.ll
index 500ebb139811a..df1ef74b86ae3 100644
--- a/llvm/test/CodeGen/X86/preserve_nonecc_call.ll
+++ b/llvm/test/CodeGen/X86/preserve_nonecc_call.ll
@@ -55,6 +55,24 @@ define preserve_nonecc void @caller2(ptr %a) {
   ret void
 }
 
+%S = type { [3 x i64] }
+declare preserve_nonecc void @callee_byval(ptr byval(%S) align 8)
+
+; Preserve_none caller with a byval argument should not be lowered as a
+; sibling tail call on X86.
+define preserve_nonecc void @caller_byval(ptr %a) {
+; CHECK-LABEL: caller_byval:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    pushq %rax
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    callq callee_byval at PLT
+; CHECK-NEXT:    popq %rax
+; CHECK-NEXT:    .cfi_def_cfa_offset 8
+; CHECK-NEXT:    retq
+  tail call preserve_nonecc void @callee_byval(ptr byval(%S) align 8 %a)
+  ret void
+}
+
 ; Preserve_none function can use more registers to pass parameters.
 declare preserve_nonecc i64 @callee_with_many_param2(i64 %a1, i64 %a2, i64 %a3, i64 %a4, i64 %a5, i64 %a6, i64 %a7, i64 %a8, i64 %a9, i64 %a10, i64 %a11)
 define preserve_nonecc i64 @callee_with_many_param(i64 %a1, i64 %a2, i64 %a3, i64 %a4, i64 %a5, i64 %a6, i64 %a7, i64 %a8, i64 %a9, i64 %a10, i64 %a11, i64 %a12) {

``````````

</details>


https://github.com/llvm/llvm-project/pull/192300