[llvm] [WebAssembly] Disallow tail calls with byval arguments (PR #125142)

[Thomas Lively via llvm-commits]

https://github.com/tlively created https://github.com/llvm/llvm-project/pull/125142

WebAssembly disallows tail calls with stack-allocated arguments because they tail calls leave no way to fix up the stack pointer after the call. WebAssembly also passes `byval` arguments on the stack, but we were not considering them when deciding whether a tail call should be allowed. Properly disallow tail calls in the presence of byval arguments and fix the existing test that should have caught this.

Fixes #124443.

>From 53f0accda525806ac222aacfcc7d524a7ae8fb9b Mon Sep 17 00:00:00 2001
From: Thomas Lively <tlively at google.com>
Date: Thu, 30 Jan 2025 16:19:28 -0800
Subject: [PATCH] [WebAssembly] Disallow tail calls with byval arguments

WebAssembly disallows tail calls with stack-allocated arguments because
they tail calls leave no way to fix up the stack pointer after the call.
WebAssembly also passes `byval` arguments on the stack, but we were not
considering them when deciding whether a tail call should be allowed.
Properly disallow tail calls in the presence of byval arguments and fix
the existing test that should have caught this.

Fixes #124443.
---
 llvm/lib/Target/WebAssembly/WebAssemblyISelLowering.cpp | 7 +++++++
 llvm/test/CodeGen/WebAssembly/tailcall.ll               | 7 ++++---
 2 files changed, 11 insertions(+), 3 deletions(-)

diff --git a/llvm/lib/Target/WebAssembly/WebAssemblyISelLowering.cpp b/llvm/lib/Target/WebAssembly/WebAssemblyISelLowering.cpp
index 02db1b142a22b5..4cc06f057a5760 100644
--- a/llvm/lib/Target/WebAssembly/WebAssemblyISelLowering.cpp
+++ b/llvm/lib/Target/WebAssembly/WebAssemblyISelLowering.cpp
@@ -1211,6 +1211,13 @@ WebAssemblyTargetLowering::LowerCall(CallLoweringInfo &CLI,
         }
       }
     }
+    // Similarly, we cannot tail call with byval arguments.
+    for (unsigned I = 0; I < CLI.Outs.size(); ++I) {
+      const ISD::OutputArg &Out = CLI.Outs[I];
+      if (Out.Flags.isByVal() && Out.Flags.getByValSize() != 0)
+        NoTail(
+            "WebAssembly does not support tail calling with stack arguments");
+    }
   }
 
   SmallVectorImpl<ISD::InputArg> &Ins = CLI.Ins;
diff --git a/llvm/test/CodeGen/WebAssembly/tailcall.ll b/llvm/test/CodeGen/WebAssembly/tailcall.ll
index 84bd142462e37e..a1eeaf9f2e77e4 100644
--- a/llvm/test/CodeGen/WebAssembly/tailcall.ll
+++ b/llvm/test/CodeGen/WebAssembly/tailcall.ll
@@ -296,12 +296,13 @@ define i32 @mismatched_byval(ptr %x) {
 ; CHECK-NEXT:    global.set __stack_pointer, $pop7
 ; CHECK-NEXT:    i32.load $push0=, 0($0)
 ; CHECK-NEXT:    i32.store 12($1), $pop0
+; CHECK-NEXT:    i32.const $push5=, 12
+; CHECK-NEXT:    i32.add $push6=, $1, $pop5
+; CHECK-NEXT:    call $0=, quux, $pop6
 ; CHECK-NEXT:    i32.const $push3=, 16
 ; CHECK-NEXT:    i32.add $push4=, $1, $pop3
 ; CHECK-NEXT:    global.set __stack_pointer, $pop4
-; CHECK-NEXT:    i32.const $push5=, 12
-; CHECK-NEXT:    i32.add $push6=, $1, $pop5
-; CHECK-NEXT:    return_call quux, $pop6
+; CHECK-NEXT:    return $0
   %v = tail call i32 @quux(ptr byval(i32) %x)
   ret i32 %v
 }