[llvm] [X86] Fix getStackProbeSymbolName returning non-callable values on Windows (PR #210800)

[via llvm-commits]

https://github.com/nullDxe updated https://github.com/llvm/llvm-project/pull/210800

>From b4987cb7e87259f0d4e661a8ee0ffc480219b2bc Mon Sep 17 00:00:00 2001
From: nullDxe <307256015+nullDxe at users.noreply.github.com>
Date: Mon, 20 Jul 2026 17:02:15 -0300
Subject: [PATCH] [X86] Fix getStackProbeSymbolName returning non-callable
 values on Windows

On Windows hasInlineStackProbe() returns false (Windows probes via __chkstk, not inline), but getStackProbeSymbolName() still returns the probe-stack attribute value verbatim, so a value that isn't a function name becomes the call target. probe-stack="inline-asm" (set by rustc on x86_64-pc-windows-msvc) emits `callq "inline-asm"`, an undefined symbol; probe-stack="" reaches the mangler and hits assert(!Name.empty()) in Mangler::getNameWithPrefix, or SIGSEGVs in GetSymbolFromOperand without assertions.

Skip both values and fall through to the existing RTLIB lookup, which resolves the right symbol per target (__chkstk, _chkstk, ___chkstk_ms). For probe-stack="" this also flips hasStackProbeSymbol() to true, giving large static frames a prologue probe, which matches the Windows ABI and the no-attribute case.
---
 llvm/lib/Target/X86/X86ISelLowering.cpp       | 16 ++++--
 .../X86/probe-stack-dynalloca-windows.ll      | 49 +++++++++++++++++++
 2 files changed, 62 insertions(+), 3 deletions(-)
 create mode 100644 llvm/test/CodeGen/X86/probe-stack-dynalloca-windows.ll

diff --git a/llvm/lib/Target/X86/X86ISelLowering.cpp b/llvm/lib/Target/X86/X86ISelLowering.cpp
index 1bf1faf1b55a1..11cbf639b98f8 100644
--- a/llvm/lib/Target/X86/X86ISelLowering.cpp
+++ b/llvm/lib/Target/X86/X86ISelLowering.cpp
@@ -64599,9 +64599,19 @@ X86TargetLowering::getStackProbeSymbolName(const MachineFunction &MF) const {
   if (hasInlineStackProbe(MF))
     return "";
 
-  // If the function specifically requests stack probes, emit them.
-  if (MF.getFunction().hasFnAttribute("probe-stack"))
-    return MF.getFunction().getFnAttribute("probe-stack").getValueAsString();
+  // If the function specifically requests a named stack probe function, use it.
+  if (MF.getFunction().hasFnAttribute("probe-stack")) {
+    StringRef Value =
+        MF.getFunction().getFnAttribute("probe-stack").getValueAsString();
+    // "inline-asm" is a sentinel value requesting inline stack probes rather
+    // than a function call. hasInlineStackProbe() already handles this and
+    // returned false (e.g. because Windows targets use their own mechanism).
+    // Don't return the sentinel as a literal symbol name; fall through to
+    // the platform default. Also skip empty values, which would crash
+    // downstream in the mangler.
+    if (Value != "inline-asm" && !Value.empty())
+      return Value;
+  }
 
   // Generally, if we aren't on Windows, the platform ABI does not include
   // support for stack probes, so don't emit them.
diff --git a/llvm/test/CodeGen/X86/probe-stack-dynalloca-windows.ll b/llvm/test/CodeGen/X86/probe-stack-dynalloca-windows.ll
new file mode 100644
index 0000000000000..ee156889e488a
--- /dev/null
+++ b/llvm/test/CodeGen/X86/probe-stack-dynalloca-windows.ll
@@ -0,0 +1,49 @@
+; RUN: llc -O2 < %s -o /dev/null
+; RUN: llc -O2 < %s | FileCheck %s
+;
+; Test that dynamic allocas on Windows with probe-stack attributes that are
+; not callable symbol names correctly fall back to the platform stack probe.
+;
+; probe-stack=inline-asm is a sentinel requesting inline probing, but
+; hasInlineStackProbe() returns false on Windows (Windows uses __chkstk).
+; Previously, getStackProbeSymbolName() returned the sentinel as a literal
+; symbol name, emitting callq "inline-asm".
+;
+; probe-stack="" (empty) flowed through unchecked and crashed in
+; Mangler::getNameWithPrefix via X86MCInstLower::GetSymbolFromOperand.
+
+target datalayout = "e-m:w-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-pc-windows-msvc"
+
+; The "inline-asm" sentinel must resolve to __chkstk, not "inline-asm".
+; CHECK-LABEL: test_inline_asm_sentinel:
+; CHECK-NOT:   inline-asm
+; CHECK:       callq __chkstk
+define ptr @test_inline_asm_sentinel(i64 %n) #0 {
+entry:
+  %cmp = icmp ugt i64 %n, 0
+  br i1 %cmp, label %alloc, label %exit
+alloc:
+  %buf = alloca i8, i64 %n, align 16
+  ret ptr %buf
+exit:
+  ret ptr null
+}
+
+; An empty probe-stack value must not crash; should also resolve to __chkstk.
+; CHECK-LABEL: test_empty_probe_stack:
+; CHECK-NOT:   inline-asm
+; CHECK:       callq __chkstk
+define ptr @test_empty_probe_stack(i64 %n) #1 {
+entry:
+  %cmp = icmp ugt i64 %n, 0
+  br i1 %cmp, label %alloc, label %exit
+alloc:
+  %buf = alloca i8, i64 %n, align 16
+  ret ptr %buf
+exit:
+  ret ptr null
+}
+
+attributes #0 = { "probe-stack"="inline-asm" }
+attributes #1 = { "probe-stack"="" }