[llvm] [SelectionDAG] Respect no-builtins attribute in SMULO expansion (PR #191020)

[Cameron Taggart via llvm-commits]

https://github.com/cataggar updated https://github.com/llvm/llvm-project/pull/191020

>From 4f8a0ff918698a6359a835069a1c5e68cced38ff Mon Sep 17 00:00:00 2001
From: Cameron Taggart <cataggar at microsoft.com>
Date: Wed, 8 Apr 2026 17:27:13 +0100
Subject: [PATCH] [SelectionDAG] Respect no-builtins attribute in SMULO
 expansion

When expanding ISD::SMULO for types that require a libcall (e.g. i128
on WebAssembly), the legalizer now checks for the "no-builtins"
function attribute. If present, it expands inline instead of emitting
a call to __muloti4.

This prevents infinite recursion when a custom implementation of
__muloti4 (under a different name) is compiled with no-builtins, as
the overflow intrinsic introduced by instcombine would otherwise be
lowered back to a call to __muloti4.

Fixes https://github.com/llvm/llvm-project/issues/189173

Assisted-by: GitHub Copilot
---
 .../SelectionDAG/LegalizeIntegerTypes.cpp     |  9 +++++---
 .../WebAssembly/muloti4-no-builtins.ll        | 21 +++++++++++++++++++
 2 files changed, 27 insertions(+), 3 deletions(-)
 create mode 100644 llvm/test/CodeGen/WebAssembly/muloti4-no-builtins.ll

diff --git a/llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp b/llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp
index 35e61b78e99a2..eed88eb3cdd47 100644
--- a/llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp
@@ -5301,11 +5301,14 @@ void DAGTypeLegalizer::ExpandIntRes_XMULO(SDNode *N,
   RTLIB::Libcall LC = RTLIB::getMULO(VT);
   RTLIB::LibcallImpl LCImpl = DAG.getLibcalls().getLibcallImpl(LC);
 
-  // If we don't have the libcall or if the function we are compiling is the
-  // implementation of the expected libcall (avoid inf-loop), expand inline.
+  // If we don't have the libcall, if the function we are compiling is the
+  // implementation of the expected libcall (avoid inf-loop), or if the
+  // function has the no-builtins attribute (it may be a custom implementation
+  // of the libcall under a different name), expand inline.
   if (LCImpl == RTLIB::Unsupported ||
       RTLIB::RuntimeLibcallsInfo::getLibcallImplName(LCImpl) ==
-          DAG.getMachineFunction().getName()) {
+          DAG.getMachineFunction().getName() ||
+      DAG.getMachineFunction().getFunction().hasFnAttribute("no-builtins")) {
     // FIXME: This is not an optimal expansion, but better than crashing.
     SDValue MulLo, MulHi;
     TLI.forceExpandWideMUL(DAG, dl, /*Signed=*/true, N->getOperand(0),
diff --git a/llvm/test/CodeGen/WebAssembly/muloti4-no-builtins.ll b/llvm/test/CodeGen/WebAssembly/muloti4-no-builtins.ll
new file mode 100644
index 0000000000000..b75bf4d16fb58
--- /dev/null
+++ b/llvm/test/CodeGen/WebAssembly/muloti4-no-builtins.ll
@@ -0,0 +1,21 @@
+; RUN: llc -asm-verbose=false < %s -wasm-keep-registers | FileCheck %s
+
+; Test that 128-bit smul.with.overflow does not emit a call to __muloti4
+; when the function has the "no-builtins" attribute. This avoids infinite
+; recursion when the function is a custom implementation of __muloti4.
+; See https://github.com/llvm/llvm-project/issues/189173
+
+target triple = "wasm32-unknown-unknown"
+
+define i128 @custom_muloti4(i128 %a, i128 %b) nounwind "no-builtins" {
+entry:
+  %smul = tail call { i128, i1 } @llvm.smul.with.overflow.i128(i128 %a, i128 %b)
+  %cmp = extractvalue { i128, i1 } %smul, 1
+  %smul.result = extractvalue { i128, i1 } %smul, 0
+  %X = select i1 %cmp, i128 %smul.result, i128 42
+  ret i128 %X
+}
+
+; CHECK-NOT: call __muloti4
+
+declare { i128, i1 } @llvm.smul.with.overflow.i128(i128, i128) nounwind readnone