[llvm] [X86] Report an error for FP environment accesses that are not 256 bits (PR #228841)

[Doha Nam via llvm-commits]

https://github.com/waroad created https://github.com/llvm/llvm-project/pull/228841

LowerGET_FPENV_MEM and LowerSET_FPENV_MEM assume the 256-bit X86 FP
environment layout (x87 environment plus MXCSR) and only checked the width
with an assertion. The IR verifier accepts llvm.get.fpenv/llvm.set.fpenv on
any integer type. With a narrower type, release builds therefore wrote 32
bytes into a smaller stack slot (for i64, fnstenv/stmxcsr overwrote the
return address).

Emit an error for other widths instead, following the approach used for
missing floating-point state libcalls.

Fixes #228830.

Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>


>From f6b51765eb7adb2da8be1f8d446f1eda21c1c293 Mon Sep 17 00:00:00 2001
From: waroad <dohanam2 at illinois.edu>
Date: Sun, 4 Oct 2026 01:41:05 -0500
Subject: [PATCH] [X86] Report an error for FP environment accesses that are
 not 256 bits

LowerGET_FPENV_MEM and LowerSET_FPENV_MEM assume the 256-bit X86 FP
environment layout (x87 environment plus MXCSR) and only checked the width
with an assertion. The IR verifier accepts llvm.get.fpenv/llvm.set.fpenv on
any integer type. With a narrower type, release builds therefore wrote 32
bytes into a smaller stack slot (for i64, fnstenv/stmxcsr overwrote the
return address).

Emit an error for other widths instead, following the approach used for
missing floating-point state libcalls.

Fixes #228830.

Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
---
 llvm/lib/Target/X86/X86ISelLowering.cpp     | 21 ++++++++++++++---
 llvm/test/CodeGen/X86/fpenv-invalid-type.ll | 25 +++++++++++++++++++++
 2 files changed, 43 insertions(+), 3 deletions(-)
 create mode 100644 llvm/test/CodeGen/X86/fpenv-invalid-type.ll

diff --git a/llvm/lib/Target/X86/X86ISelLowering.cpp b/llvm/lib/Target/X86/X86ISelLowering.cpp
index ccd53f8aaaffe..1968ec3a37f29 100644
--- a/llvm/lib/Target/X86/X86ISelLowering.cpp
+++ b/llvm/lib/Target/X86/X86ISelLowering.cpp
@@ -29466,7 +29466,20 @@ SDValue X86TargetLowering::LowerSET_ROUNDING(SDValue Op,
 
 const unsigned X87StateSize = 28;
 const unsigned FPStateSize = 32;
-[[maybe_unused]] const unsigned FPStateSizeInBits = FPStateSize * 8;
+const unsigned FPStateSizeInBits = FPStateSize * 8;
+
+// The X86 FP environment is 256 bits (x87 environment plus MXCSR). Report an
+// error for other widths rather than reading or writing past the memory
+// operand.
+static bool checkFPStateWidth(SDValue Op, EVT MemVT, SelectionDAG &DAG) {
+  if (MemVT.getSizeInBits() == FPStateSizeInBits)
+    return true;
+  DAG.getContext()->emitError(Twine("unsupported type for ") +
+                              Op->getOperationName(&DAG) +
+                              ": the X86 floating-point environment is " +
+                              Twine(FPStateSizeInBits) + " bits");
+  return false;
+}
 
 SDValue X86TargetLowering::LowerGET_FPENV_MEM(SDValue Op,
                                               SelectionDAG &DAG) const {
@@ -29476,7 +29489,8 @@ SDValue X86TargetLowering::LowerGET_FPENV_MEM(SDValue Op,
   SDValue Ptr = Op->getOperand(1);
   auto *Node = cast<FPStateAccessSDNode>(Op);
   EVT MemVT = Node->getMemoryVT();
-  assert(MemVT.getSizeInBits() == FPStateSizeInBits);
+  if (!checkFPStateWidth(Op, MemVT, DAG))
+    return Chain;
   MachineMemOperand *MMO = cast<FPStateAccessSDNode>(Op)->getMemOperand();
 
   // Get x87 state, if it presents.
@@ -29542,7 +29556,8 @@ SDValue X86TargetLowering::LowerSET_FPENV_MEM(SDValue Op,
   SDValue Ptr = Op->getOperand(1);
   auto *Node = cast<FPStateAccessSDNode>(Op);
   EVT MemVT = Node->getMemoryVT();
-  assert(MemVT.getSizeInBits() == FPStateSizeInBits);
+  if (!checkFPStateWidth(Op, MemVT, DAG))
+    return Chain;
   MachineMemOperand *MMO = cast<FPStateAccessSDNode>(Op)->getMemOperand();
   return createSetFPEnvNodes(Ptr, Chain, DL, MemVT, MMO, DAG, Subtarget);
 }
diff --git a/llvm/test/CodeGen/X86/fpenv-invalid-type.ll b/llvm/test/CodeGen/X86/fpenv-invalid-type.ll
new file mode 100644
index 0000000000000..2734e6f17dd1e
--- /dev/null
+++ b/llvm/test/CodeGen/X86/fpenv-invalid-type.ll
@@ -0,0 +1,25 @@
+; RUN: not llc -mtriple=x86_64-unknown-linux-gnu -filetype=null %s 2>&1 | FileCheck %s
+; RUN: not llc -mtriple=i686-unknown-linux-gnu -filetype=null %s 2>&1 | FileCheck %s
+
+; The X86 floating-point environment is 256 bits. Other widths are rejected
+; with an error instead of reading or writing past the memory operand.
+; https://github.com/llvm/llvm-project/issues/228830
+
+; CHECK: error: unsupported type for get_fpenv_mem: the X86 floating-point environment is 256 bits
+define i64 @get_fpenv_i64() nounwind {
+  %env = call i64 @llvm.get.fpenv.i64()
+  ret i64 %env
+}
+
+; CHECK: error: unsupported type for set_fpenv_mem: the X86 floating-point environment is 256 bits
+define void @set_fpenv_i128(i128 %env) nounwind {
+  call void @llvm.set.fpenv.i128(i128 %env)
+  ret void
+}
+
+; CHECK: error: unsupported type for get_fpenv_mem: the X86 floating-point environment is 256 bits
+define void @get_fpenv_i512(ptr %p) nounwind {
+  %env = call i512 @llvm.get.fpenv.i512()
+  store i512 %env, ptr %p
+  ret void
+}