[clang] [llvm] Minimal support of floating-point operand bundles (PR #135658)
Serge Pavlov via llvm-commits
llvm-commits at lists.llvm.org
Sun Nov 2 00:47:40 PDT 2025
https://github.com/spavloff updated https://github.com/llvm/llvm-project/pull/135658
>From 8faeccabb1433d62651a8837b6a9e02d24c898f8 Mon Sep 17 00:00:00 2001
From: Serge Pavlov <sepavloff at gmail.com>
Date: Mon, 14 Apr 2025 12:51:43 +0700
Subject: [PATCH 1/9] Minimal support of floating-point operand bundles
This is a lite version of https://github.com/llvm/llvm-project/pull/109798,
where code changes are minimized to facilitate discussion about the
implementation. The motivations and ideas behind the new floating-point
operation support are described in that PR and in the discussion
https://discourse.llvm.org/t/rfc-change-of-strict-fp-operation-representation-in-ir/85021.
There are concerns that the proposed changes are too invasive and a new
approach is required to make the transition smoother.
This implementation is essentially a subset of PR109798, where
everything beyond the minimum is removed. It tries to build eventually
the same implementation as that PR but in different steps.
The patch does not attempt to modify the existing implementation based
on the constrained intrinsics. Instead it introduces a new one using
operand bundles. This new implementation initially has very limited
functionality, which latter will be extended and finally can replace the
existing one.
This PR introduces the notion of floating-point operation, this is an
intrinsic, that is listed in the file "FloatingPointOps.def". These have
two additional properties:
1. In the strict environment (a function with strictfp attribute) calls
to these operations acquire side effect, now it is read/write access
to inaccessible memory, just as constrained intrinsics do.
2. Calls to these operations may have floating-point operand bundles.
There are two kinds of such bundles, tagged with "fp.control" and
"fp.except", which are used to carry additional information about
control modes and exception handling. Initially the set of control
modes consists of rounding mode only.
The set of operations enlisted in "FloatingPointOps.def" and in
"ConstrainedOps.def" are completely independent, an intrinsic may be in
one list or in both. The set of floating-point operations is expected to
grow and finally all FP intrinsics will be available in the new
implementation. In this patch set of intrinsics in
"FloatingPointOps.def" is minimum necessary for tests.
---
llvm/docs/LangRef.rst | 51 ++++-
llvm/docs/ReleaseNotes.md | 1 +
llvm/include/llvm/IR/FPEnv.h | 18 ++
llvm/include/llvm/IR/FloatingPointOps.def | 24 +++
llvm/include/llvm/IR/IRBuilder.h | 45 ++--
llvm/include/llvm/IR/InstrTypes.h | 16 ++
llvm/include/llvm/IR/IntrinsicInst.h | 8 +
llvm/include/llvm/IR/LLVMContext.h | 5 +-
llvm/include/llvm/Support/ModRef.h | 5 +
llvm/lib/IR/FPEnv.cpp | 64 ++++++
llvm/lib/IR/IRBuilder.cpp | 75 +++++++
llvm/lib/IR/Instructions.cpp | 83 +++++++-
llvm/lib/IR/IntrinsicInst.cpp | 10 +
llvm/lib/IR/LLVMContext.cpp | 14 ++
llvm/lib/IR/Verifier.cpp | 31 ++-
.../Bitcode/operand-bundles-bc-analyzer.ll | 2 +
llvm/test/Verifier/fp-intrinsics.ll | 91 ++++++++
llvm/unittests/IR/IRBuilderTest.cpp | 197 ++++++++++++++++++
18 files changed, 717 insertions(+), 23 deletions(-)
create mode 100644 llvm/include/llvm/IR/FloatingPointOps.def
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 54c7d0fdfbf18..2e121a00cf3a4 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -3199,6 +3199,51 @@ A "convergencectrl" operand bundle is only valid on a ``convergent`` operation.
When present, the operand bundle must contain exactly one value of token type.
See the :doc:`ConvergentOperations` document for details.
+.. _ob_fp:
+
+Floating-point Operand Bundles
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+These operand bundles are used for calls that involve floating-point
+operations and interact with :ref:`floating-point environment <floatenv>` or
+depend on floating-point options, such as rounding mode, denormal modes, etc.
+There are two kinds of such operand bundles, which represent the value of
+floating-point control modes and the treatment of status bits respectively.
+
+An operand bundle tagged with "fp.control" contains information about the
+control modes used for the operation execution. Operands specified in this
+bundle represent particular options. Currently, only rounding mode is supported.
+It is represented by a metadata string value, which specifies the rounding mode
+to be used for the operation evaluation. Possible values are:
+
+::
+
+ "rtz" - toward zero
+ "rte" - to nearest, ties to even
+ "rtp" - toward positive infinity
+ "rtn" - toward negative infinity
+ "rmm" - to nearest, ties away from zero
+ "dyn" - rounding mode is taken from control register
+
+If "fp.control" is absent, the default rounding rounding mode is taken from the
+control register (dynamic rounding). In the particular case of
+:ref:`default floating-point environment <floatenv>`, it must be rounding to
+nearest, ties to even.
+
+An operand bundle tagged with "fp.except" may be associated with operations
+that can read or write floating-point exception flags. It contains a single
+metadata string value, which can have one of the following values:
+
+::
+
+ "ignore"
+ "strict"
+ "maytrap"
+
+It has the same meaning as the corresponding argument in
+:ref:`constrained intrinsics <constrainedfp>`.
+
+
.. _moduleasm:
Module-Level Inline Assembly
@@ -3949,9 +3994,9 @@ round-to-nearest rounding mode, and subnormals are assumed to be preserved.
Running LLVM code in an environment where these assumptions are not met
typically leads to undefined behavior. The ``strictfp`` and ``denormal-fp-math``
attributes as well as :ref:`Constrained Floating-Point Intrinsics
-<constrainedfp>` can be used to weaken LLVM's assumptions and ensure defined
-behavior in non-default floating-point environments; see their respective
-documentation for details.
+<constrainedfp>` or :ref:`floating-point operand bundles<ob_fp>` can be used to
+weaken LLVM's assumptions and ensure defined behavior in non-default
+floating-point environments; see their respective documentation for details.
.. _floatnan:
diff --git a/llvm/docs/ReleaseNotes.md b/llvm/docs/ReleaseNotes.md
index 49158fb4217b6..8f8f6b9594581 100644
--- a/llvm/docs/ReleaseNotes.md
+++ b/llvm/docs/ReleaseNotes.md
@@ -67,6 +67,7 @@ Changes to the LLVM IR
Instead, the `align` attribute should be placed on the pointer (or vector of
pointers) argument.
* A `load atomic` may now be used with vector types on x86.
+* Floating-point operand bundles have been added.
Changes to LLVM infrastructure
------------------------------
diff --git a/llvm/include/llvm/IR/FPEnv.h b/llvm/include/llvm/IR/FPEnv.h
index 38395b15c8c09..78a5d6d5b75f5 100644
--- a/llvm/include/llvm/IR/FPEnv.h
+++ b/llvm/include/llvm/IR/FPEnv.h
@@ -49,20 +49,38 @@ enum ExceptionBehavior : uint8_t {
/// metadata.
LLVM_ABI std::optional<RoundingMode> convertStrToRoundingMode(StringRef);
+/// Returns a valid RoundingMode enumerator given a string that is used as
+/// rounding mode specifier in operand bundles.
+std::optional<RoundingMode> convertBundleToRoundingMode(StringRef);
+
/// For any RoundingMode enumerator, returns a string valid as input in
/// constrained intrinsic rounding mode metadata.
LLVM_ABI std::optional<StringRef> convertRoundingModeToStr(RoundingMode);
+/// For any RoundingMode enumerator, returns a string to be used in operand
+/// bundles.
+std::optional<StringRef> convertRoundingModeToBundle(RoundingMode);
+
/// Returns a valid ExceptionBehavior enumerator when given a string
/// valid as input in constrained intrinsic exception behavior metadata.
LLVM_ABI std::optional<fp::ExceptionBehavior>
convertStrToExceptionBehavior(StringRef);
+/// Returns a valid ExceptionBehavior enumerator given a string from the operand
+/// bundle argument.
+std::optional<fp::ExceptionBehavior>
+ convertBundleToExceptionBehavior(StringRef);
+
/// For any ExceptionBehavior enumerator, returns a string valid as
/// input in constrained intrinsic exception behavior metadata.
LLVM_ABI std::optional<StringRef>
convertExceptionBehaviorToStr(fp::ExceptionBehavior);
+/// Return string representing the given exception behavior for use in operand
+/// bundles
+std::optional<StringRef>
+ convertExceptionBehaviorToBundle(fp::ExceptionBehavior);
+
/// Returns true if the exception handling behavior and rounding mode
/// match what is used in the default floating point environment.
inline bool isDefaultFPEnvironment(fp::ExceptionBehavior EB, RoundingMode RM) {
diff --git a/llvm/include/llvm/IR/FloatingPointOps.def b/llvm/include/llvm/IR/FloatingPointOps.def
new file mode 100644
index 0000000000000..8567b5dbac302
--- /dev/null
+++ b/llvm/include/llvm/IR/FloatingPointOps.def
@@ -0,0 +1,24 @@
+//===- llvm/IR/FloatingPointOps.def - FP intrinsics -------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Defines set of intrinsics, which are classified as floating-point operations.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef FUNCTION
+#define FUNCTION(N,D)
+#endif
+
+// Arguments of the entries are:
+// - intrinsic function name,
+// - DAG node corresponding to the intrinsic.
+
+FUNCTION(nearbyint, FNEARBYINT)
+FUNCTION(trunc, FTRUNC)
+
+#undef FUNCTION
diff --git a/llvm/include/llvm/IR/IRBuilder.h b/llvm/include/llvm/IR/IRBuilder.h
index 972a253344ddf..eba2c6630f2d2 100644
--- a/llvm/include/llvm/IR/IRBuilder.h
+++ b/llvm/include/llvm/IR/IRBuilder.h
@@ -1000,6 +1000,16 @@ class IRBuilderBase {
FMFSource FMFSource = {},
const Twine &Name = "");
+ /// Create a call to intrinsic \p ID with \p Args, mangled using \p Types and
+ /// with operand bundles.
+ /// If \p FMFSource is provided, copy fast-math-flags from that instruction to
+ /// the intrinsic.
+ CallInst *CreateIntrinsic(Intrinsic::ID ID, ArrayRef<Type *> Types,
+ ArrayRef<Value *> Args,
+ ArrayRef<OperandBundleDef> OpBundles,
+ Instruction *FMFSource = nullptr,
+ const Twine &Name = "");
+
/// Create a call to non-overloaded intrinsic \p ID with \p Args. If
/// \p FMFSource is provided, copy fast-math-flags from that instruction to
/// the intrinsic.
@@ -2511,24 +2521,13 @@ class IRBuilderBase {
CallInst *CreateCall(FunctionType *FTy, Value *Callee,
ArrayRef<Value *> Args = {}, const Twine &Name = "",
MDNode *FPMathTag = nullptr) {
- CallInst *CI = CallInst::Create(FTy, Callee, Args, DefaultOperandBundles);
- if (IsFPConstrained)
- setConstrainedFPCallAttr(CI);
- if (isa<FPMathOperator>(CI))
- setFPAttrs(CI, FPMathTag, FMF);
- return Insert(CI, Name);
+ return CreateCall(FTy, Callee, Args, DefaultOperandBundles, Name,
+ FPMathTag);
}
CallInst *CreateCall(FunctionType *FTy, Value *Callee, ArrayRef<Value *> Args,
ArrayRef<OperandBundleDef> OpBundles,
- const Twine &Name = "", MDNode *FPMathTag = nullptr) {
- CallInst *CI = CallInst::Create(FTy, Callee, Args, OpBundles);
- if (IsFPConstrained)
- setConstrainedFPCallAttr(CI);
- if (isa<FPMathOperator>(CI))
- setFPAttrs(CI, FPMathTag, FMF);
- return Insert(CI, Name);
- }
+ const Twine &Name = "", MDNode *FPMathTag = nullptr);
CallInst *CreateCall(FunctionCallee Callee, ArrayRef<Value *> Args = {},
const Twine &Name = "", MDNode *FPMathTag = nullptr) {
@@ -2767,6 +2766,24 @@ class IRBuilderBase {
/// assumption on the provided pointer.
LLVM_ABI CallInst *CreateDereferenceableAssumption(Value *PtrValue,
Value *SizeValue);
+
+ /// Create an operand bundle in the provided bundle set to represent given FP
+ /// rounding mode.
+ ///
+ /// If the rounding mode is not defined, adds the default rounding mode,
+ /// stored in this builder object.
+ void
+ createFPRoundingBundle(SmallVectorImpl<OperandBundleDef> &Bundles,
+ std::optional<RoundingMode> Rounding = std::nullopt);
+
+ /// Create an operand bundle in the provided bundle set to represent FP
+ /// exception behavior.
+ ///
+ /// If the exception behavior is not defined, adds the default behavior,
+ /// stored in this builder object.
+ void createFPExceptionBundle(
+ SmallVectorImpl<OperandBundleDef> &Bundles,
+ std::optional<fp::ExceptionBehavior> Except = std::nullopt);
};
/// This provides a uniform API for creating instructions and inserting
diff --git a/llvm/include/llvm/IR/InstrTypes.h b/llvm/include/llvm/IR/InstrTypes.h
index 9f56779a9b935..b22230713bba5 100644
--- a/llvm/include/llvm/IR/InstrTypes.h
+++ b/llvm/include/llvm/IR/InstrTypes.h
@@ -25,6 +25,7 @@
#include "llvm/IR/CallingConv.h"
#include "llvm/IR/DerivedTypes.h"
#include "llvm/IR/FMF.h"
+#include "llvm/IR/FPEnv.h"
#include "llvm/IR/Function.h"
#include "llvm/IR/Instruction.h"
#include "llvm/IR/LLVMContext.h"
@@ -1093,6 +1094,13 @@ template <typename InputTy> class OperandBundleDefT {
using OperandBundleDef = OperandBundleDefT<Value *>;
using ConstOperandBundleDef = OperandBundleDefT<const Value *>;
+void addFPRoundingBundle(LLVMContext &Ctx,
+ SmallVectorImpl<OperandBundleDef> &Bundles,
+ RoundingMode Rounding);
+void addFPExceptionBundle(LLVMContext &Ctx,
+ SmallVectorImpl<OperandBundleDef> &Bundles,
+ fp::ExceptionBehavior Except);
+
//===----------------------------------------------------------------------===//
// CallBase Class
//===----------------------------------------------------------------------===//
@@ -1152,6 +1160,8 @@ class CallBase : public Instruction {
/// number of extra operands.
LLVM_ABI unsigned getNumSubclassExtraOperandsDynamic() const;
+ MemoryEffects getFloatingPointMemoryEffects() const;
+
public:
using Instruction::getContext;
@@ -2162,6 +2172,12 @@ class CallBase : public Instruction {
return false;
}
+ /// Return rounding mode specified by operand bundles.
+ RoundingMode getRoundingMode() const;
+
+ /// Return exception behavior specified by operand bundles.
+ std::optional<fp::ExceptionBehavior> getExceptionBehavior() const;
+
/// Used to keep track of an operand bundle. See the main comment on
/// OperandBundleUser above.
struct BundleOpInfo {
diff --git a/llvm/include/llvm/IR/IntrinsicInst.h b/llvm/include/llvm/IR/IntrinsicInst.h
index 0622bfae2c845..c35a85ec1e74a 100644
--- a/llvm/include/llvm/IR/IntrinsicInst.h
+++ b/llvm/include/llvm/IR/IntrinsicInst.h
@@ -129,6 +129,14 @@ class IntrinsicInst : public CallInst {
/// course of IR transformations
LLVM_ABI static bool mayLowerToFunctionCall(Intrinsic::ID IID);
+ /// Check if \p ID represents a function that may access FP environment and
+ /// may have FP operand bundles.
+ ///
+ /// Access to FP environment means that in the strict FP environment the
+ /// function has read/write memory effect, which is used to maintain proper
+ /// instructions ordering.
+ static bool isFloatingPointOperation(Intrinsic::ID IID);
+
/// Methods for support type inquiry through isa, cast, and dyn_cast:
static bool classof(const CallInst *I) {
auto *F = dyn_cast_or_null<Function>(I->getCalledOperand());
diff --git a/llvm/include/llvm/IR/LLVMContext.h b/llvm/include/llvm/IR/LLVMContext.h
index 5972dcb637dfa..437e3d89fea1a 100644
--- a/llvm/include/llvm/IR/LLVMContext.h
+++ b/llvm/include/llvm/IR/LLVMContext.h
@@ -98,8 +98,9 @@ class LLVMContext {
OB_kcfi = 8, // "kcfi"
OB_convergencectrl = 9, // "convergencectrl"
OB_align = 10, // "align"
- OB_LastBundleID = OB_align // Marker for last bundle ID
- };
+ OB_fp_control = 11, // "fp.control"
+ OB_fp_except = 12, // "fp.except"
+ OB_LastBundleID = OB_fp_except // Marker for last bundle ID };
/// getMDKindID - Return a unique non-zero ID for the specified metadata kind.
/// This ID is uniqued across modules in the current LLVMContext.
diff --git a/llvm/include/llvm/Support/ModRef.h b/llvm/include/llvm/Support/ModRef.h
index 71f3b5bcb9c2b..578753871a2d4 100644
--- a/llvm/include/llvm/Support/ModRef.h
+++ b/llvm/include/llvm/Support/ModRef.h
@@ -235,6 +235,11 @@ template <typename LocationEnum> class MemoryEffectsBase {
return getWithoutLoc(Location::InaccessibleMem).doesNotAccessMemory();
}
+ /// Whether this function accesses inaccessible memory.
+ bool doesAccessInaccessibleMem() const {
+ return isModOrRefSet(getModRef(Location::InaccessibleMem));
+ }
+
/// Whether this function only (at most) accesses errno memory.
bool onlyAccessesErrnoMem() const {
return getWithoutLoc(Location::ErrnoMem).doesNotAccessMemory();
diff --git a/llvm/lib/IR/FPEnv.cpp b/llvm/lib/IR/FPEnv.cpp
index 67f21d3756e93..d48bebeb6d2b7 100644
--- a/llvm/lib/IR/FPEnv.cpp
+++ b/llvm/lib/IR/FPEnv.cpp
@@ -34,6 +34,17 @@ std::optional<RoundingMode> convertStrToRoundingMode(StringRef RoundingArg) {
.Default(std::nullopt);
}
+std::optional<RoundingMode> convertBundleToRoundingMode(StringRef RoundingArg) {
+ return StringSwitch<std::optional<RoundingMode>>(RoundingArg)
+ .Case("dyn", RoundingMode::Dynamic)
+ .Case("rte", RoundingMode::NearestTiesToEven)
+ .Case("rmm", RoundingMode::NearestTiesToAway)
+ .Case("rtn", RoundingMode::TowardNegative)
+ .Case("rtp", RoundingMode::TowardPositive)
+ .Case("rtz", RoundingMode::TowardZero)
+ .Default(std::nullopt);
+}
+
std::optional<StringRef> convertRoundingModeToStr(RoundingMode UseRounding) {
std::optional<StringRef> RoundingStr;
switch (UseRounding) {
@@ -61,6 +72,33 @@ std::optional<StringRef> convertRoundingModeToStr(RoundingMode UseRounding) {
return RoundingStr;
}
+std::optional<StringRef> convertRoundingModeToBundle(RoundingMode UseRounding) {
+ std::optional<StringRef> RoundingStr;
+ switch (UseRounding) {
+ case RoundingMode::Dynamic:
+ RoundingStr = "dyn";
+ break;
+ case RoundingMode::NearestTiesToEven:
+ RoundingStr = "rte";
+ break;
+ case RoundingMode::NearestTiesToAway:
+ RoundingStr = "rmm";
+ break;
+ case RoundingMode::TowardNegative:
+ RoundingStr = "rtn";
+ break;
+ case RoundingMode::TowardPositive:
+ RoundingStr = "rtp";
+ break;
+ case RoundingMode::TowardZero:
+ RoundingStr = "rtz";
+ break;
+ default:
+ break;
+ }
+ return RoundingStr;
+}
+
std::optional<fp::ExceptionBehavior>
convertStrToExceptionBehavior(StringRef ExceptionArg) {
return StringSwitch<std::optional<fp::ExceptionBehavior>>(ExceptionArg)
@@ -70,6 +108,15 @@ convertStrToExceptionBehavior(StringRef ExceptionArg) {
.Default(std::nullopt);
}
+std::optional<fp::ExceptionBehavior>
+convertBundleToExceptionBehavior(StringRef ExceptionArg) {
+ return StringSwitch<std::optional<fp::ExceptionBehavior>>(ExceptionArg)
+ .Case("ignore", fp::ebIgnore)
+ .Case("maytrap", fp::ebMayTrap)
+ .Case("strict", fp::ebStrict)
+ .Default(std::nullopt);
+}
+
std::optional<StringRef>
convertExceptionBehaviorToStr(fp::ExceptionBehavior UseExcept) {
std::optional<StringRef> ExceptStr;
@@ -87,6 +134,23 @@ convertExceptionBehaviorToStr(fp::ExceptionBehavior UseExcept) {
return ExceptStr;
}
+std::optional<StringRef>
+convertExceptionBehaviorToBundle(fp::ExceptionBehavior UseExcept) {
+ std::optional<StringRef> ExceptStr;
+ switch (UseExcept) {
+ case fp::ebStrict:
+ ExceptStr = "strict";
+ break;
+ case fp::ebIgnore:
+ ExceptStr = "ignore";
+ break;
+ case fp::ebMayTrap:
+ ExceptStr = "maytrap";
+ break;
+ }
+ return ExceptStr;
+}
+
Intrinsic::ID getConstrainedIntrinsicID(const Instruction &Instr) {
Intrinsic::ID IID = Intrinsic::not_intrinsic;
switch (Instr.getOpcode()) {
diff --git a/llvm/lib/IR/IRBuilder.cpp b/llvm/lib/IR/IRBuilder.cpp
index 88dbd176e0d3f..7ad78fc4190ea 100644
--- a/llvm/lib/IR/IRBuilder.cpp
+++ b/llvm/lib/IR/IRBuilder.cpp
@@ -113,6 +113,67 @@ IRBuilderBase::createCallHelper(Function *Callee, ArrayRef<Value *> Ops,
return CI;
}
+CallInst *IRBuilderBase::CreateCall(FunctionType *FTy, Value *Callee,
+ ArrayRef<Value *> Args,
+ ArrayRef<OperandBundleDef> OpBundles,
+ const Twine &Name, MDNode *FPMathTag) {
+ assert(std::count_if(OpBundles.begin(), OpBundles.end(),
+ [](const OperandBundleDef &Item) {
+ return Item.getTag() == "fp.control";
+ }) <= 1);
+ assert(std::count_if(OpBundles.begin(), OpBundles.end(),
+ [](const OperandBundleDef &Item) {
+ return Item.getTag() == "fp.except";
+ }) <= 1);
+
+ ArrayRef<OperandBundleDef> ActualBundlesRef = OpBundles;
+ SmallVector<OperandBundleDef, 2> ActualBundles;
+
+ // If the builder is in strictfp mode and has non-default options (like
+ // non-dynamic rounding), add corresponding operand bundle. If such bundle is
+ // already present, assume it overwrites defaults.
+ bool doesTheCallAccessFPEnv = false;
+ if (IsFPConstrained) {
+ if (const auto *Func = dyn_cast<Function>(Callee)) {
+ if (Intrinsic::ID ID = Func->getIntrinsicID()) {
+ if (IntrinsicInst::isFloatingPointOperation(ID)) {
+ doesTheCallAccessFPEnv = true;
+ bool NeedRound = DefaultConstrainedRounding != RoundingMode::Dynamic;
+ bool NeedExcept = DefaultConstrainedExcept != fp::ebStrict;
+ for (const auto &Item : OpBundles) {
+ if (NeedRound && Item.getTag() == "fp.control")
+ NeedRound = false;
+ else if (NeedExcept && Item.getTag() == "fp.except")
+ NeedExcept = false;
+ ActualBundles.push_back(Item);
+ }
+ if (NeedRound)
+ createFPRoundingBundle(ActualBundles, DefaultConstrainedRounding);
+ if (NeedExcept)
+ createFPExceptionBundle(ActualBundles, DefaultConstrainedExcept);
+ ActualBundlesRef = ActualBundles;
+ }
+ }
+ }
+ }
+
+ // If the call accesses FPE, update memory effects accordingly.
+ CallInst *CI = CallInst::Create(FTy, Callee, Args, OpBundles);
+ if (doesTheCallAccessFPEnv) {
+ MemoryEffects ME = MemoryEffects::inaccessibleMemOnly();
+ if (CI->getAttributes().hasFnAttr(Attribute::Memory))
+ ME |= CI->getAttributes().getMemoryEffects();
+ auto A = Attribute::getWithMemoryEffects(getContext(), ME);
+ CI->addFnAttr(A);
+ }
+
+ if (IsFPConstrained)
+ setConstrainedFPCallAttr(CI);
+ if (isa<FPMathOperator>(CI))
+ setFPAttrs(CI, FPMathTag, FMF);
+ return Insert(CI, Name);
+}
+
static Value *CreateVScaleMultiple(IRBuilderBase &B, Type *Ty, uint64_t Scale) {
Value *VScale = B.CreateVScale(Ty);
if (Scale == 1)
@@ -1292,6 +1353,20 @@ CallInst *IRBuilderBase::CreateDereferenceableAssumption(Value *PtrValue,
{DereferenceableOpB});
}
+void IRBuilderBase::createFPRoundingBundle(
+ SmallVectorImpl<OperandBundleDef> &Bundles,
+ std::optional<RoundingMode> Rounding) {
+ addFPRoundingBundle(Context, Bundles,
+ Rounding.value_or(DefaultConstrainedRounding));
+}
+
+void IRBuilderBase::createFPExceptionBundle(
+ SmallVectorImpl<OperandBundleDef> &Bundles,
+ std::optional<fp::ExceptionBehavior> Except) {
+ addFPExceptionBundle(Context, Bundles,
+ Except.value_or(DefaultConstrainedExcept));
+}
+
IRBuilderDefaultInserter::~IRBuilderDefaultInserter() = default;
IRBuilderCallbackInserter::~IRBuilderCallbackInserter() = default;
IRBuilderFolder::~IRBuilderFolder() = default;
diff --git a/llvm/lib/IR/Instructions.cpp b/llvm/lib/IR/Instructions.cpp
index 3b8fde8aff45f..e63bd170fa800 100644
--- a/llvm/lib/IR/Instructions.cpp
+++ b/llvm/lib/IR/Instructions.cpp
@@ -618,9 +618,10 @@ bool CallBase::hasReadingOperandBundles() const {
// Implementation note: this is a conservative implementation of operand
// bundle semantics, where *any* non-assume operand bundle (other than
// ptrauth) forces a callsite to be at least readonly.
- return hasOperandBundlesOtherThan({LLVMContext::OB_ptrauth,
- LLVMContext::OB_kcfi,
- LLVMContext::OB_convergencectrl}) &&
+ return hasOperandBundlesOtherThan(
+ {LLVMContext::OB_ptrauth, LLVMContext::OB_kcfi,
+ LLVMContext::OB_fp_control, LLVMContext::OB_fp_except,
+ LLVMContext::OB_convergencectrl}) &&
getIntrinsicID() != Intrinsic::assume;
}
@@ -628,14 +629,70 @@ bool CallBase::hasClobberingOperandBundles() const {
return hasOperandBundlesOtherThan(
{LLVMContext::OB_deopt, LLVMContext::OB_funclet,
LLVMContext::OB_ptrauth, LLVMContext::OB_kcfi,
+ LLVMContext::OB_fp_control, LLVMContext::OB_fp_except,
LLVMContext::OB_convergencectrl}) &&
getIntrinsicID() != Intrinsic::assume;
}
+RoundingMode CallBase::getRoundingMode() const {
+ // Try reading rounding mode from FP bundle.
+ std::optional<RoundingMode> RM;
+ if (auto RoundingBundle = getOperandBundle(LLVMContext::OB_fp_control)) {
+ Value *V = RoundingBundle->Inputs.front();
+ Metadata *MD = cast<MetadataAsValue>(V)->getMetadata();
+ RM = convertBundleToRoundingMode(cast<MDString>(MD)->getString());
+ }
+ if (RM)
+ return *RM;
+
+ // No FP bundle, try to guess from the current mode.
+ if (getParent())
+ if (auto *F = getFunction(); F)
+ return F->getAttributes().hasFnAttr(Attribute::StrictFP)
+ ? RoundingMode::Dynamic
+ : RoundingMode::NearestTiesToEven;
+
+ // Isolated call. Assume default environment.
+ return RoundingMode::NearestTiesToEven;
+}
+
+std::optional<fp::ExceptionBehavior> CallBase::getExceptionBehavior() const {
+ // Try determining exception behavior from FP bundle.
+ std::optional<fp::ExceptionBehavior> EB;
+ if (auto ExceptionBundle = getOperandBundle(LLVMContext::OB_fp_except)) {
+ Value *V = ExceptionBundle->Inputs.front();
+ Metadata *MD = cast<MetadataAsValue>(V)->getMetadata();
+ EB = convertBundleToExceptionBehavior(cast<MDString>(MD)->getString());
+ }
+ if (EB)
+ return *EB;
+
+ // No FP bundle, try to guess from the current mode.
+ if (getParent())
+ if (auto *F = getFunction(); F)
+ return F->getAttributes().hasFnAttr(Attribute::StrictFP) ? fp::ebStrict
+ : fp::ebIgnore;
+
+ // Isolated call. Assume default environment.
+ return fp::ebIgnore;
+}
+
+MemoryEffects CallBase::getFloatingPointMemoryEffects() const {
+ if (Intrinsic::ID IntrID = getIntrinsicID())
+ if (const BasicBlock *BB = getParent())
+ if (const Function *F = BB->getParent())
+ if (F->hasFnAttribute(Attribute::StrictFP))
+ if (IntrinsicInst::isFloatingPointOperation(IntrID)) {
+ return MemoryEffects::inaccessibleMemOnly();
+ }
+ return MemoryEffects::none();
+}
+
MemoryEffects CallBase::getMemoryEffects() const {
MemoryEffects ME = getAttributes().getMemoryEffects();
if (auto *Fn = dyn_cast<Function>(getCalledOperand())) {
MemoryEffects FnME = Fn->getMemoryEffects();
+ FnME |= getFloatingPointMemoryEffects();
if (hasOperandBundles()) {
// TODO: Add a method to get memory effects for operand bundles instead.
if (hasReadingOperandBundles())
@@ -744,6 +801,26 @@ bool CallBase::hasArgumentWithAdditionalReturnCaptureComponents() const {
return false;
}
+void llvm::addFPRoundingBundle(LLVMContext &Ctx,
+ SmallVectorImpl<OperandBundleDef> &Bundles,
+ RoundingMode Rounding) {
+ std::optional<StringRef> RndStr = convertRoundingModeToBundle(Rounding);
+ assert(RndStr && "Garbage rounding mode!");
+ auto *RoundingMDS = MDString::get(Ctx, *RndStr);
+ auto *RM = MetadataAsValue::get(Ctx, RoundingMDS);
+ Bundles.emplace_back("fp.control", RM);
+}
+
+void llvm::addFPExceptionBundle(LLVMContext &Ctx,
+ SmallVectorImpl<OperandBundleDef> &Bundles,
+ fp::ExceptionBehavior Except) {
+ std::optional<StringRef> ExcStr = convertExceptionBehaviorToBundle(Except);
+ assert(ExcStr && "Garbage exception behavior!");
+ auto *ExceptMDS = MDString::get(Ctx, *ExcStr);
+ auto *EB = MetadataAsValue::get(Ctx, ExceptMDS);
+ Bundles.emplace_back("fp.except", EB);
+}
+
//===----------------------------------------------------------------------===//
// CallInst Implementation
//===----------------------------------------------------------------------===//
diff --git a/llvm/lib/IR/IntrinsicInst.cpp b/llvm/lib/IR/IntrinsicInst.cpp
index 23a4d1b5c615e..c00f3bb8fd6d8 100644
--- a/llvm/lib/IR/IntrinsicInst.cpp
+++ b/llvm/lib/IR/IntrinsicInst.cpp
@@ -66,6 +66,16 @@ bool IntrinsicInst::mayLowerToFunctionCall(Intrinsic::ID IID) {
}
}
+bool IntrinsicInst::isFloatingPointOperation(Intrinsic::ID IID) {
+ switch (IID) {
+#define FUNCTION(NAME, D) case Intrinsic::NAME:
+#include "llvm/IR/FloatingPointOps.def"
+ return true;
+ default:
+ return false;
+ }
+}
+
//===----------------------------------------------------------------------===//
/// DbgVariableIntrinsic - This is the common base class for debug info
/// intrinsics for variables.
diff --git a/llvm/lib/IR/LLVMContext.cpp b/llvm/lib/IR/LLVMContext.cpp
index 335c210c10e1a..7bde465ef390f 100644
--- a/llvm/lib/IR/LLVMContext.cpp
+++ b/llvm/lib/IR/LLVMContext.cpp
@@ -55,6 +55,10 @@ static StringRef knownBundleName(unsigned BundleTagID) {
return "convergencectrl";
case LLVMContext::OB_align:
return "align";
+ case LLVMContext::OB_fp_control:
+ return "fp.control";
+ case LLVMContext::OB_fp_except:
+ return "fp.except";
default:
llvm_unreachable("unknown bundle id");
}
@@ -84,6 +88,16 @@ LLVMContext::LLVMContext() : pImpl(new LLVMContextImpl(*this)) {
assert(Entry->second == BundleTagID && "operand bundle id drifted!");
}
+ auto *RoundingEntry = pImpl->getOrInsertBundleTag("fp.control");
+ assert(RoundingEntry->second == LLVMContext::OB_fp_control &&
+ "fp.control operand bundle id drifted!");
+ (void)RoundingEntry;
+
+ auto *ExceptionEntry = pImpl->getOrInsertBundleTag("fp.except");
+ assert(ExceptionEntry->second == LLVMContext::OB_fp_except &&
+ "fp.except operand bundle id drifted!");
+ (void)ExceptionEntry;
+
SyncScope::ID SingleThreadSSID =
pImpl->getOrInsertSyncScopeID("singlethread");
assert(SingleThreadSSID == SyncScope::SingleThread &&
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 7917712846990..d9f15d57f2338 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -3909,7 +3909,8 @@ void Verifier::visitCallBase(CallBase &Call) {
FoundGCTransitionBundle = false, FoundCFGuardTargetBundle = false,
FoundPreallocatedBundle = false, FoundGCLiveBundle = false,
FoundPtrauthBundle = false, FoundKCFIBundle = false,
- FoundAttachedCallBundle = false;
+ FoundAttachedCallBundle = false, FoundFpeControlBundle = false,
+ FoundFpeExceptBundle = false;
for (unsigned i = 0, e = Call.getNumOperandBundles(); i < e; ++i) {
OperandBundleUse BU = Call.getOperandBundleAt(i);
uint32_t Tag = BU.getTagID();
@@ -3972,6 +3973,34 @@ void Verifier::visitCallBase(CallBase &Call) {
"Multiple \"clang.arc.attachedcall\" operand bundles", Call);
FoundAttachedCallBundle = true;
verifyAttachedCallBundle(Call, BU);
+ } else if (Tag == LLVMContext::OB_fp_control) {
+ Check(!FoundFpeControlBundle, "Multiple fp.control operand bundles",
+ Call);
+ Check(BU.Inputs.size() == 1,
+ "Expected exactly one fp.control bundle operand", Call);
+ auto *V = dyn_cast<MetadataAsValue>(BU.Inputs.front());
+ Check(V, "Value of fp.control bundle operand must be a metadata", Call);
+ auto *MDS = dyn_cast<MDString>(V->getMetadata());
+ Check(MDS, "Value of fp.control bundle operand must be a string", Call);
+ auto RM = convertBundleToRoundingMode(MDS->getString());
+ Check(RM.has_value(),
+ "Value of fp.control bundle operand is not a correct rounding mode",
+ Call);
+ FoundFpeControlBundle = true;
+ } else if (Tag == LLVMContext::OB_fp_except) {
+ Check(!FoundFpeExceptBundle, "Multiple fp.except operand bundles", Call);
+ Check(BU.Inputs.size() == 1,
+ "Expected exactly one fp.except bundle operand", Call);
+ auto *V = dyn_cast<MetadataAsValue>(BU.Inputs.front());
+ Check(V, "Value of fp.except bundle operand must be a metadata", Call);
+ auto *MDS = dyn_cast<MDString>(V->getMetadata());
+ Check(MDS, "Value of fp.except bundle operand must be a string", Call);
+ auto EB = convertBundleToExceptionBehavior(MDS->getString());
+ Check(EB.has_value(),
+ "Value of fp.except bundle operand is not a correct exception "
+ "behavior",
+ Call);
+ FoundFpeExceptBundle = true;
}
}
diff --git a/llvm/test/Bitcode/operand-bundles-bc-analyzer.ll b/llvm/test/Bitcode/operand-bundles-bc-analyzer.ll
index 5628e17b4936e..107867675050b 100644
--- a/llvm/test/Bitcode/operand-bundles-bc-analyzer.ll
+++ b/llvm/test/Bitcode/operand-bundles-bc-analyzer.ll
@@ -14,6 +14,8 @@
; CHECK-NEXT: <OPERAND_BUNDLE_TAG
; CHECK-NEXT: <OPERAND_BUNDLE_TAG
; CHECK-NEXT: <OPERAND_BUNDLE_TAG
+; CHECK-NEXT: <OPERAND_BUNDLE_TAG
+; CHECK-NEXT: <OPERAND_BUNDLE_TAG
; CHECK-NEXT: </OPERAND_BUNDLE_TAGS_BLOCK
; CHECK: <FUNCTION_BLOCK
diff --git a/llvm/test/Verifier/fp-intrinsics.ll b/llvm/test/Verifier/fp-intrinsics.ll
index 4934843d5a2ed..12b0fca97e5af 100644
--- a/llvm/test/Verifier/fp-intrinsics.ll
+++ b/llvm/test/Verifier/fp-intrinsics.ll
@@ -51,4 +51,95 @@ entry:
ret double %fadd
}
+; Test multiple fp.control bundles.
+; CHECK-NEXT: Multiple fp.control operand bundles
+; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.control"(metadata !"rtz"), "fp.control"(metadata !"rtz") ]
+define double @f6(double %a) #0 {
+entry:
+ %ftrunc = call double @llvm.trunc.f64(double %a) #0 [ "fp.control"(metadata !"rtz"), "fp.control"(metadata !"rtz") ]
+ ret double %ftrunc
+}
+
+; Test fp.control bundle that has more than one operands.
+; CHECK-NEXT: Expected exactly one fp.control bundle operand
+; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.control"(metadata !"rtz", metadata !"rte") ]
+define double @f7(double %a) #0 {
+entry:
+ %ftrunc = call double @llvm.trunc.f64(double %a) #0 [ "fp.control"(metadata !"rtz", metadata !"rte") ]
+ ret double %ftrunc
+}
+
+; Test fp.control bundle that has non-metadata operand.
+; CHECK-NEXT: Value of fp.control bundle operand must be a metadata
+; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.control"(i32 0) ]
+define double @f8(double %a) #0 {
+entry:
+ %ftrunc = call double @llvm.trunc.f64(double %a) #0 [ "fp.control"(i32 0) ]
+ ret double %ftrunc
+}
+
+; Test fp.control bundle that has non-string operand.
+; CHECK-NEXT: Value of fp.control bundle operand must be a string
+; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.control"(metadata i64 3) ]
+define double @f9(double %a) #0 {
+entry:
+ %ftrunc = call double @llvm.trunc.f64(double %a) #0 [ "fp.control"(metadata !{i64 3}) ]
+ ret double %ftrunc
+}
+
+; Test fp.control bundle that specifies incorrect value.
+; CHECK-NEXT: Value of fp.control bundle operand is not a correct rounding mode
+; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.control"(metadata !"qqq") ]
+define double @f10(double %a) #0 {
+entry:
+ %ftrunc = call double @llvm.trunc.f64(double %a) #0 [ "fp.control"(metadata !"qqq") ]
+ ret double %ftrunc
+}
+
+; Test multiple fp.except bundles.
+; CHECK-NEXT: Multiple fp.except operand bundles
+; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.except"(metadata !"strict"), "fp.except"(metadata !"strict") ]
+define double @f11(double %a) #0 {
+entry:
+ %ftrunc = call double @llvm.trunc.f64(double %a) #0 [ "fp.except"(metadata !"strict"), "fp.except"(metadata !"strict") ]
+ ret double %ftrunc
+}
+
+; Test fp.except bundle that has more than one operands.
+; CHECK-NEXT: Expected exactly one fp.except bundle operand
+; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.except"(metadata !"strict", metadata !"strict") ]
+define double @f12(double %a) #0 {
+entry:
+ %ftrunc = call double @llvm.trunc.f64(double %a) #0 [ "fp.except"(metadata !"strict", metadata !"strict") ]
+ ret double %ftrunc
+}
+
+; Test fp.except bundle that has non-metadata operand.
+; CHECK-NEXT: Value of fp.except bundle operand must be a metadata
+; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.except"(i32 0) ]
+define double @f13(double %a) #0 {
+entry:
+ %ftrunc = call double @llvm.trunc.f64(double %a) #0 [ "fp.except"(i32 0) ]
+ ret double %ftrunc
+}
+
+; Test fp.except bundle that has non-string operand.
+; CHECK-NEXT: Value of fp.except bundle operand must be a string
+; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.except"(metadata i64 3) ]
+define double @f14(double %a) #0 {
+entry:
+ %ftrunc = call double @llvm.trunc.f64(double %a) #0 [ "fp.except"(metadata !{i64 3}) ]
+ ret double %ftrunc
+}
+
+; Test fp.except bundle that specifies incorrect value.
+; CHECK-NEXT: Value of fp.except bundle operand is not a correct exception behavior
+; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.except"(metadata !"qqq") ]
+define double @f15(double %a) #0 {
+entry:
+ %ftrunc = call double @llvm.trunc.f64(double %a) #0 [ "fp.except"(metadata !"qqq") ]
+ ret double %ftrunc
+}
+
+
attributes #0 = { strictfp }
diff --git a/llvm/unittests/IR/IRBuilderTest.cpp b/llvm/unittests/IR/IRBuilderTest.cpp
index 4d5bbe971a060..f881f85fce6ae 100644
--- a/llvm/unittests/IR/IRBuilderTest.cpp
+++ b/llvm/unittests/IR/IRBuilderTest.cpp
@@ -457,6 +457,203 @@ TEST_F(IRBuilderTest, ConstrainedFPFunctionCall) {
EXPECT_FALSE(verifyModule(*M));
}
+TEST_F(IRBuilderTest, FPBundlesDefault) {
+ IRBuilder<> Builder(BB);
+ GlobalVariable *GVDouble = new GlobalVariable(
+ *M, Type::getDoubleTy(Ctx), true, GlobalValue::ExternalLinkage, nullptr);
+ Value *FnArg = Builder.CreateLoad(GVDouble->getValueType(), GVDouble);
+ Function *Fn = Intrinsic::getOrInsertDeclaration(
+ M.get(), Intrinsic::nearbyint, {Type::getDoubleTy(Ctx)});
+
+ // A floating-point operation does not have side effects in default
+ // environment even.
+ {
+ Value *V = Builder.CreateCall(Fn, {FnArg});
+ auto *I = cast<IntrinsicInst>(V);
+ EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value());
+ EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value());
+ EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID());
+ EXPECT_EQ(RoundingMode::NearestTiesToEven, I->getRoundingMode());
+ EXPECT_EQ(fp::ebIgnore, I->getExceptionBehavior());
+ MemoryEffects ME = I->getMemoryEffects();
+ EXPECT_TRUE(ME.doesNotAccessMemory());
+ }
+
+ // Check call with FP bundles, rounding is set to default value.
+ // nearbyint(%x) [ "fp.control" (metadata !"rte") ]
+ {
+ SmallVector<OperandBundleDef, 1> Bundles;
+ llvm::addFPRoundingBundle(Ctx, Bundles, RoundingMode::NearestTiesToEven);
+ Value *V = Builder.CreateCall(Fn, {FnArg}, Bundles);
+ auto *I = cast<IntrinsicInst>(V);
+ EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value());
+ EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value());
+ EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID());
+ EXPECT_EQ(RoundingMode::NearestTiesToEven, I->getRoundingMode());
+ EXPECT_EQ(fp::ebIgnore, I->getExceptionBehavior());
+ MemoryEffects ME = I->getMemoryEffects();
+ EXPECT_TRUE(ME.doesNotAccessMemory());
+ }
+
+ // Check call with FP bundles, exception behavior is set to default value.
+ // nearbyint(%x) [ "fp.except" (metadata !"ignore") ]
+ {
+ SmallVector<OperandBundleDef, 1> Bundles;
+ llvm::addFPExceptionBundle(Ctx, Bundles, fp::ebIgnore);
+ Value *V = Builder.CreateCall(Fn, {FnArg}, Bundles);
+ auto *I = cast<IntrinsicInst>(V);
+ EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value());
+ EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value());
+ EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID());
+ EXPECT_EQ(RoundingMode::NearestTiesToEven, I->getRoundingMode());
+ EXPECT_EQ(fp::ebIgnore, I->getExceptionBehavior());
+ MemoryEffects ME = I->getMemoryEffects();
+ EXPECT_TRUE(ME.doesNotAccessMemory());
+ }
+
+ // Check call with FP bundles, both rounding mode and exception behavior are
+ // set.
+ // nearbyint(%x) [ "fp.except" (metadata !"ignore") ]
+ {
+ SmallVector<OperandBundleDef, 1> Bundles;
+ llvm::addFPRoundingBundle(Ctx, Bundles, RoundingMode::NearestTiesToEven);
+ llvm::addFPExceptionBundle(Ctx, Bundles, fp::ebIgnore);
+ Value *V = Builder.CreateCall(Fn, {FnArg}, Bundles);
+ auto *I = cast<IntrinsicInst>(V);
+ EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value());
+ EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value());
+ EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID());
+ EXPECT_EQ(RoundingMode::NearestTiesToEven, I->getRoundingMode());
+ EXPECT_EQ(fp::ebIgnore, I->getExceptionBehavior());
+ MemoryEffects ME = I->getMemoryEffects();
+ EXPECT_TRUE(ME.doesNotAccessMemory());
+ }
+}
+
+TEST_F(IRBuilderTest, FPBundlesStrict) {
+ F->addFnAttr(Attribute::StrictFP);
+
+ IRBuilder<> Builder(BB);
+ Builder.setDefaultConstrainedExcept(fp::ebStrict);
+ Builder.setDefaultConstrainedRounding(RoundingMode::TowardZero);
+ Builder.setIsFPConstrained(true);
+
+ GlobalVariable *GVDouble = new GlobalVariable(
+ *M, Type::getDoubleTy(Ctx), true, GlobalValue::ExternalLinkage, nullptr);
+ Value *FnArg = Builder.CreateLoad(GVDouble->getValueType(), GVDouble);
+ Function *Fn = Intrinsic::getOrInsertDeclaration(
+ M.get(), Intrinsic::nearbyint, {Type::getDoubleTy(Ctx)});
+
+ // A floating-point operation has side effects in strictfp environment even
+ // if it has no FP bundles.
+ {
+ Value *V = Builder.CreateCall(Fn, {FnArg});
+ auto *I = cast<IntrinsicInst>(V);
+ EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value());
+ EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value());
+ EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID());
+ EXPECT_EQ(RoundingMode::Dynamic, I->getRoundingMode());
+ EXPECT_EQ(fp::ebStrict, I->getExceptionBehavior());
+ MemoryEffects ME = I->getMemoryEffects();
+ EXPECT_TRUE(ME.doesAccessInaccessibleMem());
+ }
+
+ // Check call with FP bundles, with default (dynamic) rounding mode
+ // nearbyint(%x) [ "fp.control" (metadata !"dyn") ]
+ {
+ SmallVector<OperandBundleDef, 1> Bundles;
+ llvm::addFPRoundingBundle(Ctx, Bundles, RoundingMode::Dynamic);
+ Value *V = Builder.CreateCall(Fn, {FnArg}, Bundles);
+ auto *I = cast<IntrinsicInst>(V);
+ EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value());
+ EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value());
+ EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID());
+ EXPECT_EQ(RoundingMode::Dynamic, I->getRoundingMode());
+ EXPECT_EQ(fp::ebStrict, I->getExceptionBehavior());
+ MemoryEffects ME = I->getMemoryEffects();
+ EXPECT_TRUE(ME.doesAccessInaccessibleMem());
+ }
+
+ // Check call with FP bundles, with specific rounding mode
+ // nearbyint(%x) [ "fp.control" (metadata !"rtz") ]
+ {
+ SmallVector<OperandBundleDef, 1> Bundles;
+ llvm::addFPRoundingBundle(Ctx, Bundles, RoundingMode::TowardZero);
+ Value *V = Builder.CreateCall(Fn, {FnArg}, Bundles);
+ auto *I = cast<IntrinsicInst>(V);
+ EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value());
+ EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value());
+ EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID());
+ EXPECT_EQ(RoundingMode::TowardZero, I->getRoundingMode());
+ EXPECT_EQ(fp::ebStrict, I->getExceptionBehavior());
+ MemoryEffects ME = I->getMemoryEffects();
+ EXPECT_TRUE(ME.doesAccessInaccessibleMem());
+ }
+
+ // Check call with FP bundles, exception behavior is set to default value.
+ // nearbyint(%x) [ "fp.except" (metadata !"strict") ]
+ {
+ SmallVector<OperandBundleDef, 1> Bundles;
+ llvm::addFPExceptionBundle(Ctx, Bundles, fp::ebStrict);
+ Value *V = Builder.CreateCall(Fn, {FnArg}, Bundles);
+ auto *I = cast<IntrinsicInst>(V);
+ EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value());
+ EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value());
+ EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID());
+ EXPECT_EQ(RoundingMode::Dynamic, I->getRoundingMode());
+ EXPECT_EQ(fp::ebStrict, I->getExceptionBehavior());
+ MemoryEffects ME = I->getMemoryEffects();
+ EXPECT_TRUE(ME.doesAccessInaccessibleMem());
+ }
+
+ // Check call with FP bundles, exception behavior is set to specific value.
+ // nearbyint(%x) [ "fp.except" (metadata !"ignore") ]
+ {
+ SmallVector<OperandBundleDef, 1> Bundles;
+ llvm::addFPExceptionBundle(Ctx, Bundles, fp::ebIgnore);
+ Value *V = Builder.CreateCall(Fn, {FnArg}, Bundles);
+ auto *I = cast<IntrinsicInst>(V);
+ EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value());
+ EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value());
+ EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID());
+ EXPECT_EQ(RoundingMode::Dynamic, I->getRoundingMode());
+ EXPECT_EQ(fp::ebIgnore, I->getExceptionBehavior());
+ MemoryEffects ME = I->getMemoryEffects();
+ EXPECT_TRUE(ME.doesAccessInaccessibleMem());
+ }
+
+ // Check call with both FP bundles.
+ // nearbyint(%x) [ "fp.control" (metadata !"rtz"),
+ // "fp.except" (metadata !"ignore") ]
+ {
+ SmallVector<OperandBundleDef, 1> Bundles;
+ llvm::addFPRoundingBundle(Ctx, Bundles, RoundingMode::NearestTiesToEven);
+ llvm::addFPExceptionBundle(Ctx, Bundles, fp::ebIgnore);
+ Value *V = Builder.CreateCall(Fn, {FnArg}, Bundles);
+ auto *I = cast<IntrinsicInst>(V);
+ EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value());
+ EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value());
+ EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID());
+ EXPECT_EQ(RoundingMode::NearestTiesToEven, I->getRoundingMode());
+ EXPECT_EQ(fp::ebIgnore, I->getExceptionBehavior());
+ MemoryEffects ME = I->getMemoryEffects();
+ EXPECT_TRUE(ME.doesAccessInaccessibleMem());
+ }
+
+ // Function calls, that do not depend on FP options, does not have
+ // memory effects.
+ {
+ Function *Fn = Intrinsic::getOrInsertDeclaration(M.get(), Intrinsic::fabs,
+ {Type::getDoubleTy(Ctx)});
+ Value *V = Builder.CreateCall(Fn, {FnArg});
+ auto *I = cast<IntrinsicInst>(V);
+ EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value());
+ EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value());
+ MemoryEffects ME = I->getMemoryEffects();
+ EXPECT_TRUE(ME.doesNotAccessMemory());
+ }
+}
+
TEST_F(IRBuilderTest, Lifetime) {
IRBuilder<> Builder(BB);
AllocaInst *Var1 = Builder.CreateAlloca(Builder.getInt8Ty());
>From 1c47c38a37ed772c37056fd74acc814aa162ad34 Mon Sep 17 00:00:00 2001
From: Serge Pavlov <sepavloff at gmail.com>
Date: Wed, 16 Apr 2025 00:03:49 +0700
Subject: [PATCH 2/9] Fix test
---
clang/test/CodeGen/strictfp-elementwise-builtins.cpp | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/clang/test/CodeGen/strictfp-elementwise-builtins.cpp b/clang/test/CodeGen/strictfp-elementwise-builtins.cpp
index b250512efc5c7..532686512b0d4 100644
--- a/clang/test/CodeGen/strictfp-elementwise-builtins.cpp
+++ b/clang/test/CodeGen/strictfp-elementwise-builtins.cpp
@@ -210,7 +210,7 @@ float4 strict_elementwise_rint(float4 a) {
// CHECK-LABEL: define dso_local noundef <4 x float> @_Z28strict_elementwise_nearbyintDv4_f
// CHECK-SAME: (<4 x float> noundef [[A:%.*]]) local_unnamed_addr #[[ATTR2]] {
// CHECK-NEXT: entry:
-// CHECK-NEXT: [[ELT_NEARBYINT:%.*]] = tail call <4 x float> @llvm.nearbyint.v4f32(<4 x float> [[A]]) #[[ATTR4]]
+// CHECK-NEXT: [[ELT_NEARBYINT:%.*]] = tail call <4 x float> @llvm.nearbyint.v4f32(<4 x float> [[A]]) #[[ATTR5:[0-9]+]]
// CHECK-NEXT: ret <4 x float> [[ELT_NEARBYINT]]
//
float4 strict_elementwise_nearbyint(float4 a) {
@@ -300,7 +300,7 @@ float4 strict_elementwise_atan2(float4 a, float4 b) {
// CHECK-LABEL: define dso_local noundef <4 x float> @_Z24strict_elementwise_truncDv4_f
// CHECK-SAME: (<4 x float> noundef [[A:%.*]]) local_unnamed_addr #[[ATTR2]] {
// CHECK-NEXT: entry:
-// CHECK-NEXT: [[ELT_TRUNC:%.*]] = tail call <4 x float> @llvm.trunc.v4f32(<4 x float> [[A]]) #[[ATTR4]]
+// CHECK-NEXT: [[ELT_TRUNC:%.*]] = tail call <4 x float> @llvm.trunc.v4f32(<4 x float> [[A]]) #[[ATTR5]]
// CHECK-NEXT: ret <4 x float> [[ELT_TRUNC]]
//
float4 strict_elementwise_trunc(float4 a) {
>From 4407af42ae335824dfb48199b78b4b10bbd19a4e Mon Sep 17 00:00:00 2001
From: Serge Pavlov <sepavloff at gmail.com>
Date: Thu, 17 Apr 2025 14:00:55 +0700
Subject: [PATCH 3/9] Set MemoryEffects of call only if called function is
nomem
It make call close to constrained calls, where memory effects of the
calls do not define memory effects. It helps migration to operand
bundles.
---
llvm/lib/IR/IRBuilder.cpp | 9 ++++-----
1 file changed, 4 insertions(+), 5 deletions(-)
diff --git a/llvm/lib/IR/IRBuilder.cpp b/llvm/lib/IR/IRBuilder.cpp
index 7ad78fc4190ea..09e71208b224e 100644
--- a/llvm/lib/IR/IRBuilder.cpp
+++ b/llvm/lib/IR/IRBuilder.cpp
@@ -132,12 +132,13 @@ CallInst *IRBuilderBase::CreateCall(FunctionType *FTy, Value *Callee,
// If the builder is in strictfp mode and has non-default options (like
// non-dynamic rounding), add corresponding operand bundle. If such bundle is
// already present, assume it overwrites defaults.
- bool doesTheCallAccessFPEnv = false;
+ bool NeedUpdateMemoryEffects = false;
if (IsFPConstrained) {
if (const auto *Func = dyn_cast<Function>(Callee)) {
if (Intrinsic::ID ID = Func->getIntrinsicID()) {
if (IntrinsicInst::isFloatingPointOperation(ID)) {
- doesTheCallAccessFPEnv = true;
+ MemoryEffects FME = Func->getMemoryEffects();
+ NeedUpdateMemoryEffects = !FME.doesAccessInaccessibleMem();
bool NeedRound = DefaultConstrainedRounding != RoundingMode::Dynamic;
bool NeedExcept = DefaultConstrainedExcept != fp::ebStrict;
for (const auto &Item : OpBundles) {
@@ -159,10 +160,8 @@ CallInst *IRBuilderBase::CreateCall(FunctionType *FTy, Value *Callee,
// If the call accesses FPE, update memory effects accordingly.
CallInst *CI = CallInst::Create(FTy, Callee, Args, OpBundles);
- if (doesTheCallAccessFPEnv) {
+ if (NeedUpdateMemoryEffects) {
MemoryEffects ME = MemoryEffects::inaccessibleMemOnly();
- if (CI->getAttributes().hasFnAttr(Attribute::Memory))
- ME |= CI->getAttributes().getMemoryEffects();
auto A = Attribute::getWithMemoryEffects(getContext(), ME);
CI->addFnAttr(A);
}
>From be57027ccfda1b71c0bb2852566567ddd1739bd2 Mon Sep 17 00:00:00 2001
From: Serge Pavlov <sepavloff at gmail.com>
Date: Thu, 17 Apr 2025 15:30:17 +0700
Subject: [PATCH 4/9] Fix method signature
---
llvm/include/llvm/IR/InstrTypes.h | 6 +++---
llvm/lib/IR/Instructions.cpp | 2 +-
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/llvm/include/llvm/IR/InstrTypes.h b/llvm/include/llvm/IR/InstrTypes.h
index b22230713bba5..b1fcc6fc56c4d 100644
--- a/llvm/include/llvm/IR/InstrTypes.h
+++ b/llvm/include/llvm/IR/InstrTypes.h
@@ -2172,11 +2172,11 @@ class CallBase : public Instruction {
return false;
}
- /// Return rounding mode specified by operand bundles.
+ /// Return rounding mode specified for this call.
RoundingMode getRoundingMode() const;
- /// Return exception behavior specified by operand bundles.
- std::optional<fp::ExceptionBehavior> getExceptionBehavior() const;
+ /// Return exception behavior specified for this call.
+ fp::ExceptionBehavior getExceptionBehavior() const;
/// Used to keep track of an operand bundle. See the main comment on
/// OperandBundleUser above.
diff --git a/llvm/lib/IR/Instructions.cpp b/llvm/lib/IR/Instructions.cpp
index e63bd170fa800..25c78e4c1a99e 100644
--- a/llvm/lib/IR/Instructions.cpp
+++ b/llvm/lib/IR/Instructions.cpp
@@ -656,7 +656,7 @@ RoundingMode CallBase::getRoundingMode() const {
return RoundingMode::NearestTiesToEven;
}
-std::optional<fp::ExceptionBehavior> CallBase::getExceptionBehavior() const {
+fp::ExceptionBehavior CallBase::getExceptionBehavior() const {
// Try determining exception behavior from FP bundle.
std::optional<fp::ExceptionBehavior> EB;
if (auto ExceptionBundle = getOperandBundle(LLVMContext::OB_fp_except)) {
>From 552275b1b6e2e98db2509d110707cd522aaec794 Mon Sep 17 00:00:00 2001
From: Serge Pavlov <sepavloff at gmail.com>
Date: Fri, 18 Apr 2025 01:55:35 +0700
Subject: [PATCH 5/9] Organize checks in Verifier
---
llvm/lib/IR/Verifier.cpp | 52 ++++++++++++++++++-----------
llvm/test/Verifier/fp-intrinsics.ll | 22 ++++++------
2 files changed, 44 insertions(+), 30 deletions(-)
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index d9f15d57f2338..c7e350651245d 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -3974,32 +3974,46 @@ void Verifier::visitCallBase(CallBase &Call) {
FoundAttachedCallBundle = true;
verifyAttachedCallBundle(Call, BU);
} else if (Tag == LLVMContext::OB_fp_control) {
- Check(!FoundFpeControlBundle, "Multiple fp.control operand bundles",
- Call);
- Check(BU.Inputs.size() == 1,
- "Expected exactly one fp.control bundle operand", Call);
- auto *V = dyn_cast<MetadataAsValue>(BU.Inputs.front());
- Check(V, "Value of fp.control bundle operand must be a metadata", Call);
- auto *MDS = dyn_cast<MDString>(V->getMetadata());
- Check(MDS, "Value of fp.control bundle operand must be a string", Call);
- auto RM = convertBundleToRoundingMode(MDS->getString());
- Check(RM.has_value(),
- "Value of fp.control bundle operand is not a correct rounding mode",
+ Check(!FoundFpeControlBundle, "Multiple \"fp.control\" operand bundles",
Call);
+ bool FoundRoundingMode = false;
+ for (auto &U : BU.Inputs) {
+ Value *V = U.get();
+ Check(isa<MetadataAsValue>(V),
+ "Value of a \"fp.control\" bundle operand must be a metadata",
+ Call);
+ Metadata *MD = cast<MetadataAsValue>(V)->getMetadata();
+ Check(isa<MDString>(MD),
+ "Value of a \"fp.control\" bundle operand must be a string",
+ Call);
+ StringRef Item = cast<MDString>(MD)->getString();
+ if (convertBundleToRoundingMode(Item)) {
+ Check(!FoundRoundingMode, "Rounding mode is specified more that once",
+ Call);
+ FoundRoundingMode = true;
+ } else {
+ CheckFailed("Unrecognized value in \"fp.control\" bundle operand",
+ Call);
+ }
+ }
FoundFpeControlBundle = true;
} else if (Tag == LLVMContext::OB_fp_except) {
- Check(!FoundFpeExceptBundle, "Multiple fp.except operand bundles", Call);
+ Check(!FoundFpeExceptBundle, "Multiple \"fp.except\" operand bundles",
+ Call);
Check(BU.Inputs.size() == 1,
- "Expected exactly one fp.except bundle operand", Call);
+ "Expected exactly one \"fp.except\" bundle operand", Call);
auto *V = dyn_cast<MetadataAsValue>(BU.Inputs.front());
- Check(V, "Value of fp.except bundle operand must be a metadata", Call);
+ Check(V, "Value of a \"fp.except\" bundle operand must be a metadata",
+ Call);
auto *MDS = dyn_cast<MDString>(V->getMetadata());
- Check(MDS, "Value of fp.except bundle operand must be a string", Call);
- auto EB = convertBundleToExceptionBehavior(MDS->getString());
- Check(EB.has_value(),
- "Value of fp.except bundle operand is not a correct exception "
- "behavior",
+ Check(MDS, "Value of a \"fp.except\" bundle operand must be a string",
Call);
+ auto EB = convertBundleToExceptionBehavior(MDS->getString());
+ Check(
+ EB.has_value(),
+ "Value of a \"fp.except\" bundle operand is not a correct exception "
+ "behavior",
+ Call);
FoundFpeExceptBundle = true;
}
}
diff --git a/llvm/test/Verifier/fp-intrinsics.ll b/llvm/test/Verifier/fp-intrinsics.ll
index 12b0fca97e5af..1e1df1b0da96a 100644
--- a/llvm/test/Verifier/fp-intrinsics.ll
+++ b/llvm/test/Verifier/fp-intrinsics.ll
@@ -52,7 +52,7 @@ entry:
}
; Test multiple fp.control bundles.
-; CHECK-NEXT: Multiple fp.control operand bundles
+; CHECK-NEXT: Multiple "fp.control" operand bundles
; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.control"(metadata !"rtz"), "fp.control"(metadata !"rtz") ]
define double @f6(double %a) #0 {
entry:
@@ -60,8 +60,8 @@ entry:
ret double %ftrunc
}
-; Test fp.control bundle that has more than one operands.
-; CHECK-NEXT: Expected exactly one fp.control bundle operand
+; Test fp.control bundle that has more than one rounding mode specification.
+; CHECK-NEXT: Rounding mode is specified more that once
; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.control"(metadata !"rtz", metadata !"rte") ]
define double @f7(double %a) #0 {
entry:
@@ -70,7 +70,7 @@ entry:
}
; Test fp.control bundle that has non-metadata operand.
-; CHECK-NEXT: Value of fp.control bundle operand must be a metadata
+; CHECK-NEXT: Value of a "fp.control" bundle operand must be a metadata
; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.control"(i32 0) ]
define double @f8(double %a) #0 {
entry:
@@ -79,7 +79,7 @@ entry:
}
; Test fp.control bundle that has non-string operand.
-; CHECK-NEXT: Value of fp.control bundle operand must be a string
+; CHECK-NEXT: Value of a "fp.control" bundle operand must be a string
; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.control"(metadata i64 3) ]
define double @f9(double %a) #0 {
entry:
@@ -88,7 +88,7 @@ entry:
}
; Test fp.control bundle that specifies incorrect value.
-; CHECK-NEXT: Value of fp.control bundle operand is not a correct rounding mode
+; CHECK-NEXT: Unrecognized value in "fp.control" bundle operand
; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.control"(metadata !"qqq") ]
define double @f10(double %a) #0 {
entry:
@@ -97,7 +97,7 @@ entry:
}
; Test multiple fp.except bundles.
-; CHECK-NEXT: Multiple fp.except operand bundles
+; CHECK-NEXT: Multiple "fp.except" operand bundles
; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.except"(metadata !"strict"), "fp.except"(metadata !"strict") ]
define double @f11(double %a) #0 {
entry:
@@ -106,7 +106,7 @@ entry:
}
; Test fp.except bundle that has more than one operands.
-; CHECK-NEXT: Expected exactly one fp.except bundle operand
+; CHECK-NEXT: Expected exactly one "fp.except" bundle operand
; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.except"(metadata !"strict", metadata !"strict") ]
define double @f12(double %a) #0 {
entry:
@@ -115,7 +115,7 @@ entry:
}
; Test fp.except bundle that has non-metadata operand.
-; CHECK-NEXT: Value of fp.except bundle operand must be a metadata
+; CHECK-NEXT: Value of a "fp.except" bundle operand must be a metadata
; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.except"(i32 0) ]
define double @f13(double %a) #0 {
entry:
@@ -124,7 +124,7 @@ entry:
}
; Test fp.except bundle that has non-string operand.
-; CHECK-NEXT: Value of fp.except bundle operand must be a string
+; CHECK-NEXT: Value of a "fp.except" bundle operand must be a string
; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.except"(metadata i64 3) ]
define double @f14(double %a) #0 {
entry:
@@ -133,7 +133,7 @@ entry:
}
; Test fp.except bundle that specifies incorrect value.
-; CHECK-NEXT: Value of fp.except bundle operand is not a correct exception behavior
+; CHECK-NEXT: Value of a "fp.except" bundle operand is not a correct exception behavior
; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.except"(metadata !"qqq") ]
define double @f15(double %a) #0 {
entry:
>From fcb5a4c5b3cb2a3dcc6745a122722ba6a8ccc815 Mon Sep 17 00:00:00 2001
From: Serge Pavlov <sepavloff at gmail.com>
Date: Sat, 19 Apr 2025 02:05:28 +0700
Subject: [PATCH 6/9] Small changes in documentation
---
llvm/docs/LangRef.rst | 13 +++++++------
1 file changed, 7 insertions(+), 6 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 2e121a00cf3a4..2073f888faa3d 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -3213,8 +3213,9 @@ floating-point control modes and the treatment of status bits respectively.
An operand bundle tagged with "fp.control" contains information about the
control modes used for the operation execution. Operands specified in this
bundle represent particular options. Currently, only rounding mode is supported.
-It is represented by a metadata string value, which specifies the rounding mode
-to be used for the operation evaluation. Possible values are:
+
+Rounding mode is represented by a metadata string value, which specifies the
+the mode used for the operation evaluation. Possible values are:
::
@@ -3225,10 +3226,10 @@ to be used for the operation evaluation. Possible values are:
"rmm" - to nearest, ties away from zero
"dyn" - rounding mode is taken from control register
-If "fp.control" is absent, the default rounding rounding mode is taken from the
-control register (dynamic rounding). In the particular case of
-:ref:`default floating-point environment <floatenv>`, it must be rounding to
-nearest, ties to even.
+Only one such value may be specified. If "fp.control" is absent, the default
+rounding rounding mode is taken from the control register (dynamic rounding).
+In the particular case of :ref:`default floating-point environment <floatenv>`,
+the operation uses rounding to nearest, ties to even.
An operand bundle tagged with "fp.except" may be associated with operations
that can read or write floating-point exception flags. It contains a single
>From a01d4ad08d23948fb549defc8b82ae1cd6bacdce Mon Sep 17 00:00:00 2001
From: Serge Pavlov <sepavloff at gmail.com>
Date: Wed, 23 Jul 2025 13:16:45 +0700
Subject: [PATCH 7/9] Update according to the obtained feedback
- Bundles are renamed: "fp.control" -> "fp.round",
- Corrected documentation,
- DAG nodes STRICT_* are used to lower the supported intrinsics,
- Constrained properties in IRBuilder are reused for handling FP bundles,
- DAG gets methods for lowering FP operations,
- IR dump prints FP memory effects,
- Fixed some problems in IRBuilderBase::CreateCall,
- Added a new check to Verifier,
- Added tests for checking code generation.
---
clang/lib/CodeGen/CodeGenFunction.cpp | 2 +-
llvm/docs/LangRef.rst | 53 ++---
llvm/include/llvm/IR/FloatingPointOps.def | 16 +-
llvm/include/llvm/IR/IRBuilder.h | 35 ++--
llvm/include/llvm/IR/InstrTypes.h | 22 ++-
llvm/include/llvm/IR/LLVMContext.h | 3 +-
.../SelectionDAG/SelectionDAGBuilder.cpp | 86 +++++++-
.../SelectionDAG/SelectionDAGBuilder.h | 3 +
llvm/lib/IR/AsmWriter.cpp | 25 +++
llvm/lib/IR/FPEnv.cpp | 24 +--
llvm/lib/IR/IRBuilder.cpp | 87 +++++----
llvm/lib/IR/Instructions.cpp | 67 ++++---
llvm/lib/IR/LLVMContext.cpp | 10 +-
llvm/lib/IR/Verifier.cpp | 24 ++-
.../CodeGen/X86/fp-strict-scalar-round.ll | 184 ++++++++++++++++++
llvm/test/CodeGen/X86/vec-strict-512.ll | 36 ++++
llvm/test/Verifier/fp-intrinsics.ll | 47 +++--
llvm/unittests/IR/IRBuilderTest.cpp | 91 ++++++---
18 files changed, 616 insertions(+), 199 deletions(-)
diff --git a/clang/lib/CodeGen/CodeGenFunction.cpp b/clang/lib/CodeGen/CodeGenFunction.cpp
index 88628530cf66b..02d8cb638087a 100644
--- a/clang/lib/CodeGen/CodeGenFunction.cpp
+++ b/clang/lib/CodeGen/CodeGenFunction.cpp
@@ -1089,7 +1089,7 @@ void CodeGenFunction::StartFunction(GlobalDecl GD, QualType RetTy,
if ((FD && (FD->UsesFPIntrin() || FD->hasAttr<StrictFPAttr>())) ||
(!FD && (FPExceptionBehavior != llvm::fp::ebIgnore ||
RM != llvm::RoundingMode::NearestTiesToEven))) {
- Builder.setIsFPConstrained(true);
+ Builder.setIsFPConstrained(true, false);
Fn->addFnAttr(llvm::Attribute::StrictFP);
}
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 2073f888faa3d..5462c497c5baa 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -3207,43 +3207,48 @@ Floating-point Operand Bundles
These operand bundles are used for calls that involve floating-point
operations and interact with :ref:`floating-point environment <floatenv>` or
depend on floating-point options, such as rounding mode, denormal modes, etc.
-There are two kinds of such operand bundles, which represent the value of
-floating-point control modes and the treatment of status bits respectively.
-An operand bundle tagged with "fp.control" contains information about the
-control modes used for the operation execution. Operands specified in this
-bundle represent particular options. Currently, only rounding mode is supported.
-
-Rounding mode is represented by a metadata string value, which specifies the
-the mode used for the operation evaluation. Possible values are:
+An operand bundle tagged with "fp.round" contains information about the
+rounding mode used for the operation execution (the effective rounding mode).
+This mode is represented by a metadata string value and specifies the mode used
+for the operation evaluation. Possible values are:
::
- "rtz" - toward zero
- "rte" - to nearest, ties to even
- "rtp" - toward positive infinity
- "rtn" - toward negative infinity
- "rmm" - to nearest, ties away from zero
- "dyn" - rounding mode is taken from control register
+ "towardzero"
+ "tonearest"
+ "upward"
+ "downward"
+ "tonearestaway"
+ "dynamic"
-Only one such value may be specified. If "fp.control" is absent, the default
-rounding rounding mode is taken from the control register (dynamic rounding).
-In the particular case of :ref:`default floating-point environment <floatenv>`,
-the operation uses rounding to nearest, ties to even.
+Only one value may be specified. If the "fp.round" bundle is absent, and
+the operation depends on rounding mode, the default behavior is to use the value
+from a control register (dynamic rounding). In the specific case of
+:ref:`default floating-point environment <floatenv>`, the register is assumed to
+set rounding to nearest, ties to even.
An operand bundle tagged with "fp.except" may be associated with operations
-that can read or write floating-point exception flags. It contains a single
-metadata string value, which can have one of the following values:
+that can raise floating-point exceptions. It contains a single metadata string
+value, which can have one of the following:
::
"ignore"
"strict"
- "maytrap"
-
-It has the same meaning as the corresponding argument in
-:ref:`constrained intrinsics <constrainedfp>`.
+If the argument is "ignore", floating-point exceptions raised by the call are not
+intended to be observed. Optimization transformations may reorder such operations
+or omit them in some cases. For example, if the call result is unused, the call
+may be removed, even if it could raise exceptions. This is the only permitted value
+in the :ref:`default floating-point environment <floatenv>`.
+
+If the argument of "fp.except" is "strict", all transformations must preserve the
+floating-point exception semantics of the original code. Any exception that would
+have been raised by the original code must also be raised by the transformed code
+unless it can be proven unobservable. No new observable floating-point exceptions
+may be introduced. This value may only be used only in functions with
+``strictfp`` attribute.
.. _moduleasm:
diff --git a/llvm/include/llvm/IR/FloatingPointOps.def b/llvm/include/llvm/IR/FloatingPointOps.def
index 8567b5dbac302..465be751a0087 100644
--- a/llvm/include/llvm/IR/FloatingPointOps.def
+++ b/llvm/include/llvm/IR/FloatingPointOps.def
@@ -10,15 +10,21 @@
//
//===----------------------------------------------------------------------===//
+// Describes floating-point operation.
+// Arguments of the entries are:
+// N - intrinsic function name,
+// D - DAG node corresponding to the intrinsic.
#ifndef FUNCTION
#define FUNCTION(N,D)
#endif
-// Arguments of the entries are:
-// - intrinsic function name,
-// - DAG node corresponding to the intrinsic.
+// Describes floating-point operation, which lowers to STRICT_* nodes in DAG.
+#ifndef LEGACY_DAG
+#define LEGACY_DAG(N,D) FUNCTION(N,D)
+#endif
-FUNCTION(nearbyint, FNEARBYINT)
-FUNCTION(trunc, FTRUNC)
+LEGACY_DAG(nearbyint, FNEARBYINT)
+LEGACY_DAG(trunc, FTRUNC)
#undef FUNCTION
+#undef LEGACY_DAG
diff --git a/llvm/include/llvm/IR/IRBuilder.h b/llvm/include/llvm/IR/IRBuilder.h
index eba2c6630f2d2..6494cf26cb3a0 100644
--- a/llvm/include/llvm/IR/IRBuilder.h
+++ b/llvm/include/llvm/IR/IRBuilder.h
@@ -153,8 +153,8 @@ class IRBuilderBase {
FastMathFlags FMF;
bool IsFPConstrained = false;
- fp::ExceptionBehavior DefaultConstrainedExcept = fp::ebStrict;
- RoundingMode DefaultConstrainedRounding = RoundingMode::Dynamic;
+ fp::ExceptionBehavior DefaultConstrainedExcept = fp::ebIgnore;
+ RoundingMode DefaultConstrainedRounding = RoundingMode::NearestTiesToEven;
ArrayRef<OperandBundleDef> DefaultOperandBundles;
@@ -348,7 +348,18 @@ class IRBuilderBase {
/// enabled the CreateF<op>() calls instead create constrained
/// floating point intrinsic calls. Fast math flags are unaffected
/// by this setting.
- void setIsFPConstrained(bool IsCon) { IsFPConstrained = IsCon; }
+ void setIsFPConstrained(bool IsCon, bool AndReset = true) {
+ if (AndReset) {
+ if (IsCon) {
+ setDefaultConstrainedRounding(RoundingMode::Dynamic);
+ setDefaultConstrainedExcept(fp::ebStrict);
+ } else {
+ setDefaultConstrainedRounding(RoundingMode::NearestTiesToEven);
+ setDefaultConstrainedExcept(fp::ebIgnore);
+ }
+ }
+ IsFPConstrained = IsCon;
+ }
/// Query for the use of constrained floating point math
bool getIsFPConstrained() { return IsFPConstrained; }
@@ -2767,23 +2778,21 @@ class IRBuilderBase {
LLVM_ABI CallInst *CreateDereferenceableAssumption(Value *PtrValue,
Value *SizeValue);
- /// Create an operand bundle in the provided bundle set to represent given FP
- /// rounding mode.
+ /// Create an operand bundle in the provided bundle set to represent the given
+ /// floating-point rounding mode.
///
/// If the rounding mode is not defined, adds the default rounding mode,
/// stored in this builder object.
- void
- createFPRoundingBundle(SmallVectorImpl<OperandBundleDef> &Bundles,
- std::optional<RoundingMode> Rounding = std::nullopt);
+ void createRoundingBundle(SmallVectorImpl<OperandBundleDef> &Bundles,
+ RoundingMode RM);
- /// Create an operand bundle in the provided bundle set to represent FP
- /// exception behavior.
+ /// Create an operand bundle in the provided bundle set to represent the given
+ /// floating-point exception behavior.
///
/// If the exception behavior is not defined, adds the default behavior,
/// stored in this builder object.
- void createFPExceptionBundle(
- SmallVectorImpl<OperandBundleDef> &Bundles,
- std::optional<fp::ExceptionBehavior> Except = std::nullopt);
+ void createExceptionBundle(SmallVectorImpl<OperandBundleDef> &Bundles,
+ fp::ExceptionBehavior Except);
};
/// This provides a uniform API for creating instructions and inserting
diff --git a/llvm/include/llvm/IR/InstrTypes.h b/llvm/include/llvm/IR/InstrTypes.h
index b1fcc6fc56c4d..bd764472ab58a 100644
--- a/llvm/include/llvm/IR/InstrTypes.h
+++ b/llvm/include/llvm/IR/InstrTypes.h
@@ -1094,12 +1094,17 @@ template <typename InputTy> class OperandBundleDefT {
using OperandBundleDef = OperandBundleDefT<Value *>;
using ConstOperandBundleDef = OperandBundleDefT<const Value *>;
-void addFPRoundingBundle(LLVMContext &Ctx,
- SmallVectorImpl<OperandBundleDef> &Bundles,
- RoundingMode Rounding);
-void addFPExceptionBundle(LLVMContext &Ctx,
- SmallVectorImpl<OperandBundleDef> &Bundles,
- fp::ExceptionBehavior Except);
+/// Add a bundle with tag "fp.round" and the specified rounding to the given
+/// bundle set.
+void addRoundingBundle(LLVMContext &Ctx,
+ SmallVectorImpl<OperandBundleDef> &Bundles,
+ RoundingMode Rounding);
+
+/// Add a bundle with tag "fp.except" and the specified exception behavior to
+/// the given bundle set.
+void addExceptionBundle(LLVMContext &Ctx,
+ SmallVectorImpl<OperandBundleDef> &Bundles,
+ fp::ExceptionBehavior Except);
//===----------------------------------------------------------------------===//
// CallBase Class
@@ -1160,6 +1165,7 @@ class CallBase : public Instruction {
/// number of extra operands.
LLVM_ABI unsigned getNumSubclassExtraOperandsDynamic() const;
+ /// Get memory effects specific to floating-point operations.
MemoryEffects getFloatingPointMemoryEffects() const;
public:
@@ -2172,10 +2178,10 @@ class CallBase : public Instruction {
return false;
}
- /// Return rounding mode specified for this call.
+ /// Return the effective rounding mode for this call.
RoundingMode getRoundingMode() const;
- /// Return exception behavior specified for this call.
+ /// Return the effective exception behavior for this call.
fp::ExceptionBehavior getExceptionBehavior() const;
/// Used to keep track of an operand bundle. See the main comment on
diff --git a/llvm/include/llvm/IR/LLVMContext.h b/llvm/include/llvm/IR/LLVMContext.h
index 437e3d89fea1a..4e125e8583ffa 100644
--- a/llvm/include/llvm/IR/LLVMContext.h
+++ b/llvm/include/llvm/IR/LLVMContext.h
@@ -98,9 +98,10 @@ class LLVMContext {
OB_kcfi = 8, // "kcfi"
OB_convergencectrl = 9, // "convergencectrl"
OB_align = 10, // "align"
- OB_fp_control = 11, // "fp.control"
+ OB_fp_round = 11, // "fp.round"
OB_fp_except = 12, // "fp.except"
OB_LastBundleID = OB_fp_except // Marker for last bundle ID };
+ };
/// getMDKindID - Return a unique non-zero ID for the specified metadata kind.
/// This ID is uniqued across modules in the current LLVMContext.
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index a52265055c88a..dd2c3922d3122 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -6817,9 +6817,7 @@ void SelectionDAGBuilder::visitIntrinsicCall(const CallInst &I,
case Intrinsic::exp10:
case Intrinsic::floor:
case Intrinsic::ceil:
- case Intrinsic::trunc:
case Intrinsic::rint:
- case Intrinsic::nearbyint:
case Intrinsic::round:
case Intrinsic::roundeven:
case Intrinsic::canonicalize: {
@@ -6841,9 +6839,7 @@ void SelectionDAGBuilder::visitIntrinsicCall(const CallInst &I,
case Intrinsic::exp10: Opcode = ISD::FEXP10; break;
case Intrinsic::floor: Opcode = ISD::FFLOOR; break;
case Intrinsic::ceil: Opcode = ISD::FCEIL; break;
- case Intrinsic::trunc: Opcode = ISD::FTRUNC; break;
case Intrinsic::rint: Opcode = ISD::FRINT; break;
- case Intrinsic::nearbyint: Opcode = ISD::FNEARBYINT; break;
case Intrinsic::round: Opcode = ISD::FROUND; break;
case Intrinsic::roundeven: Opcode = ISD::FROUNDEVEN; break;
case Intrinsic::canonicalize: Opcode = ISD::FCANONICALIZE; break;
@@ -6978,6 +6974,11 @@ void SelectionDAGBuilder::visitIntrinsicCall(const CallInst &I,
#include "llvm/IR/VPIntrinsics.def"
visitVectorPredicationIntrinsic(cast<VPIntrinsic>(I));
return;
+#define FUNCTION(NAME, DAGN) \
+ case Intrinsic::NAME: \
+ visitFPOperation(I, ISD::DAGN); \
+ break;
+#include "llvm/IR/FloatingPointOps.def"
case Intrinsic::fptrunc_round: {
// Get the last argument, the metadata and convert it to an integer in the
// call
@@ -8359,6 +8360,33 @@ void SelectionDAGBuilder::pushFPOpOutChain(SDValue Result,
}
}
+void SelectionDAGBuilder::pushOutChain(SDValue Result,
+ fp::ExceptionBehavior EB) {
+ assert(Result.getNode()->getNumValues() == 2);
+
+ // Push node to the appropriate list so that future instructions can be
+ // chained up correctly.
+ SDValue OutChain = Result.getValue(1);
+ switch (EB) {
+ case fp::ExceptionBehavior::ebIgnore:
+ // The only reason why ebIgnore nodes still need to be chained is that
+ // they might depend on the current rounding mode, and therefore must
+ // not be moved across instruction that may change that mode.
+ [[fallthrough]];
+ case fp::ExceptionBehavior::ebMayTrap:
+ // These must not be moved across calls or instructions that may change
+ // floating-point exception masks.
+ PendingConstrainedFP.push_back(OutChain);
+ break;
+ case fp::ExceptionBehavior::ebStrict:
+ // These must not be moved across calls or instructions that may change
+ // floating-point exception masks or read floating-point exception flags.
+ // In addition, they cannot be optimized out even if unused.
+ PendingConstrainedFPStrict.push_back(OutChain);
+ break;
+ }
+}
+
void SelectionDAGBuilder::visitConstrainedFPIntrinsic(
const ConstrainedFPIntrinsic &FPI) {
SDLoc sdl = getCurSDLoc();
@@ -9425,6 +9453,56 @@ bool SelectionDAGBuilder::visitBinaryFloatCall(const CallInst &I,
return true;
}
+bool SelectionDAGBuilder::visitFPOperation(const CallInst &I, unsigned Opcode) {
+ // We already checked this call's prototype; verify it doesn't modify errno.
+ MemoryEffects ME = I.getMemoryEffects();
+ if (!ME.onlyAccessesInaccessibleMem())
+ return false;
+
+ SmallVector<SDValue, 4> Operands;
+ bool HasChain = ME.doesAccessInaccessibleMem();
+ if (HasChain)
+ Operands.push_back(getRoot());
+ for (auto &Arg : I.args())
+ Operands.push_back(getValue(Arg));
+
+ const TargetLowering &TLI = DAG.getTargetLoweringInfo();
+ EVT VT = TLI.getValueType(DAG.getDataLayout(), I.getType(), true);
+ SDVTList NodeVT;
+ if (HasChain)
+ NodeVT = DAG.getVTList(VT, MVT::Other);
+ else
+ NodeVT = DAG.getVTList(VT);
+
+ SDNodeFlags Flags;
+ fp::ExceptionBehavior EB = I.getExceptionBehavior();
+ if (EB == fp::ExceptionBehavior::ebIgnore)
+ Flags.setNoFPExcept(true);
+ if (auto *FPOp = dyn_cast<FPMathOperator>(&I))
+ Flags.copyFMF(*FPOp);
+
+ // Temporary solution: use STRICT_* nodes.
+ if (HasChain)
+ switch (Opcode) {
+ default:
+ break;
+#define LEGACY_DAG(NAME, DAGN) \
+ case ISD::DAGN: \
+ Opcode = ISD::STRICT_##DAGN; \
+ break;
+#include "llvm/IR/FloatingPointOps.def"
+ }
+
+ SDLoc sdl = getCurSDLoc();
+ SDValue Result = DAG.getNode(Opcode, sdl, NodeVT, Operands, Flags);
+ if (HasChain)
+ pushOutChain(Result, EB);
+
+ SDValue FPResult = Result.getValue(0);
+ setValue(&I, FPResult);
+ return true;
+}
+
void SelectionDAGBuilder::visitCall(const CallInst &I) {
// Handle inline assembly differently.
if (I.isInlineAsm()) {
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h
index 47e19f77a15e7..47b8fec2c23aa 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h
@@ -379,6 +379,8 @@ class SelectionDAGBuilder {
/// Evict any dangling debug information, attempting to salvage it first.
void resolveOrClearDbgInfo();
+ void pushOutChain(SDValue Result, fp::ExceptionBehavior EB);
+
SDValue getValue(const Value *V);
SDValue getNonRegisterValue(const Value *V);
@@ -628,6 +630,7 @@ class SelectionDAGBuilder {
bool visitStrNLenCall(const CallInst &I);
bool visitUnaryFloatCall(const CallInst &I, unsigned Opcode);
bool visitBinaryFloatCall(const CallInst &I, unsigned Opcode);
+ bool visitFPOperation(const CallInst &I, unsigned Opcode);
void visitAtomicLoad(const LoadInst &I);
void visitAtomicStore(const StoreInst &I);
void visitLoadFromSwiftError(const LoadInst &I);
diff --git a/llvm/lib/IR/AsmWriter.cpp b/llvm/lib/IR/AsmWriter.cpp
index 95d954f6b8174..3ed0018b41faa 100644
--- a/llvm/lib/IR/AsmWriter.cpp
+++ b/llvm/lib/IR/AsmWriter.cpp
@@ -4350,6 +4350,31 @@ void AssemblyWriter::printInfoComment(const Value &V) {
if (PrintInstAddrs)
Out << " ; " << &V;
+
+ if (auto *CI = dyn_cast<CallInst>(&V))
+ if (Intrinsic::ID IID = CI->getIntrinsicID())
+ if (IntrinsicInst::isFloatingPointOperation(IID))
+ if (const BasicBlock *BB = CI->getParent())
+ if (const Function *F = BB->getParent())
+ if (F->hasFnAttribute(Attribute::StrictFP)) {
+ MemoryEffects ME = CI->getMemoryEffects();
+ ModRefInfo MR = ME.getModRef(IRMemLocation::InaccessibleMem);
+ Out << " ; fpe=[";
+ switch (MR) {
+ case ModRefInfo::NoModRef:
+ break;
+ case ModRefInfo::Ref:
+ Out << "r";
+ break;
+ case ModRefInfo::Mod:
+ Out << "w";
+ break;
+ case ModRefInfo::ModRef:
+ Out << "rw";
+ break;
+ }
+ Out << "]";
+ }
}
static void maybePrintCallAddrSpace(const Value *Operand, const Instruction *I,
diff --git a/llvm/lib/IR/FPEnv.cpp b/llvm/lib/IR/FPEnv.cpp
index d48bebeb6d2b7..3a24c379faf65 100644
--- a/llvm/lib/IR/FPEnv.cpp
+++ b/llvm/lib/IR/FPEnv.cpp
@@ -36,12 +36,12 @@ std::optional<RoundingMode> convertStrToRoundingMode(StringRef RoundingArg) {
std::optional<RoundingMode> convertBundleToRoundingMode(StringRef RoundingArg) {
return StringSwitch<std::optional<RoundingMode>>(RoundingArg)
- .Case("dyn", RoundingMode::Dynamic)
- .Case("rte", RoundingMode::NearestTiesToEven)
- .Case("rmm", RoundingMode::NearestTiesToAway)
- .Case("rtn", RoundingMode::TowardNegative)
- .Case("rtp", RoundingMode::TowardPositive)
- .Case("rtz", RoundingMode::TowardZero)
+ .Case("dynamic", RoundingMode::Dynamic)
+ .Case("tonearest", RoundingMode::NearestTiesToEven)
+ .Case("tonearestaway", RoundingMode::NearestTiesToAway)
+ .Case("downward", RoundingMode::TowardNegative)
+ .Case("upward", RoundingMode::TowardPositive)
+ .Case("towardzero", RoundingMode::TowardZero)
.Default(std::nullopt);
}
@@ -76,22 +76,22 @@ std::optional<StringRef> convertRoundingModeToBundle(RoundingMode UseRounding) {
std::optional<StringRef> RoundingStr;
switch (UseRounding) {
case RoundingMode::Dynamic:
- RoundingStr = "dyn";
+ RoundingStr = "dynamic";
break;
case RoundingMode::NearestTiesToEven:
- RoundingStr = "rte";
+ RoundingStr = "tonearest";
break;
case RoundingMode::NearestTiesToAway:
- RoundingStr = "rmm";
+ RoundingStr = "tonearestaway";
break;
case RoundingMode::TowardNegative:
- RoundingStr = "rtn";
+ RoundingStr = "downward";
break;
case RoundingMode::TowardPositive:
- RoundingStr = "rtp";
+ RoundingStr = "upward";
break;
case RoundingMode::TowardZero:
- RoundingStr = "rtz";
+ RoundingStr = "towardzero";
break;
default:
break;
diff --git a/llvm/lib/IR/IRBuilder.cpp b/llvm/lib/IR/IRBuilder.cpp
index 09e71208b224e..105cf628a3e54 100644
--- a/llvm/lib/IR/IRBuilder.cpp
+++ b/llvm/lib/IR/IRBuilder.cpp
@@ -117,49 +117,54 @@ CallInst *IRBuilderBase::CreateCall(FunctionType *FTy, Value *Callee,
ArrayRef<Value *> Args,
ArrayRef<OperandBundleDef> OpBundles,
const Twine &Name, MDNode *FPMathTag) {
- assert(std::count_if(OpBundles.begin(), OpBundles.end(),
- [](const OperandBundleDef &Item) {
- return Item.getTag() == "fp.control";
- }) <= 1);
- assert(std::count_if(OpBundles.begin(), OpBundles.end(),
- [](const OperandBundleDef &Item) {
- return Item.getTag() == "fp.except";
- }) <= 1);
-
+ // Operand bundles can be extended by floating-point bundles. In this case
+ // we need a copy of OpBundles, because ArrayRef is immutable.
ArrayRef<OperandBundleDef> ActualBundlesRef = OpBundles;
SmallVector<OperandBundleDef, 2> ActualBundles;
- // If the builder is in strictfp mode and has non-default options (like
- // non-dynamic rounding), add corresponding operand bundle. If such bundle is
- // already present, assume it overwrites defaults.
bool NeedUpdateMemoryEffects = false;
- if (IsFPConstrained) {
- if (const auto *Func = dyn_cast<Function>(Callee)) {
- if (Intrinsic::ID ID = Func->getIntrinsicID()) {
- if (IntrinsicInst::isFloatingPointOperation(ID)) {
+ if (const auto *Func = dyn_cast<Function>(Callee))
+ if (Intrinsic::ID ID = Func->getIntrinsicID())
+ if (IntrinsicInst::isFloatingPointOperation(ID)) {
+ // If the builder has non-default floating-point options, add
+ // corresponding operand bundle unless a bundle with such tag is already
+ // present.
+ bool NeedRounding;
+ bool NeedExceptions;
+ if (IsFPConstrained) {
+ NeedRounding = DefaultConstrainedRounding != RoundingMode::Dynamic;
+ NeedExceptions = DefaultConstrainedExcept != fp::ebStrict;
+ } else {
+ NeedRounding =
+ DefaultConstrainedRounding != RoundingMode::NearestTiesToEven;
+ NeedExceptions = false;
+ assert(DefaultConstrainedExcept == fp::ebIgnore &&
+ "FP exception in default mode must be ignored");
+ }
+ if (NeedRounding || NeedExceptions) {
+ for (const auto &Bundle : OpBundles) {
+ if (NeedRounding && Bundle.getTag() == "fp.round")
+ NeedRounding = false;
+ if (NeedExceptions && Bundle.getTag() == "fp.except")
+ NeedExceptions = false;
+ }
+ if (NeedRounding || NeedExceptions) {
+ ActualBundles.append(OpBundles.begin(), OpBundles.end());
+ if (NeedRounding)
+ createRoundingBundle(ActualBundles, DefaultConstrainedRounding);
+ if (NeedExceptions)
+ createExceptionBundle(ActualBundles, DefaultConstrainedExcept);
+ ActualBundlesRef = ActualBundles;
+ }
+ }
+ if (IsFPConstrained) {
MemoryEffects FME = Func->getMemoryEffects();
NeedUpdateMemoryEffects = !FME.doesAccessInaccessibleMem();
- bool NeedRound = DefaultConstrainedRounding != RoundingMode::Dynamic;
- bool NeedExcept = DefaultConstrainedExcept != fp::ebStrict;
- for (const auto &Item : OpBundles) {
- if (NeedRound && Item.getTag() == "fp.control")
- NeedRound = false;
- else if (NeedExcept && Item.getTag() == "fp.except")
- NeedExcept = false;
- ActualBundles.push_back(Item);
- }
- if (NeedRound)
- createFPRoundingBundle(ActualBundles, DefaultConstrainedRounding);
- if (NeedExcept)
- createFPExceptionBundle(ActualBundles, DefaultConstrainedExcept);
- ActualBundlesRef = ActualBundles;
}
}
- }
- }
// If the call accesses FPE, update memory effects accordingly.
- CallInst *CI = CallInst::Create(FTy, Callee, Args, OpBundles);
+ CallInst *CI = CallInst::Create(FTy, Callee, Args, ActualBundlesRef);
if (NeedUpdateMemoryEffects) {
MemoryEffects ME = MemoryEffects::inaccessibleMemOnly();
auto A = Attribute::getWithMemoryEffects(getContext(), ME);
@@ -1352,18 +1357,14 @@ CallInst *IRBuilderBase::CreateDereferenceableAssumption(Value *PtrValue,
{DereferenceableOpB});
}
-void IRBuilderBase::createFPRoundingBundle(
- SmallVectorImpl<OperandBundleDef> &Bundles,
- std::optional<RoundingMode> Rounding) {
- addFPRoundingBundle(Context, Bundles,
- Rounding.value_or(DefaultConstrainedRounding));
+void IRBuilderBase::createRoundingBundle(
+ SmallVectorImpl<OperandBundleDef> &Bundles, RoundingMode RM) {
+ addRoundingBundle(Context, Bundles, RM);
}
-void IRBuilderBase::createFPExceptionBundle(
- SmallVectorImpl<OperandBundleDef> &Bundles,
- std::optional<fp::ExceptionBehavior> Except) {
- addFPExceptionBundle(Context, Bundles,
- Except.value_or(DefaultConstrainedExcept));
+void IRBuilderBase::createExceptionBundle(
+ SmallVectorImpl<OperandBundleDef> &Bundles, fp::ExceptionBehavior Except) {
+ addExceptionBundle(Context, Bundles, Except);
}
IRBuilderDefaultInserter::~IRBuilderDefaultInserter() = default;
diff --git a/llvm/lib/IR/Instructions.cpp b/llvm/lib/IR/Instructions.cpp
index 25c78e4c1a99e..034c0bc714d65 100644
--- a/llvm/lib/IR/Instructions.cpp
+++ b/llvm/lib/IR/Instructions.cpp
@@ -620,7 +620,7 @@ bool CallBase::hasReadingOperandBundles() const {
// ptrauth) forces a callsite to be at least readonly.
return hasOperandBundlesOtherThan(
{LLVMContext::OB_ptrauth, LLVMContext::OB_kcfi,
- LLVMContext::OB_fp_control, LLVMContext::OB_fp_except,
+ LLVMContext::OB_fp_round, LLVMContext::OB_fp_except,
LLVMContext::OB_convergencectrl}) &&
getIntrinsicID() != Intrinsic::assume;
}
@@ -629,25 +629,23 @@ bool CallBase::hasClobberingOperandBundles() const {
return hasOperandBundlesOtherThan(
{LLVMContext::OB_deopt, LLVMContext::OB_funclet,
LLVMContext::OB_ptrauth, LLVMContext::OB_kcfi,
- LLVMContext::OB_fp_control, LLVMContext::OB_fp_except,
+ LLVMContext::OB_fp_round, LLVMContext::OB_fp_except,
LLVMContext::OB_convergencectrl}) &&
getIntrinsicID() != Intrinsic::assume;
}
RoundingMode CallBase::getRoundingMode() const {
- // Try reading rounding mode from FP bundle.
- std::optional<RoundingMode> RM;
- if (auto RoundingBundle = getOperandBundle(LLVMContext::OB_fp_control)) {
+ // Try reading rounding mode from operand bundle.
+ if (auto RoundingBundle = getOperandBundle(LLVMContext::OB_fp_round)) {
Value *V = RoundingBundle->Inputs.front();
Metadata *MD = cast<MetadataAsValue>(V)->getMetadata();
- RM = convertBundleToRoundingMode(cast<MDString>(MD)->getString());
+ if (auto RM = convertBundleToRoundingMode(cast<MDString>(MD)->getString()))
+ return *RM;
}
- if (RM)
- return *RM;
- // No FP bundle, try to guess from the current mode.
- if (getParent())
- if (auto *F = getFunction(); F)
+ // No FP bundle, try to guess from attributes of the current function.
+ if (const BasicBlock *BB = getParent())
+ if (const Function *F = BB->getParent())
return F->getAttributes().hasFnAttr(Attribute::StrictFP)
? RoundingMode::Dynamic
: RoundingMode::NearestTiesToEven;
@@ -658,18 +656,17 @@ RoundingMode CallBase::getRoundingMode() const {
fp::ExceptionBehavior CallBase::getExceptionBehavior() const {
// Try determining exception behavior from FP bundle.
- std::optional<fp::ExceptionBehavior> EB;
if (auto ExceptionBundle = getOperandBundle(LLVMContext::OB_fp_except)) {
Value *V = ExceptionBundle->Inputs.front();
Metadata *MD = cast<MetadataAsValue>(V)->getMetadata();
- EB = convertBundleToExceptionBehavior(cast<MDString>(MD)->getString());
+ if (auto EB =
+ convertBundleToExceptionBehavior(cast<MDString>(MD)->getString()))
+ return *EB;
}
- if (EB)
- return *EB;
- // No FP bundle, try to guess from the current mode.
- if (getParent())
- if (auto *F = getFunction(); F)
+ // No FP bundle, try to guess from attributes of the current function.
+ if (const BasicBlock *BB = getParent())
+ if (const Function *F = BB->getParent())
return F->getAttributes().hasFnAttr(Attribute::StrictFP) ? fp::ebStrict
: fp::ebIgnore;
@@ -681,10 +678,12 @@ MemoryEffects CallBase::getFloatingPointMemoryEffects() const {
if (Intrinsic::ID IntrID = getIntrinsicID())
if (const BasicBlock *BB = getParent())
if (const Function *F = BB->getParent())
- if (F->hasFnAttribute(Attribute::StrictFP))
- if (IntrinsicInst::isFloatingPointOperation(IntrID)) {
- return MemoryEffects::inaccessibleMemOnly();
- }
+ if (F->hasFnAttribute(Attribute::StrictFP) &&
+ IntrinsicInst::isFloatingPointOperation(IntrID)) {
+ // Floating-point operations in strictfp function always have side
+ // effect at least because they can raise exceptions.
+ return MemoryEffects::inaccessibleMemOnly();
+ }
return MemoryEffects::none();
}
@@ -801,19 +800,29 @@ bool CallBase::hasArgumentWithAdditionalReturnCaptureComponents() const {
return false;
}
-void llvm::addFPRoundingBundle(LLVMContext &Ctx,
- SmallVectorImpl<OperandBundleDef> &Bundles,
- RoundingMode Rounding) {
+void llvm::addRoundingBundle(LLVMContext &Ctx,
+ SmallVectorImpl<OperandBundleDef> &Bundles,
+ RoundingMode Rounding) {
+ assert(std::find_if(Bundles.begin(), Bundles.end(),
+ [](const OperandBundleDef &B) {
+ return B.getTag() == "fp.round";
+ }) == Bundles.end() &&
+ "Bundle 'fp.round' already exists");
std::optional<StringRef> RndStr = convertRoundingModeToBundle(Rounding);
assert(RndStr && "Garbage rounding mode!");
auto *RoundingMDS = MDString::get(Ctx, *RndStr);
auto *RM = MetadataAsValue::get(Ctx, RoundingMDS);
- Bundles.emplace_back("fp.control", RM);
+ Bundles.emplace_back("fp.round", RM);
}
-void llvm::addFPExceptionBundle(LLVMContext &Ctx,
- SmallVectorImpl<OperandBundleDef> &Bundles,
- fp::ExceptionBehavior Except) {
+void llvm::addExceptionBundle(LLVMContext &Ctx,
+ SmallVectorImpl<OperandBundleDef> &Bundles,
+ fp::ExceptionBehavior Except) {
+ assert(std::find_if(Bundles.begin(), Bundles.end(),
+ [](const OperandBundleDef &B) {
+ return B.getTag() == "fp.except";
+ }) == Bundles.end() &&
+ "Bundle 'fp.except' already exists");
std::optional<StringRef> ExcStr = convertExceptionBehaviorToBundle(Except);
assert(ExcStr && "Garbage exception behavior!");
auto *ExceptMDS = MDString::get(Ctx, *ExcStr);
diff --git a/llvm/lib/IR/LLVMContext.cpp b/llvm/lib/IR/LLVMContext.cpp
index 7bde465ef390f..6f7e97b2eff86 100644
--- a/llvm/lib/IR/LLVMContext.cpp
+++ b/llvm/lib/IR/LLVMContext.cpp
@@ -55,8 +55,8 @@ static StringRef knownBundleName(unsigned BundleTagID) {
return "convergencectrl";
case LLVMContext::OB_align:
return "align";
- case LLVMContext::OB_fp_control:
- return "fp.control";
+ case LLVMContext::OB_fp_round:
+ return "fp.round";
case LLVMContext::OB_fp_except:
return "fp.except";
default:
@@ -88,9 +88,9 @@ LLVMContext::LLVMContext() : pImpl(new LLVMContextImpl(*this)) {
assert(Entry->second == BundleTagID && "operand bundle id drifted!");
}
- auto *RoundingEntry = pImpl->getOrInsertBundleTag("fp.control");
- assert(RoundingEntry->second == LLVMContext::OB_fp_control &&
- "fp.control operand bundle id drifted!");
+ auto *RoundingEntry = pImpl->getOrInsertBundleTag("fp.round");
+ assert(RoundingEntry->second == LLVMContext::OB_fp_round &&
+ "fp.round operand bundle id drifted!");
(void)RoundingEntry;
auto *ExceptionEntry = pImpl->getOrInsertBundleTag("fp.except");
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index c7e350651245d..129295bf6dc48 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -3909,7 +3909,7 @@ void Verifier::visitCallBase(CallBase &Call) {
FoundGCTransitionBundle = false, FoundCFGuardTargetBundle = false,
FoundPreallocatedBundle = false, FoundGCLiveBundle = false,
FoundPtrauthBundle = false, FoundKCFIBundle = false,
- FoundAttachedCallBundle = false, FoundFpeControlBundle = false,
+ FoundAttachedCallBundle = false, FoundFpeRoundBundle = false,
FoundFpeExceptBundle = false;
for (unsigned i = 0, e = Call.getNumOperandBundles(); i < e; ++i) {
OperandBundleUse BU = Call.getOperandBundleAt(i);
@@ -3973,30 +3973,29 @@ void Verifier::visitCallBase(CallBase &Call) {
"Multiple \"clang.arc.attachedcall\" operand bundles", Call);
FoundAttachedCallBundle = true;
verifyAttachedCallBundle(Call, BU);
- } else if (Tag == LLVMContext::OB_fp_control) {
- Check(!FoundFpeControlBundle, "Multiple \"fp.control\" operand bundles",
+ } else if (Tag == LLVMContext::OB_fp_round) {
+ Check(!FoundFpeRoundBundle, "Multiple \"fp.round\" operand bundles",
Call);
bool FoundRoundingMode = false;
for (auto &U : BU.Inputs) {
Value *V = U.get();
Check(isa<MetadataAsValue>(V),
- "Value of a \"fp.control\" bundle operand must be a metadata",
+ "Value of a \"fp.round\" bundle operand must be a metadata",
Call);
Metadata *MD = cast<MetadataAsValue>(V)->getMetadata();
Check(isa<MDString>(MD),
- "Value of a \"fp.control\" bundle operand must be a string",
- Call);
+ "Value of a \"fp.round\" bundle operand must be a string", Call);
StringRef Item = cast<MDString>(MD)->getString();
if (convertBundleToRoundingMode(Item)) {
Check(!FoundRoundingMode, "Rounding mode is specified more that once",
Call);
FoundRoundingMode = true;
} else {
- CheckFailed("Unrecognized value in \"fp.control\" bundle operand",
+ CheckFailed("Unrecognized value in \"fp.round\" bundle operand",
Call);
}
}
- FoundFpeControlBundle = true;
+ FoundFpeRoundBundle = true;
} else if (Tag == LLVMContext::OB_fp_except) {
Check(!FoundFpeExceptBundle, "Multiple \"fp.except\" operand bundles",
Call);
@@ -4014,6 +4013,15 @@ void Verifier::visitCallBase(CallBase &Call) {
"Value of a \"fp.except\" bundle operand is not a correct exception "
"behavior",
Call);
+ if (EB && *EB != fp::ebIgnore)
+ if (const BasicBlock *BB = Call.getParent())
+ if (const Function *F = BB->getParent()) {
+ bool StrictFP = F->hasFnAttribute(Attribute::StrictFP);
+ Check(StrictFP,
+ "Value of a \"fp.except\" bundle operand in "
+ "default mode must be \"ignore\"",
+ Call);
+ }
FoundFpeExceptBundle = true;
}
}
diff --git a/llvm/test/CodeGen/X86/fp-strict-scalar-round.ll b/llvm/test/CodeGen/X86/fp-strict-scalar-round.ll
index 13f890ae6e191..35c066afb9c13 100644
--- a/llvm/test/CodeGen/X86/fp-strict-scalar-round.ll
+++ b/llvm/test/CodeGen/X86/fp-strict-scalar-round.ll
@@ -250,6 +250,46 @@ define float @ftrunc32(float %f) #0 {
ret float %res
}
+define float @ftrunc32ob(float %f) #0 {
+; SSE41-X86-LABEL: ftrunc32ob:
+; SSE41-X86: # %bb.0:
+; SSE41-X86-NEXT: pushl %eax
+; SSE41-X86-NEXT: .cfi_def_cfa_offset 8
+; SSE41-X86-NEXT: movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; SSE41-X86-NEXT: roundss $11, %xmm0, %xmm0
+; SSE41-X86-NEXT: movss %xmm0, (%esp)
+; SSE41-X86-NEXT: flds (%esp)
+; SSE41-X86-NEXT: wait
+; SSE41-X86-NEXT: popl %eax
+; SSE41-X86-NEXT: .cfi_def_cfa_offset 4
+; SSE41-X86-NEXT: retl
+;
+; SSE41-X64-LABEL: ftrunc32ob:
+; SSE41-X64: # %bb.0:
+; SSE41-X64-NEXT: roundss $11, %xmm0, %xmm0
+; SSE41-X64-NEXT: retq
+;
+; AVX-X86-LABEL: ftrunc32ob:
+; AVX-X86: # %bb.0:
+; AVX-X86-NEXT: pushl %eax
+; AVX-X86-NEXT: .cfi_def_cfa_offset 8
+; AVX-X86-NEXT: vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX-X86-NEXT: vroundss $11, %xmm0, %xmm0, %xmm0
+; AVX-X86-NEXT: vmovss %xmm0, (%esp)
+; AVX-X86-NEXT: flds (%esp)
+; AVX-X86-NEXT: wait
+; AVX-X86-NEXT: popl %eax
+; AVX-X86-NEXT: .cfi_def_cfa_offset 4
+; AVX-X86-NEXT: retl
+;
+; AVX-X64-LABEL: ftrunc32ob:
+; AVX-X64: # %bb.0:
+; AVX-X64-NEXT: vroundss $11, %xmm0, %xmm0, %xmm0
+; AVX-X64-NEXT: retq
+ %res = call float @llvm.trunc.f32(float %f) [ "fp.except"(metadata !"strict") ]
+ ret float %res
+}
+
define double @ftruncf64(double %f) #0 {
; SSE41-X86-LABEL: ftruncf64:
; SSE41-X86: # %bb.0:
@@ -303,6 +343,58 @@ define double @ftruncf64(double %f) #0 {
ret double %res
}
+define double @ftruncf64ob(double %f) #0 {
+; SSE41-X86-LABEL: ftruncf64ob:
+; SSE41-X86: # %bb.0:
+; SSE41-X86-NEXT: pushl %ebp
+; SSE41-X86-NEXT: .cfi_def_cfa_offset 8
+; SSE41-X86-NEXT: .cfi_offset %ebp, -8
+; SSE41-X86-NEXT: movl %esp, %ebp
+; SSE41-X86-NEXT: .cfi_def_cfa_register %ebp
+; SSE41-X86-NEXT: andl $-8, %esp
+; SSE41-X86-NEXT: subl $8, %esp
+; SSE41-X86-NEXT: movsd {{.*#+}} xmm0 = mem[0],zero
+; SSE41-X86-NEXT: roundsd $11, %xmm0, %xmm0
+; SSE41-X86-NEXT: movsd %xmm0, (%esp)
+; SSE41-X86-NEXT: fldl (%esp)
+; SSE41-X86-NEXT: wait
+; SSE41-X86-NEXT: movl %ebp, %esp
+; SSE41-X86-NEXT: popl %ebp
+; SSE41-X86-NEXT: .cfi_def_cfa %esp, 4
+; SSE41-X86-NEXT: retl
+;
+; SSE41-X64-LABEL: ftruncf64ob:
+; SSE41-X64: # %bb.0:
+; SSE41-X64-NEXT: roundsd $11, %xmm0, %xmm0
+; SSE41-X64-NEXT: retq
+;
+; AVX-X86-LABEL: ftruncf64ob:
+; AVX-X86: # %bb.0:
+; AVX-X86-NEXT: pushl %ebp
+; AVX-X86-NEXT: .cfi_def_cfa_offset 8
+; AVX-X86-NEXT: .cfi_offset %ebp, -8
+; AVX-X86-NEXT: movl %esp, %ebp
+; AVX-X86-NEXT: .cfi_def_cfa_register %ebp
+; AVX-X86-NEXT: andl $-8, %esp
+; AVX-X86-NEXT: subl $8, %esp
+; AVX-X86-NEXT: vmovsd {{.*#+}} xmm0 = mem[0],zero
+; AVX-X86-NEXT: vroundsd $11, %xmm0, %xmm0, %xmm0
+; AVX-X86-NEXT: vmovsd %xmm0, (%esp)
+; AVX-X86-NEXT: fldl (%esp)
+; AVX-X86-NEXT: wait
+; AVX-X86-NEXT: movl %ebp, %esp
+; AVX-X86-NEXT: popl %ebp
+; AVX-X86-NEXT: .cfi_def_cfa %esp, 4
+; AVX-X86-NEXT: retl
+;
+; AVX-X64-LABEL: ftruncf64ob:
+; AVX-X64: # %bb.0:
+; AVX-X64-NEXT: vroundsd $11, %xmm0, %xmm0, %xmm0
+; AVX-X64-NEXT: retq
+ %res = call double @llvm.trunc.f64(double %f) [ "fp.except"(metadata !"strict") ]
+ ret double %res
+}
+
define float @frint32(float %f) #0 {
; SSE41-X86-LABEL: frint32:
; SSE41-X86: # %bb.0:
@@ -441,6 +533,46 @@ define float @fnearbyint32(float %f) #0 {
ret float %res
}
+define float @fnearbyint32ob(float %f) #0 {
+; SSE41-X86-LABEL: fnearbyint32ob:
+; SSE41-X86: # %bb.0:
+; SSE41-X86-NEXT: pushl %eax
+; SSE41-X86-NEXT: .cfi_def_cfa_offset 8
+; SSE41-X86-NEXT: movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; SSE41-X86-NEXT: roundss $12, %xmm0, %xmm0
+; SSE41-X86-NEXT: movss %xmm0, (%esp)
+; SSE41-X86-NEXT: flds (%esp)
+; SSE41-X86-NEXT: wait
+; SSE41-X86-NEXT: popl %eax
+; SSE41-X86-NEXT: .cfi_def_cfa_offset 4
+; SSE41-X86-NEXT: retl
+;
+; SSE41-X64-LABEL: fnearbyint32ob:
+; SSE41-X64: # %bb.0:
+; SSE41-X64-NEXT: roundss $12, %xmm0, %xmm0
+; SSE41-X64-NEXT: retq
+;
+; AVX-X86-LABEL: fnearbyint32ob:
+; AVX-X86: # %bb.0:
+; AVX-X86-NEXT: pushl %eax
+; AVX-X86-NEXT: .cfi_def_cfa_offset 8
+; AVX-X86-NEXT: vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX-X86-NEXT: vroundss $12, %xmm0, %xmm0, %xmm0
+; AVX-X86-NEXT: vmovss %xmm0, (%esp)
+; AVX-X86-NEXT: flds (%esp)
+; AVX-X86-NEXT: wait
+; AVX-X86-NEXT: popl %eax
+; AVX-X86-NEXT: .cfi_def_cfa_offset 4
+; AVX-X86-NEXT: retl
+;
+; AVX-X64-LABEL: fnearbyint32ob:
+; AVX-X64: # %bb.0:
+; AVX-X64-NEXT: vroundss $12, %xmm0, %xmm0, %xmm0
+; AVX-X64-NEXT: retq
+ %res = call float @llvm.nearbyint.f32(float %f) [ "fp.round"(metadata !"dynamic"), "fp.except"(metadata !"strict") ]
+ ret float %res
+}
+
define double @fnearbyintf64(double %f) #0 {
; SSE41-X86-LABEL: fnearbyintf64:
; SSE41-X86: # %bb.0:
@@ -495,6 +627,58 @@ define double @fnearbyintf64(double %f) #0 {
ret double %res
}
+define double @fnearbyint64ob(double %f) #0 {
+; SSE41-X86-LABEL: fnearbyint64ob:
+; SSE41-X86: # %bb.0:
+; SSE41-X86-NEXT: pushl %ebp
+; SSE41-X86-NEXT: .cfi_def_cfa_offset 8
+; SSE41-X86-NEXT: .cfi_offset %ebp, -8
+; SSE41-X86-NEXT: movl %esp, %ebp
+; SSE41-X86-NEXT: .cfi_def_cfa_register %ebp
+; SSE41-X86-NEXT: andl $-8, %esp
+; SSE41-X86-NEXT: subl $8, %esp
+; SSE41-X86-NEXT: movsd {{.*#+}} xmm0 = mem[0],zero
+; SSE41-X86-NEXT: roundsd $12, %xmm0, %xmm0
+; SSE41-X86-NEXT: movsd %xmm0, (%esp)
+; SSE41-X86-NEXT: fldl (%esp)
+; SSE41-X86-NEXT: wait
+; SSE41-X86-NEXT: movl %ebp, %esp
+; SSE41-X86-NEXT: popl %ebp
+; SSE41-X86-NEXT: .cfi_def_cfa %esp, 4
+; SSE41-X86-NEXT: retl
+;
+; SSE41-X64-LABEL: fnearbyint64ob:
+; SSE41-X64: # %bb.0:
+; SSE41-X64-NEXT: roundsd $12, %xmm0, %xmm0
+; SSE41-X64-NEXT: retq
+;
+; AVX-X86-LABEL: fnearbyint64ob:
+; AVX-X86: # %bb.0:
+; AVX-X86-NEXT: pushl %ebp
+; AVX-X86-NEXT: .cfi_def_cfa_offset 8
+; AVX-X86-NEXT: .cfi_offset %ebp, -8
+; AVX-X86-NEXT: movl %esp, %ebp
+; AVX-X86-NEXT: .cfi_def_cfa_register %ebp
+; AVX-X86-NEXT: andl $-8, %esp
+; AVX-X86-NEXT: subl $8, %esp
+; AVX-X86-NEXT: vmovsd {{.*#+}} xmm0 = mem[0],zero
+; AVX-X86-NEXT: vroundsd $12, %xmm0, %xmm0, %xmm0
+; AVX-X86-NEXT: vmovsd %xmm0, (%esp)
+; AVX-X86-NEXT: fldl (%esp)
+; AVX-X86-NEXT: wait
+; AVX-X86-NEXT: movl %ebp, %esp
+; AVX-X86-NEXT: popl %ebp
+; AVX-X86-NEXT: .cfi_def_cfa %esp, 4
+; AVX-X86-NEXT: retl
+;
+; AVX-X64-LABEL: fnearbyint64ob:
+; AVX-X64: # %bb.0:
+; AVX-X64-NEXT: vroundsd $12, %xmm0, %xmm0, %xmm0
+; AVX-X64-NEXT: retq
+ %res = call double @llvm.nearbyint.f64(double %f) [ "fp.round"(metadata !"dynamic"), "fp.except"(metadata !"strict") ]
+ ret double %res
+}
+
define float @fround32(float %f) #0 {
; SSE41-X86-LABEL: fround32:
; SSE41-X86: # %bb.0:
diff --git a/llvm/test/CodeGen/X86/vec-strict-512.ll b/llvm/test/CodeGen/X86/vec-strict-512.ll
index 2cafd74af4953..5bd78d99c078b 100644
--- a/llvm/test/CodeGen/X86/vec-strict-512.ll
+++ b/llvm/test/CodeGen/X86/vec-strict-512.ll
@@ -231,6 +231,15 @@ define <16 x float> @strict_vector_ftrunc_v16f32(<16 x float> %f) #0 {
ret <16 x float> %res
}
+define <16 x float> @strict_vector_ftrunc_v16f32ob(<16 x float> %f) #0 {
+; CHECK-LABEL: strict_vector_ftrunc_v16f32ob:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vrndscaleps $11, %zmm0, %zmm0
+; CHECK-NEXT: ret{{[l|q]}}
+ %res = call <16 x float> @llvm.trunc.v16f32(<16 x float> %f) [ "fp.except"(metadata !"strict") ]
+ ret <16 x float> %res
+}
+
define <8 x double> @strict_vector_ftrunc_v8f64(<8 x double> %f) #0 {
; CHECK-LABEL: strict_vector_ftrunc_v8f64:
; CHECK: # %bb.0:
@@ -240,6 +249,15 @@ define <8 x double> @strict_vector_ftrunc_v8f64(<8 x double> %f) #0 {
ret <8 x double> %res
}
+define <8 x double> @strict_vector_ftrunc_v8f64ob(<8 x double> %f) #0 {
+; CHECK-LABEL: strict_vector_ftrunc_v8f64ob:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vrndscalepd $11, %zmm0, %zmm0
+; CHECK-NEXT: ret{{[l|q]}}
+ %res = call <8 x double> @llvm.trunc.v8f64(<8 x double> %f) [ "fp.except"(metadata !"strict") ]
+ ret <8 x double> %res
+}
+
define <16 x float> @strict_vector_frint_v16f32(<16 x float> %f) #0 {
; CHECK-LABEL: strict_vector_frint_v16f32:
; CHECK: # %bb.0:
@@ -270,6 +288,15 @@ define <16 x float> @strict_vector_fnearbyint_v16f32(<16 x float> %f) #0 {
ret <16 x float> %res
}
+define <16 x float> @strict_vector_fnearbyint_v16f32ob(<16 x float> %f) #0 {
+; CHECK-LABEL: strict_vector_fnearbyint_v16f32ob:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vrndscaleps $12, %zmm0, %zmm0
+; CHECK-NEXT: ret{{[l|q]}}
+ %res = call <16 x float> @llvm.nearbyint.v16f32(<16 x float> %f) [ "fp.round"(metadata !"dynamic"), "fp.except"(metadata !"strict") ]
+ ret <16 x float> %res
+}
+
define <8 x double> @strict_vector_fnearbyint_v8f64(<8 x double> %f) #0 {
; CHECK-LABEL: strict_vector_fnearbyint_v8f64:
; CHECK: # %bb.0:
@@ -280,4 +307,13 @@ define <8 x double> @strict_vector_fnearbyint_v8f64(<8 x double> %f) #0 {
ret <8 x double> %res
}
+define <8 x double> @strict_vector_fnearbyint_v8f64ob(<8 x double> %f) #0 {
+; CHECK-LABEL: strict_vector_fnearbyint_v8f64ob:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vrndscalepd $12, %zmm0, %zmm0
+; CHECK-NEXT: ret{{[l|q]}}
+ %res = call <8 x double> @llvm.nearbyint.v8f64(<8 x double> %f) [ "fp.round"(metadata !"dynamic"), "fp.except"(metadata !"strict") ]
+ ret <8 x double> %res
+}
+
attributes #0 = { strictfp }
diff --git a/llvm/test/Verifier/fp-intrinsics.ll b/llvm/test/Verifier/fp-intrinsics.ll
index 1e1df1b0da96a..f96d519870aaa 100644
--- a/llvm/test/Verifier/fp-intrinsics.ll
+++ b/llvm/test/Verifier/fp-intrinsics.ll
@@ -51,48 +51,48 @@ entry:
ret double %fadd
}
-; Test multiple fp.control bundles.
-; CHECK-NEXT: Multiple "fp.control" operand bundles
-; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.control"(metadata !"rtz"), "fp.control"(metadata !"rtz") ]
+; Test multiple fp.round bundles.
+; CHECK-NEXT: Multiple "fp.round" operand bundles
+; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.round"(metadata !"towardzero"), "fp.round"(metadata !"towardzero") ]
define double @f6(double %a) #0 {
entry:
- %ftrunc = call double @llvm.trunc.f64(double %a) #0 [ "fp.control"(metadata !"rtz"), "fp.control"(metadata !"rtz") ]
+ %ftrunc = call double @llvm.trunc.f64(double %a) #0 [ "fp.round"(metadata !"towardzero"), "fp.round"(metadata !"towardzero") ]
ret double %ftrunc
}
-; Test fp.control bundle that has more than one rounding mode specification.
+; Test fp.round bundle that has more than one rounding mode specification.
; CHECK-NEXT: Rounding mode is specified more that once
-; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.control"(metadata !"rtz", metadata !"rte") ]
+; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.round"(metadata !"towardzero", metadata !"tonearest") ]
define double @f7(double %a) #0 {
entry:
- %ftrunc = call double @llvm.trunc.f64(double %a) #0 [ "fp.control"(metadata !"rtz", metadata !"rte") ]
+ %ftrunc = call double @llvm.trunc.f64(double %a) #0 [ "fp.round"(metadata !"towardzero", metadata !"tonearest") ]
ret double %ftrunc
}
-; Test fp.control bundle that has non-metadata operand.
-; CHECK-NEXT: Value of a "fp.control" bundle operand must be a metadata
-; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.control"(i32 0) ]
+; Test fp.round bundle that has non-metadata operand.
+; CHECK-NEXT: Value of a "fp.round" bundle operand must be a metadata
+; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.round"(i32 0) ]
define double @f8(double %a) #0 {
entry:
- %ftrunc = call double @llvm.trunc.f64(double %a) #0 [ "fp.control"(i32 0) ]
+ %ftrunc = call double @llvm.trunc.f64(double %a) #0 [ "fp.round"(i32 0) ]
ret double %ftrunc
}
-; Test fp.control bundle that has non-string operand.
-; CHECK-NEXT: Value of a "fp.control" bundle operand must be a string
-; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.control"(metadata i64 3) ]
+; Test fp.round bundle that has non-string operand.
+; CHECK-NEXT: Value of a "fp.round" bundle operand must be a string
+; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.round"(metadata i64 3) ]
define double @f9(double %a) #0 {
entry:
- %ftrunc = call double @llvm.trunc.f64(double %a) #0 [ "fp.control"(metadata !{i64 3}) ]
+ %ftrunc = call double @llvm.trunc.f64(double %a) #0 [ "fp.round"(metadata !{i64 3}) ]
ret double %ftrunc
}
-; Test fp.control bundle that specifies incorrect value.
-; CHECK-NEXT: Unrecognized value in "fp.control" bundle operand
-; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.control"(metadata !"qqq") ]
+; Test fp.round bundle that specifies incorrect value.
+; CHECK-NEXT: Unrecognized value in "fp.round" bundle operand
+; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.round"(metadata !"qqq") ]
define double @f10(double %a) #0 {
entry:
- %ftrunc = call double @llvm.trunc.f64(double %a) #0 [ "fp.control"(metadata !"qqq") ]
+ %ftrunc = call double @llvm.trunc.f64(double %a) #0 [ "fp.round"(metadata !"qqq") ]
ret double %ftrunc
}
@@ -141,5 +141,14 @@ entry:
ret double %ftrunc
}
+; Test fp.except bundle in default mode cannot be other than "ignore".
+; CHECK-NEXT: Value of a "fp.except" bundle operand in default mode must be "ignore"
+; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) [ "fp.except"(metadata !"strict") ]
+define double @f16(double %a) {
+entry:
+ %ftrunc = call double @llvm.trunc.f64(double %a) [ "fp.except"(metadata !"strict") ]
+ ret double %ftrunc
+}
+
attributes #0 = { strictfp }
diff --git a/llvm/unittests/IR/IRBuilderTest.cpp b/llvm/unittests/IR/IRBuilderTest.cpp
index f881f85fce6ae..a1da8b1b5f3f9 100644
--- a/llvm/unittests/IR/IRBuilderTest.cpp
+++ b/llvm/unittests/IR/IRBuilderTest.cpp
@@ -272,7 +272,9 @@ TEST_F(IRBuilderTest, ConstrainedFP) {
// See if we get constrained intrinsics instead of non-constrained
// instructions.
- Builder.setIsFPConstrained(true);
+ Builder.setDefaultConstrainedRounding(RoundingMode::Dynamic);
+ Builder.setDefaultConstrainedExcept(fp::ebStrict);
+ Builder.setIsFPConstrained(true, false);
auto Parent = BB->getParent();
Parent->addFnAttr(Attribute::StrictFP);
@@ -470,7 +472,7 @@ TEST_F(IRBuilderTest, FPBundlesDefault) {
{
Value *V = Builder.CreateCall(Fn, {FnArg});
auto *I = cast<IntrinsicInst>(V);
- EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value());
+ EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_round).has_value());
EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value());
EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID());
EXPECT_EQ(RoundingMode::NearestTiesToEven, I->getRoundingMode());
@@ -480,13 +482,13 @@ TEST_F(IRBuilderTest, FPBundlesDefault) {
}
// Check call with FP bundles, rounding is set to default value.
- // nearbyint(%x) [ "fp.control" (metadata !"rte") ]
+ // nearbyint(%x) [ "fp.round" (metadata !"tonearest") ]
{
SmallVector<OperandBundleDef, 1> Bundles;
- llvm::addFPRoundingBundle(Ctx, Bundles, RoundingMode::NearestTiesToEven);
+ llvm::addRoundingBundle(Ctx, Bundles, RoundingMode::NearestTiesToEven);
Value *V = Builder.CreateCall(Fn, {FnArg}, Bundles);
auto *I = cast<IntrinsicInst>(V);
- EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value());
+ EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_round).has_value());
EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value());
EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID());
EXPECT_EQ(RoundingMode::NearestTiesToEven, I->getRoundingMode());
@@ -499,10 +501,10 @@ TEST_F(IRBuilderTest, FPBundlesDefault) {
// nearbyint(%x) [ "fp.except" (metadata !"ignore") ]
{
SmallVector<OperandBundleDef, 1> Bundles;
- llvm::addFPExceptionBundle(Ctx, Bundles, fp::ebIgnore);
+ llvm::addExceptionBundle(Ctx, Bundles, fp::ebIgnore);
Value *V = Builder.CreateCall(Fn, {FnArg}, Bundles);
auto *I = cast<IntrinsicInst>(V);
- EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value());
+ EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_round).has_value());
EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value());
EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID());
EXPECT_EQ(RoundingMode::NearestTiesToEven, I->getRoundingMode());
@@ -513,14 +515,15 @@ TEST_F(IRBuilderTest, FPBundlesDefault) {
// Check call with FP bundles, both rounding mode and exception behavior are
// set.
- // nearbyint(%x) [ "fp.except" (metadata !"ignore") ]
+ // nearbyint(%x) [ "fp.round" (metadata !"tonearest"),
+ // "fp.except" (metadata !"ignore") ]
{
SmallVector<OperandBundleDef, 1> Bundles;
- llvm::addFPRoundingBundle(Ctx, Bundles, RoundingMode::NearestTiesToEven);
- llvm::addFPExceptionBundle(Ctx, Bundles, fp::ebIgnore);
+ llvm::addRoundingBundle(Ctx, Bundles, RoundingMode::NearestTiesToEven);
+ llvm::addExceptionBundle(Ctx, Bundles, fp::ebIgnore);
Value *V = Builder.CreateCall(Fn, {FnArg}, Bundles);
auto *I = cast<IntrinsicInst>(V);
- EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value());
+ EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_round).has_value());
EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value());
EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID());
EXPECT_EQ(RoundingMode::NearestTiesToEven, I->getRoundingMode());
@@ -528,6 +531,40 @@ TEST_F(IRBuilderTest, FPBundlesDefault) {
MemoryEffects ME = I->getMemoryEffects();
EXPECT_TRUE(ME.doesNotAccessMemory());
}
+
+ // If the builder object specifies a rounding mode, the resulting call will
+ // include the corresponding "fp.round" operand bundle.
+ {
+ Builder.setDefaultConstrainedRounding(RoundingMode::TowardNegative);
+ SmallVector<OperandBundleDef, 1> Bundles;
+ Value *V = Builder.CreateCall(Fn, {FnArg}, Bundles);
+ auto *I = cast<IntrinsicInst>(V);
+ EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_round).has_value());
+ EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value());
+ EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID());
+ EXPECT_EQ(RoundingMode::TowardNegative, I->getRoundingMode());
+ EXPECT_EQ(fp::ebIgnore, I->getExceptionBehavior());
+ MemoryEffects ME = I->getMemoryEffects();
+ EXPECT_TRUE(ME.doesNotAccessMemory());
+ }
+
+ // If the builder object specifies a rounding mode but the operand bundles
+ // already contain an "fp.round" bundle, the builder's specified mode
+ // is ignored.
+ {
+ Builder.setDefaultConstrainedRounding(RoundingMode::TowardNegative);
+ SmallVector<OperandBundleDef, 1> Bundles;
+ llvm::addRoundingBundle(Ctx, Bundles, RoundingMode::TowardZero);
+ Value *V = Builder.CreateCall(Fn, {FnArg}, Bundles);
+ auto *I = cast<IntrinsicInst>(V);
+ EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_round).has_value());
+ EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value());
+ EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID());
+ EXPECT_EQ(RoundingMode::TowardZero, I->getRoundingMode());
+ EXPECT_EQ(fp::ebIgnore, I->getExceptionBehavior());
+ MemoryEffects ME = I->getMemoryEffects();
+ EXPECT_TRUE(ME.doesNotAccessMemory());
+ }
}
TEST_F(IRBuilderTest, FPBundlesStrict) {
@@ -549,7 +586,7 @@ TEST_F(IRBuilderTest, FPBundlesStrict) {
{
Value *V = Builder.CreateCall(Fn, {FnArg});
auto *I = cast<IntrinsicInst>(V);
- EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value());
+ EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_round).has_value());
EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value());
EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID());
EXPECT_EQ(RoundingMode::Dynamic, I->getRoundingMode());
@@ -559,13 +596,13 @@ TEST_F(IRBuilderTest, FPBundlesStrict) {
}
// Check call with FP bundles, with default (dynamic) rounding mode
- // nearbyint(%x) [ "fp.control" (metadata !"dyn") ]
+ // nearbyint(%x) [ "fp.round" (metadata !"dynamic") ]
{
SmallVector<OperandBundleDef, 1> Bundles;
- llvm::addFPRoundingBundle(Ctx, Bundles, RoundingMode::Dynamic);
+ llvm::addRoundingBundle(Ctx, Bundles, RoundingMode::Dynamic);
Value *V = Builder.CreateCall(Fn, {FnArg}, Bundles);
auto *I = cast<IntrinsicInst>(V);
- EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value());
+ EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_round).has_value());
EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value());
EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID());
EXPECT_EQ(RoundingMode::Dynamic, I->getRoundingMode());
@@ -575,13 +612,13 @@ TEST_F(IRBuilderTest, FPBundlesStrict) {
}
// Check call with FP bundles, with specific rounding mode
- // nearbyint(%x) [ "fp.control" (metadata !"rtz") ]
+ // nearbyint(%x) [ "fp.round" (metadata !"towardzero") ]
{
SmallVector<OperandBundleDef, 1> Bundles;
- llvm::addFPRoundingBundle(Ctx, Bundles, RoundingMode::TowardZero);
+ llvm::addRoundingBundle(Ctx, Bundles, RoundingMode::TowardZero);
Value *V = Builder.CreateCall(Fn, {FnArg}, Bundles);
auto *I = cast<IntrinsicInst>(V);
- EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value());
+ EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_round).has_value());
EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value());
EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID());
EXPECT_EQ(RoundingMode::TowardZero, I->getRoundingMode());
@@ -594,10 +631,10 @@ TEST_F(IRBuilderTest, FPBundlesStrict) {
// nearbyint(%x) [ "fp.except" (metadata !"strict") ]
{
SmallVector<OperandBundleDef, 1> Bundles;
- llvm::addFPExceptionBundle(Ctx, Bundles, fp::ebStrict);
+ llvm::addExceptionBundle(Ctx, Bundles, fp::ebStrict);
Value *V = Builder.CreateCall(Fn, {FnArg}, Bundles);
auto *I = cast<IntrinsicInst>(V);
- EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value());
+ EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_round).has_value());
EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value());
EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID());
EXPECT_EQ(RoundingMode::Dynamic, I->getRoundingMode());
@@ -610,10 +647,10 @@ TEST_F(IRBuilderTest, FPBundlesStrict) {
// nearbyint(%x) [ "fp.except" (metadata !"ignore") ]
{
SmallVector<OperandBundleDef, 1> Bundles;
- llvm::addFPExceptionBundle(Ctx, Bundles, fp::ebIgnore);
+ llvm::addExceptionBundle(Ctx, Bundles, fp::ebIgnore);
Value *V = Builder.CreateCall(Fn, {FnArg}, Bundles);
auto *I = cast<IntrinsicInst>(V);
- EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value());
+ EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_round).has_value());
EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value());
EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID());
EXPECT_EQ(RoundingMode::Dynamic, I->getRoundingMode());
@@ -623,15 +660,15 @@ TEST_F(IRBuilderTest, FPBundlesStrict) {
}
// Check call with both FP bundles.
- // nearbyint(%x) [ "fp.control" (metadata !"rtz"),
+ // nearbyint(%x) [ "fp.round" (metadata !"towardzero"),
// "fp.except" (metadata !"ignore") ]
{
SmallVector<OperandBundleDef, 1> Bundles;
- llvm::addFPRoundingBundle(Ctx, Bundles, RoundingMode::NearestTiesToEven);
- llvm::addFPExceptionBundle(Ctx, Bundles, fp::ebIgnore);
+ llvm::addRoundingBundle(Ctx, Bundles, RoundingMode::NearestTiesToEven);
+ llvm::addExceptionBundle(Ctx, Bundles, fp::ebIgnore);
Value *V = Builder.CreateCall(Fn, {FnArg}, Bundles);
auto *I = cast<IntrinsicInst>(V);
- EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value());
+ EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_round).has_value());
EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value());
EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID());
EXPECT_EQ(RoundingMode::NearestTiesToEven, I->getRoundingMode());
@@ -648,7 +685,7 @@ TEST_F(IRBuilderTest, FPBundlesStrict) {
Value *V = Builder.CreateCall(Fn, {FnArg});
auto *I = cast<IntrinsicInst>(V);
EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value());
- EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value());
+ EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_round).has_value());
MemoryEffects ME = I->getMemoryEffects();
EXPECT_TRUE(ME.doesNotAccessMemory());
}
>From 6149fa1b0eef9f3b7cafaac715b13e726beeca8b Mon Sep 17 00:00:00 2001
From: Serge Pavlov <sepavloff at gmail.com>
Date: Tue, 21 Oct 2025 23:58:04 +0700
Subject: [PATCH 8/9] Address review comments
- Add IRBuilder::resetModeToStrictFP instead of adding extra argument to
setIsFPConstrained.
- Change wording in the Release Notes.
- Modify printing MemoryEffects of FP operation.
- Fix IRBuilder::CreateCall.
- Clean up unit test.
---
clang/lib/CodeGen/CodeGenFunction.cpp | 2 +-
llvm/docs/ReleaseNotes.md | 2 +
llvm/include/llvm/IR/FPEnv.h | 5 ++
llvm/include/llvm/IR/IRBuilder.h | 22 ++++---
llvm/lib/CodeGen/AtomicExpandPass.cpp | 4 +-
llvm/lib/CodeGen/HardwareLoops.cpp | 6 +-
.../SelectionDAG/SelectionDAGBuilder.cpp | 29 +--------
.../SelectionDAG/SelectionDAGBuilder.h | 2 -
llvm/lib/IR/AsmWriter.cpp | 55 +++++++++-------
llvm/lib/IR/FPEnv.cpp | 33 ++++------
llvm/lib/IR/IRBuilder.cpp | 31 ++++-----
.../Target/AMDGPU/AMDGPUAtomicOptimizer.cpp | 2 +-
llvm/lib/Target/AMDGPU/AMDGPULibCalls.cpp | 2 +-
.../Transforms/Utils/LibCallsShrinkWrap.cpp | 2 +-
llvm/lib/Transforms/Utils/LowerAtomic.cpp | 2 +-
llvm/test/Other/print-fp-memeffects.ll | 15 +++++
llvm/unittests/IR/IRBuilderTest.cpp | 63 +++++++++++++++----
17 files changed, 154 insertions(+), 123 deletions(-)
create mode 100644 llvm/test/Other/print-fp-memeffects.ll
diff --git a/clang/lib/CodeGen/CodeGenFunction.cpp b/clang/lib/CodeGen/CodeGenFunction.cpp
index 02d8cb638087a..88628530cf66b 100644
--- a/clang/lib/CodeGen/CodeGenFunction.cpp
+++ b/clang/lib/CodeGen/CodeGenFunction.cpp
@@ -1089,7 +1089,7 @@ void CodeGenFunction::StartFunction(GlobalDecl GD, QualType RetTy,
if ((FD && (FD->UsesFPIntrin() || FD->hasAttr<StrictFPAttr>())) ||
(!FD && (FPExceptionBehavior != llvm::fp::ebIgnore ||
RM != llvm::RoundingMode::NearestTiesToEven))) {
- Builder.setIsFPConstrained(true, false);
+ Builder.setIsFPConstrained(true);
Fn->addFnAttr(llvm::Attribute::StrictFP);
}
diff --git a/llvm/docs/ReleaseNotes.md b/llvm/docs/ReleaseNotes.md
index 8f8f6b9594581..f9f2ade978712 100644
--- a/llvm/docs/ReleaseNotes.md
+++ b/llvm/docs/ReleaseNotes.md
@@ -68,6 +68,8 @@ Changes to the LLVM IR
pointers) argument.
* A `load atomic` may now be used with vector types on x86.
* Floating-point operand bundles have been added.
+* Calls to floating-point intrinsics can have operand bundles "fp.round" and
+ "fp.except", which specify effective rounding mode and exception behavior.
Changes to LLVM infrastructure
------------------------------
diff --git a/llvm/include/llvm/IR/FPEnv.h b/llvm/include/llvm/IR/FPEnv.h
index 78a5d6d5b75f5..6f95b7c688a9d 100644
--- a/llvm/include/llvm/IR/FPEnv.h
+++ b/llvm/include/llvm/IR/FPEnv.h
@@ -81,6 +81,11 @@ LLVM_ABI std::optional<StringRef>
std::optional<StringRef>
convertExceptionBehaviorToBundle(fp::ExceptionBehavior);
+inline raw_ostream &operator<<(raw_ostream &OS, fp::ExceptionBehavior EB) {
+ OS << convertExceptionBehaviorToBundle(EB).value_or("invalid");
+ return OS;
+}
+
/// Returns true if the exception handling behavior and rounding mode
/// match what is used in the default floating point environment.
inline bool isDefaultFPEnvironment(fp::ExceptionBehavior EB, RoundingMode RM) {
diff --git a/llvm/include/llvm/IR/IRBuilder.h b/llvm/include/llvm/IR/IRBuilder.h
index 6494cf26cb3a0..e66094eae2582 100644
--- a/llvm/include/llvm/IR/IRBuilder.h
+++ b/llvm/include/llvm/IR/IRBuilder.h
@@ -348,17 +348,19 @@ class IRBuilderBase {
/// enabled the CreateF<op>() calls instead create constrained
/// floating point intrinsic calls. Fast math flags are unaffected
/// by this setting.
- void setIsFPConstrained(bool IsCon, bool AndReset = true) {
- if (AndReset) {
- if (IsCon) {
- setDefaultConstrainedRounding(RoundingMode::Dynamic);
- setDefaultConstrainedExcept(fp::ebStrict);
- } else {
- setDefaultConstrainedRounding(RoundingMode::NearestTiesToEven);
- setDefaultConstrainedExcept(fp::ebIgnore);
- }
+ void setIsFPConstrained(bool IsCon) { IsFPConstrained = IsCon; }
+
+ /// Enable/Disable use of constrained floating point math and reset FP options
+ /// according to the selected mode.
+ void resetModeToStrictFP(bool IsCon) {
+ if (IsCon) {
+ setDefaultConstrainedRounding(RoundingMode::Dynamic);
+ setDefaultConstrainedExcept(fp::ebStrict);
+ } else {
+ setDefaultConstrainedRounding(RoundingMode::NearestTiesToEven);
+ setDefaultConstrainedExcept(fp::ebIgnore);
}
- IsFPConstrained = IsCon;
+ setIsFPConstrained(IsCon);
}
/// Query for the use of constrained floating point math
diff --git a/llvm/lib/CodeGen/AtomicExpandPass.cpp b/llvm/lib/CodeGen/AtomicExpandPass.cpp
index 53f1cfe24a68d..9af4b3d40c6fb 100644
--- a/llvm/lib/CodeGen/AtomicExpandPass.cpp
+++ b/llvm/lib/CodeGen/AtomicExpandPass.cpp
@@ -159,7 +159,7 @@ struct ReplacementIRBuilder
SetInsertPoint(I);
this->CollectMetadataToCopy(I, {LLVMContext::MD_pcsections});
if (BB->getParent()->getAttributes().hasFnAttr(Attribute::StrictFP))
- this->setIsFPConstrained(true);
+ this->resetModeToStrictFP(true);
MMRAMD = I->getMetadata(LLVMContext::MD_mmra);
}
@@ -1716,7 +1716,7 @@ bool AtomicExpandImpl::tryExpandAtomicCmpXchg(AtomicCmpXchgInst *CI) {
bool llvm::expandAtomicRMWToCmpXchg(AtomicRMWInst *AI,
CreateCmpXchgInstFun CreateCmpXchg) {
ReplacementIRBuilder Builder(AI, AI->getDataLayout());
- Builder.setIsFPConstrained(
+ Builder.resetModeToStrictFP(
AI->getFunction()->hasFnAttribute(Attribute::StrictFP));
// FIXME: If FP exceptions are observable, we should force them off for the
diff --git a/llvm/lib/CodeGen/HardwareLoops.cpp b/llvm/lib/CodeGen/HardwareLoops.cpp
index 628d8cc8621f2..7a7512aeb1edf 100644
--- a/llvm/lib/CodeGen/HardwareLoops.cpp
+++ b/llvm/lib/CodeGen/HardwareLoops.cpp
@@ -503,7 +503,7 @@ Value *HardwareLoop::InitLoopCount() {
Value* HardwareLoop::InsertIterationSetup(Value *LoopCountInit) {
IRBuilder<> Builder(BeginBB->getTerminator());
if (BeginBB->getParent()->getAttributes().hasFnAttr(Attribute::StrictFP))
- Builder.setIsFPConstrained(true);
+ Builder.resetModeToStrictFP(true);
Type *Ty = LoopCountInit->getType();
bool UsePhi = UsePHICounter || Opts.ForcePhi;
Intrinsic::ID ID = UseLoopGuard
@@ -537,7 +537,7 @@ void HardwareLoop::InsertLoopDec() {
IRBuilder<> CondBuilder(ExitBranch);
if (ExitBranch->getParent()->getParent()->getAttributes().hasFnAttr(
Attribute::StrictFP))
- CondBuilder.setIsFPConstrained(true);
+ CondBuilder.resetModeToStrictFP(true);
Value *Ops[] = { LoopDecrement };
Value *NewCond = CondBuilder.CreateIntrinsic(Intrinsic::loop_decrement,
@@ -560,7 +560,7 @@ Instruction* HardwareLoop::InsertLoopRegDec(Value *EltsRem) {
IRBuilder<> CondBuilder(ExitBranch);
if (ExitBranch->getParent()->getParent()->getAttributes().hasFnAttr(
Attribute::StrictFP))
- CondBuilder.setIsFPConstrained(true);
+ CondBuilder.resetModeToStrictFP(true);
Value *Ops[] = { EltsRem, LoopDecrement };
Value *Call = CondBuilder.CreateIntrinsic(Intrinsic::loop_decrement_reg,
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index dd2c3922d3122..f55b627d9f9e3 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -8360,33 +8360,6 @@ void SelectionDAGBuilder::pushFPOpOutChain(SDValue Result,
}
}
-void SelectionDAGBuilder::pushOutChain(SDValue Result,
- fp::ExceptionBehavior EB) {
- assert(Result.getNode()->getNumValues() == 2);
-
- // Push node to the appropriate list so that future instructions can be
- // chained up correctly.
- SDValue OutChain = Result.getValue(1);
- switch (EB) {
- case fp::ExceptionBehavior::ebIgnore:
- // The only reason why ebIgnore nodes still need to be chained is that
- // they might depend on the current rounding mode, and therefore must
- // not be moved across instruction that may change that mode.
- [[fallthrough]];
- case fp::ExceptionBehavior::ebMayTrap:
- // These must not be moved across calls or instructions that may change
- // floating-point exception masks.
- PendingConstrainedFP.push_back(OutChain);
- break;
- case fp::ExceptionBehavior::ebStrict:
- // These must not be moved across calls or instructions that may change
- // floating-point exception masks or read floating-point exception flags.
- // In addition, they cannot be optimized out even if unused.
- PendingConstrainedFPStrict.push_back(OutChain);
- break;
- }
-}
-
void SelectionDAGBuilder::visitConstrainedFPIntrinsic(
const ConstrainedFPIntrinsic &FPI) {
SDLoc sdl = getCurSDLoc();
@@ -9496,7 +9469,7 @@ bool SelectionDAGBuilder::visitFPOperation(const CallInst &I, unsigned Opcode) {
SDLoc sdl = getCurSDLoc();
SDValue Result = DAG.getNode(Opcode, sdl, NodeVT, Operands, Flags);
if (HasChain)
- pushOutChain(Result, EB);
+ pushFPOpOutChain(Result, EB);
SDValue FPResult = Result.getValue(0);
setValue(&I, FPResult);
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h
index 47b8fec2c23aa..16f311b70fc17 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h
@@ -379,8 +379,6 @@ class SelectionDAGBuilder {
/// Evict any dangling debug information, attempting to salvage it first.
void resolveOrClearDbgInfo();
- void pushOutChain(SDValue Result, fp::ExceptionBehavior EB);
-
SDValue getValue(const Value *V);
SDValue getNonRegisterValue(const Value *V);
diff --git a/llvm/lib/IR/AsmWriter.cpp b/llvm/lib/IR/AsmWriter.cpp
index 3ed0018b41faa..996c5a6a3074c 100644
--- a/llvm/lib/IR/AsmWriter.cpp
+++ b/llvm/lib/IR/AsmWriter.cpp
@@ -106,6 +106,10 @@ static cl::opt<bool> PreserveAssemblyUseListOrder(
"preserve-ll-uselistorder", cl::Hidden, cl::init(false),
cl::desc("Preserve use-list order when writing LLVM assembly."));
+static cl::opt<bool> PrintFPMemoryEffects(
+ "print-fp-memory-effects", cl::Hidden,
+ cl::desc("Pretty print floating-point memory effects when dumping"));
+
// Make virtual table appear in this compilation unit.
AssemblyAnnotationWriter::~AssemblyAnnotationWriter() = default;
@@ -4321,6 +4325,25 @@ void AssemblyWriter::printGCRelocateComment(const GCRelocateInst &Relocate) {
Out << ")";
}
+static void printFPMemoryEffects(raw_ostream &Out, MemoryEffects ME) {
+ ModRefInfo MR = ME.getModRef(IRMemLocation::InaccessibleMem);
+ Out << " ; fpe=[";
+ switch (MR) {
+ case ModRefInfo::NoModRef:
+ break;
+ case ModRefInfo::Ref:
+ Out << "r";
+ break;
+ case ModRefInfo::Mod:
+ Out << "w";
+ break;
+ case ModRefInfo::ModRef:
+ Out << "rw";
+ break;
+ }
+ Out << "]";
+}
+
/// printInfoComment - Print a little comment after the instruction indicating
/// which slot it occupies.
void AssemblyWriter::printInfoComment(const Value &V) {
@@ -4351,30 +4374,14 @@ void AssemblyWriter::printInfoComment(const Value &V) {
if (PrintInstAddrs)
Out << " ; " << &V;
- if (auto *CI = dyn_cast<CallInst>(&V))
- if (Intrinsic::ID IID = CI->getIntrinsicID())
- if (IntrinsicInst::isFloatingPointOperation(IID))
- if (const BasicBlock *BB = CI->getParent())
- if (const Function *F = BB->getParent())
- if (F->hasFnAttribute(Attribute::StrictFP)) {
- MemoryEffects ME = CI->getMemoryEffects();
- ModRefInfo MR = ME.getModRef(IRMemLocation::InaccessibleMem);
- Out << " ; fpe=[";
- switch (MR) {
- case ModRefInfo::NoModRef:
- break;
- case ModRefInfo::Ref:
- Out << "r";
- break;
- case ModRefInfo::Mod:
- Out << "w";
- break;
- case ModRefInfo::ModRef:
- Out << "rw";
- break;
- }
- Out << "]";
- }
+ if (PrintFPMemoryEffects) {
+ if (auto *CI = dyn_cast<CallInst>(&V))
+ if (Intrinsic::ID IID = CI->getIntrinsicID())
+ if (const Function *F = CI->getFunction())
+ if (IntrinsicInst::isFloatingPointOperation(IID) &&
+ F->hasFnAttribute(Attribute::StrictFP))
+ printFPMemoryEffects(Out, CI->getMemoryEffects());
+ }
}
static void maybePrintCallAddrSpace(const Value *Operand, const Instruction *I,
diff --git a/llvm/lib/IR/FPEnv.cpp b/llvm/lib/IR/FPEnv.cpp
index 3a24c379faf65..8e595652d373a 100644
--- a/llvm/lib/IR/FPEnv.cpp
+++ b/llvm/lib/IR/FPEnv.cpp
@@ -76,27 +76,20 @@ std::optional<StringRef> convertRoundingModeToBundle(RoundingMode UseRounding) {
std::optional<StringRef> RoundingStr;
switch (UseRounding) {
case RoundingMode::Dynamic:
- RoundingStr = "dynamic";
- break;
+ return "dynamic";
case RoundingMode::NearestTiesToEven:
- RoundingStr = "tonearest";
- break;
+ return "tonearest";
case RoundingMode::NearestTiesToAway:
- RoundingStr = "tonearestaway";
- break;
+ return "tonearestaway";
case RoundingMode::TowardNegative:
- RoundingStr = "downward";
- break;
+ return "downward";
case RoundingMode::TowardPositive:
- RoundingStr = "upward";
- break;
+ return "upward";
case RoundingMode::TowardZero:
- RoundingStr = "towardzero";
- break;
+ return "towardzero";
default:
- break;
+ return std::nullopt;
}
- return RoundingStr;
}
std::optional<fp::ExceptionBehavior>
@@ -139,16 +132,14 @@ convertExceptionBehaviorToBundle(fp::ExceptionBehavior UseExcept) {
std::optional<StringRef> ExceptStr;
switch (UseExcept) {
case fp::ebStrict:
- ExceptStr = "strict";
- break;
+ return "strict";
case fp::ebIgnore:
- ExceptStr = "ignore";
- break;
+ return "ignore";
case fp::ebMayTrap:
- ExceptStr = "maytrap";
- break;
+ return "maytrap";
+ default:
+ return std::nullopt;
}
- return ExceptStr;
}
Intrinsic::ID getConstrainedIntrinsicID(const Instruction &Instr) {
diff --git a/llvm/lib/IR/IRBuilder.cpp b/llvm/lib/IR/IRBuilder.cpp
index 105cf628a3e54..59d67cee07c9a 100644
--- a/llvm/lib/IR/IRBuilder.cpp
+++ b/llvm/lib/IR/IRBuilder.cpp
@@ -126,7 +126,7 @@ CallInst *IRBuilderBase::CreateCall(FunctionType *FTy, Value *Callee,
if (const auto *Func = dyn_cast<Function>(Callee))
if (Intrinsic::ID ID = Func->getIntrinsicID())
if (IntrinsicInst::isFloatingPointOperation(ID)) {
- // If the builder has non-default floating-point options, add
+ // If the builder specifies non-default floating-point options, add
// corresponding operand bundle unless a bundle with such tag is already
// present.
bool NeedRounding;
@@ -141,23 +141,24 @@ CallInst *IRBuilderBase::CreateCall(FunctionType *FTy, Value *Callee,
assert(DefaultConstrainedExcept == fp::ebIgnore &&
"FP exception in default mode must be ignored");
}
+ // Options specified by bundles have higher precedence.
+ for (const auto &Bundle : OpBundles) {
+ if (NeedRounding && Bundle.getTag() == "fp.round")
+ NeedRounding = false;
+ if (NeedExceptions && Bundle.getTag() == "fp.except")
+ NeedExceptions = false;
+ }
if (NeedRounding || NeedExceptions) {
- for (const auto &Bundle : OpBundles) {
- if (NeedRounding && Bundle.getTag() == "fp.round")
- NeedRounding = false;
- if (NeedExceptions && Bundle.getTag() == "fp.except")
- NeedExceptions = false;
- }
- if (NeedRounding || NeedExceptions) {
- ActualBundles.append(OpBundles.begin(), OpBundles.end());
- if (NeedRounding)
- createRoundingBundle(ActualBundles, DefaultConstrainedRounding);
- if (NeedExceptions)
- createExceptionBundle(ActualBundles, DefaultConstrainedExcept);
- ActualBundlesRef = ActualBundles;
- }
+ ActualBundles.append(OpBundles.begin(), OpBundles.end());
+ if (NeedRounding)
+ createRoundingBundle(ActualBundles, DefaultConstrainedRounding);
+ if (NeedExceptions)
+ createExceptionBundle(ActualBundles, DefaultConstrainedExcept);
+ ActualBundlesRef = ActualBundles;
}
if (IsFPConstrained) {
+ // Due to potential reading FP exception bits, in strictfp mode the
+ // memory effects must include read/write access to FPE.
MemoryEffects FME = Func->getMemoryEffects();
NeedUpdateMemoryEffects = !FME.doesAccessInaccessibleMem();
}
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
index 0a163f8dc7f6b..01200b818d9d3 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
@@ -649,7 +649,7 @@ void AMDGPUAtomicOptimizerImpl::optimizeAtomic(Instruction &I,
IRBuilder<> B(&I);
if (AtomicRMWInst::isFPOperation(Op)) {
- B.setIsFPConstrained(I.getFunction()->hasFnAttribute(Attribute::StrictFP));
+ B.resetModeToStrictFP(I.getFunction()->hasFnAttribute(Attribute::StrictFP));
}
// If we are in a pixel shader, because of how we have to mask out helper
diff --git a/llvm/lib/Target/AMDGPU/AMDGPULibCalls.cpp b/llvm/lib/Target/AMDGPU/AMDGPULibCalls.cpp
index aa755344d3325..57e05fc2969b9 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPULibCalls.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPULibCalls.cpp
@@ -654,7 +654,7 @@ bool AMDGPULibCalls::fold(CallInst *CI) {
IRBuilder<> B(CI);
if (CI->isStrictFP())
- B.setIsFPConstrained(true);
+ B.resetModeToStrictFP(true);
if (FPMathOperator *FPOp = dyn_cast<FPMathOperator>(CI)) {
// Under unsafe-math, evaluate calls if possible.
diff --git a/llvm/lib/Transforms/Utils/LibCallsShrinkWrap.cpp b/llvm/lib/Transforms/Utils/LibCallsShrinkWrap.cpp
index fca09c678eba4..8cf57bbc6ad78 100644
--- a/llvm/lib/Transforms/Utils/LibCallsShrinkWrap.cpp
+++ b/llvm/lib/Transforms/Utils/LibCallsShrinkWrap.cpp
@@ -103,7 +103,7 @@ class LibCallsShrinkWrap : public InstVisitor<LibCallsShrinkWrap> {
if (!Arg->getType()->isFloatTy())
V = ConstantFoldCastInstruction(Instruction::FPExt, V, Arg->getType());
if (BBBuilder.GetInsertBlock()->getParent()->hasFnAttribute(Attribute::StrictFP))
- BBBuilder.setIsFPConstrained(true);
+ BBBuilder.resetModeToStrictFP(true);
return BBBuilder.CreateFCmp(Cmp, Arg, V);
}
diff --git a/llvm/lib/Transforms/Utils/LowerAtomic.cpp b/llvm/lib/Transforms/Utils/LowerAtomic.cpp
index e8b06415d4062..b0c788fa74a5d 100644
--- a/llvm/lib/Transforms/Utils/LowerAtomic.cpp
+++ b/llvm/lib/Transforms/Utils/LowerAtomic.cpp
@@ -124,7 +124,7 @@ Value *llvm::buildAtomicRMWValue(AtomicRMWInst::BinOp Op,
bool llvm::lowerAtomicRMWInst(AtomicRMWInst *RMWI) {
IRBuilder<> Builder(RMWI);
- Builder.setIsFPConstrained(
+ Builder.resetModeToStrictFP(
RMWI->getFunction()->hasFnAttribute(Attribute::StrictFP));
Value *Ptr = RMWI->getPointerOperand();
diff --git a/llvm/test/Other/print-fp-memeffects.ll b/llvm/test/Other/print-fp-memeffects.ll
new file mode 100644
index 0000000000000..937e31456b26b
--- /dev/null
+++ b/llvm/test/Other/print-fp-memeffects.ll
@@ -0,0 +1,15 @@
+; RUN: opt %s -print-fp-memory-effects -S | FileCheck %s --match-full-lines
+
+define float @test_00(float %x) {
+ %res = call float @llvm.nearbyint.f32(float %x) [ "fpe.round"(metadata !"upward"), "fpe.except"(metadata !"ignore") ]
+ ret float %res
+; CHECK-LABEL: define float @test_00({{.*}}
+; CHECK: {{.*}} = call float @llvm.nearbyint.f32({{.*}}]
+}
+
+define float @test_01(float %x) strictfp {
+ %res = call float @llvm.nearbyint.f32(float %x) [ "fpe.round"(metadata !"upward"), "fpe.except"(metadata !"strict") ]
+ ret float %res
+; CHECK-LABEL: define float @test_01({{.*}}
+; CHECK: {{.*}} = call float @llvm.nearbyint.f32({{.*}}] ; fpe=[rw]
+}
diff --git a/llvm/unittests/IR/IRBuilderTest.cpp b/llvm/unittests/IR/IRBuilderTest.cpp
index a1da8b1b5f3f9..96f43848a9394 100644
--- a/llvm/unittests/IR/IRBuilderTest.cpp
+++ b/llvm/unittests/IR/IRBuilderTest.cpp
@@ -274,7 +274,7 @@ TEST_F(IRBuilderTest, ConstrainedFP) {
// instructions.
Builder.setDefaultConstrainedRounding(RoundingMode::Dynamic);
Builder.setDefaultConstrainedExcept(fp::ebStrict);
- Builder.setIsFPConstrained(true, false);
+ Builder.setIsFPConstrained(true);
auto Parent = BB->getParent();
Parent->addFnAttr(Attribute::StrictFP);
@@ -442,12 +442,12 @@ TEST_F(IRBuilderTest, ConstrainedFPFunctionCall) {
Function::Create(FTy, Function::ExternalLinkage, "", M.get());
BasicBlock *CalleeBB = BasicBlock::Create(Ctx, "", Callee);
IRBuilder<> CalleeBuilder(CalleeBB);
- CalleeBuilder.setIsFPConstrained(true);
+ CalleeBuilder.resetModeToStrictFP(true);
CalleeBuilder.setConstrainedFPFunctionAttr();
CalleeBuilder.CreateRetVoid();
// Now call the empty constrained FP function.
- Builder.setIsFPConstrained(true);
+ Builder.resetModeToStrictFP(true);
Builder.setConstrainedFPFunctionAttr();
CallInst *FCall = Builder.CreateCall(Callee, {});
@@ -571,9 +571,6 @@ TEST_F(IRBuilderTest, FPBundlesStrict) {
F->addFnAttr(Attribute::StrictFP);
IRBuilder<> Builder(BB);
- Builder.setDefaultConstrainedExcept(fp::ebStrict);
- Builder.setDefaultConstrainedRounding(RoundingMode::TowardZero);
- Builder.setIsFPConstrained(true);
GlobalVariable *GVDouble = new GlobalVariable(
*M, Type::getDoubleTy(Ctx), true, GlobalValue::ExternalLinkage, nullptr);
@@ -581,6 +578,8 @@ TEST_F(IRBuilderTest, FPBundlesStrict) {
Function *Fn = Intrinsic::getOrInsertDeclaration(
M.get(), Intrinsic::nearbyint, {Type::getDoubleTy(Ctx)});
+ Builder.resetModeToStrictFP(true);
+
// A floating-point operation has side effects in strictfp environment even
// if it has no FP bundles.
{
@@ -595,7 +594,8 @@ TEST_F(IRBuilderTest, FPBundlesStrict) {
EXPECT_TRUE(ME.doesAccessInaccessibleMem());
}
- // Check call with FP bundles, with default (dynamic) rounding mode
+ // A call may have FP bundles, in this case it has a bundle that specifies
+ // default (dynamic) rounding mode.
// nearbyint(%x) [ "fp.round" (metadata !"dynamic") ]
{
SmallVector<OperandBundleDef, 1> Bundles;
@@ -611,7 +611,7 @@ TEST_F(IRBuilderTest, FPBundlesStrict) {
EXPECT_TRUE(ME.doesAccessInaccessibleMem());
}
- // Check call with FP bundles, with specific rounding mode
+ // FP bundle may specify a non-default rounding mode.
// nearbyint(%x) [ "fp.round" (metadata !"towardzero") ]
{
SmallVector<OperandBundleDef, 1> Bundles;
@@ -627,7 +627,7 @@ TEST_F(IRBuilderTest, FPBundlesStrict) {
EXPECT_TRUE(ME.doesAccessInaccessibleMem());
}
- // Check call with FP bundles, exception behavior is set to default value.
+ // FP bundles may specify exception behavior, here it is the default value.
// nearbyint(%x) [ "fp.except" (metadata !"strict") ]
{
SmallVector<OperandBundleDef, 1> Bundles;
@@ -643,7 +643,7 @@ TEST_F(IRBuilderTest, FPBundlesStrict) {
EXPECT_TRUE(ME.doesAccessInaccessibleMem());
}
- // Check call with FP bundles, exception behavior is set to specific value.
+ // FP bundles may specify exception behavior, including non-default values.
// nearbyint(%x) [ "fp.except" (metadata !"ignore") ]
{
SmallVector<OperandBundleDef, 1> Bundles;
@@ -659,19 +659,56 @@ TEST_F(IRBuilderTest, FPBundlesStrict) {
EXPECT_TRUE(ME.doesAccessInaccessibleMem());
}
- // Check call with both FP bundles.
+ // FP bundles may specify both rounding mode and exception behavior, here is
+ // the case of default values.
+ // nearbyint(%x) [ "fp.round" (metadata !"dynamic"),
+ // "fp.except" (metadata !"strict") ]
+ {
+ SmallVector<OperandBundleDef, 1> Bundles;
+ llvm::addRoundingBundle(Ctx, Bundles, RoundingMode::Dynamic);
+ llvm::addExceptionBundle(Ctx, Bundles, fp::ebStrict);
+ Value *V = Builder.CreateCall(Fn, {FnArg}, Bundles);
+ auto *I = cast<IntrinsicInst>(V);
+ EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_round).has_value());
+ EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value());
+ EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID());
+ EXPECT_EQ(RoundingMode::Dynamic, I->getRoundingMode());
+ EXPECT_EQ(fp::ebStrict, I->getExceptionBehavior());
+ MemoryEffects ME = I->getMemoryEffects();
+ EXPECT_TRUE(ME.doesAccessInaccessibleMem());
+ }
+
+ // FP bundles may specify both rounding mode and exception behavior, here is
+ // the case of non-default values.
// nearbyint(%x) [ "fp.round" (metadata !"towardzero"),
// "fp.except" (metadata !"ignore") ]
{
SmallVector<OperandBundleDef, 1> Bundles;
- llvm::addRoundingBundle(Ctx, Bundles, RoundingMode::NearestTiesToEven);
+ llvm::addRoundingBundle(Ctx, Bundles, RoundingMode::TowardZero);
llvm::addExceptionBundle(Ctx, Bundles, fp::ebIgnore);
Value *V = Builder.CreateCall(Fn, {FnArg}, Bundles);
auto *I = cast<IntrinsicInst>(V);
EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_round).has_value());
EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value());
EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID());
- EXPECT_EQ(RoundingMode::NearestTiesToEven, I->getRoundingMode());
+ EXPECT_EQ(RoundingMode::TowardZero, I->getRoundingMode());
+ EXPECT_EQ(fp::ebIgnore, I->getExceptionBehavior());
+ MemoryEffects ME = I->getMemoryEffects();
+ EXPECT_TRUE(ME.doesAccessInaccessibleMem());
+ }
+
+ // FP bundles are added when IRBuilder specifies rounding or exception
+ // behavior, non-default for the selected mode.
+ Builder.setDefaultConstrainedExcept(fp::ebIgnore);
+ Builder.setDefaultConstrainedRounding(RoundingMode::TowardZero);
+ Builder.setIsFPConstrained(true);
+ {
+ Value *V = Builder.CreateCall(Fn, {FnArg});
+ auto *I = cast<IntrinsicInst>(V);
+ EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_round).has_value());
+ EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value());
+ EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID());
+ EXPECT_EQ(RoundingMode::TowardZero, I->getRoundingMode());
EXPECT_EQ(fp::ebIgnore, I->getExceptionBehavior());
MemoryEffects ME = I->getMemoryEffects();
EXPECT_TRUE(ME.doesAccessInaccessibleMem());
>From 1842ac0ce10ce0ea1da8574e015a85faaeab26ce Mon Sep 17 00:00:00 2001
From: Serge Pavlov <sepavloff at gmail.com>
Date: Sun, 2 Nov 2025 14:46:55 +0700
Subject: [PATCH 9/9] Fix warnings
---
llvm/lib/IR/FPEnv.cpp | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/llvm/lib/IR/FPEnv.cpp b/llvm/lib/IR/FPEnv.cpp
index 8e595652d373a..e7ed603898f6d 100644
--- a/llvm/lib/IR/FPEnv.cpp
+++ b/llvm/lib/IR/FPEnv.cpp
@@ -73,7 +73,6 @@ std::optional<StringRef> convertRoundingModeToStr(RoundingMode UseRounding) {
}
std::optional<StringRef> convertRoundingModeToBundle(RoundingMode UseRounding) {
- std::optional<StringRef> RoundingStr;
switch (UseRounding) {
case RoundingMode::Dynamic:
return "dynamic";
@@ -129,7 +128,6 @@ convertExceptionBehaviorToStr(fp::ExceptionBehavior UseExcept) {
std::optional<StringRef>
convertExceptionBehaviorToBundle(fp::ExceptionBehavior UseExcept) {
- std::optional<StringRef> ExceptStr;
switch (UseExcept) {
case fp::ebStrict:
return "strict";
@@ -137,9 +135,8 @@ convertExceptionBehaviorToBundle(fp::ExceptionBehavior UseExcept) {
return "ignore";
case fp::ebMayTrap:
return "maytrap";
- default:
- return std::nullopt;
}
+ return std::nullopt;
}
Intrinsic::ID getConstrainedIntrinsicID(const Instruction &Instr) {
More information about the llvm-commits
mailing list