[Mlir-commits] [mlir] [mlir][arith][RFC] Add new strict FP handling in Arith (PR #209287)
Andy Kaylor
llvmlistbot at llvm.org
Mon Jul 13 12:54:39 PDT 2026
https://github.com/andykaylor created https://github.com/llvm/llvm-project/pull/209287
This is a proposed implementation of strict floating-point handling for Arith and Math dialects. If this direction is accepted, I will split the individual commits in this PR out for proper review. I have combined them here so that the entire design can be considered as a whole.
This is a follow-up to my prior [RFC on Discourse](https://discourse.llvm.org/t/rfc-mlir-new-representation-of-floating-point-constraints/91123) proposing a new MLIR representation of floating-point constraints. Based on the feedback I received in https://github.com/llvm/llvm-project/pull/205158, I am moving the floating-point environment attribute out to the CIR and Arith dialects and lowering these to existing operations in the LLVM dialect that continue to map directly to the current form of LLVM IR.
This PR shows the Arith and Math implementation. I have posted a separate [PR for the CIR dialect changes](https://github.com/llvm/llvm-project/pull/209275).
This is intended to replace the existing rounding-mode handling in the Arith dialect. I have left the old handling in place here but added comments that it should be considered deprecated. I am open to suggestions on how that should be handled.
The implementation here has one significant difference from the CIR implementation. In the CIR implementation, I am lowering operations with the Fenv attribute to general intrinsic calls to the constrained floating-point intrinsics. Here I have added specialized constrained operations to the LLVM dialect where they were missing and lower to those. I did this to be consistent with the existing Arith rounding-mode handling, though I think it may be more consistent with the LLVM dialect design philosophy to remove the specialized constrained operations there entirely and to lower to the generalized intrinsic calls as I have done in CIR.
Assisted-by: Cursor / claude-opus-4.8
>From 67b0393ec879f961ddec5c55efe65ba00d095b6a Mon Sep 17 00:00:00 2001
From: Andy Kaylor <akaylor at nvidia.com>
Date: Thu, 2 Jul 2026 16:44:22 -0700
Subject: [PATCH 1/7] [mlir][llvm] Add more constrained FP operations
This change adds special constrained forms of transcendental operations
that may lower to contrained fp intrinsic calls. These will be used by
the Math and CIR dialects in later changes.
WIP! Needs a test.
---
.../mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td | 68 +++++++++++++++++++
.../LLVMIR/Import/intrinsic-unregistered.ll | 10 +--
2 files changed, 73 insertions(+), 5 deletions(-)
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td
index 76a3afad9a360..b912cef1386dd 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td
@@ -504,6 +504,42 @@ class LLVM_ConstrainedIntr<string mnem, int numArgs,
}];
}
+// Convenience base class for constrained intrinsics that take a single
+// floating-point operand of the same type as the result and support a rounding
+// mode (e.g. the transcendental math functions).
+class LLVM_ConstrainedUnaryIntrRM<string mnem>
+ : LLVM_ConstrainedIntr<mnem, /*numArgs=*/1,
+ /*overloadedResult=*/1, /*overloadedOperands=*/[],
+ /*hasRoundingMode=*/1, [SameOperandsAndResultType]> {
+ let assemblyFormat = [{
+ $arg_0 $roundingmode $fpExceptionBehavior attr-dict `:` type($arg_0)
+ }];
+}
+
+// Convenience base class for constrained intrinsics that take a single
+// floating-point operand of the same type as the result and do not support a
+// rounding mode (e.g. the rounding math functions such as `ceil`).
+class LLVM_ConstrainedUnaryIntrNoRM<string mnem>
+ : LLVM_ConstrainedIntr<mnem, /*numArgs=*/1,
+ /*overloadedResult=*/1, /*overloadedOperands=*/[],
+ /*hasRoundingMode=*/0, [SameOperandsAndResultType]> {
+ let assemblyFormat = [{
+ $arg_0 $fpExceptionBehavior attr-dict `:` type($arg_0)
+ }];
+}
+
+// Convenience base class for constrained intrinsics that take two
+// floating-point operands of the same type as the result and support a rounding
+// mode (e.g. `pow` and `atan2`).
+class LLVM_ConstrainedBinaryIntrRM<string mnem>
+ : LLVM_ConstrainedIntr<mnem, /*numArgs=*/2,
+ /*overloadedResult=*/1, /*overloadedOperands=*/[],
+ /*hasRoundingMode=*/1, [SameOperandsAndResultType]> {
+ let assemblyFormat = [{
+ $arg_0 `,` $arg_1 $roundingmode $fpExceptionBehavior attr-dict `:` type($arg_0)
+ }];
+}
+
def LLVM_ConstrainedFAddIntr
: LLVM_ConstrainedIntr<"fadd", /*numArgs=*/2,
/*overloadedResult=*/1, /*overloadedOperands=*/[],
@@ -567,6 +603,38 @@ def LLVM_ConstrainedFMulAddIntr
}];
}
+// Constrained intrinsics for math library functions. These correspond to the
+// like-named operations in the Math dialect and are produced when the source
+// operation carries a `#arith.fenv` floating-point environment attribute.
+
+// Unary transcendental functions (support a rounding mode).
+def LLVM_ConstrainedSqrtIntr : LLVM_ConstrainedUnaryIntrRM<"sqrt">;
+def LLVM_ConstrainedSinIntr : LLVM_ConstrainedUnaryIntrRM<"sin">;
+def LLVM_ConstrainedCosIntr : LLVM_ConstrainedUnaryIntrRM<"cos">;
+def LLVM_ConstrainedTanIntr : LLVM_ConstrainedUnaryIntrRM<"tan">;
+def LLVM_ConstrainedASinIntr : LLVM_ConstrainedUnaryIntrRM<"asin">;
+def LLVM_ConstrainedACosIntr : LLVM_ConstrainedUnaryIntrRM<"acos">;
+def LLVM_ConstrainedATanIntr : LLVM_ConstrainedUnaryIntrRM<"atan">;
+def LLVM_ConstrainedSinhIntr : LLVM_ConstrainedUnaryIntrRM<"sinh">;
+def LLVM_ConstrainedCoshIntr : LLVM_ConstrainedUnaryIntrRM<"cosh">;
+def LLVM_ConstrainedTanhIntr : LLVM_ConstrainedUnaryIntrRM<"tanh">;
+def LLVM_ConstrainedExpIntr : LLVM_ConstrainedUnaryIntrRM<"exp">;
+def LLVM_ConstrainedExp2Intr : LLVM_ConstrainedUnaryIntrRM<"exp2">;
+def LLVM_ConstrainedLogIntr : LLVM_ConstrainedUnaryIntrRM<"log">;
+def LLVM_ConstrainedLog10Intr : LLVM_ConstrainedUnaryIntrRM<"log10">;
+def LLVM_ConstrainedLog2Intr : LLVM_ConstrainedUnaryIntrRM<"log2">;
+
+// Unary rounding functions (do not support a rounding mode).
+def LLVM_ConstrainedCeilIntr : LLVM_ConstrainedUnaryIntrNoRM<"ceil">;
+def LLVM_ConstrainedFloorIntr : LLVM_ConstrainedUnaryIntrNoRM<"floor">;
+def LLVM_ConstrainedRoundIntr : LLVM_ConstrainedUnaryIntrNoRM<"round">;
+def LLVM_ConstrainedRoundEvenIntr : LLVM_ConstrainedUnaryIntrNoRM<"roundeven">;
+def LLVM_ConstrainedTruncIntr : LLVM_ConstrainedUnaryIntrNoRM<"trunc">;
+
+// Binary functions (support a rounding mode).
+def LLVM_ConstrainedPowIntr : LLVM_ConstrainedBinaryIntrRM<"pow">;
+def LLVM_ConstrainedATan2Intr : LLVM_ConstrainedBinaryIntrRM<"atan2">;
+
def LLVM_ConstrainedUIToFP
: LLVM_ConstrainedIntr<"uitofp", /*numArgs=*/1,
/*overloadedResult=*/1, /*overloadedOperands=*/[0],
diff --git a/mlir/test/Target/LLVMIR/Import/intrinsic-unregistered.ll b/mlir/test/Target/LLVMIR/Import/intrinsic-unregistered.ll
index da58d48d995be..556803f3a0dbb 100644
--- a/mlir/test/Target/LLVMIR/Import/intrinsic-unregistered.ll
+++ b/mlir/test/Target/LLVMIR/Import/intrinsic-unregistered.ll
@@ -85,14 +85,14 @@ define signext i32 @test_intrin_arg_attr(i32 signext %a) nounwind {
; imported as `llvm.mlir.metadata_as_value` ops wrapping the corresponding
; `#llvm.md_string` attribute.
-declare float @llvm.experimental.constrained.sqrt.f32(float, metadata, metadata)
+declare float @llvm.experimental.constrained.nearbyint.f32(float, metadata, metadata)
-; CHECK-LABEL: llvm.func @constrained_sqrt
-define float @constrained_sqrt(float %a) {
+; CHECK-LABEL: llvm.func @constrained_nearbyint
+define float @constrained_nearbyint(float %a) {
; CHECK: %[[RM:.*]] = llvm.mlir.metadata_as_value #llvm.md_string<"round.tonearest">
; CHECK: %[[EB:.*]] = llvm.mlir.metadata_as_value #llvm.md_string<"fpexcept.strict">
- ; CHECK: %{{.*}} = llvm.call_intrinsic "llvm.experimental.constrained.sqrt.f32"(%{{.*}}, %[[RM]], %[[EB]]) : (f32, !llvm.metadata, !llvm.metadata) -> f32
- %r = call float @llvm.experimental.constrained.sqrt.f32(float %a, metadata !"round.tonearest", metadata !"fpexcept.strict")
+ ; CHECK: %{{.*}} = llvm.call_intrinsic "llvm.experimental.constrained.nearbyint.f32"(%{{.*}}, %[[RM]], %[[EB]]) : (f32, !llvm.metadata, !llvm.metadata) -> f32
+ %r = call float @llvm.experimental.constrained.nearbyint.f32(float %a, metadata !"round.tonearest", metadata !"fpexcept.strict")
ret float %r
}
>From 382229113f7af27560812b61787ee28e78f6c375 Mon Sep 17 00:00:00 2001
From: Andy Kaylor <akaylor at nvidia.com>
Date: Fri, 26 Jun 2026 08:47:24 -0700
Subject: [PATCH 2/7] [mlir][arith] Introduce fenv attribute to arith dialect
This adds a new attribute, arith::FenvAttr, and adds it to all operations
that currently support arith::RoundingModeAttr. The fenv attribute is
intended as a replacement for the rounding mode attribute, extending it
to also describe exception behavior requirements.
WIP!
---
.../mlir/Dialect/Arith/IR/ArithBase.td | 148 +++++++++++++++++-
.../include/mlir/Dialect/Arith/IR/ArithOps.td | 89 ++++++++++-
.../Dialect/Arith/IR/ArithOpsInterfaces.td | 83 ++++++++++
mlir/include/mlir/Dialect/Math/IR/MathOps.td | 53 ++++++-
.../ArithCommon/AttrToLLVMConverter.cpp | 2 +
.../Dialect/Arith/IR/ArithCanonicalization.td | 40 +++--
.../Dialect/Arith/Transforms/ExpandOps.cpp | 14 +-
.../Conversion/ArithToLLVM/arith-to-llvm.mlir | 2 +
mlir/test/Dialect/Arith/invalid.mlir | 8 +
mlir/test/Dialect/Arith/ops.mlir | 23 +++
10 files changed, 432 insertions(+), 30 deletions(-)
diff --git a/mlir/include/mlir/Dialect/Arith/IR/ArithBase.td b/mlir/include/mlir/Dialect/Arith/IR/ArithBase.td
index 985ae01008002..4132a4cfb3c55 100644
--- a/mlir/include/mlir/Dialect/Arith/IR/ArithBase.td
+++ b/mlir/include/mlir/Dialect/Arith/IR/ArithBase.td
@@ -186,12 +186,158 @@ def Arith_RTowardZero // Round toward 0
: I32EnumAttrCase<"toward_zero", 3>;
def Arith_RToNearestTiesAwayFromZero // Round to nearest, ties away from zero
: I32EnumAttrCase<"to_nearest_away", 4>;
+def Arith_RDynamic // Rounding mode unknown at compile time
+ : I32EnumAttrCase<"unknown", 7>; // Mirrors `llvm::RoundingMode::Dynamic`.
def Arith_RoundingModeAttr : I32EnumAttr<
"RoundingMode", "Floating point rounding mode",
[Arith_RToNearestTiesToEven, Arith_RDownward, Arith_RUpward,
- Arith_RTowardZero, Arith_RToNearestTiesAwayFromZero]> {
+ Arith_RTowardZero, Arith_RToNearestTiesAwayFromZero, Arith_RDynamic]> {
let cppNamespace = "::mlir::arith";
}
+//===----------------------------------------------------------------------===//
+// Floating-point environment enums (used by the `#arith.fenv` attribute)
+//===----------------------------------------------------------------------===//
+
+def FPExceptionMode : I32EnumAttr<
+ "FPExceptionMode", "floating-point exception mode", [
+ I32EnumAttrCase<"Unknown", 0, "unknown">,
+ I32EnumAttrCase<"Masked", 1, "masked">,
+ I32EnumAttrCase<"Unmasked", 2, "unmasked">
+]> {
+ let description = [{
+ The known floating-point exception mode at the point the instruction is
+ executed. If the actual exception mode differs from this value, the
+ behavior is undefined.
+ }];
+ let cppNamespace = "::mlir::arith";
+ let genSpecializedAttr = 0;
+}
+
+//===----------------------------------------------------------------------===//
+// FenvAttr
+//===----------------------------------------------------------------------===//
+
+def Arith_FenvAttr : AttrDef<Arith_Dialect, "Fenv"> {
+ let mnemonic = "fenv";
+ let summary = "Describes floating-point environment constraints";
+ let description = [{
+ The `#arith.fenv` attribute describes constraints on the floating-point
+ handling of a floating-point operation. It can be attached to
+ floating-point operations to capture rounding and exception behavior. All
+ of its parameters are optional.
+
+ - `dynamic_rounding_mode`: the known dynamic rounding mode at the point the
+ instruction is executed. If the actual dynamic rounding mode when the
+ instruction is executed does not match the assumed value, the behavior is
+ undefined.
+ - `except_mode`: the known exception mode at the point the instruction is
+ executed. If the actual state of exception masking when the instruction is
+ executed does not match this value, the behavior is undefined.
+ - `strict_except`: if `false`, any case that would produce an FP exception
+ produces it non-deterministically instead (i.e. it may or may not occur).
+ This means the FP status is written non-deterministically, and, if
+ exceptions are unmasked, the instruction traps non-deterministically.
+
+ When this attribute is present on an operation, the operation is lowered to
+ the corresponding `llvm.experimental.constrained.*` intrinsic during
+ translation to LLVM IR.
+
+ The default setting for each parameter (`unknown` for
+ `dynamic_rounding_mode`, `masked` for exception_mode, and `false` for
+ `strict_except`) is canonically represented as an unset parameter.
+ Constructing an attribute whose parameters are all set to their defaults
+ therefore yields the same canonical, empty attribute as `#arith.fenv<>`.
+
+ Carrying the (possibly empty) attribute is *not* the same as having no
+ attribute at all. An operation that carries the attribute is lowered to the
+ corresponding constrained intrinsic operation in LLVM, even when every
+ parameter is at its default value. An operation with no `#arith.fenv`
+ attribute is lowered to the regular, unconstrained operation.
+
+ Examples:
+ ```mlir
+ fenv<dynamic_rounding_mode = to_nearest_even>
+ fenv<except_mode = unmasked>
+ fenv<strict_except = true>
+ fenv<dynamic_rounding_mode = upward, except_mode = unknown, strict_except = true>
+ ```
+ }];
+
+ let parameters = (ins
+ OptionalParameter<
+ "std::optional<::mlir::arith::RoundingMode>">:$dynamic_rounding_mode,
+ OptionalParameter<"std::optional<::mlir::arith::FPExceptionMode>">:$except_mode,
+ OptionalParameter<"mlir::BoolAttr">:$strict_except);
+
+ let assemblyFormat = [{
+ `<` struct($dynamic_rounding_mode, $except_mode, $strict_except) `>`
+ }];
+
+ // Replace the default builder so that constructing the attribute normalizes
+ // any parameter that is set to its default value to "unset". This keeps a
+ // single canonical representation for the default floating-point environment.
+ // Because `skipDefaultBuilders` suppresses the auto-synthesized const builder
+ // call, provide it explicitly. The `DefaultValuedAttr` form below uses a null
+ // attribute as its default, so this simply forwards its argument (a null
+ // attribute when the operation does not carry a `#arith.fenv` attribute).
+ let skipDefaultBuilders = 1;
+ let constBuilderCall = "$0";
+ let builders = [
+ AttrBuilder<(ins
+ CArg<"std::optional<::mlir::arith::RoundingMode>",
+ "std::nullopt">:$dynamicRoundingMode,
+ CArg<"std::optional<::mlir::arith::FPExceptionMode>",
+ "std::nullopt">:$exceptMode,
+ CArg<"mlir::BoolAttr", "{}">:$strictExcept), [{
+ if (dynamicRoundingMode == ::mlir::arith::RoundingMode::unknown)
+ dynamicRoundingMode = std::nullopt;
+ if (exceptMode == ::mlir::arith::FPExceptionMode::Masked)
+ exceptMode = std::nullopt;
+ if (strictExcept && !strictExcept.getValue())
+ strictExcept = {};
+ return Base::get($_ctxt, dynamicRoundingMode, exceptMode, strictExcept);
+ }]>
+ ];
+
+ let extraClassDeclaration = [{
+ /// The default dynamic rounding mode used when `dynamic_rounding_mode` is
+ /// unset.
+ static constexpr ::mlir::arith::RoundingMode
+ getDefaultDynamicRoundingMode() {
+ return ::mlir::arith::RoundingMode::unknown;
+ }
+
+ /// The default exception mode used when `except_mode` is unset.
+ static constexpr ::mlir::arith::FPExceptionMode getDefaultExceptionMode() {
+ return ::mlir::arith::FPExceptionMode::Masked;
+ }
+
+ /// Returns the effective dynamic rounding mode, substituting the default
+ /// when unset.
+ ::mlir::arith::RoundingMode getDynamicRoundingModeOrDefault() const {
+ return getDynamicRoundingMode().value_or(getDefaultDynamicRoundingMode());
+ }
+ /// Returns the effective exception mode, substituting the default when
+ /// unset.
+ ::mlir::arith::FPExceptionMode getExceptionModeOrDefault() const {
+ return getExceptMode().value_or(getDefaultExceptionMode());
+ }
+ /// Returns the effective `strict_except` flag.
+ bool getStrictExceptOrDefault() const {
+ ::mlir::BoolAttr attr = getStrictExcept();
+ return attr ? attr.getValue() : false;
+ }
+
+ /// Returns true if every parameter is at its default value, i.e. the
+ /// attribute describes the default floating-point environment and imposes
+ /// no constraints. Because the builder normalizes defaults to "unset", this
+ /// is equivalent to all parameters being unset.
+ bool isDefault() const {
+ return !getDynamicRoundingMode() && !getExceptMode() && !getStrictExcept();
+ }
+ }];
+}
+
#endif // ARITH_BASE
diff --git a/mlir/include/mlir/Dialect/Arith/IR/ArithOps.td b/mlir/include/mlir/Dialect/Arith/IR/ArithOps.td
index 423948c8734af..c383e8d8cd1b9 100644
--- a/mlir/include/mlir/Dialect/Arith/IR/ArithOps.td
+++ b/mlir/include/mlir/Dialect/Arith/IR/ArithOps.td
@@ -93,6 +93,11 @@ class Arith_FloatBinaryOp<string mnemonic, list<Trait> traits = []> :
// Base class for floating point binary operations with an optional rounding
// mode.
+//
+// Deprecated: use `Arith_FloatBinaryOpWithFenv` instead. This class is retained
+// temporarily while floating-point operations migrate to the `#arith.fenv`
+// floating-point environment attribute, and will be removed once the migration
+// is complete.
class Arith_FloatBinaryOpWithRoundingMode<string mnemonic,
list<Trait> traits = []> :
Arith_FloatBinaryOp<mnemonic,
@@ -128,6 +133,50 @@ class Arith_FloatBinaryOpWithRoundingMode<string mnemonic,
attr-dict `:` type($result) }];
}
+// Base class for floating point binary operations that can carry an optional
+// floating-point environment (`#arith.fenv`) attribute. It extends
+// `Arith_FloatBinaryOpWithRoundingMode` with the `fenv` attribute and the
+// `FPEnvConstrainedOpInterface`, which provides a uniform API for querying the
+// floating-point environment constraints.
+class Arith_FloatBinaryOpWithFenv<string mnemonic, list<Trait> traits = []> :
+ Arith_FloatBinaryOpWithRoundingMode<mnemonic,
+ !listconcat([DeclareOpInterfaceMethods<FPEnvConstrainedOpInterface>],
+ traits)> {
+ let arguments = (ins FloatLike:$lhs, FloatLike:$rhs,
+ DefaultValuedAttr<
+ Arith_FastMathAttr, "::mlir::arith::FastMathFlags::none">:$fastmath,
+ OptionalAttr<Arith_RoundingModeAttr>:$roundingmode,
+ OptionalAttr<Arith_FenvAttr>:$fenv);
+ // With two trailing optional attributes (`roundingmode` and `fenv`), the
+ // generated builders no longer provide C++ default arguments for them, so the
+ // convenience builders must pass both explicitly.
+ let builders = [
+ OpBuilder<(ins "Value":$lhs, "Value":$rhs,
+ CArg<"::mlir::arith::FastMathFlags",
+ "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+ build($_builder, $_state, lhs, rhs, fastmath,
+ ::mlir::arith::RoundingModeAttr{}, ::mlir::arith::FenvAttr{});
+ }]>,
+ OpBuilder<(ins "Value":$lhs, "Value":$rhs,
+ "::mlir::arith::FastMathFlagsAttr":$fastmath), [{
+ build($_builder, $_state, lhs, rhs, fastmath,
+ ::mlir::arith::RoundingModeAttr{}, ::mlir::arith::FenvAttr{});
+ }]>,
+ OpBuilder<(ins "Type":$type, "Value":$lhs, "Value":$rhs,
+ CArg<"::mlir::arith::FastMathFlags",
+ "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+ build($_builder, $_state, type, lhs, rhs,
+ ::mlir::arith::FastMathFlagsAttr::get(
+ $_builder.getContext(), fastmath),
+ ::mlir::arith::RoundingModeAttr{}, ::mlir::arith::FenvAttr{});
+ }]>,
+ ];
+ let assemblyFormat = [{ $lhs `,` $rhs ($roundingmode^)?
+ (`fastmath` `` $fastmath^)?
+ (`fenv` `` $fenv^)?
+ attr-dict `:` type($result) }];
+}
+
// Checks that tensor input and outputs have identical shapes. This is stricker
// than the verification done in `SameOperandsAndResultShape` that allows for
// tensor dimensions to be 'compatible' (e.g., dynamic dimensions being
@@ -1089,7 +1138,7 @@ def Arith_FlushDenormalsOp : Arith_FloatUnaryOp<"flush_denormals"> {
// AddFOp
//===----------------------------------------------------------------------===//
-def Arith_AddFOp : Arith_FloatBinaryOpWithRoundingMode<"addf", [Commutative]> {
+def Arith_AddFOp : Arith_FloatBinaryOpWithFenv<"addf", [Commutative]> {
let summary = "floating point addition operation";
let description = [{
The `addf` operation takes two operands and returns one result, each of
@@ -1120,7 +1169,7 @@ def Arith_AddFOp : Arith_FloatBinaryOpWithRoundingMode<"addf", [Commutative]> {
// SubFOp
//===----------------------------------------------------------------------===//
-def Arith_SubFOp : Arith_FloatBinaryOpWithRoundingMode<"subf"> {
+def Arith_SubFOp : Arith_FloatBinaryOpWithFenv<"subf"> {
let summary = "floating point subtraction operation";
let description = [{
The `subf` operation takes two operands and returns one result, each of
@@ -1276,7 +1325,7 @@ def Arith_MinUIOp : Arith_TotalIntBinaryOp<"minui",
// MulFOp
//===----------------------------------------------------------------------===//
-def Arith_MulFOp : Arith_FloatBinaryOpWithRoundingMode<"mulf", [Commutative]> {
+def Arith_MulFOp : Arith_FloatBinaryOpWithFenv<"mulf", [Commutative]> {
let summary = "floating point multiplication operation";
let description = [{
The `mulf` operation takes two operands and returns one result, each of
@@ -1308,7 +1357,7 @@ def Arith_MulFOp : Arith_FloatBinaryOpWithRoundingMode<"mulf", [Commutative]> {
// DivFOp
//===----------------------------------------------------------------------===//
-def Arith_DivFOp : Arith_FloatBinaryOpWithRoundingMode<"divf"> {
+def Arith_DivFOp : Arith_FloatBinaryOpWithFenv<"divf"> {
let summary = "floating point division operation";
let description = [{
The `divf` operation takes two operands and returns one result, each of
@@ -1566,10 +1615,17 @@ def Arith_TruncFOp :
[Pure, SameOperandsAndResultShape, SameInputOutputTensorDims,
DeclareOpInterfaceMethods<ArithRoundingModeInterface>,
DeclareOpInterfaceMethods<ArithFastMathInterface>,
+ DeclareOpInterfaceMethods<FPEnvConstrainedOpInterface>,
DeclareOpInterfaceMethods<CastOpInterface>]>,
Arguments<(ins FloatLike:$in,
+ // Deprecated: represent the rounding mode through the `fenv`
+ // attribute instead. This attribute is retained temporarily
+ // while floating-point operations migrate to the
+ // `#arith.fenv` floating-point environment attribute, and
+ // will be removed once the migration is complete.
OptionalAttr<Arith_RoundingModeAttr>:$roundingmode,
- OptionalAttr<Arith_FastMathAttr>:$fastmath)>,
+ OptionalAttr<Arith_FastMathAttr>:$fastmath,
+ OptionalAttr<Arith_FenvAttr>:$fenv)>,
Results<(outs FloatLike:$out)> {
let summary = "cast from floating-point to narrower floating-point";
let description = [{
@@ -1589,6 +1645,7 @@ def Arith_TruncFOp :
let hasVerifier = 1;
let assemblyFormat = [{ $in ($roundingmode^)?
(`fastmath` `` $fastmath^)?
+ (`fenv` `` $fenv^)?
attr-dict `:` type($in) `to` type($out) }];
}
@@ -1601,10 +1658,17 @@ def Arith_ConvertFOp :
[Pure, SameOperandsAndResultShape, SameInputOutputTensorDims,
DeclareOpInterfaceMethods<ArithRoundingModeInterface>,
DeclareOpInterfaceMethods<ArithFastMathInterface>,
+ DeclareOpInterfaceMethods<FPEnvConstrainedOpInterface>,
DeclareOpInterfaceMethods<CastOpInterface>]>,
Arguments<(ins FloatLike:$in,
+ // Deprecated: represent the rounding mode through the `fenv`
+ // attribute instead. This attribute is retained temporarily
+ // while floating-point operations migrate to the
+ // `#arith.fenv` floating-point environment attribute, and
+ // will be removed once the migration is complete.
OptionalAttr<Arith_RoundingModeAttr>:$roundingmode,
- OptionalAttr<Arith_FastMathAttr>:$fastmath)>,
+ OptionalAttr<Arith_FastMathAttr>:$fastmath,
+ OptionalAttr<Arith_FenvAttr>:$fenv)>,
Results<(outs FloatLike:$out)> {
let summary = "cast between floating-point types of the same bitwidth";
let description = [{
@@ -1621,6 +1685,7 @@ def Arith_ConvertFOp :
let hasVerifier = 1;
let assemblyFormat = [{ $in ($roundingmode^)?
(`fastmath` `` $fastmath^)?
+ (`fenv` `` $fenv^)?
attr-dict `:` type($in) `to` type($out) }];
}
@@ -1633,10 +1698,17 @@ def Arith_ScalingTruncFOp
[Pure, SameInputOutputTensorDims,
DeclareOpInterfaceMethods<ArithRoundingModeInterface>,
DeclareOpInterfaceMethods<ArithFastMathInterface>,
+ DeclareOpInterfaceMethods<FPEnvConstrainedOpInterface>,
DeclareOpInterfaceMethods<CastOpInterface>]>,
Arguments<(ins FloatLike:$in, FloatLike:$scale,
+ // Deprecated: represent the rounding mode through the `fenv`
+ // attribute instead. This attribute is retained temporarily while
+ // floating-point operations migrate to the `#arith.fenv`
+ // floating-point environment attribute, and will be removed once the
+ // migration is complete.
OptionalAttr<Arith_RoundingModeAttr>:$roundingmode,
- OptionalAttr<Arith_FastMathAttr>:$fastmath)>,
+ OptionalAttr<Arith_FastMathAttr>:$fastmath,
+ OptionalAttr<Arith_FenvAttr>:$fenv)>,
Results<(outs FloatLike:$out)> {
let summary = "Downcasts input floating point values using provided scales "
"values following OCP MXFP Spec";
@@ -1690,7 +1762,8 @@ def Arith_ScalingTruncFOp
}];
let hasVerifier = 1;
let assemblyFormat =
- [{ $in `,` $scale ($roundingmode^)? (`fastmath` `` $fastmath^)? attr-dict `:`
+ [{ $in `,` $scale ($roundingmode^)? (`fastmath` `` $fastmath^)?
+ (`fenv` `` $fenv^)? attr-dict `:`
type($in) `,` type($scale) `to` type($out)}];
}
diff --git a/mlir/include/mlir/Dialect/Arith/IR/ArithOpsInterfaces.td b/mlir/include/mlir/Dialect/Arith/IR/ArithOpsInterfaces.td
index d1b8e250cdb59..204bb32f89f4a 100644
--- a/mlir/include/mlir/Dialect/Arith/IR/ArithOpsInterfaces.td
+++ b/mlir/include/mlir/Dialect/Arith/IR/ArithOpsInterfaces.td
@@ -186,4 +186,87 @@ def ArithRoundingModeInterface : OpInterface<"ArithRoundingModeInterface"> {
];
}
+def FPEnvConstrainedOpInterface : OpInterface<"FPEnvConstrainedOpInterface"> {
+ let description = [{
+ This interface is implemented by floating-point operations that can carry an
+ optional `#arith.fenv` attribute describing constraints on their
+ floating-point environment. It provides a uniform API for querying the
+ attribute and its individual fields.
+
+ When the attribute is present, the operation is lowered to the corresponding
+ `llvm.experimental.constrained.*` intrinsic during translation to LLVM IR.
+ }];
+
+ let cppNamespace = "::mlir::arith";
+
+ let verify = [{
+ auto fenvOp = ::llvm::cast<::mlir::arith::FPEnvConstrainedOpInterface>($_op);
+ if (fenvOp.getFenvAttr()) {
+ if (auto roundingOp =
+ ::llvm::dyn_cast<::mlir::arith::ArithRoundingModeInterface>($_op))
+ if (roundingOp.getRoundingModeAttr())
+ return $_op->emitOpError("The `roundingmode` attribute is deprecated."
+ " Represent the rounding mode through the "
+ "`fenv` attribute instead.");
+ }
+ return ::mlir::success();
+ }];
+
+ let methods = [
+ InterfaceMethod<
+ /*desc=*/ [{Returns the `#arith.fenv` attribute attached to the
+ operation, or a null attribute if none is present.}],
+ /*returnType=*/ "::mlir::arith::FenvAttr",
+ /*methodName=*/ "getFenvAttr",
+ /*args=*/ (ins),
+ /*methodBody=*/ [{}],
+ /*defaultImpl=*/ [{
+ auto op = cast<ConcreteOp>(this->getOperation());
+ return op.getFenvAttr();
+ }]
+ >,
+ InterfaceMethod<
+ /*desc=*/ [{Returns the effective known dynamic rounding mode
+ requested by the attribute, substituting the default
+ (`unknown`) when the attribute is absent or does not set
+ the field.}],
+ /*returnType=*/ "::mlir::arith::RoundingMode",
+ /*methodName=*/ "getFenvDynamicRoundingMode",
+ /*args=*/ (ins),
+ /*methodBody=*/ [{}],
+ /*defaultImpl=*/ [{
+ ::mlir::arith::FenvAttr attr = $_op.getFenvAttr();
+ return attr ? attr.getDynamicRoundingModeOrDefault()
+ : ::mlir::arith::FenvAttr::getDefaultDynamicRoundingMode();
+ }]
+ >,
+ InterfaceMethod<
+ /*desc=*/ [{Returns the effective known exception mode requested by
+ the attribute, substituting the default (`masked`) when
+ the attribute is absent or does not set the field.}],
+ /*returnType=*/ "::mlir::arith::FPExceptionMode",
+ /*methodName=*/ "getFenvExceptionMode",
+ /*args=*/ (ins),
+ /*methodBody=*/ [{}],
+ /*defaultImpl=*/ [{
+ ::mlir::arith::FenvAttr attr = $_op.getFenvAttr();
+ return attr ? attr.getExceptionModeOrDefault()
+ : ::mlir::arith::FenvAttr::getDefaultExceptionMode();
+ }]
+ >,
+ InterfaceMethod<
+ /*desc=*/ [{Returns the effective strict-exception flag requested
+ by the attribute.}],
+ /*returnType=*/ "bool",
+ /*methodName=*/ "getFenvStrictExcept",
+ /*args=*/ (ins),
+ /*methodBody=*/ [{}],
+ /*defaultImpl=*/ [{
+ ::mlir::arith::FenvAttr attr = $_op.getFenvAttr();
+ return attr ? attr.getStrictExceptOrDefault() : false;
+ }]
+ >
+ ];
+}
+
#endif // ARITH_OPS_INTERFACES
diff --git a/mlir/include/mlir/Dialect/Math/IR/MathOps.td b/mlir/include/mlir/Dialect/Math/IR/MathOps.td
index 926bac1de2b0d..21c49484610a0 100644
--- a/mlir/include/mlir/Dialect/Math/IR/MathOps.td
+++ b/mlir/include/mlir/Dialect/Math/IR/MathOps.td
@@ -112,6 +112,11 @@ class Math_FloatTernaryOp<string mnemonic, list<Trait> traits = []> :
// Base class for floating point ternary operations with an optional rounding
// mode.
+//
+// Deprecated: use `Math_FloatTernaryOpWithFenv` instead. This class is retained
+// temporarily while floating-point operations migrate to the `#arith.fenv`
+// floating-point environment attribute, and will be removed once the migration
+// is complete.
class Math_FloatTernaryOpWithRoundingMode<string mnemonic,
list<Trait> traits = []> :
Math_FloatTernaryOp<mnemonic,
@@ -147,6 +152,52 @@ class Math_FloatTernaryOpWithRoundingMode<string mnemonic,
attr-dict `:` type($result) }];
}
+// Base class for floating point ternary operations that can carry an optional
+// floating-point environment (`#arith.fenv`) attribute. It extends
+// `Math_FloatTernaryOpWithRoundingMode` with the `fenv` attribute and the
+// `FPEnvConstrainedOpInterface`, which provides a uniform API for querying the
+// floating-point environment constraints.
+class Math_FloatTernaryOpWithFenv<string mnemonic, list<Trait> traits = []> :
+ Math_FloatTernaryOpWithRoundingMode<mnemonic,
+ !listconcat([DeclareOpInterfaceMethods<FPEnvConstrainedOpInterface>],
+ traits)> {
+ let arguments = (ins FloatLike:$a, FloatLike:$b, FloatLike:$c,
+ DefaultValuedAttr<Arith_FastMathAttr,
+ "::mlir::arith::FastMathFlags::none">:$fastmath,
+ // Deprecated: represent the rounding mode through the `fenv` attribute
+ // instead.
+ OptionalAttr<Arith_RoundingModeAttr>:$roundingmode,
+ OptionalAttr<Arith_FenvAttr>:$fenv);
+ // With two trailing optional attributes (`roundingmode` and `fenv`), the
+ // generated builders no longer provide C++ default arguments for them, so the
+ // convenience builders must pass both explicitly.
+ let builders = [
+ OpBuilder<(ins "Value":$a, "Value":$b, "Value":$c,
+ CArg<"::mlir::arith::FastMathFlags",
+ "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+ build($_builder, $_state, a, b, c, fastmath,
+ ::mlir::arith::RoundingModeAttr{}, ::mlir::arith::FenvAttr{});
+ }]>,
+ OpBuilder<(ins "Value":$a, "Value":$b, "Value":$c,
+ "::mlir::arith::FastMathFlagsAttr":$fastmath), [{
+ build($_builder, $_state, a, b, c, fastmath,
+ ::mlir::arith::RoundingModeAttr{}, ::mlir::arith::FenvAttr{});
+ }]>,
+ OpBuilder<(ins "Type":$type, "Value":$a, "Value":$b, "Value":$c,
+ CArg<"::mlir::arith::FastMathFlags",
+ "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+ build($_builder, $_state, type, a, b, c,
+ ::mlir::arith::FastMathFlagsAttr::get(
+ $_builder.getContext(), fastmath),
+ ::mlir::arith::RoundingModeAttr{}, ::mlir::arith::FenvAttr{});
+ }]>,
+ ];
+ let assemblyFormat = [{ $a `,` $b `,` $c ($roundingmode^)?
+ (`fastmath` `` $fastmath^)?
+ (`fenv` `` $fenv^)?
+ attr-dict `:` type($result) }];
+}
+
//===----------------------------------------------------------------------===//
// AbsFOp
//===----------------------------------------------------------------------===//
@@ -787,7 +838,7 @@ def Math_FloorOp : Math_FloatUnaryOp<"floor"> {
// FmaOp
//===----------------------------------------------------------------------===//
-def Math_FmaOp : Math_FloatTernaryOpWithRoundingMode<"fma"> {
+def Math_FmaOp : Math_FloatTernaryOpWithFenv<"fma"> {
let summary = "floating point fused multipy-add operation";
let description = [{
The `fma` operation takes three operands and returns one result, each of
diff --git a/mlir/lib/Conversion/ArithCommon/AttrToLLVMConverter.cpp b/mlir/lib/Conversion/ArithCommon/AttrToLLVMConverter.cpp
index cf60a048f782c..b3f5705601cfb 100644
--- a/mlir/lib/Conversion/ArithCommon/AttrToLLVMConverter.cpp
+++ b/mlir/lib/Conversion/ArithCommon/AttrToLLVMConverter.cpp
@@ -62,6 +62,8 @@ mlir::arith::convertArithRoundingModeToLLVM(arith::RoundingMode roundingMode) {
return LLVM::RoundingMode::TowardZero;
case arith::RoundingMode::upward:
return LLVM::RoundingMode::TowardPositive;
+ case arith::RoundingMode::unknown:
+ return LLVM::RoundingMode::Dynamic;
}
llvm_unreachable("Unhandled rounding mode");
}
diff --git a/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td b/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td
index 86c9b6d3551d8..74f1b2ab44dde 100644
--- a/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td
+++ b/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td
@@ -486,17 +486,21 @@ def TruncIShrSIToTrunciShrUI :
// TruncFOp
//===----------------------------------------------------------------------===//
-// truncf(sitofp(x)) -> sitofp(x) if default rounding mode.
+// truncf(sitofp(x)) -> sitofp(x) if default rounding mode and no fenv
+// constraints.
def TruncFSIToFPToSIToFP :
- Pat<(Arith_TruncFOp:$tr (Arith_SIToFPOp:$fp $x), $rmf, $fmf),
+ Pat<(Arith_TruncFOp:$tr (Arith_SIToFPOp:$fp $x), $rmf, $fmf, $fenv),
(Arith_SIToFPOp $x),
- [(Constraint<CPred<"$0 == nullptr">, "default rounding mode"> $rmf)]>;
+ [(Constraint<CPred<"$0 == nullptr">, "default rounding mode"> $rmf),
+ (Constraint<CPred<"$0 == nullptr">, "no fenv constraints"> $fenv)]>;
-// truncf(uitofp(x)) -> uitofp(x) if default rounding mode.
+// truncf(uitofp(x)) -> uitofp(x) if default rounding mode and no fenv
+// constraints.
def TruncFUIToFPToUIToFP :
- Pat<(Arith_TruncFOp:$tr (Arith_UIToFPOp:$fp $x, $nneg), $rmf, $fmf),
+ Pat<(Arith_TruncFOp:$tr (Arith_UIToFPOp:$fp $x, $nneg), $rmf, $fmf, $fenv),
(Arith_UIToFPOp $x, $nneg),
- [(Constraint<CPred<"$0 == nullptr">, "default rounding mode"> $rmf)]>;
+ [(Constraint<CPred<"$0 == nullptr">, "default rounding mode"> $rmf),
+ (Constraint<CPred<"$0 == nullptr">, "no fenv constraints"> $fenv)]>;
//===----------------------------------------------------------------------===//
// SIToFPOp
@@ -534,10 +538,12 @@ def IsNegZeroFloat :
// TODO: Verify if this canonicalization is safe when a rounding mode is
// specified. For the moment, bail on custom rounding modes.
def SubFOfNegZero :
- Pat<(Arith_SubFOp (ConstantLikeMatcher AnyAttr:$c), $x, $fmf, $rm),
+ Pat<(Arith_SubFOp (ConstantLikeMatcher AnyAttr:$c), $x, $fmf, $rm, $fenv),
(Arith_NegFOp $x, $fmf),
[(IsNegZeroFloat $c),
- (Constraint<CPred<"$0 == nullptr">, "default rounding mode"> $rm)]>;
+ (Constraint<CPred<"$0 == nullptr">, "default rounding mode"> $rm),
+ (Constraint<CPred<"$0 == nullptr">,
+ "default floating-point environment"> $fenv)]>;
//===----------------------------------------------------------------------===//
// MulFOp
@@ -547,10 +553,13 @@ def SubFOfNegZero :
// TODO: Verify if this canonicalization is safe when a rounding mode is
// specified. For the moment, bail on custom rounding modes.
def MulFOfNegF :
- Pat<(Arith_MulFOp (Arith_NegFOp $x, $_), (Arith_NegFOp $y, $_), $fmf, $rm),
- (Arith_MulFOp $x, $y, $fmf, $rm),
+ Pat<(Arith_MulFOp (Arith_NegFOp $x, $_), (Arith_NegFOp $y, $_), $fmf, $rm,
+ $fenv),
+ (Arith_MulFOp $x, $y, $fmf, $rm, $fenv),
[(TypesMatch $x, $y),
- (Constraint<CPred<"$0 == nullptr">, "default rounding mode"> $rm)]>;
+ (Constraint<CPred<"$0 == nullptr">, "default rounding mode"> $rm),
+ (Constraint<CPred<"$0 == nullptr">,
+ "default floating-point environment"> $fenv)]>;
//===----------------------------------------------------------------------===//
// DivFOp
@@ -560,9 +569,12 @@ def MulFOfNegF :
// TODO: Verify if this canonicalization is safe when a rounding mode is
// specified. For the moment, bail on custom rounding modes.
def DivFOfNegF :
- Pat<(Arith_DivFOp (Arith_NegFOp $x, $_), (Arith_NegFOp $y, $_), $fmf, $rm),
- (Arith_DivFOp $x, $y, $fmf, $rm),
+ Pat<(Arith_DivFOp (Arith_NegFOp $x, $_), (Arith_NegFOp $y, $_), $fmf, $rm,
+ $fenv),
+ (Arith_DivFOp $x, $y, $fmf, $rm, $fenv),
[(TypesMatch $x, $y),
- (Constraint<CPred<"$0 == nullptr">, "default rounding mode"> $rm)]>;
+ (Constraint<CPred<"$0 == nullptr">, "default rounding mode"> $rm),
+ (Constraint<CPred<"$0 == nullptr">,
+ "default floating-point environment"> $fenv)]>;
#endif // ARITH_PATTERNS
diff --git a/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp b/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
index c9217c57a5f25..85c6d82cffe5c 100644
--- a/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
+++ b/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
@@ -485,7 +485,7 @@ struct F8E8M0ExtFOpConverter : public OpRewritePattern<arith::ExtFOp> {
Value result = arith::BitcastOp::create(b, f32Ty, f32Bits);
if (resultETy.getIntOrFloatBitWidth() < 32) {
result = arith::TruncFOp::create(b, resultTy, result, nullptr,
- op.getFastmathAttr());
+ op.getFastmathAttr(), nullptr);
} else if (resultETy.getIntOrFloatBitWidth() > 32) {
result = arith::ExtFOp::create(b, resultTy, result, op.getFastmathAttr());
}
@@ -649,8 +649,9 @@ struct F8E8M0TruncFOpConverter : public OpRewritePattern<arith::TruncFOp> {
if (operandETy.getIntOrFloatBitWidth() < 32) {
operand = arith::ExtFOp::create(b, f32Ty, operand, op.getFastmathAttr());
} else if (operandETy.getIntOrFloatBitWidth() > 32) {
- operand = arith::TruncFOp::create(
- b, f32Ty, operand, op.getRoundingmodeAttr(), op.getFastmathAttr());
+ operand = arith::TruncFOp::create(b, f32Ty, operand,
+ op.getRoundingmodeAttr(),
+ op.getFastmathAttr(), op.getFenvAttr());
}
Value f32Bits = arith::BitcastOp::create(b, i32Ty, operand);
Value cF32MantissaWidth = createConst(op->getLoc(), i32Ty, 23, rewriter);
@@ -676,7 +677,7 @@ struct ScalingExtFOpConverter : public OpRewritePattern<arith::ScalingExtFOp> {
scaleETy = b.getF8E8M0Type();
scaleTy = cloneToShapedType(scaleTy, scaleETy);
scaleOperand = arith::TruncFOp::create(b, scaleTy, scaleOperand, nullptr,
- op.getFastmathAttr());
+ op.getFastmathAttr(), nullptr);
}
// Catch scale types like f8E5M2.
if (!llvm::isa<Float8E8M0FNUType>(scaleETy)) {
@@ -718,7 +719,7 @@ struct ScalingTruncFOpConverter
scaleETy = b.getF8E8M0Type();
scaleTy = cloneToShapedType(scaleTy, scaleETy);
scaleOperand = arith::TruncFOp::create(b, scaleTy, scaleOperand, nullptr,
- op.getFastmathAttr());
+ op.getFastmathAttr(), nullptr);
}
if (!llvm::isa<Float8E8M0FNUType>(scaleETy)) {
return rewriter.notifyMatchFailure(
@@ -734,7 +735,8 @@ struct ScalingTruncFOpConverter
Value result = arith::DivFOp::create(b, inputOperand, scaleOperand,
op.getFastmathAttr());
Value resultCast = arith::TruncFOp::create(
- b, resultTy, result, op.getRoundingmodeAttr(), op.getFastmathAttr());
+ b, resultTy, result, op.getRoundingmodeAttr(), op.getFastmathAttr(),
+ op.getFenvAttr());
rewriter.replaceOp(op, resultCast);
return success();
}
diff --git a/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir b/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
index 43f21561e6544..5d24ab17ee8c3 100644
--- a/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
+++ b/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
@@ -396,6 +396,8 @@ func.func @experimental_constrained_fptrunc(%arg0 : f64) {
%3 = arith.truncf %arg0 toward_zero : f64 to f32
// CHECK-NEXT: = llvm.intr.experimental.constrained.fptrunc {{.*}} tonearestaway ignore : f64 to f32
%4 = arith.truncf %arg0 to_nearest_away : f64 to f32
+// CHECK-NEXT: = llvm.intr.experimental.constrained.fptrunc {{.*}} dynamic ignore : f64 to f32
+ %5 = arith.truncf %arg0 unknown : f64 to f32
return
}
diff --git a/mlir/test/Dialect/Arith/invalid.mlir b/mlir/test/Dialect/Arith/invalid.mlir
index 49f55e855663d..4829402c4e6a9 100644
--- a/mlir/test/Dialect/Arith/invalid.mlir
+++ b/mlir/test/Dialect/Arith/invalid.mlir
@@ -1152,3 +1152,11 @@ func.func @convertf_int_output(%arg0 : f32) {
%0 = arith.convertf %arg0 : f32 to i32
return
}
+
+// -----
+
+func.func @fenv_with_rounding_mode(%arg0 : f32, %arg1 : f32) {
+ // expected-error @+1 {{'arith.addf' op The `roundingmode` attribute is deprecated. Represent the rounding mode through the `fenv` attribute instead.}}
+ %0 = arith.addf %arg0, %arg1 to_nearest_even fenv<except_mode = unmasked> : f32
+ return
+}
diff --git a/mlir/test/Dialect/Arith/ops.mlir b/mlir/test/Dialect/Arith/ops.mlir
index 76aef203a64fb..ba416c78fc3df 100644
--- a/mlir/test/Dialect/Arith/ops.mlir
+++ b/mlir/test/Dialect/Arith/ops.mlir
@@ -1305,6 +1305,29 @@ func.func @roundingmode(%arg0: f32, %arg1: f32) {
return
}
+// CHECK-LABEL: @fenv
+func.func @fenv(%arg0: f32, %arg1: f32) {
+// CHECK: {{.*}} = arith.addf %arg0, %arg1 fenv<dynamic_rounding_mode = upward> : f32
+ %0 = arith.addf %arg0, %arg1 fenv<dynamic_rounding_mode = upward> : f32
+// CHECK: {{.*}} = arith.subf %arg0, %arg1 fenv<except_mode = unmasked, strict_except = true> : f32
+ %1 = arith.subf %arg0, %arg1 fenv<except_mode = unmasked, strict_except = true> : f32
+// CHECK: {{.*}} = arith.mulf %arg0, %arg1 fenv<dynamic_rounding_mode = downward, except_mode = unknown> : f32
+ %2 = arith.mulf %arg0, %arg1 fenv<dynamic_rounding_mode = downward, except_mode = unknown> : f32
+// CHECK: {{.*}} = arith.divf %arg0, %arg1 fenv<> : f32
+ %3 = arith.divf %arg0, %arg1 fenv<> : f32
+
+ // Parameters at their default values are normalized to an unset (canonical
+ // empty) attribute.
+// CHECK: {{.*}} = arith.addf %arg0, %arg1 fenv<> : f32
+ %4 = arith.addf %arg0, %arg1 fenv<dynamic_rounding_mode = unknown> : f32
+// CHECK: {{.*}} = arith.subf %arg0, %arg1 fenv<> : f32
+ %5 = arith.subf %arg0, %arg1 fenv<except_mode = masked> : f32
+// CHECK: {{.*}} = arith.mulf %arg0, %arg1 fenv<except_mode = unmasked> : f32
+ %6 = arith.mulf %arg0, %arg1 fenv<except_mode = unmasked, strict_except = false> : f32
+
+ return
+}
+
// CHECK-LABEL: @select_tensor
func.func @select_tensor(%arg0 : tensor<8xi1>, %arg1 : tensor<8xi32>, %arg2 : tensor<8xi32>) -> tensor<8xi32> {
// CHECK: = arith.select %{{.*}}, %{{.*}}, %{{.*}} : tensor<8xi1>, tensor<8xi32>
>From 95bf042c894277b2c0c106427500dc0994a2a3e4 Mon Sep 17 00:00:00 2001
From: Andy Kaylor <akaylor at nvidia.com>
Date: Fri, 26 Jun 2026 17:45:32 -0700
Subject: [PATCH 3/7] [mlir][arith] Lower operations with the fenv attribute
This change adds handling to lower operations in the Arith or Math dialects
with the fenv attribute to constrained fp operations in the LLVM dialect.
WIP!
---
.../ArithCommon/AttrToLLVMConverter.h | 59 ++++++++++--
.../ArithCommon/AttrToLLVMConverter.cpp | 15 +++
.../Conversion/ArithToLLVM/ArithToLLVM.cpp | 92 ++++++++++++++-----
mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp | 23 +++--
.../Dialect/Arith/Transforms/ExpandOps.cpp | 6 +-
.../Conversion/ArithToLLVM/arith-to-llvm.mlir | 80 ++++++++++++++++
.../Conversion/MathToLLVM/math-to-llvm.mlir | 18 ++++
mlir/test/Dialect/Arith/expand-ops.mlir | 13 +++
8 files changed, 259 insertions(+), 47 deletions(-)
diff --git a/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h b/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h
index feb74c86e349f..31d94daf4bb92 100644
--- a/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h
+++ b/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h
@@ -45,6 +45,18 @@ convertArithRoundingModeAttrToLLVM(arith::RoundingModeAttr roundingModeAttr);
LLVM::FPExceptionBehaviorAttr
getLLVMDefaultFPExceptionBehavior(MLIRContext &context);
+/// Maps an arith floating-point exception mode and strict-exception flag, as
+/// described by an `#arith.fenv` attribute, to the corresponding LLVM
+/// constrained-intrinsic exception behavior:
+///
+/// masked + strict_except=false -> fpexcept.ignore
+/// masked + strict_except=true -> fpexcept.strict
+/// unmasked|unknown + strict_except=false -> fpexcept.maytrap
+/// unmasked|unknown + strict_except=true -> fpexcept.strict
+LLVM::FPExceptionBehavior
+convertArithFPExceptionBehaviorToLLVM(arith::FPExceptionMode exceptionMode,
+ bool strictExcept);
+
// Attribute converter that populates a NamedAttrList by removing the fastmath
// attribute from the source operation attributes, and replacing it with an
// equivalent LLVM fastmath attribute.
@@ -145,25 +157,52 @@ class AttrConverterConstrainedFPToLLVM {
AttrConverterConstrainedFPToLLVM(SourceOp srcOp) {
// Copy the source attributes.
convertedAttr = NamedAttrList{srcOp->getAttrs()};
+ MLIRContext *ctx = srcOp->getContext();
+
+ // The floating-point environment may be described either by the deprecated
+ // `roundingmode` attribute or by the `#arith.fenv` attribute. Collect both
+ // (the verifier guarantees they are never set at the same time) and remove
+ // them from the attributes carried over to the target op.
+ auto roundingModeAttr = dyn_cast_if_present<arith::RoundingModeAttr>(
+ convertedAttr.erase(srcOp.getRoundingModeAttrName()));
+ arith::FenvAttr fenvAttr = srcOp.getFenvAttr();
+ convertedAttr.erase(srcOp.getFenvAttrName());
+
+ // Determine the rounding mode. The `fenv` attribute takes precedence; when
+ // neither source carries one, fall back to the default dynamic rounding
+ // mode.
+ [[maybe_unused]] arith::RoundingMode roundingMode =
+ arith::FenvAttr::getDefaultDynamicRoundingMode();
+ if (fenvAttr)
+ roundingMode = fenvAttr.getDynamicRoundingModeOrDefault();
+ else if (roundingModeAttr)
+ roundingMode = roundingModeAttr.getValue();
if constexpr (TargetOp::template hasTrait<
LLVM::RoundingModeOpInterface::Trait>()) {
- // Get the name of the rounding mode attribute.
- StringRef arithAttrName = srcOp.getRoundingModeAttrName();
- // Remove the source attribute.
- auto arithAttr =
- cast<arith::RoundingModeAttr>(convertedAttr.erase(arithAttrName));
- // Set the target attribute.
- convertedAttr.set(TargetOp::getRoundingModeAttrName(),
- convertArithRoundingModeAttrToLLVM(arithAttr));
+ convertedAttr.set(
+ TargetOp::getRoundingModeAttrName(),
+ LLVM::RoundingModeAttr::get(
+ ctx, convertArithRoundingModeToLLVM(roundingMode)));
}
// Constrained intrinsics (llvm.intr.experimental.constrained.*) do not
// support fastmath flags. Remove the arith fastmath attribute if present.
if constexpr (SourceOp::template hasTrait<
arith::ArithFastMathInterface::Trait>())
convertedAttr.erase(srcOp.getFastMathAttrName());
- convertedAttr.set(TargetOp::getFPExceptionBehaviorAttrName(),
- getLLVMDefaultFPExceptionBehavior(*srcOp->getContext()));
+
+ // Determine the exception behavior from the `fenv` attribute, defaulting to
+ // `ignore` when no environment is specified (e.g. only a `roundingmode`
+ // attribute is present).
+ LLVM::FPExceptionBehavior exceptionBehavior =
+ LLVM::FPExceptionBehavior::Ignore;
+ if (fenvAttr)
+ exceptionBehavior = convertArithFPExceptionBehaviorToLLVM(
+ fenvAttr.getExceptionModeOrDefault(),
+ fenvAttr.getStrictExceptOrDefault());
+ convertedAttr.set(
+ TargetOp::getFPExceptionBehaviorAttrName(),
+ LLVM::FPExceptionBehaviorAttr::get(ctx, exceptionBehavior));
}
ArrayRef<NamedAttribute> getAttrs() const { return convertedAttr.getAttrs(); }
diff --git a/mlir/lib/Conversion/ArithCommon/AttrToLLVMConverter.cpp b/mlir/lib/Conversion/ArithCommon/AttrToLLVMConverter.cpp
index b3f5705601cfb..b964acdf7a137 100644
--- a/mlir/lib/Conversion/ArithCommon/AttrToLLVMConverter.cpp
+++ b/mlir/lib/Conversion/ArithCommon/AttrToLLVMConverter.cpp
@@ -81,3 +81,18 @@ mlir::arith::getLLVMDefaultFPExceptionBehavior(MLIRContext &context) {
return LLVM::FPExceptionBehaviorAttr::get(&context,
LLVM::FPExceptionBehavior::Ignore);
}
+
+LLVM::FPExceptionBehavior mlir::arith::convertArithFPExceptionBehaviorToLLVM(
+ arith::FPExceptionMode exceptionMode, bool strictExcept) {
+ // A strict-exception requirement always maps to `strict`, which preserves
+ // both the side effects and the exact exception semantics regardless of the
+ // exception mode.
+ if (strictExcept)
+ return LLVM::FPExceptionBehavior::Strict;
+ // Without the strict requirement, a masked environment may ignore exceptions,
+ // while an unmasked or unknown environment must conservatively assume that
+ // traps may occur.
+ if (exceptionMode == arith::FPExceptionMode::Masked)
+ return LLVM::FPExceptionBehavior::Ignore;
+ return LLVM::FPExceptionBehavior::MayTrap;
+}
diff --git a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
index 6bf0fe85bb62f..fed991c7074d2 100644
--- a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
+++ b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
@@ -29,20 +29,21 @@ using namespace mlir;
namespace {
-/// Lowering pattern that matches only when the source op's rounding mode
-/// presence agrees with `HasRoundingMode`. This allows registering two
-/// instances of the same pattern for one source op: one that handles the
-/// unconstrained case (no rounding mode, lowering to a regular LLVM op) and
-/// one that handles the constrained case (rounding mode present, lowering to
-/// a constrained LLVM intrinsic).
+/// Lowering pattern that matches only when the source op's constrained
+/// floating-point environment presence agrees with `IsConstrained`. A source
+/// op is considered constrained when it carries either the deprecated
+/// `roundingmode` attribute or the `#arith.fenv` attribute. This allows
+/// registering two instances of the same pattern for one source op: one that
+/// handles the unconstrained case (lowering to a regular LLVM op) and one that
+/// handles the constrained case (lowering to a constrained LLVM intrinsic).
///
-/// * `HasRoundingMode`: the pattern matches if and only if the source op has
-/// a rounding mode attribute.
+/// * `IsConstrained`: the pattern matches if and only if the source op carries
+/// a `roundingmode` or `#arith.fenv` attribute.
/// * `AttrConvert`: attribute converter to translate source attributes to
/// target attributes.
/// * `FailOnUnsupportedFP`: whether to fail if the source op has unsupported
/// floating point types.
-template <typename SourceOp, typename TargetOp, bool HasRoundingMode,
+template <typename SourceOp, typename TargetOp, bool IsConstrained,
template <typename, typename> typename AttrConvert =
AttrConvertPassThrough,
bool FailOnUnsupportedFP = false>
@@ -56,7 +57,8 @@ struct ConstrainedVectorConvertToLLVMPattern
LogicalResult
matchAndRewrite(SourceOp op, typename SourceOp::Adaptor adaptor,
ConversionPatternRewriter &rewriter) const override {
- if (HasRoundingMode != static_cast<bool>(op.getRoundingModeAttr()))
+ bool opIsConstrained = op.getRoundingModeAttr() || op.getFenvAttr();
+ if (IsConstrained != opIsConstrained)
return failure();
return VectorConvertToLLVMPattern<
SourceOp, TargetOp, AttrConvert,
@@ -89,11 +91,11 @@ struct IdentityBitcastLowering final
using AddFOpLowering =
ConstrainedVectorConvertToLLVMPattern<arith::AddFOp, LLVM::FAddOp,
- /*HasRoundingMode=*/false,
+ /*IsConstrained=*/false,
arith::AttrConvertFastMathToLLVM,
/*FailOnUnsupportedFP=*/true>;
using ConstrainedAddFOpLowering = ConstrainedVectorConvertToLLVMPattern<
- arith::AddFOp, LLVM::ConstrainedFAddIntr, /*HasRoundingMode=*/true,
+ arith::AddFOp, LLVM::ConstrainedFAddIntr, /*IsConstrained=*/true,
arith::AttrConverterConstrainedFPToLLVM, /*FailOnUnsupportedFP=*/true>;
using AddIOpLowering =
VectorConvertToLLVMPattern<arith::AddIOp, LLVM::AddOp,
@@ -103,11 +105,11 @@ using BitcastOpLowering =
VectorConvertToLLVMPattern<arith::BitcastOp, LLVM::BitcastOp>;
using DivFOpLowering =
ConstrainedVectorConvertToLLVMPattern<arith::DivFOp, LLVM::FDivOp,
- /*HasRoundingMode=*/false,
+ /*IsConstrained=*/false,
arith::AttrConvertFastMathToLLVM,
/*FailOnUnsupportedFP=*/true>;
using ConstrainedDivFOpLowering = ConstrainedVectorConvertToLLVMPattern<
- arith::DivFOp, LLVM::ConstrainedFDivIntr, /*HasRoundingMode=*/true,
+ arith::DivFOp, LLVM::ConstrainedFDivIntr, /*IsConstrained=*/true,
arith::AttrConverterConstrainedFPToLLVM, /*FailOnUnsupportedFP=*/true>;
using DivSIOpLowering =
VectorConvertToLLVMPattern<arith::DivSIOp, LLVM::SDivOp>;
@@ -155,11 +157,11 @@ using MinUIOpLowering =
VectorConvertToLLVMPattern<arith::MinUIOp, LLVM::UMinOp>;
using MulFOpLowering =
ConstrainedVectorConvertToLLVMPattern<arith::MulFOp, LLVM::FMulOp,
- /*HasRoundingMode=*/false,
+ /*IsConstrained=*/false,
arith::AttrConvertFastMathToLLVM,
/*FailOnUnsupportedFP=*/true>;
using ConstrainedMulFOpLowering = ConstrainedVectorConvertToLLVMPattern<
- arith::MulFOp, LLVM::ConstrainedFMulIntr, /*HasRoundingMode=*/true,
+ arith::MulFOp, LLVM::ConstrainedFMulIntr, /*IsConstrained=*/true,
arith::AttrConverterConstrainedFPToLLVM, /*FailOnUnsupportedFP=*/true>;
using MulIOpLowering =
VectorConvertToLLVMPattern<arith::MulIOp, LLVM::MulOp,
@@ -190,22 +192,22 @@ using SIToFPOpLowering =
VectorConvertToLLVMPattern<arith::SIToFPOp, LLVM::SIToFPOp>;
using SubFOpLowering =
ConstrainedVectorConvertToLLVMPattern<arith::SubFOp, LLVM::FSubOp,
- /*HasRoundingMode=*/false,
+ /*IsConstrained=*/false,
arith::AttrConvertFastMathToLLVM,
/*FailOnUnsupportedFP=*/true>;
using ConstrainedSubFOpLowering = ConstrainedVectorConvertToLLVMPattern<
- arith::SubFOp, LLVM::ConstrainedFSubIntr, /*HasRoundingMode=*/true,
+ arith::SubFOp, LLVM::ConstrainedFSubIntr, /*IsConstrained=*/true,
arith::AttrConverterConstrainedFPToLLVM, /*FailOnUnsupportedFP=*/true>;
using SubIOpLowering =
VectorConvertToLLVMPattern<arith::SubIOp, LLVM::SubOp,
arith::AttrConvertOverflowToLLVM>;
using TruncFOpLowering =
ConstrainedVectorConvertToLLVMPattern<arith::TruncFOp, LLVM::FPTruncOp,
- /*HasRoundingMode=*/false,
+ /*IsConstrained=*/false,
AttrConvertPassThrough,
/*FailOnUnsupportedFP=*/true>;
using ConstrainedTruncFOpLowering = ConstrainedVectorConvertToLLVMPattern<
- arith::TruncFOp, LLVM::ConstrainedFPTruncIntr, /*HasRoundingMode=*/true,
+ arith::TruncFOp, LLVM::ConstrainedFPTruncIntr, /*IsConstrained=*/true,
arith::AttrConverterConstrainedFPToLLVM, /*FailOnUnsupportedFP=*/true>;
using TruncIOpLowering =
VectorConvertToLLVMPattern<arith::TruncIOp, LLVM::TruncOp,
@@ -300,6 +302,12 @@ struct CmpFOpLowering : public ConvertOpToLLVMPattern<arith::CmpFOp> {
/// Extends to f32 via llvm.fpext, then truncates to the target type via
/// llvm.fptrunc. This handles bf16 <-> f16, which is the only same-bitwidth
/// pair of LLVM-supported FP types.
+///
+/// When the op carries a constrained floating-point environment (through the
+/// deprecated `roundingmode` attribute or the `#arith.fenv` attribute), the two
+/// steps lower to the constrained intrinsics instead: the widening `fpext` is
+/// exact and only carries the exception behavior, while the narrowing `fptrunc`
+/// carries both the rounding mode and the exception behavior.
struct ConvertFOpLowering : public ConvertOpToLLVMPattern<arith::ConvertFOp> {
using ConvertOpToLLVMPattern::ConvertOpToLLVMPattern;
@@ -322,12 +330,33 @@ struct ConvertFOpLowering : public ConvertOpToLLVMPattern<arith::ConvertFOp> {
if (!convertedType)
return rewriter.notifyMatchFailure(op, "failed to convert result type");
+ // Derive the constrained floating-point environment, if any, from either
+ // the deprecated `roundingmode` attribute or the `#arith.fenv` attribute.
+ // The rounding mode only applies to the narrowing step; the exception
+ // behavior, when present, applies to both steps.
+ MLIRContext *ctx = rewriter.getContext();
+ LLVM::RoundingModeAttr roundingModeAttr;
+ LLVM::FPExceptionBehaviorAttr exceptionBehaviorAttr;
+ if (arith::FenvAttr fenvAttr = op.getFenvAttr()) {
+ roundingModeAttr = LLVM::RoundingModeAttr::get(
+ ctx, arith::convertArithRoundingModeToLLVM(
+ fenvAttr.getDynamicRoundingModeOrDefault()));
+ exceptionBehaviorAttr = LLVM::FPExceptionBehaviorAttr::get(
+ ctx, arith::convertArithFPExceptionBehaviorToLLVM(
+ fenvAttr.getExceptionModeOrDefault(),
+ fenvAttr.getStrictExceptOrDefault()));
+ } else if (arith::RoundingModeAttr rmAttr = op.getRoundingmodeAttr()) {
+ roundingModeAttr = arith::convertArithRoundingModeAttrToLLVM(rmAttr);
+ exceptionBehaviorAttr = arith::getLLVMDefaultFPExceptionBehavior(*ctx);
+ }
+
Value input = adaptor.getIn();
Location loc = op.getLoc();
if (!isa<LLVM::LLVMArrayType>(input.getType())) {
- rewriter.replaceOp(op,
- emitConversion(rewriter, loc, input, convertedType));
+ rewriter.replaceOp(op, emitConversion(rewriter, loc, input, convertedType,
+ roundingModeAttr,
+ exceptionBehaviorAttr));
return success();
}
@@ -337,20 +366,33 @@ struct ConvertFOpLowering : public ConvertOpToLLVMPattern<arith::ConvertFOp> {
return LLVM::detail::handleMultidimensionalVectors(
op.getOperation(), adaptor.getOperands(), *getTypeConverter(),
[&](Type llvm1DVectorTy, ValueRange operands) -> Value {
- return emitConversion(rewriter, loc, operands.front(),
- llvm1DVectorTy);
+ return emitConversion(rewriter, loc, operands.front(), llvm1DVectorTy,
+ roundingModeAttr, exceptionBehaviorAttr);
},
rewriter);
}
private:
static Value emitConversion(ConversionPatternRewriter &rewriter, Location loc,
- Value input, Type targetType) {
+ Value input, Type targetType,
+ LLVM::RoundingModeAttr roundingModeAttr,
+ LLVM::FPExceptionBehaviorAttr
+ exceptionBehaviorAttr) {
Type f32Scalar = Float32Type::get(rewriter.getContext());
Type f32Ty = f32Scalar;
if (auto vecTy = dyn_cast<VectorType>(targetType))
f32Ty = VectorType::get(vecTy.getShape(), f32Scalar);
+ // A constrained floating-point environment was requested: emit the
+ // constrained intrinsics carrying the rounding mode and exception behavior.
+ if (exceptionBehaviorAttr) {
+ Value ext = LLVM::ConstrainedFPExtIntr::create(rewriter, loc, f32Ty, input,
+ exceptionBehaviorAttr);
+ return LLVM::ConstrainedFPTruncIntr::create(rewriter, loc, targetType, ext,
+ roundingModeAttr,
+ exceptionBehaviorAttr);
+ }
+
Value ext = LLVM::FPExtOp::create(rewriter, loc, f32Ty, input);
return LLVM::FPTruncOp::create(rewriter, loc, targetType, ext);
}
diff --git a/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp b/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp
index 1ed1ffd005d49..aca6973ab5956 100644
--- a/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp
+++ b/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp
@@ -37,13 +37,15 @@ using ConvertFMFMathToLLVMPattern =
VectorConvertToLLVMPattern<SourceOp, TargetOp, ConvertFastMath,
FailOnUnsupportedFP>;
-/// Lowering pattern that matches only when the source op's rounding mode
-/// presence agrees with `HasRoundingMode`. Mirrors the helper of the same
-/// name in `mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp`. This lets us
-/// register two patterns for one math op: an unconstrained one that lowers
-/// to a regular LLVM op, and a constrained one (rounding mode present) that
-/// lowers to an `llvm.intr.experimental.constrained.*` intrinsic.
-template <typename SourceOp, typename TargetOp, bool HasRoundingMode,
+/// Lowering pattern that matches only when the source op's constrained
+/// floating-point environment presence agrees with `IsConstrained`. A source
+/// op is considered constrained when it carries either the deprecated
+/// `roundingmode` attribute or the `#arith.fenv` attribute. Mirrors the helper
+/// of the same name in `mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp`. This
+/// lets us register two patterns for one math op: an unconstrained one that
+/// lowers to a regular LLVM op, and a constrained one that lowers to an
+/// `llvm.intr.experimental.constrained.*` intrinsic.
+template <typename SourceOp, typename TargetOp, bool IsConstrained,
template <typename, typename> typename AttrConvert =
AttrConvertPassThrough,
bool FailOnUnsupportedFP = true>
@@ -57,7 +59,8 @@ struct ConstrainedVectorConvertToLLVMPattern
LogicalResult
matchAndRewrite(SourceOp op, typename SourceOp::Adaptor adaptor,
ConversionPatternRewriter &rewriter) const override {
- if (HasRoundingMode != static_cast<bool>(op.getRoundingModeAttr()))
+ bool opIsConstrained = op.getRoundingModeAttr() || op.getFenvAttr();
+ if (IsConstrained != opIsConstrained)
return failure();
return VectorConvertToLLVMPattern<
SourceOp, TargetOp, AttrConvert,
@@ -84,11 +87,11 @@ using FloorOpLowering =
ConvertFMFMathToLLVMPattern<math::FloorOp, LLVM::FFloorOp>;
using FmaOpLowering =
ConstrainedVectorConvertToLLVMPattern<math::FmaOp, LLVM::FMAOp,
- /*HasRoundingMode=*/false,
+ /*IsConstrained=*/false,
ConvertFastMath,
/*FailOnUnsupportedFP=*/true>;
using ConstrainedFmaOpLowering = ConstrainedVectorConvertToLLVMPattern<
- math::FmaOp, LLVM::ConstrainedFMAIntr, /*HasRoundingMode=*/true,
+ math::FmaOp, LLVM::ConstrainedFMAIntr, /*IsConstrained=*/true,
arith::AttrConverterConstrainedFPToLLVM, /*FailOnUnsupportedFP=*/true>;
using Log10OpLowering =
ConvertFMFMathToLLVMPattern<math::Log10Op, LLVM::Log10Op>;
diff --git a/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp b/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
index 85c6d82cffe5c..b104a3b9711e9 100644
--- a/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
+++ b/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
@@ -719,7 +719,8 @@ struct ScalingTruncFOpConverter
scaleETy = b.getF8E8M0Type();
scaleTy = cloneToShapedType(scaleTy, scaleETy);
scaleOperand = arith::TruncFOp::create(b, scaleTy, scaleOperand, nullptr,
- op.getFastmathAttr(), nullptr);
+ op.getFastmathAttr(),
+ op.getFenvAttr());
}
if (!llvm::isa<Float8E8M0FNUType>(scaleETy)) {
return rewriter.notifyMatchFailure(
@@ -733,7 +734,8 @@ struct ScalingTruncFOpConverter
scaleOperand =
arith::ExtFOp::create(b, inputTy, scaleOperand, op.getFastmathAttr());
Value result = arith::DivFOp::create(b, inputOperand, scaleOperand,
- op.getFastmathAttr());
+ op.getFastmathAttr(), nullptr,
+ op.getFenvAttr());
Value resultCast = arith::TruncFOp::create(
b, resultTy, result, op.getRoundingmodeAttr(), op.getFastmathAttr(),
op.getFenvAttr());
diff --git a/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir b/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
index 5d24ab17ee8c3..20e92d826c6b3 100644
--- a/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
+++ b/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
@@ -465,6 +465,47 @@ func.func @constrained_addf_with_fastmath(%arg0 : f64, %arg1 : f64) {
// -----
+// Verify that the `#arith.fenv` attribute lowers to a constrained intrinsic,
+// mapping the dynamic rounding mode and the exception behavior. The exception
+// behavior depends on both `except_mode` and `strict_except`:
+// masked + strict_except=false -> ignore
+// masked + strict_except=true -> strict
+// unmasked|unknown + strict_except=false -> maytrap
+// unmasked|unknown + strict_except=true -> strict
+// CHECK-LABEL: experimental_constrained_fenv
+func.func @experimental_constrained_fenv(%arg0 : f64, %arg1 : f64) {
+// An empty fenv uses all defaults: dynamic rounding and ignored exceptions.
+// CHECK-NEXT: = llvm.intr.experimental.constrained.fadd %arg0, %arg1 dynamic ignore
+ %0 = arith.addf %arg0, %arg1 fenv<> : f64
+// masked (default) + non-strict (default) -> ignore
+// CHECK-NEXT: = llvm.intr.experimental.constrained.fadd %arg0, %arg1 tonearest ignore
+ %1 = arith.addf %arg0, %arg1 fenv<dynamic_rounding_mode = to_nearest_even> : f64
+// masked + strict -> strict
+// CHECK-NEXT: = llvm.intr.experimental.constrained.fadd %arg0, %arg1 upward strict
+ %2 = arith.addf %arg0, %arg1 fenv<dynamic_rounding_mode = upward, strict_except = true> : f64
+// unmasked + non-strict -> maytrap
+// CHECK-NEXT: = llvm.intr.experimental.constrained.fadd %arg0, %arg1 downward maytrap
+ %3 = arith.addf %arg0, %arg1 fenv<dynamic_rounding_mode = downward, except_mode = unmasked> : f64
+// unknown + non-strict -> maytrap
+// CHECK-NEXT: = llvm.intr.experimental.constrained.fadd %arg0, %arg1 towardzero maytrap
+ %4 = arith.addf %arg0, %arg1 fenv<dynamic_rounding_mode = toward_zero, except_mode = unknown> : f64
+// unmasked + strict -> strict
+// CHECK-NEXT: = llvm.intr.experimental.constrained.fadd %arg0, %arg1 tonearestaway strict
+ %5 = arith.addf %arg0, %arg1 fenv<dynamic_rounding_mode = to_nearest_away, except_mode = unmasked, strict_except = true> : f64
+ return
+}
+
+// -----
+
+// CHECK-LABEL: experimental_constrained_fptrunc_fenv
+func.func @experimental_constrained_fptrunc_fenv(%arg0 : f64) {
+// CHECK-NEXT: = llvm.intr.experimental.constrained.fptrunc %arg0 upward maytrap : f64 to f32
+ %0 = arith.truncf %arg0 fenv<dynamic_rounding_mode = upward, except_mode = unmasked> : f64 to f32
+ return
+}
+
+// -----
+
// CHECK-LABEL: @convertf_f16_to_bf16
func.func @convertf_f16_to_bf16(%arg0 : f16) -> bf16 {
// CHECK-NEXT: %[[EXT:.*]] = llvm.fpext %arg0 : f16 to f32
@@ -498,6 +539,45 @@ func.func @convertf_vector(%arg0 : vector<2xf16>) -> vector<2xbf16> {
// -----
+// A constrained floating-point environment lowers both conversion steps to the
+// constrained intrinsics: the widening `fpext` is exact and only carries the
+// exception behavior, while the narrowing `fptrunc` carries both the rounding
+// mode and the exception behavior.
+// CHECK-LABEL: @convertf_fenv
+func.func @convertf_fenv(%arg0 : f16) -> bf16 {
+// CHECK-NEXT: %[[EXT:.*]] = llvm.intr.experimental.constrained.fpext %arg0 maytrap : f16 to f32
+// CHECK-NEXT: %[[TRUNC:.*]] = llvm.intr.experimental.constrained.fptrunc %[[EXT]] upward maytrap : f32 to bf16
+ %0 = arith.convertf %arg0 fenv<dynamic_rounding_mode = upward, except_mode = unmasked> : f16 to bf16
+// CHECK-NEXT: return %[[TRUNC]]
+ return %0 : bf16
+}
+
+// -----
+
+// The deprecated `roundingmode` attribute also lowers to the constrained
+// intrinsics, using the default (`ignore`) exception behavior.
+// CHECK-LABEL: @convertf_roundingmode
+func.func @convertf_roundingmode(%arg0 : bf16) -> f16 {
+// CHECK-NEXT: %[[EXT:.*]] = llvm.intr.experimental.constrained.fpext %arg0 ignore : bf16 to f32
+// CHECK-NEXT: %[[TRUNC:.*]] = llvm.intr.experimental.constrained.fptrunc %[[EXT]] tonearest ignore : f32 to f16
+ %0 = arith.convertf %arg0 to_nearest_even : bf16 to f16
+// CHECK-NEXT: return %[[TRUNC]]
+ return %0 : f16
+}
+
+// -----
+
+// CHECK-LABEL: @convertf_fenv_vector
+func.func @convertf_fenv_vector(%arg0 : vector<2xf16>) -> vector<2xbf16> {
+// CHECK-NEXT: %[[EXT:.*]] = llvm.intr.experimental.constrained.fpext %arg0 strict : vector<2xf16> to vector<2xf32>
+// CHECK-NEXT: %[[TRUNC:.*]] = llvm.intr.experimental.constrained.fptrunc %[[EXT]] towardzero strict : vector<2xf32> to vector<2xbf16>
+ %0 = arith.convertf %arg0 fenv<dynamic_rounding_mode = toward_zero, strict_except = true> : vector<2xf16> to vector<2xbf16>
+// CHECK-NEXT: return %[[TRUNC]]
+ return %0 : vector<2xbf16>
+}
+
+// -----
+
// Check sign and zero extension and truncation of integers.
// CHECK-LABEL: @integer_extension_and_truncation
func.func @integer_extension_and_truncation(%arg0 : i3) {
diff --git a/mlir/test/Conversion/MathToLLVM/math-to-llvm.mlir b/mlir/test/Conversion/MathToLLVM/math-to-llvm.mlir
index 638261bc6bde3..e0a89893eeb3a 100644
--- a/mlir/test/Conversion/MathToLLVM/math-to-llvm.mlir
+++ b/mlir/test/Conversion/MathToLLVM/math-to-llvm.mlir
@@ -692,3 +692,21 @@ func.func @constrained_fma_with_fastmath(%a : f64, %b : f64, %c : f64) {
%0 = math.fma %a, %b, %c to_nearest_even fastmath<fast> : f64
return
}
+
+// -----
+
+// The `#arith.fenv` attribute lowers to the constrained fma intrinsic, mapping
+// the dynamic rounding mode and the exception behavior derived from
+// `except_mode` and `strict_except`.
+// CHECK-LABEL: func @experimental_constrained_fma_fenv
+func.func @experimental_constrained_fma_fenv(%a : f64, %b : f64, %c : f64) {
+ // CHECK-NEXT: llvm.intr.experimental.constrained.fma %{{.*}}, %{{.*}}, %{{.*}} dynamic ignore : f64
+ %0 = math.fma %a, %b, %c fenv<> : f64
+ // masked + strict -> strict
+ // CHECK-NEXT: llvm.intr.experimental.constrained.fma %{{.*}}, %{{.*}}, %{{.*}} tonearest strict : f64
+ %1 = math.fma %a, %b, %c fenv<dynamic_rounding_mode = to_nearest_even, strict_except = true> : f64
+ // unknown + non-strict -> maytrap
+ // CHECK-NEXT: llvm.intr.experimental.constrained.fma %{{.*}}, %{{.*}}, %{{.*}} towardzero maytrap : f64
+ %2 = math.fma %a, %b, %c fenv<dynamic_rounding_mode = toward_zero, except_mode = unknown> : f64
+ return
+}
diff --git a/mlir/test/Dialect/Arith/expand-ops.mlir b/mlir/test/Dialect/Arith/expand-ops.mlir
index 75c4de2168761..ca0de33533ec1 100644
--- a/mlir/test/Dialect/Arith/expand-ops.mlir
+++ b/mlir/test/Dialect/Arith/expand-ops.mlir
@@ -349,6 +349,19 @@ func.func @scaling_truncf_propagate_rounding_mode_fast_math(%arg0 : vector<4xf16
// -----
+func.func @scaling_truncf_propagate_fenv(%arg0 : vector<4xf16>, %arg1: vector<4xf16>) -> vector<4xf6E3M2FN> {
+ %0 = arith.scaling_truncf %arg0, %arg1 fenv<dynamic_rounding_mode = upward, except_mode = unmasked, strict_except = true> : vector<4xf16>, vector<4xf16> to vector<4xf6E3M2FN>
+ return %0 : vector<4xf6E3M2FN>
+}
+// SCHECK-LABEL: @scaling_truncf_propagate_fenv
+// SCHECK: %[[SCALEF8:.+]] = arith.truncf %arg1 fenv<dynamic_rounding_mode = upward, except_mode = unmasked, strict_except = true> : vector<4xf16> to vector<4xf8E8M0FNU>
+// SCHECK: %[[SCALEINTY:.+]] = arith.extf %[[SCALEF8]] : vector<4xf8E8M0FNU> to vector<4xf16>
+// SCHECK: %[[DIVF:.+]] = arith.divf %arg0, %[[SCALEINTY]] fenv<dynamic_rounding_mode = upward, except_mode = unmasked, strict_except = true> : vector<4xf16>
+// SCHECK: %[[TRUNCF:.+]] = arith.truncf [[_:%[a-zA-Z0-9_]+]] fenv<dynamic_rounding_mode = upward, except_mode = unmasked, strict_except = true> : vector<4xf16> to vector<4xf6E3M2FN>
+// SCHECK: return %[[TRUNCF]] : vector<4xf6E3M2FN>
+
+// -----
+
func.func @scaling_truncf_f16_to_f4E2M1FN_using_f16_scales(%arg0: f16, %arg1 : f16) -> f4E2M1FN {
%0 = arith.scaling_truncf %arg0, %arg1 : f16, f16 to f4E2M1FN
return %0 : f4E2M1FN
>From dc79401c1926a5f4bc205dff9965ee4a06c23f5b Mon Sep 17 00:00:00 2001
From: Andy Kaylor <akaylor at nvidia.com>
Date: Thu, 2 Jul 2026 16:46:41 -0700
Subject: [PATCH 4/7] [mlir][math] Add strict fp support to more operations
This change adds the arith.fenv attribute to transcendental operations
that may be affected by rounding mode or floating-point exception
constraints.
---
.../ArithCommon/AttrToLLVMConverter.h | 20 ++-
mlir/include/mlir/Dialect/Math/IR/MathOps.td | 169 +++++++++++++++---
mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp | 160 ++++++++++++++---
.../Conversion/MathToLLVM/math-to-llvm.mlir | 87 +++++++++
mlir/test/Dialect/Math/ops.mlir | 18 ++
5 files changed, 396 insertions(+), 58 deletions(-)
diff --git a/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h b/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h
index 31d94daf4bb92..007c62481372c 100644
--- a/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h
+++ b/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h
@@ -159,12 +159,11 @@ class AttrConverterConstrainedFPToLLVM {
convertedAttr = NamedAttrList{srcOp->getAttrs()};
MLIRContext *ctx = srcOp->getContext();
- // The floating-point environment may be described either by the deprecated
- // `roundingmode` attribute or by the `#arith.fenv` attribute. Collect both
- // (the verifier guarantees they are never set at the same time) and remove
- // them from the attributes carried over to the target op.
- auto roundingModeAttr = dyn_cast_if_present<arith::RoundingModeAttr>(
- convertedAttr.erase(srcOp.getRoundingModeAttrName()));
+ // The floating-point environment is described by the `#arith.fenv`
+ // attribute and, on operations that predate it, may also be described by
+ // the deprecated `roundingmode` attribute. Collect both (the verifier
+ // guarantees they are never set at the same time) and remove them from the
+ // attributes carried over to the target op.
arith::FenvAttr fenvAttr = srcOp.getFenvAttr();
convertedAttr.erase(srcOp.getFenvAttrName());
@@ -173,10 +172,15 @@ class AttrConverterConstrainedFPToLLVM {
// mode.
[[maybe_unused]] arith::RoundingMode roundingMode =
arith::FenvAttr::getDefaultDynamicRoundingMode();
+ if constexpr (SourceOp::template hasTrait<
+ arith::ArithRoundingModeInterface::Trait>()) {
+ auto roundingModeAttr = dyn_cast_if_present<arith::RoundingModeAttr>(
+ convertedAttr.erase(srcOp.getRoundingModeAttrName()));
+ if (!fenvAttr && roundingModeAttr)
+ roundingMode = roundingModeAttr.getValue();
+ }
if (fenvAttr)
roundingMode = fenvAttr.getDynamicRoundingModeOrDefault();
- else if (roundingModeAttr)
- roundingMode = roundingModeAttr.getValue();
if constexpr (TargetOp::template hasTrait<
LLVM::RoundingModeOpInterface::Trait>()) {
diff --git a/mlir/include/mlir/Dialect/Math/IR/MathOps.td b/mlir/include/mlir/Dialect/Math/IR/MathOps.td
index 21c49484610a0..8c86436e514d4 100644
--- a/mlir/include/mlir/Dialect/Math/IR/MathOps.td
+++ b/mlir/include/mlir/Dialect/Math/IR/MathOps.td
@@ -67,6 +67,65 @@ class Math_FloatUnaryOp<string mnemonic, list<Trait> traits = []> :
attr-dict `:` type($result) }];
}
+// Base class for unary floating point operations that can carry an optional
+// floating-point environment (`#arith.fenv`) attribute.
+class Math_FloatUnaryOpWithFenv<string mnemonic, list<Trait> traits = []> :
+ Math_FloatUnaryOp<mnemonic,
+ !listconcat([DeclareOpInterfaceMethods<FPEnvConstrainedOpInterface>],
+ traits)> {
+ let arguments = (ins FloatLike:$operand,
+ DefaultValuedAttr<Arith_FastMathAttr,
+ "::mlir::arith::FastMathFlags::none">:$fastmath,
+ OptionalAttr<Arith_FenvAttr>:$fenv);
+ // With a trailing optional `fenv` attribute following the `fastmath`
+ // attribute, the generated builders no longer provide C++ default arguments,
+ // so the convenience builders that predate the `fenv` attribute must be
+ // provided explicitly.
+ let builders = [
+ OpBuilder<(ins "::mlir::Value":$operand,
+ CArg<"::mlir::arith::FastMathFlags",
+ "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+ build($_builder, $_state, operand,
+ ::mlir::arith::FastMathFlagsAttr::get(
+ $_builder.getContext(), fastmath),
+ ::mlir::arith::FenvAttr{});
+ }]>,
+ OpBuilder<(ins "::mlir::Value":$operand,
+ "::mlir::arith::FastMathFlagsAttr":$fastmath), [{
+ build($_builder, $_state, operand, fastmath, ::mlir::arith::FenvAttr{});
+ }]>,
+ OpBuilder<(ins "::mlir::Type":$result, "::mlir::Value":$operand,
+ CArg<"::mlir::arith::FastMathFlags",
+ "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+ build($_builder, $_state, result, operand,
+ ::mlir::arith::FastMathFlagsAttr::get(
+ $_builder.getContext(), fastmath),
+ ::mlir::arith::FenvAttr{});
+ }]>,
+ OpBuilder<(ins "::mlir::Type":$result, "::mlir::Value":$operand,
+ "::mlir::arith::FastMathFlagsAttr":$fastmath), [{
+ build($_builder, $_state, result, operand, fastmath,
+ ::mlir::arith::FenvAttr{});
+ }]>,
+ OpBuilder<(ins "::mlir::TypeRange":$resultTypes, "::mlir::Value":$operand,
+ CArg<"::mlir::arith::FastMathFlags",
+ "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+ build($_builder, $_state, resultTypes, operand,
+ ::mlir::arith::FastMathFlagsAttr::get(
+ $_builder.getContext(), fastmath),
+ ::mlir::arith::FenvAttr{});
+ }]>,
+ OpBuilder<(ins "::mlir::TypeRange":$resultTypes, "::mlir::Value":$operand,
+ "::mlir::arith::FastMathFlagsAttr":$fastmath), [{
+ build($_builder, $_state, resultTypes, operand, fastmath,
+ ::mlir::arith::FenvAttr{});
+ }]>,
+ ];
+ let assemblyFormat = [{ $operand (`fastmath` `` $fastmath^)?
+ (`fenv` `` $fenv^)?
+ attr-dict `:` type($result) }];
+}
+
// Base class for binary math operations on integer types. Require two
// operands and one result of the same type. This type can be an integer
// type, vector or tensor thereof.
@@ -94,6 +153,67 @@ class Math_FloatBinaryOp<string mnemonic, list<Trait> traits = []> :
attr-dict `:` type($result) }];
}
+// Base class for binary floating point operations that can carry an optional
+// floating-point environment (`#arith.fenv`) attribute.
+class Math_FloatBinaryOpWithFenv<string mnemonic, list<Trait> traits = []> :
+ Math_FloatBinaryOp<mnemonic,
+ !listconcat([DeclareOpInterfaceMethods<FPEnvConstrainedOpInterface>],
+ traits)> {
+ let arguments = (ins FloatLike:$lhs, FloatLike:$rhs,
+ DefaultValuedAttr<Arith_FastMathAttr,
+ "::mlir::arith::FastMathFlags::none">:$fastmath,
+ OptionalAttr<Arith_FenvAttr>:$fenv);
+ // With a trailing optional `fenv` attribute following the `fastmath`
+ // attribute, the generated builders no longer provide C++ default arguments,
+ // so the convenience builders that predate the `fenv` attribute must be
+ // provided explicitly.
+ let builders = [
+ OpBuilder<(ins "::mlir::Value":$lhs, "::mlir::Value":$rhs,
+ CArg<"::mlir::arith::FastMathFlags",
+ "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+ build($_builder, $_state, lhs, rhs,
+ ::mlir::arith::FastMathFlagsAttr::get(
+ $_builder.getContext(), fastmath),
+ ::mlir::arith::FenvAttr{});
+ }]>,
+ OpBuilder<(ins "::mlir::Value":$lhs, "::mlir::Value":$rhs,
+ "::mlir::arith::FastMathFlagsAttr":$fastmath), [{
+ build($_builder, $_state, lhs, rhs, fastmath, ::mlir::arith::FenvAttr{});
+ }]>,
+ OpBuilder<(ins "::mlir::Type":$result, "::mlir::Value":$lhs,
+ "::mlir::Value":$rhs,
+ CArg<"::mlir::arith::FastMathFlags",
+ "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+ build($_builder, $_state, result, lhs, rhs,
+ ::mlir::arith::FastMathFlagsAttr::get(
+ $_builder.getContext(), fastmath),
+ ::mlir::arith::FenvAttr{});
+ }]>,
+ OpBuilder<(ins "::mlir::Type":$result, "::mlir::Value":$lhs,
+ "::mlir::Value":$rhs, "::mlir::arith::FastMathFlagsAttr":$fastmath), [{
+ build($_builder, $_state, result, lhs, rhs, fastmath,
+ ::mlir::arith::FenvAttr{});
+ }]>,
+ OpBuilder<(ins "::mlir::TypeRange":$resultTypes, "::mlir::Value":$lhs,
+ "::mlir::Value":$rhs,
+ CArg<"::mlir::arith::FastMathFlags",
+ "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+ build($_builder, $_state, resultTypes, lhs, rhs,
+ ::mlir::arith::FastMathFlagsAttr::get(
+ $_builder.getContext(), fastmath),
+ ::mlir::arith::FenvAttr{});
+ }]>,
+ OpBuilder<(ins "::mlir::TypeRange":$resultTypes, "::mlir::Value":$lhs,
+ "::mlir::Value":$rhs, "::mlir::arith::FastMathFlagsAttr":$fastmath), [{
+ build($_builder, $_state, resultTypes, lhs, rhs, fastmath,
+ ::mlir::arith::FenvAttr{});
+ }]>,
+ ];
+ let assemblyFormat = [{ $lhs `,` $rhs (`fastmath` `` $fastmath^)?
+ (`fenv` `` $fenv^)?
+ attr-dict `:` type($result) }];
+}
+
// Base class for floating point ternary operations. Require three operands and
// one result of the same type. This type can be a floating point type, vector
// or tensor thereof.
@@ -153,10 +273,7 @@ class Math_FloatTernaryOpWithRoundingMode<string mnemonic,
}
// Base class for floating point ternary operations that can carry an optional
-// floating-point environment (`#arith.fenv`) attribute. It extends
-// `Math_FloatTernaryOpWithRoundingMode` with the `fenv` attribute and the
-// `FPEnvConstrainedOpInterface`, which provides a uniform API for querying the
-// floating-point environment constraints.
+// floating-point environment (`#arith.fenv`) attribute.
class Math_FloatTernaryOpWithFenv<string mnemonic, list<Trait> traits = []> :
Math_FloatTernaryOpWithRoundingMode<mnemonic,
!listconcat([DeclareOpInterfaceMethods<FPEnvConstrainedOpInterface>],
@@ -271,7 +388,7 @@ def Math_AcoshOp : Math_FloatUnaryOp<"acosh">{
// AsinOp
//===----------------------------------------------------------------------===//
-def Math_AsinOp : Math_FloatUnaryOp<"asin">{
+def Math_AsinOp : Math_FloatUnaryOpWithFenv<"asin">{
let summary = "arcus sine of the given value";
let description = [{
Syntax:
@@ -325,7 +442,7 @@ def Math_AsinhOp : Math_FloatUnaryOp<"asinh">{
// AtanOp
//===----------------------------------------------------------------------===//
-def Math_AtanOp : Math_FloatUnaryOp<"atan">{
+def Math_AtanOp : Math_FloatUnaryOpWithFenv<"atan">{
let summary = "arcus tangent of the given value";
let description = [{
The `atan` operation computes the arcus tangent of a given value. It takes
@@ -373,7 +490,7 @@ def Math_AtanhOp : Math_FloatUnaryOp<"atanh">{
// Atan2Op
//===----------------------------------------------------------------------===//
-def Math_Atan2Op : Math_FloatBinaryOp<"atan2">{
+def Math_Atan2Op : Math_FloatBinaryOpWithFenv<"atan2">{
let summary = "2-argument arcus tangent of the given values";
let description = [{
The `atan2` operation takes two operands and returns one result, all of
@@ -425,7 +542,7 @@ def Math_CbrtOp : Math_FloatUnaryOp<"cbrt"> {
// CeilOp
//===----------------------------------------------------------------------===//
-def Math_CeilOp : Math_FloatUnaryOp<"ceil"> {
+def Math_CeilOp : Math_FloatUnaryOpWithFenv<"ceil"> {
let summary = "ceiling of the specified value";
let description = [{
The `ceil` operation computes the ceiling of a given value. It takes one
@@ -500,7 +617,7 @@ def Math_CopySignOp : Math_FloatBinaryOp<"copysign"> {
// CosOp
//===----------------------------------------------------------------------===//
-def Math_CosOp : Math_FloatUnaryOp<"cos"> {
+def Math_CosOp : Math_FloatUnaryOpWithFenv<"cos"> {
let summary = "cosine of the specified value";
let description = [{
The `cos` operation computes the cosine of a given value. It takes one
@@ -521,7 +638,7 @@ def Math_CosOp : Math_FloatUnaryOp<"cos"> {
// AcosOp
//===----------------------------------------------------------------------===//
-def Math_AcosOp : Math_FloatUnaryOp<"acos"> {
+def Math_AcosOp : Math_FloatUnaryOpWithFenv<"acos"> {
let summary = "arcus cosine of the specified value";
let description = [{
The `acos` operation computes the arcus cosine of a given value. It takes one
@@ -542,7 +659,7 @@ def Math_AcosOp : Math_FloatUnaryOp<"acos"> {
// CoshOp
//===----------------------------------------------------------------------===//
-def Math_CoshOp : Math_FloatUnaryOp<"cosh"> {
+def Math_CoshOp : Math_FloatUnaryOpWithFenv<"cosh"> {
let summary = "hyperbolic cosine of the specified value";
let description = [{
The `cosh` operation computes the hyperbolic cosine. It takes one operand
@@ -563,7 +680,7 @@ def Math_CoshOp : Math_FloatUnaryOp<"cosh"> {
// SinOp
//===----------------------------------------------------------------------===//
-def Math_SinOp : Math_FloatUnaryOp<"sin"> {
+def Math_SinOp : Math_FloatUnaryOpWithFenv<"sin"> {
let summary = "sine of the specified value";
let description = [{
The `sin` operation computes the sine of a given value. It takes one
@@ -584,7 +701,7 @@ def Math_SinOp : Math_FloatUnaryOp<"sin"> {
// SinhOp
//===----------------------------------------------------------------------===//
-def Math_SinhOp : Math_FloatUnaryOp<"sinh"> {
+def Math_SinhOp : Math_FloatUnaryOpWithFenv<"sinh"> {
let summary = "hyperbolic sine of the specified value";
let description = [{
The `sinh` operation computes the hyperbolic sine. It takes one operand
@@ -750,7 +867,7 @@ def Math_ErfcOp : Math_FloatUnaryOp<"erfc"> {
// ExpOp
//===----------------------------------------------------------------------===//
-def Math_ExpOp : Math_FloatUnaryOp<"exp"> {
+def Math_ExpOp : Math_FloatUnaryOpWithFenv<"exp"> {
let summary = "base-e exponential of the specified value";
let description = [{
The `exp` operation takes one operand of floating point type (i.e., scalar,
@@ -771,7 +888,7 @@ def Math_ExpOp : Math_FloatUnaryOp<"exp"> {
// Exp2Op
//===----------------------------------------------------------------------===//
-def Math_Exp2Op : Math_FloatUnaryOp<"exp2"> {
+def Math_Exp2Op : Math_FloatUnaryOpWithFenv<"exp2"> {
let summary = "base-2 exponential of the specified value";
let description = [{
@@ -816,7 +933,7 @@ def Math_ExpM1Op : Math_FloatUnaryOp<"expm1"> {
// FloorOp
//===----------------------------------------------------------------------===//
-def Math_FloorOp : Math_FloatUnaryOp<"floor"> {
+def Math_FloorOp : Math_FloatUnaryOpWithFenv<"floor"> {
let summary = "floor of the specified value";
let description = [{
The `floor` operation computes the floor of a given value. It takes one
@@ -965,7 +1082,7 @@ def Math_IsNormalOp : Math_FloatClassificationOp<"isnormal"> {
// LogOp
//===----------------------------------------------------------------------===//
-def Math_LogOp : Math_FloatUnaryOp<"log"> {
+def Math_LogOp : Math_FloatUnaryOpWithFenv<"log"> {
let summary = "base-e logarithm of the specified value";
let description = [{
@@ -987,7 +1104,7 @@ def Math_LogOp : Math_FloatUnaryOp<"log"> {
// Log10Op
//===----------------------------------------------------------------------===//
-def Math_Log10Op : Math_FloatUnaryOp<"log10"> {
+def Math_Log10Op : Math_FloatUnaryOpWithFenv<"log10"> {
let summary = "base-10 logarithm of the specified value";
let description = [{
@@ -1033,7 +1150,7 @@ def Math_Log1pOp : Math_FloatUnaryOp<"log1p"> {
// Log2Op
//===----------------------------------------------------------------------===//
-def Math_Log2Op : Math_FloatUnaryOp<"log2"> {
+def Math_Log2Op : Math_FloatUnaryOpWithFenv<"log2"> {
let summary = "base-2 logarithm of the specified value";
let description = [{
@@ -1055,7 +1172,7 @@ def Math_Log2Op : Math_FloatUnaryOp<"log2"> {
// PowFOp
//===----------------------------------------------------------------------===//
-def Math_PowFOp : Math_FloatBinaryOp<"powf"> {
+def Math_PowFOp : Math_FloatBinaryOpWithFenv<"powf"> {
let summary = "floating point raised to the power of operation";
let description = [{
The `powf` operation takes two operands of floating point type (i.e.,
@@ -1097,7 +1214,7 @@ def Math_RsqrtOp : Math_FloatUnaryOp<"rsqrt"> {
// SqrtOp
//===----------------------------------------------------------------------===//
-def Math_SqrtOp : Math_FloatUnaryOp<"sqrt"> {
+def Math_SqrtOp : Math_FloatUnaryOpWithFenv<"sqrt"> {
let summary = "sqrt of the specified value";
let description = [{
The `sqrt` operation computes the square root. It takes one operand of
@@ -1118,7 +1235,7 @@ def Math_SqrtOp : Math_FloatUnaryOp<"sqrt"> {
// TanOp
//===----------------------------------------------------------------------===//
-def Math_TanOp : Math_FloatUnaryOp<"tan"> {
+def Math_TanOp : Math_FloatUnaryOpWithFenv<"tan"> {
let summary = "tangent of the specified value";
let description = [{
The `tan` operation computes the tangent. It takes one operand
@@ -1139,7 +1256,7 @@ def Math_TanOp : Math_FloatUnaryOp<"tan"> {
// TanhOp
//===----------------------------------------------------------------------===//
-def Math_TanhOp : Math_FloatUnaryOp<"tanh"> {
+def Math_TanhOp : Math_FloatUnaryOpWithFenv<"tanh"> {
let summary = "hyperbolic tangent of the specified value";
let description = [{
The `tanh` operation computes the hyperbolic tangent. It takes one operand
@@ -1160,7 +1277,7 @@ def Math_TanhOp : Math_FloatUnaryOp<"tanh"> {
// RoundEvenOp
//===----------------------------------------------------------------------===//
-def Math_RoundEvenOp : Math_FloatUnaryOp<"roundeven"> {
+def Math_RoundEvenOp : Math_FloatUnaryOpWithFenv<"roundeven"> {
let summary = "round of the specified value with halfway cases to even";
let description = [{
The `roundeven` operation returns the operand rounded to the nearest integer
@@ -1184,7 +1301,7 @@ def Math_RoundEvenOp : Math_FloatUnaryOp<"roundeven"> {
// RoundOp
//===----------------------------------------------------------------------===//
-def Math_RoundOp : Math_FloatUnaryOp<"round"> {
+def Math_RoundOp : Math_FloatUnaryOpWithFenv<"round"> {
let summary = "round of the specified value";
let description = [{
The `round` operation returns the operand rounded to the nearest integer
@@ -1208,7 +1325,7 @@ def Math_RoundOp : Math_FloatUnaryOp<"round"> {
// TruncOp
//===----------------------------------------------------------------------===//
-def Math_TruncOp : Math_FloatUnaryOp<"trunc"> {
+def Math_TruncOp : Math_FloatUnaryOpWithFenv<"trunc"> {
let summary = "trunc of the specified value";
let description = [{
The `trunc` operation returns the operand rounded to the nearest integer
diff --git a/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp b/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp
index aca6973ab5956..4786a3f0cc17e 100644
--- a/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp
+++ b/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp
@@ -59,7 +59,13 @@ struct ConstrainedVectorConvertToLLVMPattern
LogicalResult
matchAndRewrite(SourceOp op, typename SourceOp::Adaptor adaptor,
ConversionPatternRewriter &rewriter) const override {
- bool opIsConstrained = op.getRoundingModeAttr() || op.getFenvAttr();
+ bool opIsConstrained = static_cast<bool>(op.getFenvAttr());
+ // Operations that predate the `#arith.fenv` attribute may also carry the
+ // deprecated `roundingmode` attribute.
+ if constexpr (SourceOp::template hasTrait<
+ arith::ArithRoundingModeInterface::Trait>())
+ opIsConstrained =
+ opIsConstrained || static_cast<bool>(op.getRoundingModeAttr());
if (IsConstrained != opIsConstrained)
return failure();
return VectorConvertToLLVMPattern<
@@ -68,23 +74,62 @@ struct ConstrainedVectorConvertToLLVMPattern
}
};
+// Convenience alias for the pattern that lowers a math op carrying no
+// floating-point environment constraint to a regular LLVM intrinsic op,
+// converting the fastmath flags.
+template <typename SourceOp, typename TargetOp>
+using ConvertUnconstrainedMathToLLVMPattern =
+ ConstrainedVectorConvertToLLVMPattern<SourceOp, TargetOp,
+ /*IsConstrained=*/false, ConvertFastMath,
+ /*FailOnUnsupportedFP=*/true>;
+
+// Convenience alias for the pattern that lowers a math op carrying a
+// floating-point environment constraint (the `#arith.fenv` attribute) to the
+// matching `llvm.intr.experimental.constrained.*` intrinsic.
+template <typename SourceOp, typename TargetOp>
+using ConvertConstrainedMathToLLVMPattern =
+ ConstrainedVectorConvertToLLVMPattern<
+ SourceOp, TargetOp, /*IsConstrained=*/true,
+ arith::AttrConverterConstrainedFPToLLVM, /*FailOnUnsupportedFP=*/true>;
+
using AbsFOpLowering =
ConvertFMFMathToLLVMPattern<math::AbsFOp, LLVM::FAbsOp,
/*FailOnUnsupportedFP=*/true>;
-using CeilOpLowering = ConvertFMFMathToLLVMPattern<math::CeilOp, LLVM::FCeilOp>;
+using CeilOpLowering =
+ ConvertUnconstrainedMathToLLVMPattern<math::CeilOp, LLVM::FCeilOp>;
+using ConstrainedCeilOpLowering =
+ ConvertConstrainedMathToLLVMPattern<math::CeilOp, LLVM::ConstrainedCeilIntr>;
using CopySignOpLowering =
ConvertFMFMathToLLVMPattern<math::CopySignOp, LLVM::CopySignOp>;
-using CosOpLowering = ConvertFMFMathToLLVMPattern<math::CosOp, LLVM::CosOp>;
-using CoshOpLowering = ConvertFMFMathToLLVMPattern<math::CoshOp, LLVM::CoshOp>;
-using AcosOpLowering = ConvertFMFMathToLLVMPattern<math::AcosOp, LLVM::ACosOp>;
+using CosOpLowering =
+ ConvertUnconstrainedMathToLLVMPattern<math::CosOp, LLVM::CosOp>;
+using ConstrainedCosOpLowering =
+ ConvertConstrainedMathToLLVMPattern<math::CosOp, LLVM::ConstrainedCosIntr>;
+using CoshOpLowering =
+ ConvertUnconstrainedMathToLLVMPattern<math::CoshOp, LLVM::CoshOp>;
+using ConstrainedCoshOpLowering =
+ ConvertConstrainedMathToLLVMPattern<math::CoshOp, LLVM::ConstrainedCoshIntr>;
+using AcosOpLowering =
+ ConvertUnconstrainedMathToLLVMPattern<math::AcosOp, LLVM::ACosOp>;
+using ConstrainedAcosOpLowering =
+ ConvertConstrainedMathToLLVMPattern<math::AcosOp, LLVM::ConstrainedACosIntr>;
using CtPopFOpLowering =
VectorConvertToLLVMPattern<math::CtPopOp, LLVM::CtPopOp,
AttrConvertPassThrough,
/*FailOnUnsupportedFP=*/true>;
-using Exp2OpLowering = ConvertFMFMathToLLVMPattern<math::Exp2Op, LLVM::Exp2Op>;
-using ExpOpLowering = ConvertFMFMathToLLVMPattern<math::ExpOp, LLVM::ExpOp>;
+using Exp2OpLowering =
+ ConvertUnconstrainedMathToLLVMPattern<math::Exp2Op, LLVM::Exp2Op>;
+using ConstrainedExp2OpLowering =
+ ConvertConstrainedMathToLLVMPattern<math::Exp2Op, LLVM::ConstrainedExp2Intr>;
+using ExpOpLowering =
+ ConvertUnconstrainedMathToLLVMPattern<math::ExpOp, LLVM::ExpOp>;
+using ConstrainedExpOpLowering =
+ ConvertConstrainedMathToLLVMPattern<math::ExpOp, LLVM::ConstrainedExpIntr>;
using FloorOpLowering =
- ConvertFMFMathToLLVMPattern<math::FloorOp, LLVM::FFloorOp>;
+ ConvertUnconstrainedMathToLLVMPattern<math::FloorOp, LLVM::FFloorOp>;
+using ConstrainedFloorOpLowering =
+ ConvertConstrainedMathToLLVMPattern<math::FloorOp,
+ LLVM::ConstrainedFloorIntr>;
using FmaOpLowering =
ConstrainedVectorConvertToLLVMPattern<math::FmaOp, LLVM::FMAOp,
/*IsConstrained=*/false,
@@ -94,27 +139,72 @@ using ConstrainedFmaOpLowering = ConstrainedVectorConvertToLLVMPattern<
math::FmaOp, LLVM::ConstrainedFMAIntr, /*IsConstrained=*/true,
arith::AttrConverterConstrainedFPToLLVM, /*FailOnUnsupportedFP=*/true>;
using Log10OpLowering =
- ConvertFMFMathToLLVMPattern<math::Log10Op, LLVM::Log10Op>;
-using Log2OpLowering = ConvertFMFMathToLLVMPattern<math::Log2Op, LLVM::Log2Op>;
-using LogOpLowering = ConvertFMFMathToLLVMPattern<math::LogOp, LLVM::LogOp>;
-using PowFOpLowering = ConvertFMFMathToLLVMPattern<math::PowFOp, LLVM::PowOp>;
+ ConvertUnconstrainedMathToLLVMPattern<math::Log10Op, LLVM::Log10Op>;
+using ConstrainedLog10OpLowering =
+ ConvertConstrainedMathToLLVMPattern<math::Log10Op,
+ LLVM::ConstrainedLog10Intr>;
+using Log2OpLowering =
+ ConvertUnconstrainedMathToLLVMPattern<math::Log2Op, LLVM::Log2Op>;
+using ConstrainedLog2OpLowering =
+ ConvertConstrainedMathToLLVMPattern<math::Log2Op, LLVM::ConstrainedLog2Intr>;
+using LogOpLowering =
+ ConvertUnconstrainedMathToLLVMPattern<math::LogOp, LLVM::LogOp>;
+using ConstrainedLogOpLowering =
+ ConvertConstrainedMathToLLVMPattern<math::LogOp, LLVM::ConstrainedLogIntr>;
+using PowFOpLowering =
+ ConvertUnconstrainedMathToLLVMPattern<math::PowFOp, LLVM::PowOp>;
+using ConstrainedPowFOpLowering =
+ ConvertConstrainedMathToLLVMPattern<math::PowFOp, LLVM::ConstrainedPowIntr>;
using FPowIOpLowering =
ConvertFMFMathToLLVMPattern<math::FPowIOp, LLVM::PowIOp>;
using RoundEvenOpLowering =
- ConvertFMFMathToLLVMPattern<math::RoundEvenOp, LLVM::RoundEvenOp>;
+ ConvertUnconstrainedMathToLLVMPattern<math::RoundEvenOp, LLVM::RoundEvenOp>;
+using ConstrainedRoundEvenOpLowering =
+ ConvertConstrainedMathToLLVMPattern<math::RoundEvenOp,
+ LLVM::ConstrainedRoundEvenIntr>;
using RoundOpLowering =
- ConvertFMFMathToLLVMPattern<math::RoundOp, LLVM::RoundOp>;
-using SinOpLowering = ConvertFMFMathToLLVMPattern<math::SinOp, LLVM::SinOp>;
-using SinhOpLowering = ConvertFMFMathToLLVMPattern<math::SinhOp, LLVM::SinhOp>;
-using ASinOpLowering = ConvertFMFMathToLLVMPattern<math::AsinOp, LLVM::ASinOp>;
-using SqrtOpLowering = ConvertFMFMathToLLVMPattern<math::SqrtOp, LLVM::SqrtOp>;
+ ConvertUnconstrainedMathToLLVMPattern<math::RoundOp, LLVM::RoundOp>;
+using ConstrainedRoundOpLowering =
+ ConvertConstrainedMathToLLVMPattern<math::RoundOp,
+ LLVM::ConstrainedRoundIntr>;
+using SinOpLowering =
+ ConvertUnconstrainedMathToLLVMPattern<math::SinOp, LLVM::SinOp>;
+using ConstrainedSinOpLowering =
+ ConvertConstrainedMathToLLVMPattern<math::SinOp, LLVM::ConstrainedSinIntr>;
+using SinhOpLowering =
+ ConvertUnconstrainedMathToLLVMPattern<math::SinhOp, LLVM::SinhOp>;
+using ConstrainedSinhOpLowering =
+ ConvertConstrainedMathToLLVMPattern<math::SinhOp, LLVM::ConstrainedSinhIntr>;
+using ASinOpLowering =
+ ConvertUnconstrainedMathToLLVMPattern<math::AsinOp, LLVM::ASinOp>;
+using ConstrainedASinOpLowering =
+ ConvertConstrainedMathToLLVMPattern<math::AsinOp, LLVM::ConstrainedASinIntr>;
+using SqrtOpLowering =
+ ConvertUnconstrainedMathToLLVMPattern<math::SqrtOp, LLVM::SqrtOp>;
+using ConstrainedSqrtOpLowering =
+ ConvertConstrainedMathToLLVMPattern<math::SqrtOp, LLVM::ConstrainedSqrtIntr>;
using FTruncOpLowering =
- ConvertFMFMathToLLVMPattern<math::TruncOp, LLVM::FTruncOp>;
-using TanOpLowering = ConvertFMFMathToLLVMPattern<math::TanOp, LLVM::TanOp>;
-using TanhOpLowering = ConvertFMFMathToLLVMPattern<math::TanhOp, LLVM::TanhOp>;
-using ATanOpLowering = ConvertFMFMathToLLVMPattern<math::AtanOp, LLVM::ATanOp>;
+ ConvertUnconstrainedMathToLLVMPattern<math::TruncOp, LLVM::FTruncOp>;
+using ConstrainedFTruncOpLowering =
+ ConvertConstrainedMathToLLVMPattern<math::TruncOp,
+ LLVM::ConstrainedTruncIntr>;
+using TanOpLowering =
+ ConvertUnconstrainedMathToLLVMPattern<math::TanOp, LLVM::TanOp>;
+using ConstrainedTanOpLowering =
+ ConvertConstrainedMathToLLVMPattern<math::TanOp, LLVM::ConstrainedTanIntr>;
+using TanhOpLowering =
+ ConvertUnconstrainedMathToLLVMPattern<math::TanhOp, LLVM::TanhOp>;
+using ConstrainedTanhOpLowering =
+ ConvertConstrainedMathToLLVMPattern<math::TanhOp, LLVM::ConstrainedTanhIntr>;
+using ATanOpLowering =
+ ConvertUnconstrainedMathToLLVMPattern<math::AtanOp, LLVM::ATanOp>;
+using ConstrainedATanOpLowering =
+ ConvertConstrainedMathToLLVMPattern<math::AtanOp, LLVM::ConstrainedATanIntr>;
using ATan2OpLowering =
- ConvertFMFMathToLLVMPattern<math::Atan2Op, LLVM::ATan2Op>;
+ ConvertUnconstrainedMathToLLVMPattern<math::Atan2Op, LLVM::ATan2Op>;
+using ConstrainedATan2OpLowering =
+ ConvertConstrainedMathToLLVMPattern<math::Atan2Op,
+ LLVM::ConstrainedATan2Intr>;
// A `CtLz/CtTz/absi(a)` is converted into `CtLz/CtTz/absi(a, false)`.
// TODO: Result and operand types match for `absi` as opposed to `ct*z`, so it
// may be better to separate the patterns.
@@ -468,37 +558,59 @@ void mlir::populateMathToLLVMConversionPatterns(
AbsFOpLowering,
AbsIOpLowering,
CeilOpLowering,
+ ConstrainedCeilOpLowering,
CopySignOpLowering,
CosOpLowering,
+ ConstrainedCosOpLowering,
CoshOpLowering,
+ ConstrainedCoshOpLowering,
AcosOpLowering,
+ ConstrainedAcosOpLowering,
CountLeadingZerosOpLowering,
CountTrailingZerosOpLowering,
CtPopFOpLowering,
Exp2OpLowering,
+ ConstrainedExp2OpLowering,
ExpM1OpLowering,
ExpOpLowering,
+ ConstrainedExpOpLowering,
FPowIOpLowering,
FloorOpLowering,
+ ConstrainedFloorOpLowering,
FmaOpLowering,
ConstrainedFmaOpLowering,
Log10OpLowering,
+ ConstrainedLog10OpLowering,
Log2OpLowering,
+ ConstrainedLog2OpLowering,
LogOpLowering,
+ ConstrainedLogOpLowering,
PowFOpLowering,
+ ConstrainedPowFOpLowering,
RoundEvenOpLowering,
+ ConstrainedRoundEvenOpLowering,
RoundOpLowering,
+ ConstrainedRoundOpLowering,
RsqrtOpLowering,
SincosOpLowering,
SinOpLowering,
+ ConstrainedSinOpLowering,
SinhOpLowering,
+ ConstrainedSinhOpLowering,
ASinOpLowering,
+ ConstrainedASinOpLowering,
SqrtOpLowering,
+ ConstrainedSqrtOpLowering,
FTruncOpLowering,
+ ConstrainedFTruncOpLowering,
TanOpLowering,
+ ConstrainedTanOpLowering,
TanhOpLowering,
+ ConstrainedTanhOpLowering,
ATanOpLowering,
- ATan2OpLowering
+ ConstrainedATanOpLowering,
+ ATan2OpLowering,
+ ConstrainedATan2OpLowering
>(converter, benefit);
// clang-format on
}
diff --git a/mlir/test/Conversion/MathToLLVM/math-to-llvm.mlir b/mlir/test/Conversion/MathToLLVM/math-to-llvm.mlir
index e0a89893eeb3a..67edabe4f7c31 100644
--- a/mlir/test/Conversion/MathToLLVM/math-to-llvm.mlir
+++ b/mlir/test/Conversion/MathToLLVM/math-to-llvm.mlir
@@ -710,3 +710,90 @@ func.func @experimental_constrained_fma_fenv(%a : f64, %b : f64, %c : f64) {
%2 = math.fma %a, %b, %c fenv<dynamic_rounding_mode = toward_zero, except_mode = unknown> : f64
return
}
+
+// -----
+
+// Unary transcendental math ops with the `#arith.fenv` attribute lower to the
+// matching constrained intrinsic, carrying a rounding mode and exception
+// behavior.
+// CHECK-LABEL: func @experimental_constrained_unary
+func.func @experimental_constrained_unary(%a : f64) {
+ // CHECK-NEXT: llvm.intr.experimental.constrained.sqrt %{{.*}} dynamic ignore : f64
+ %0 = math.sqrt %a fenv<> : f64
+ // CHECK-NEXT: llvm.intr.experimental.constrained.exp %{{.*}} tonearest strict : f64
+ %1 = math.exp %a fenv<dynamic_rounding_mode = to_nearest_even, strict_except = true> : f64
+ // CHECK-NEXT: llvm.intr.experimental.constrained.log %{{.*}} towardzero maytrap : f64
+ %2 = math.log %a fenv<dynamic_rounding_mode = toward_zero, except_mode = unknown> : f64
+ // CHECK-NEXT: llvm.intr.experimental.constrained.sin %{{.*}} dynamic ignore : f64
+ %3 = math.sin %a fenv<> : f64
+ // CHECK-NEXT: llvm.intr.experimental.constrained.cos %{{.*}} dynamic ignore : f64
+ %4 = math.cos %a fenv<> : f64
+ // CHECK-NEXT: llvm.intr.experimental.constrained.tanh %{{.*}} dynamic ignore : f64
+ %5 = math.tanh %a fenv<> : f64
+ return
+}
+
+// -----
+
+// Without the `#arith.fenv` attribute, the unconstrained lowering is used.
+// CHECK-LABEL: func @unconstrained_sqrt
+func.func @unconstrained_sqrt(%a : f64) {
+ // CHECK-NEXT: llvm.intr.sqrt(%{{.*}}) : (f64) -> f64
+ // CHECK-NOT: constrained
+ %0 = math.sqrt %a : f64
+ return
+}
+
+// -----
+
+// Rounding math ops (ceil/floor/round/roundeven/trunc) map to constrained
+// intrinsics that only carry the exception behavior (no rounding mode).
+// CHECK-LABEL: func @experimental_constrained_rounding
+func.func @experimental_constrained_rounding(%a : f64) {
+ // CHECK-NEXT: llvm.intr.experimental.constrained.ceil %{{.*}} ignore : f64
+ %0 = math.ceil %a fenv<> : f64
+ // CHECK-NEXT: llvm.intr.experimental.constrained.floor %{{.*}} maytrap : f64
+ %1 = math.floor %a fenv<except_mode = unknown> : f64
+ // CHECK-NEXT: llvm.intr.experimental.constrained.trunc %{{.*}} strict : f64
+ %2 = math.trunc %a fenv<strict_except = true> : f64
+ // CHECK-NEXT: llvm.intr.experimental.constrained.round %{{.*}} ignore : f64
+ %3 = math.round %a fenv<> : f64
+ // CHECK-NEXT: llvm.intr.experimental.constrained.roundeven %{{.*}} ignore : f64
+ %4 = math.roundeven %a fenv<> : f64
+ return
+}
+
+// -----
+
+// Binary math ops with the `#arith.fenv` attribute lower to the matching
+// constrained intrinsic.
+// CHECK-LABEL: func @experimental_constrained_binary
+func.func @experimental_constrained_binary(%a : f64, %b : f64) {
+ // CHECK-NEXT: llvm.intr.experimental.constrained.pow %{{.*}}, %{{.*}} dynamic ignore : f64
+ %0 = math.powf %a, %b fenv<> : f64
+ // CHECK-NEXT: llvm.intr.experimental.constrained.atan2 %{{.*}}, %{{.*}} tonearest ignore : f64
+ %1 = math.atan2 %a, %b fenv<dynamic_rounding_mode = to_nearest_even> : f64
+ return
+}
+
+// -----
+
+// The constrained lowering also applies element-wise to vector operands.
+// CHECK-LABEL: func @experimental_constrained_vector
+func.func @experimental_constrained_vector(%a : vector<4xf32>) {
+ // CHECK: llvm.intr.experimental.constrained.sqrt {{.*}} dynamic ignore : vector<4xf32>
+ %0 = math.sqrt %a fenv<> : vector<4xf32>
+ return
+}
+
+// -----
+
+// Constrained intrinsics do not carry fastmath flags. The fastmath attribute is
+// dropped during the lowering.
+// CHECK-LABEL: func @constrained_unary_with_fastmath
+func.func @constrained_unary_with_fastmath(%a : f64) {
+ // CHECK-NEXT: llvm.intr.experimental.constrained.sqrt %{{.*}} dynamic ignore : f64
+ // CHECK-NOT: fastmath
+ %0 = math.sqrt %a fastmath<fast> fenv<> : f64
+ return
+}
diff --git a/mlir/test/Dialect/Math/ops.mlir b/mlir/test/Dialect/Math/ops.mlir
index b33f3d1eeea97..30912a7cf597a 100644
--- a/mlir/test/Dialect/Math/ops.mlir
+++ b/mlir/test/Dialect/Math/ops.mlir
@@ -317,6 +317,24 @@ func.func @fastmath(%f: f32, %i: i32, %v: vector<4xf32>, %t: tensor<4x4x?xf32>)
return
}
+// CHECK-LABEL: func @fenv(
+// CHECK-SAME: %[[F:.*]]: f32, %[[V:.*]]: vector<4xf32>)
+func.func @fenv(%f: f32, %v: vector<4xf32>) {
+ // CHECK: math.sqrt %[[F]] fenv<> : f32
+ %0 = math.sqrt %f fenv<> : f32
+ // CHECK: math.exp %[[F]] fenv<dynamic_rounding_mode = to_nearest_even, strict_except = true> : f32
+ %1 = math.exp %f fenv<dynamic_rounding_mode = to_nearest_even, strict_except = true> : f32
+ // CHECK: math.sin %[[V]] fastmath<contract> fenv<except_mode = unknown> : vector<4xf32>
+ %2 = math.sin %v fastmath<contract> fenv<except_mode = unknown> : vector<4xf32>
+ // CHECK: math.ceil %[[F]] fenv<> : f32
+ %3 = math.ceil %f fenv<> : f32
+ // CHECK: math.powf %[[F]], %[[F]] fenv<dynamic_rounding_mode = toward_zero> : f32
+ %4 = math.powf %f, %f fenv<dynamic_rounding_mode = toward_zero> : f32
+ // CHECK: math.atan2 %[[F]], %[[F]] fenv<> : f32
+ %5 = math.atan2 %f, %f fenv<> : f32
+ return
+}
+
// CHECK-LABEL: func @fpclassify(
// CHECK-SAME: %[[F:.+]]: f32, %[[D:.+]]: f64,
// CHECK-SAME: %[[V:.+]]: vector<4xf32>, %[[T:.+]]: tensor<4x?xf32>
>From 17921ffeb45f8956a39c3cfaa127594cba0e74b7 Mon Sep 17 00:00:00 2001
From: Andy Kaylor <akaylor at nvidia.com>
Date: Thu, 2 Jul 2026 17:31:01 -0700
Subject: [PATCH 5/7] [mlir][llvm] Add a constrained version of the powi
operation
This adds a version of the powi operation that maps to the constrained
intrinsic.
WIP! Need a test.
---
mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td
index b912cef1386dd..5840f80dfe704 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td
@@ -635,6 +635,16 @@ def LLVM_ConstrainedTruncIntr : LLVM_ConstrainedUnaryIntrNoRM<"trunc">;
def LLVM_ConstrainedPowIntr : LLVM_ConstrainedBinaryIntrRM<"pow">;
def LLVM_ConstrainedATan2Intr : LLVM_ConstrainedBinaryIntrRM<"atan2">;
+def LLVM_ConstrainedPowIIntr
+ : LLVM_ConstrainedIntr<"powi", /*numArgs=*/2,
+ /*overloadedResult=*/1, /*overloadedOperands=*/[],
+ /*hasRoundingMode=*/1> {
+ let assemblyFormat = [{
+ $arg_0 `,` $arg_1 $roundingmode $fpExceptionBehavior attr-dict `:`
+ functional-type(operands, results)
+ }];
+}
+
def LLVM_ConstrainedUIToFP
: LLVM_ConstrainedIntr<"uitofp", /*numArgs=*/1,
/*overloadedResult=*/1, /*overloadedOperands=*/[0],
>From b97d67c990e481e989af4f62fd8154eaf9027b23 Mon Sep 17 00:00:00 2001
From: Andy Kaylor <akaylor at nvidia.com>
Date: Thu, 2 Jul 2026 17:32:06 -0700
Subject: [PATCH 6/7] [mlir][math] Add fenv support for the powi operation
This adds the arith.fenv attribute (optionally) to the math.powi
operation and adds handling to lower to the corresponding
constrained LLVM operation when it is present.
Note that the math.powi operation currently requires the floating-point
input operand and the integer exponent operand to have the same number
of elements if the input value is a vector. This is a mismatch with the
llvm.powi operation (and the documented requirements of the LLVM IR
llvm.powi intrinsic), and so that form can't be successfully lowered.
---
mlir/include/mlir/Dialect/Math/IR/MathOps.td | 56 ++++++++++++++++++-
mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp | 43 +++++++++++++-
mlir/lib/Dialect/Math/IR/MathOps.cpp | 5 ++
.../Conversion/MathToLLVM/math-to-llvm.mlir | 23 ++++++++
mlir/test/Dialect/Math/ops.mlir | 6 +-
5 files changed, 128 insertions(+), 5 deletions(-)
diff --git a/mlir/include/mlir/Dialect/Math/IR/MathOps.td b/mlir/include/mlir/Dialect/Math/IR/MathOps.td
index 8c86436e514d4..8d03cafcdfe5a 100644
--- a/mlir/include/mlir/Dialect/Math/IR/MathOps.td
+++ b/mlir/include/mlir/Dialect/Math/IR/MathOps.td
@@ -1350,7 +1350,8 @@ def Math_TruncOp : Math_FloatUnaryOpWithFenv<"trunc"> {
def Math_FPowIOp : Math_Op<"fpowi",
[SameOperandsAndResultShape, AllTypesMatch<["lhs", "result"]>,
- DeclareOpInterfaceMethods<ArithFastMathInterface>]> {
+ DeclareOpInterfaceMethods<ArithFastMathInterface>,
+ DeclareOpInterfaceMethods<FPEnvConstrainedOpInterface>]> {
let summary = "floating point raised to the signed integer power";
let description = [{
The `fpowi` operation takes a `base` operand of floating point type
@@ -1375,13 +1376,64 @@ def Math_FPowIOp : Math_Op<"fpowi",
// Scalar exponentiation.
%a = math.fpowi %base, %power : f64, i32
```
+
+ An optional `#arith.fenv` attribute may be attached to describe constraints
+ on the floating-point environment.
}];
let arguments = (ins FloatLike:$lhs, SignlessIntegerOrIndexLike:$rhs,
DefaultValuedAttr<Arith_FastMathAttr,
- "::mlir::arith::FastMathFlags::none">:$fastmath);
+ "::mlir::arith::FastMathFlags::none">:$fastmath,
+ OptionalAttr<Arith_FenvAttr>:$fenv);
let results = (outs FloatLike:$result);
+ // With a trailing optional `fenv` attribute following the `fastmath`
+ // attribute, the generated builders no longer provide C++ default arguments,
+ // so the convenience builders that predate the `fenv` attribute must be
+ // provided explicitly.
+ let builders = [
+ OpBuilder<(ins "::mlir::Value":$lhs, "::mlir::Value":$rhs,
+ CArg<"::mlir::arith::FastMathFlags",
+ "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+ build($_builder, $_state, lhs, rhs,
+ ::mlir::arith::FastMathFlagsAttr::get(
+ $_builder.getContext(), fastmath),
+ ::mlir::arith::FenvAttr{});
+ }]>,
+ OpBuilder<(ins "::mlir::Value":$lhs, "::mlir::Value":$rhs,
+ "::mlir::arith::FastMathFlagsAttr":$fastmath), [{
+ build($_builder, $_state, lhs, rhs, fastmath, ::mlir::arith::FenvAttr{});
+ }]>,
+ OpBuilder<(ins "::mlir::Type":$result, "::mlir::Value":$lhs,
+ "::mlir::Value":$rhs,
+ CArg<"::mlir::arith::FastMathFlags",
+ "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+ build($_builder, $_state, result, lhs, rhs,
+ ::mlir::arith::FastMathFlagsAttr::get(
+ $_builder.getContext(), fastmath),
+ ::mlir::arith::FenvAttr{});
+ }]>,
+ OpBuilder<(ins "::mlir::Type":$result, "::mlir::Value":$lhs,
+ "::mlir::Value":$rhs, "::mlir::arith::FastMathFlagsAttr":$fastmath), [{
+ build($_builder, $_state, result, lhs, rhs, fastmath,
+ ::mlir::arith::FenvAttr{});
+ }]>,
+ OpBuilder<(ins "::mlir::TypeRange":$resultTypes, "::mlir::Value":$lhs,
+ "::mlir::Value":$rhs,
+ CArg<"::mlir::arith::FastMathFlags",
+ "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+ build($_builder, $_state, resultTypes, lhs, rhs,
+ ::mlir::arith::FastMathFlagsAttr::get(
+ $_builder.getContext(), fastmath),
+ ::mlir::arith::FenvAttr{});
+ }]>,
+ OpBuilder<(ins "::mlir::TypeRange":$resultTypes, "::mlir::Value":$lhs,
+ "::mlir::Value":$rhs, "::mlir::arith::FastMathFlagsAttr":$fastmath), [{
+ build($_builder, $_state, resultTypes, lhs, rhs, fastmath,
+ ::mlir::arith::FenvAttr{});
+ }]>,
+ ];
let assemblyFormat = [{ $lhs `,` $rhs (`fastmath` `` $fastmath^)?
+ (`fenv` `` $fenv^)?
attr-dict `:` type($lhs) `,` type($rhs) }];
let hasFolder = 1;
diff --git a/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp b/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp
index 4786a3f0cc17e..84b716f578568 100644
--- a/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp
+++ b/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp
@@ -156,7 +156,7 @@ using PowFOpLowering =
using ConstrainedPowFOpLowering =
ConvertConstrainedMathToLLVMPattern<math::PowFOp, LLVM::ConstrainedPowIntr>;
using FPowIOpLowering =
- ConvertFMFMathToLLVMPattern<math::FPowIOp, LLVM::PowIOp>;
+ ConvertUnconstrainedMathToLLVMPattern<math::FPowIOp, LLVM::PowIOp>;
using RoundEvenOpLowering =
ConvertUnconstrainedMathToLLVMPattern<math::RoundEvenOp, LLVM::RoundEvenOp>;
using ConstrainedRoundEvenOpLowering =
@@ -205,6 +205,46 @@ using ATan2OpLowering =
using ConstrainedATan2OpLowering =
ConvertConstrainedMathToLLVMPattern<math::Atan2Op,
LLVM::ConstrainedATan2Intr>;
+
+// A `math.fpowi` carrying a floating-point environment (`#arith.fenv`) lowers
+// to the constrained `powi` intrinsic.
+struct ConstrainedFPowIOpLowering
+ : public ConvertOpToLLVMPattern<math::FPowIOp,
+ /*FailOnUnsupportedFP=*/true> {
+ using ConvertOpToLLVMPattern<
+ math::FPowIOp, /*FailOnUnsupportedFP=*/true>::ConvertOpToLLVMPattern;
+
+ LogicalResult
+ matchAndRewrite(math::FPowIOp op, OpAdaptor adaptor,
+ ConversionPatternRewriter &rewriter) const override {
+ if (!op.getFenvAttr())
+ return rewriter.notifyMatchFailure(
+ op, "unconstrained fpowi is handled by a separate pattern");
+
+ // `llvm.intr.experimental.constrained.powi` requires a scalar operand with
+ // an `i32` exponent.
+ Type resultType = op.getResult().getType();
+ if (!isa<FloatType>(resultType))
+ return rewriter.notifyMatchFailure(
+ op, "constrained fpowi only supports scalar operands");
+ if (!op.getRhs().getType().isInteger(32))
+ return rewriter.notifyMatchFailure(
+ op, "constrained fpowi requires an i32 exponent");
+
+ Type llvmResultType = getTypeConverter()->convertType(resultType);
+ if (!llvmResultType)
+ return failure();
+
+ arith::AttrConverterConstrainedFPToLLVM<math::FPowIOp,
+ LLVM::ConstrainedPowIIntr>
+ attrConvert(op);
+ rewriter.replaceOpWithNewOp<LLVM::ConstrainedPowIIntr>(
+ op, llvmResultType, ValueRange{adaptor.getLhs(), adaptor.getRhs()},
+ attrConvert.getAttrs());
+ return success();
+ }
+};
+
// A `CtLz/CtTz/absi(a)` is converted into `CtLz/CtTz/absi(a, false)`.
// TODO: Result and operand types match for `absi` as opposed to `ct*z`, so it
// may be better to separate the patterns.
@@ -575,6 +615,7 @@ void mlir::populateMathToLLVMConversionPatterns(
ExpOpLowering,
ConstrainedExpOpLowering,
FPowIOpLowering,
+ ConstrainedFPowIOpLowering,
FloorOpLowering,
ConstrainedFloorOpLowering,
FmaOpLowering,
diff --git a/mlir/lib/Dialect/Math/IR/MathOps.cpp b/mlir/lib/Dialect/Math/IR/MathOps.cpp
index b900cb1911759..2faf3fe3cd6b8 100644
--- a/mlir/lib/Dialect/Math/IR/MathOps.cpp
+++ b/mlir/lib/Dialect/Math/IR/MathOps.cpp
@@ -828,6 +828,11 @@ OpFoldResult math::TruncOp::fold(FoldAdaptor adaptor) {
//===----------------------------------------------------------------------===//
OpFoldResult math::FPowIOp::fold(FoldAdaptor adaptor) {
+ // Do not fold when a constrained floating-point environment is requested.
+ // In theory, we could fold this when we know the rounding mode and we can
+ // verify that the operation wouldn't raise an exception.
+ if (getFenvAttr())
+ return {};
return constFoldBinaryOpConditional<FloatAttr, IntegerAttr>(
adaptor.getOperands(),
[](const APFloat &base, const APInt &exp) -> std::optional<APFloat> {
diff --git a/mlir/test/Conversion/MathToLLVM/math-to-llvm.mlir b/mlir/test/Conversion/MathToLLVM/math-to-llvm.mlir
index 67edabe4f7c31..6e4f8fe1ffc0d 100644
--- a/mlir/test/Conversion/MathToLLVM/math-to-llvm.mlir
+++ b/mlir/test/Conversion/MathToLLVM/math-to-llvm.mlir
@@ -778,6 +778,29 @@ func.func @experimental_constrained_binary(%a : f64, %b : f64) {
// -----
+// `math.fpowi` with a floating-point environment lowers to the constrained
+// `powi` intrinsic, which takes a scalar `i32` exponent.
+// CHECK-LABEL: func @experimental_constrained_fpowi
+func.func @experimental_constrained_fpowi(%a : f64, %b : i32) {
+ // CHECK-NEXT: llvm.intr.experimental.constrained.powi %{{.*}}, %{{.*}} tonearest maytrap : (f64, i32) -> f64
+ %0 = math.fpowi %a, %b fenv<dynamic_rounding_mode = to_nearest_even, except_mode = unknown> : f64, i32
+ return
+}
+
+// -----
+
+// Without the `#arith.fenv` attribute, `math.fpowi` uses the unconstrained
+// `llvm.intr.powi` lowering.
+// CHECK-LABEL: func @unconstrained_fpowi
+func.func @unconstrained_fpowi(%a : f64, %b : i32) {
+ // CHECK-NEXT: llvm.intr.powi(%{{.*}}, %{{.*}}) : (f64, i32) -> f64
+ // CHECK-NOT: constrained
+ %0 = math.fpowi %a, %b : f64, i32
+ return
+}
+
+// -----
+
// The constrained lowering also applies element-wise to vector operands.
// CHECK-LABEL: func @experimental_constrained_vector
func.func @experimental_constrained_vector(%a : vector<4xf32>) {
diff --git a/mlir/test/Dialect/Math/ops.mlir b/mlir/test/Dialect/Math/ops.mlir
index 30912a7cf597a..fd60dd80086cb 100644
--- a/mlir/test/Dialect/Math/ops.mlir
+++ b/mlir/test/Dialect/Math/ops.mlir
@@ -318,8 +318,8 @@ func.func @fastmath(%f: f32, %i: i32, %v: vector<4xf32>, %t: tensor<4x4x?xf32>)
}
// CHECK-LABEL: func @fenv(
-// CHECK-SAME: %[[F:.*]]: f32, %[[V:.*]]: vector<4xf32>)
-func.func @fenv(%f: f32, %v: vector<4xf32>) {
+// CHECK-SAME: %[[F:.*]]: f32, %[[V:.*]]: vector<4xf32>, %[[I:.*]]: i32)
+func.func @fenv(%f: f32, %v: vector<4xf32>, %i: i32) {
// CHECK: math.sqrt %[[F]] fenv<> : f32
%0 = math.sqrt %f fenv<> : f32
// CHECK: math.exp %[[F]] fenv<dynamic_rounding_mode = to_nearest_even, strict_except = true> : f32
@@ -332,6 +332,8 @@ func.func @fenv(%f: f32, %v: vector<4xf32>) {
%4 = math.powf %f, %f fenv<dynamic_rounding_mode = toward_zero> : f32
// CHECK: math.atan2 %[[F]], %[[F]] fenv<> : f32
%5 = math.atan2 %f, %f fenv<> : f32
+ // CHECK: math.fpowi %[[F]], %[[I]] fenv<dynamic_rounding_mode = toward_zero> : f32, i32
+ %6 = math.fpowi %f, %i fenv<dynamic_rounding_mode = toward_zero> : f32, i32
return
}
>From 2c9169da738b3e760cfe2f18791b2db69a527d37 Mon Sep 17 00:00:00 2001
From: Andy Kaylor <akaylor at nvidia.com>
Date: Mon, 13 Jul 2026 12:45:07 -0700
Subject: [PATCH 7/7] Fix formatting
---
.../ArithCommon/AttrToLLVMConverter.h | 7 ++--
.../Conversion/ArithToLLVM/ArithToLLVM.cpp | 17 +++++-----
mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp | 33 ++++++++++++-------
.../Dialect/Arith/Transforms/ExpandOps.cpp | 24 +++++++-------
4 files changed, 45 insertions(+), 36 deletions(-)
diff --git a/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h b/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h
index 007c62481372c..7f4bb80cc4f07 100644
--- a/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h
+++ b/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h
@@ -184,10 +184,9 @@ class AttrConverterConstrainedFPToLLVM {
if constexpr (TargetOp::template hasTrait<
LLVM::RoundingModeOpInterface::Trait>()) {
- convertedAttr.set(
- TargetOp::getRoundingModeAttrName(),
- LLVM::RoundingModeAttr::get(
- ctx, convertArithRoundingModeToLLVM(roundingMode)));
+ convertedAttr.set(TargetOp::getRoundingModeAttrName(),
+ LLVM::RoundingModeAttr::get(
+ ctx, convertArithRoundingModeToLLVM(roundingMode)));
}
// Constrained intrinsics (llvm.intr.experimental.constrained.*) do not
// support fastmath flags. Remove the arith fastmath attribute if present.
diff --git a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
index fed991c7074d2..39672cf447816 100644
--- a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
+++ b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
@@ -373,11 +373,10 @@ struct ConvertFOpLowering : public ConvertOpToLLVMPattern<arith::ConvertFOp> {
}
private:
- static Value emitConversion(ConversionPatternRewriter &rewriter, Location loc,
- Value input, Type targetType,
- LLVM::RoundingModeAttr roundingModeAttr,
- LLVM::FPExceptionBehaviorAttr
- exceptionBehaviorAttr) {
+ static Value
+ emitConversion(ConversionPatternRewriter &rewriter, Location loc, Value input,
+ Type targetType, LLVM::RoundingModeAttr roundingModeAttr,
+ LLVM::FPExceptionBehaviorAttr exceptionBehaviorAttr) {
Type f32Scalar = Float32Type::get(rewriter.getContext());
Type f32Ty = f32Scalar;
if (auto vecTy = dyn_cast<VectorType>(targetType))
@@ -386,10 +385,10 @@ struct ConvertFOpLowering : public ConvertOpToLLVMPattern<arith::ConvertFOp> {
// A constrained floating-point environment was requested: emit the
// constrained intrinsics carrying the rounding mode and exception behavior.
if (exceptionBehaviorAttr) {
- Value ext = LLVM::ConstrainedFPExtIntr::create(rewriter, loc, f32Ty, input,
- exceptionBehaviorAttr);
- return LLVM::ConstrainedFPTruncIntr::create(rewriter, loc, targetType, ext,
- roundingModeAttr,
+ Value ext = LLVM::ConstrainedFPExtIntr::create(
+ rewriter, loc, f32Ty, input, exceptionBehaviorAttr);
+ return LLVM::ConstrainedFPTruncIntr::create(rewriter, loc, targetType,
+ ext, roundingModeAttr,
exceptionBehaviorAttr);
}
diff --git a/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp b/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp
index 84b716f578568..73528707bde49 100644
--- a/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp
+++ b/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp
@@ -80,7 +80,8 @@ struct ConstrainedVectorConvertToLLVMPattern
template <typename SourceOp, typename TargetOp>
using ConvertUnconstrainedMathToLLVMPattern =
ConstrainedVectorConvertToLLVMPattern<SourceOp, TargetOp,
- /*IsConstrained=*/false, ConvertFastMath,
+ /*IsConstrained=*/false,
+ ConvertFastMath,
/*FailOnUnsupportedFP=*/true>;
// Convenience alias for the pattern that lowers a math op carrying a
@@ -98,7 +99,8 @@ using AbsFOpLowering =
using CeilOpLowering =
ConvertUnconstrainedMathToLLVMPattern<math::CeilOp, LLVM::FCeilOp>;
using ConstrainedCeilOpLowering =
- ConvertConstrainedMathToLLVMPattern<math::CeilOp, LLVM::ConstrainedCeilIntr>;
+ ConvertConstrainedMathToLLVMPattern<math::CeilOp,
+ LLVM::ConstrainedCeilIntr>;
using CopySignOpLowering =
ConvertFMFMathToLLVMPattern<math::CopySignOp, LLVM::CopySignOp>;
using CosOpLowering =
@@ -108,11 +110,13 @@ using ConstrainedCosOpLowering =
using CoshOpLowering =
ConvertUnconstrainedMathToLLVMPattern<math::CoshOp, LLVM::CoshOp>;
using ConstrainedCoshOpLowering =
- ConvertConstrainedMathToLLVMPattern<math::CoshOp, LLVM::ConstrainedCoshIntr>;
+ ConvertConstrainedMathToLLVMPattern<math::CoshOp,
+ LLVM::ConstrainedCoshIntr>;
using AcosOpLowering =
ConvertUnconstrainedMathToLLVMPattern<math::AcosOp, LLVM::ACosOp>;
using ConstrainedAcosOpLowering =
- ConvertConstrainedMathToLLVMPattern<math::AcosOp, LLVM::ConstrainedACosIntr>;
+ ConvertConstrainedMathToLLVMPattern<math::AcosOp,
+ LLVM::ConstrainedACosIntr>;
using CtPopFOpLowering =
VectorConvertToLLVMPattern<math::CtPopOp, LLVM::CtPopOp,
AttrConvertPassThrough,
@@ -120,7 +124,8 @@ using CtPopFOpLowering =
using Exp2OpLowering =
ConvertUnconstrainedMathToLLVMPattern<math::Exp2Op, LLVM::Exp2Op>;
using ConstrainedExp2OpLowering =
- ConvertConstrainedMathToLLVMPattern<math::Exp2Op, LLVM::ConstrainedExp2Intr>;
+ ConvertConstrainedMathToLLVMPattern<math::Exp2Op,
+ LLVM::ConstrainedExp2Intr>;
using ExpOpLowering =
ConvertUnconstrainedMathToLLVMPattern<math::ExpOp, LLVM::ExpOp>;
using ConstrainedExpOpLowering =
@@ -146,7 +151,8 @@ using ConstrainedLog10OpLowering =
using Log2OpLowering =
ConvertUnconstrainedMathToLLVMPattern<math::Log2Op, LLVM::Log2Op>;
using ConstrainedLog2OpLowering =
- ConvertConstrainedMathToLLVMPattern<math::Log2Op, LLVM::ConstrainedLog2Intr>;
+ ConvertConstrainedMathToLLVMPattern<math::Log2Op,
+ LLVM::ConstrainedLog2Intr>;
using LogOpLowering =
ConvertUnconstrainedMathToLLVMPattern<math::LogOp, LLVM::LogOp>;
using ConstrainedLogOpLowering =
@@ -174,15 +180,18 @@ using ConstrainedSinOpLowering =
using SinhOpLowering =
ConvertUnconstrainedMathToLLVMPattern<math::SinhOp, LLVM::SinhOp>;
using ConstrainedSinhOpLowering =
- ConvertConstrainedMathToLLVMPattern<math::SinhOp, LLVM::ConstrainedSinhIntr>;
+ ConvertConstrainedMathToLLVMPattern<math::SinhOp,
+ LLVM::ConstrainedSinhIntr>;
using ASinOpLowering =
ConvertUnconstrainedMathToLLVMPattern<math::AsinOp, LLVM::ASinOp>;
using ConstrainedASinOpLowering =
- ConvertConstrainedMathToLLVMPattern<math::AsinOp, LLVM::ConstrainedASinIntr>;
+ ConvertConstrainedMathToLLVMPattern<math::AsinOp,
+ LLVM::ConstrainedASinIntr>;
using SqrtOpLowering =
ConvertUnconstrainedMathToLLVMPattern<math::SqrtOp, LLVM::SqrtOp>;
using ConstrainedSqrtOpLowering =
- ConvertConstrainedMathToLLVMPattern<math::SqrtOp, LLVM::ConstrainedSqrtIntr>;
+ ConvertConstrainedMathToLLVMPattern<math::SqrtOp,
+ LLVM::ConstrainedSqrtIntr>;
using FTruncOpLowering =
ConvertUnconstrainedMathToLLVMPattern<math::TruncOp, LLVM::FTruncOp>;
using ConstrainedFTruncOpLowering =
@@ -195,11 +204,13 @@ using ConstrainedTanOpLowering =
using TanhOpLowering =
ConvertUnconstrainedMathToLLVMPattern<math::TanhOp, LLVM::TanhOp>;
using ConstrainedTanhOpLowering =
- ConvertConstrainedMathToLLVMPattern<math::TanhOp, LLVM::ConstrainedTanhIntr>;
+ ConvertConstrainedMathToLLVMPattern<math::TanhOp,
+ LLVM::ConstrainedTanhIntr>;
using ATanOpLowering =
ConvertUnconstrainedMathToLLVMPattern<math::AtanOp, LLVM::ATanOp>;
using ConstrainedATanOpLowering =
- ConvertConstrainedMathToLLVMPattern<math::AtanOp, LLVM::ConstrainedATanIntr>;
+ ConvertConstrainedMathToLLVMPattern<math::AtanOp,
+ LLVM::ConstrainedATanIntr>;
using ATan2OpLowering =
ConvertUnconstrainedMathToLLVMPattern<math::Atan2Op, LLVM::ATan2Op>;
using ConstrainedATan2OpLowering =
diff --git a/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp b/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
index b104a3b9711e9..df10228ac864b 100644
--- a/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
+++ b/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
@@ -649,9 +649,9 @@ struct F8E8M0TruncFOpConverter : public OpRewritePattern<arith::TruncFOp> {
if (operandETy.getIntOrFloatBitWidth() < 32) {
operand = arith::ExtFOp::create(b, f32Ty, operand, op.getFastmathAttr());
} else if (operandETy.getIntOrFloatBitWidth() > 32) {
- operand = arith::TruncFOp::create(b, f32Ty, operand,
- op.getRoundingmodeAttr(),
- op.getFastmathAttr(), op.getFenvAttr());
+ operand =
+ arith::TruncFOp::create(b, f32Ty, operand, op.getRoundingmodeAttr(),
+ op.getFastmathAttr(), op.getFenvAttr());
}
Value f32Bits = arith::BitcastOp::create(b, i32Ty, operand);
Value cF32MantissaWidth = createConst(op->getLoc(), i32Ty, 23, rewriter);
@@ -718,9 +718,9 @@ struct ScalingTruncFOpConverter
if (scaleETy.getIntOrFloatBitWidth() >= 16) {
scaleETy = b.getF8E8M0Type();
scaleTy = cloneToShapedType(scaleTy, scaleETy);
- scaleOperand = arith::TruncFOp::create(b, scaleTy, scaleOperand, nullptr,
- op.getFastmathAttr(),
- op.getFenvAttr());
+ scaleOperand =
+ arith::TruncFOp::create(b, scaleTy, scaleOperand, nullptr,
+ op.getFastmathAttr(), op.getFenvAttr());
}
if (!llvm::isa<Float8E8M0FNUType>(scaleETy)) {
return rewriter.notifyMatchFailure(
@@ -733,12 +733,12 @@ struct ScalingTruncFOpConverter
// inputTy that is 2^scale and will also propagate NaNs
scaleOperand =
arith::ExtFOp::create(b, inputTy, scaleOperand, op.getFastmathAttr());
- Value result = arith::DivFOp::create(b, inputOperand, scaleOperand,
- op.getFastmathAttr(), nullptr,
- op.getFenvAttr());
- Value resultCast = arith::TruncFOp::create(
- b, resultTy, result, op.getRoundingmodeAttr(), op.getFastmathAttr(),
- op.getFenvAttr());
+ Value result =
+ arith::DivFOp::create(b, inputOperand, scaleOperand,
+ op.getFastmathAttr(), nullptr, op.getFenvAttr());
+ Value resultCast =
+ arith::TruncFOp::create(b, resultTy, result, op.getRoundingmodeAttr(),
+ op.getFastmathAttr(), op.getFenvAttr());
rewriter.replaceOp(op, resultCast);
return success();
}
More information about the Mlir-commits
mailing list