[llvm] [DAG] Avoid strictfp nodes in lowering of llvm.fcanonicalize (PR #177399)
Serge Pavlov via llvm-commits
llvm-commits at lists.llvm.org
Thu Jan 22 08:47:01 PST 2026
https://github.com/spavloff created https://github.com/llvm/llvm-project/pull/177399
The default lowering of llvm.fcanonicalize, implemented in #106370 and
#142105, follows the suggestions in
https://llvm.org/docs/LangRef.html#llvm-canonicalize-intrinsic and implements canonicalization as a multiplication by 1.0. The problem with this implementation is that the compiler tries to "optimize" the multiplication by removing it. To prevent from this, the existing implementation uses STRICT_FMUL for the multiplication, even in the default mode.
This solution has some drawbacks. First, STRICT_* nodes are designed to represent the side effects of floating-point operations. In the default Mode, these side effects are ignored for all operation, so the occurrence of strict node in this context is a misuse, it impedes the development of strictfp support.
Second, the solution relies on the absence of optimizations for STRICT_FMUL. However, STRICT_* nodes should also be optimized, and future development may invalidate this assumption.
This patch modifies the default expansion of llvm.fcanonicalize. Instead of using STRICT_FMUL it represents the multiplication by a special node, which is changed to FMUL immediately before the instruction selection.
>From d3acd55dd5367644c205da36c90d2277e96e9edd Mon Sep 17 00:00:00 2001
From: Serge Pavlov <sepavloff at gmail.com>
Date: Mon, 5 Jan 2026 23:14:14 +0700
Subject: [PATCH] [DAG] Avoid strictfp nodes in lowering of llvm.fcanonicalize
The default lowering of llvm.fcanonicalize, implemented in #106370 and
#142105, follows the suggestions in
https://llvm.org/docs/LangRef.html#llvm-canonicalize-intrinsic and
implements canonicalization as a multiplication by 1.0. The problem with
this implementation is that the compiler tries to "optimize" the
multiplication by removing it. To prevent from this, the existing
implementation uses STRICT_FMUL for the multiplication, even in the
default mode.
This solution has some drawbacks. First, STRICT_* nodes are designed to
represent the side effects of floating-point operations. In the default
Mode, these side effects are ignored for all operation, so the
occurrence of strict node in this context is a misuse, it impedes the
development of strictfp support.
Second, the solution relies on the absence of optimizations for
STRICT_FMUL. However, STRICT_* nodes should also be optimized, and
future development may invalidate this assumption.
This patch modifies the default expansion of llvm.fcanonicalize. Instead
of using STRICT_FMUL it represents the multiplication by a special node,
which is changed to FMUL immediately before the instruction selection.
---
llvm/include/llvm/CodeGen/ISDOpcodes.h | 5 +
llvm/lib/CodeGen/SelectionDAG/LegalizeDAG.cpp | 20 +--
.../SelectionDAG/LegalizeFloatTypes.cpp | 18 +--
.../SelectionDAG/SelectionDAGDumper.cpp | 1 +
.../CodeGen/SelectionDAG/SelectionDAGISel.cpp | 11 ++
llvm/lib/Target/X86/X86ISelLowering.cpp | 22 +--
.../CodeGen/X86/canonicalize-vars-f16-type.ll | 135 ++++++------------
7 files changed, 80 insertions(+), 132 deletions(-)
diff --git a/llvm/include/llvm/CodeGen/ISDOpcodes.h b/llvm/include/llvm/CodeGen/ISDOpcodes.h
index 2ebd2641944f5..144eb6b116702 100644
--- a/llvm/include/llvm/CodeGen/ISDOpcodes.h
+++ b/llvm/include/llvm/CodeGen/ISDOpcodes.h
@@ -540,6 +540,11 @@ enum NodeType {
/// Returns platform specific canonical encoding of a floating point number.
FCANONICALIZE,
+ /// Returns platform specific canonical encoding of a floating point number
+ /// using multiplying it by 1.0. The fist operand is the floating-point
+ /// number, the second is constant 1.0.
+ FCANONICALIZE_MUL,
+
/// Performs a check of floating point class property, defined by IEEE-754.
/// The first operand is the floating point value to check. The second operand
/// specifies the checked property and is a TargetConstant which specifies
diff --git a/llvm/lib/CodeGen/SelectionDAG/LegalizeDAG.cpp b/llvm/lib/CodeGen/SelectionDAG/LegalizeDAG.cpp
index 09d79ae208a8f..eb75d7df6fa7a 100644
--- a/llvm/lib/CodeGen/SelectionDAG/LegalizeDAG.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/LegalizeDAG.cpp
@@ -3499,24 +3499,14 @@ bool SelectionDAGLegalize::ExpandNode(SDNode *Node) {
// This implements llvm.canonicalize.f* by multiplication with 1.0, as
// suggested in
// https://llvm.org/docs/LangRef.html#llvm-canonicalize-intrinsic.
- // It uses strict_fp operations even outside a strict_fp context in order
- // to guarantee that the canonicalization is not optimized away by later
- // passes. The result chain introduced by that is intentionally ignored
- // since no ordering requirement is intended here.
-
- // Create strict multiplication by 1.0.
+ // To avoid optimization 'x*1.0 -> x', use a special node instead of FMUL.
+ // The node will be replaced by FMUL immediately before the instruction
+ // selection.
SDValue Operand = Node->getOperand(0);
EVT VT = Operand.getValueType();
SDValue One = DAG.getConstantFP(1.0, dl, VT);
- SDValue Chain = DAG.getEntryNode();
- // Propagate existing flags on canonicalize, and additionally set
- // NoFPExcept.
- SDNodeFlags CanonicalizeFlags = Node->getFlags();
- CanonicalizeFlags.setNoFPExcept(true);
- SDValue Mul = DAG.getNode(ISD::STRICT_FMUL, dl, {VT, MVT::Other},
- {Chain, Operand, One}, CanonicalizeFlags);
-
- Results.push_back(Mul);
+ SDValue NewNode = DAG.getNode(ISD::FCANONICALIZE_MUL, dl, VT, Operand, One);
+ Results.push_back(NewNode);
break;
}
case ISD::SIGN_EXTEND_INREG: {
diff --git a/llvm/lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp b/llvm/lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp
index 1e7bc757d2c58..85954798eaef0 100644
--- a/llvm/lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp
@@ -72,6 +72,8 @@ void DAGTypeLegalizer::SoftenFloatResult(SDNode *N, unsigned ResNo) {
case ISD::FABS: R = SoftenFloatRes_FABS(N); break;
case ISD::FCANONICALIZE:
R = SoftenFloatRes_FCANONICALIZE(N); break;
+ case ISD::FCANONICALIZE_MUL:
+ R = SoftenFloatRes_FMUL(N); break;
case ISD::STRICT_FMINNUM:
case ISD::FMINNUM: R = SoftenFloatRes_FMINNUM(N); break;
case ISD::STRICT_FMAXNUM:
@@ -319,22 +321,14 @@ SDValue DAGTypeLegalizer::SoftenFloatRes_FCANONICALIZE(SDNode *N) {
// This implements llvm.canonicalize.f* by multiplication with 1.0, as
// suggested in
// https://llvm.org/docs/LangRef.html#llvm-canonicalize-intrinsic.
- // It uses strict_fp operations even outside a strict_fp context in order
- // to guarantee that the canonicalization is not optimized away by later
- // passes. The result chain introduced by that is intentionally ignored
- // since no ordering requirement is intended here.
+ // To avoid optimization 'x*1.0 -> x', use a special node instead of FMUL.
+ // The node will be replaced by FMUL immediately before the instruction
+ // selection.
- // Create strict multiplication by 1.0.
SDValue Operand = N->getOperand(0);
EVT VT = Operand.getValueType();
SDValue One = DAG.getConstantFP(1.0, dl, VT);
- SDValue Chain = DAG.getEntryNode();
- // Propagate existing flags on canonicalize, and additionally set
- // NoFPExcept.
- SDNodeFlags CanonicalizeFlags = N->getFlags();
- CanonicalizeFlags.setNoFPExcept(true);
- SDValue Mul = DAG.getNode(ISD::STRICT_FMUL, dl, {VT, MVT::Other},
- {Chain, Operand, One}, CanonicalizeFlags);
+ SDValue Mul = DAG.getNode(ISD::FCANONICALIZE_MUL, dl, VT, Operand, One);
return BitConvertToInteger(Mul);
}
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp
index 965e4f61659db..f1f93b65a5092 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp
@@ -320,6 +320,7 @@ std::string SDNode::getOperationName(const SelectionDAG *G) const {
case ISD::FCOPYSIGN: return "fcopysign";
case ISD::FGETSIGN: return "fgetsign";
case ISD::FCANONICALIZE: return "fcanonicalize";
+ case ISD::FCANONICALIZE_MUL: return "fcanonicalize_mul";
case ISD::IS_FPCLASS: return "is_fpclass";
case ISD::FPOW: return "fpow";
case ISD::STRICT_FPOW: return "strict_fpow";
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp
index de9daca767388..32a1479379df3 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp
@@ -1390,6 +1390,17 @@ void SelectionDAGISel::DoInstructionSelection() {
Node = CurDAG->mutateStrictFPToFP(Node);
}
+ // Default lowering of llvm.fcanonicalize is to replace it with
+ // multiplication by 1.0. The replacement was postponed to avoid removing
+ // it due to optimizations.
+ if (Node->getOpcode() == ISD::FCANONICALIZE_MUL) {
+ SDValue Operand = Node->getOperand(0);
+ SDValue One = Node->getOperand(1);
+ EVT VT = Operand.getValueType();
+ Node = CurDAG->MorphNodeTo(Node, ISD::FMUL, CurDAG->getVTList(VT),
+ { Operand, One });
+ }
+
LLVM_DEBUG(dbgs() << "\nISEL: Starting selection on root node: ";
Node->dump(CurDAG));
diff --git a/llvm/lib/Target/X86/X86ISelLowering.cpp b/llvm/lib/Target/X86/X86ISelLowering.cpp
index dad0fa4421cda..9bbc3a0c6cb8f 100644
--- a/llvm/lib/Target/X86/X86ISelLowering.cpp
+++ b/llvm/lib/Target/X86/X86ISelLowering.cpp
@@ -715,7 +715,7 @@ X86TargetLowering::X86TargetLowering(const X86TargetMachine &TM,
setOperationAction(ISD::STRICT_FROUNDEVEN, MVT::f16, Promote);
setOperationAction(ISD::STRICT_FTRUNC, MVT::f16, Promote);
setOperationAction(ISD::STRICT_FP_ROUND, MVT::f16, Custom);
- setOperationAction(ISD::FCANONICALIZE, MVT::f16, Custom);
+ setOperationAction(ISD::FCANONICALIZE, MVT::f16, Promote);
setOperationAction(ISD::STRICT_FP_EXTEND, MVT::f32, Custom);
setOperationAction(ISD::STRICT_FP_EXTEND, MVT::f64, Custom);
@@ -1235,6 +1235,7 @@ X86TargetLowering::X86TargetLowering(const X86TargetMachine &TM,
setOperationAction(ISD::FNEG, MVT::v8f16, Custom);
setOperationAction(ISD::FABS, MVT::v8f16, Custom);
setOperationAction(ISD::FCOPYSIGN, MVT::v8f16, Custom);
+ setOperationAction(ISD::FCANONICALIZE, MVT::v8f16, Expand);
// Custom lower v2i64 and v2f64 selects.
setOperationAction(ISD::SELECT, MVT::v2f64, Custom);
@@ -1712,6 +1713,7 @@ X86TargetLowering::X86TargetLowering(const X86TargetMachine &TM,
setOperationAction(ISD::FSUB, MVT::v16f16, Expand);
setOperationAction(ISD::FMUL, MVT::v16f16, Expand);
setOperationAction(ISD::FDIV, MVT::v16f16, Expand);
+ setOperationAction(ISD::FCANONICALIZE, MVT::v16f16, Expand);
if (HasInt256) {
setOperationAction(ISD::VSELECT, MVT::v32i8, Legal);
@@ -1770,9 +1772,6 @@ X86TargetLowering::X86TargetLowering(const X86TargetMachine &TM,
setOperationAction(ISD::FP_TO_UINT, MVT::v2i1, Custom);
setOperationAction(ISD::STRICT_FP_TO_SINT, MVT::v2i1, Custom);
setOperationAction(ISD::STRICT_FP_TO_UINT, MVT::v2i1, Custom);
- setOperationAction(ISD::FCANONICALIZE, MVT::v8f16, Custom);
- setOperationAction(ISD::FCANONICALIZE, MVT::v16f16, Custom);
- setOperationAction(ISD::FCANONICALIZE, MVT::v32f16, Custom);
// There is no byte sized k-register load or store without AVX512DQ.
if (!Subtarget.hasDQI()) {
@@ -2082,7 +2081,8 @@ X86TargetLowering::X86TargetLowering(const X86TargetMachine &TM,
setOperationAction(ISD::STRICT_FP_ROUND, MVT::v16f16, Custom);
setOperationAction(ISD::FP_EXTEND, MVT::v16f32, Custom);
setOperationAction(ISD::STRICT_FP_EXTEND, MVT::v16f32, Custom);
- for (unsigned Opc : {ISD::FADD, ISD::FSUB, ISD::FMUL, ISD::FDIV})
+ for (unsigned Opc : {ISD::FADD, ISD::FSUB, ISD::FMUL, ISD::FDIV,
+ ISD::FCANONICALIZE})
setOperationPromotedToType(Opc, MVT::v32f16, MVT::v32f32);
setOperationAction(ISD::SETCC, MVT::v32f16, Custom);
@@ -33877,17 +33877,9 @@ static SDValue LowerFCanonicalize(SDValue Op, SelectionDAG &DAG) {
SDValue Operand = N->getOperand(0);
EVT VT = Operand.getValueType();
SDLoc dl(N);
-
SDValue One = DAG.getConstantFP(1.0, dl, VT);
-
- // TODO: Fix Crash for bf16 when generating strict_fmul as it
- // leads to a error : SoftPromoteHalfResult #0: t11: bf16,ch = strict_fmul t0,
- // ConstantFP:bf16<APFloat(16256)>, t5 LLVM ERROR: Do not know how to soft
- // promote this operator's result!
- SDValue Chain = DAG.getEntryNode();
- SDValue StrictFmul = DAG.getNode(ISD::STRICT_FMUL, dl, {VT, MVT::Other},
- {Chain, Operand, One});
- return StrictFmul;
+ SDValue NewNode = DAG.getNode(ISD::FCANONICALIZE_MUL, dl, VT, Operand, One);
+ return NewNode;
}
static StringRef getInstrStrFromOpNo(const SmallVectorImpl<StringRef> &AsmStrs,
diff --git a/llvm/test/CodeGen/X86/canonicalize-vars-f16-type.ll b/llvm/test/CodeGen/X86/canonicalize-vars-f16-type.ll
index 8b3aa2964db02..0a7170870bb01 100644
--- a/llvm/test/CodeGen/X86/canonicalize-vars-f16-type.ll
+++ b/llvm/test/CodeGen/X86/canonicalize-vars-f16-type.ll
@@ -1,4 +1,4 @@
-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --default-march x86_64-unknown-linux-gnu --version 5
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --tool P:arbeitspace2buildDebugbinllc.exe --default-march x86_64-unknown-linux-gnu --version 5
; RUN: llc -mattr=+sse2 -mtriple=x86_64 < %s | FileCheck %s -check-prefixes=SSE
; RUN: llc -mattr=+avx -mtriple=x86_64 < %s | FileCheck %s -check-prefixes=AVX,AVX1
; RUN: llc -mattr=+avx2 -mtriple=x86_64 < %s | FileCheck %s -check-prefixes=AVX,AVX2
@@ -9,49 +9,33 @@ define void @v_test_canonicalize__half(half addrspace(1)* %out) nounwind {
; SSE-LABEL: v_test_canonicalize__half:
; SSE: # %bb.0: # %entry
; SSE-NEXT: pushq %rbx
-; SSE-NEXT: subq $16, %rsp
; SSE-NEXT: movq %rdi, %rbx
; SSE-NEXT: pinsrw $0, (%rdi), %xmm0
; SSE-NEXT: callq __extendhfsf2 at PLT
-; SSE-NEXT: movd %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Folded Spill
-; SSE-NEXT: pinsrw $0, {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
-; SSE-NEXT: callq __extendhfsf2 at PLT
-; SSE-NEXT: mulss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 4-byte Folded Reload
+; SSE-NEXT: mulss {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
; SSE-NEXT: callq __truncsfhf2 at PLT
; SSE-NEXT: pextrw $0, %xmm0, %eax
; SSE-NEXT: movw %ax, (%rbx)
-; SSE-NEXT: addq $16, %rsp
; SSE-NEXT: popq %rbx
; SSE-NEXT: retq
;
; AVX-LABEL: v_test_canonicalize__half:
; AVX: # %bb.0: # %entry
; AVX-NEXT: pushq %rbx
-; AVX-NEXT: subq $16, %rsp
; AVX-NEXT: movq %rdi, %rbx
; AVX-NEXT: vpinsrw $0, (%rdi), %xmm0, %xmm0
; AVX-NEXT: callq __extendhfsf2 at PLT
-; AVX-NEXT: vmovd %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Folded Spill
-; AVX-NEXT: vpinsrw $0, {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0, %xmm0
-; AVX-NEXT: callq __extendhfsf2 at PLT
-; AVX-NEXT: vmulss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0, %xmm0 # 4-byte Folded Reload
+; AVX-NEXT: vmulss {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0, %xmm0
; AVX-NEXT: callq __truncsfhf2 at PLT
; AVX-NEXT: vpextrw $0, %xmm0, (%rbx)
-; AVX-NEXT: addq $16, %rsp
; AVX-NEXT: popq %rbx
; AVX-NEXT: retq
;
; AVX512-LABEL: v_test_canonicalize__half:
; AVX512: # %bb.0: # %entry
-; AVX512-NEXT: movzwl (%rdi), %eax
-; AVX512-NEXT: movzwl {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ecx
-; AVX512-NEXT: vmovd %ecx, %xmm0
+; AVX512-NEXT: vpinsrw $0, (%rdi), %xmm0, %xmm0
; AVX512-NEXT: vcvtph2ps %xmm0, %xmm0
-; AVX512-NEXT: vmovd %eax, %xmm1
-; AVX512-NEXT: vcvtph2ps %xmm1, %xmm1
-; AVX512-NEXT: vmulss %xmm0, %xmm1, %xmm0
-; AVX512-NEXT: vxorps %xmm1, %xmm1, %xmm1
-; AVX512-NEXT: vmovss {{.*#+}} xmm0 = xmm0[0],xmm1[1,2,3]
+; AVX512-NEXT: vmulss {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0, %xmm0
; AVX512-NEXT: vcvtps2ph $4, %xmm0, %xmm0
; AVX512-NEXT: vpextrw $0, %xmm0, (%rdi)
; AVX512-NEXT: retq
@@ -66,33 +50,30 @@ define half @complex_canonicalize_fmul_half(half %a, half %b) nounwind {
; SSE-LABEL: complex_canonicalize_fmul_half:
; SSE: # %bb.0: # %entry
; SSE-NEXT: pushq %rax
-; SSE-NEXT: movss %xmm1, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Spill
+; SSE-NEXT: movss %xmm1, (%rsp) # 4-byte Spill
; SSE-NEXT: callq __extendhfsf2 at PLT
-; SSE-NEXT: movss %xmm0, (%rsp) # 4-byte Spill
-; SSE-NEXT: movss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 4-byte Reload
+; SSE-NEXT: movss %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Spill
+; SSE-NEXT: movss (%rsp), %xmm0 # 4-byte Reload
; SSE-NEXT: # xmm0 = mem[0],zero,zero,zero
; SSE-NEXT: callq __extendhfsf2 at PLT
-; SSE-NEXT: movss %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Spill
-; SSE-NEXT: movss (%rsp), %xmm1 # 4-byte Reload
+; SSE-NEXT: movss %xmm0, (%rsp) # 4-byte Spill
+; SSE-NEXT: movss {{[-0-9]+}}(%r{{[sb]}}p), %xmm1 # 4-byte Reload
; SSE-NEXT: # xmm1 = mem[0],zero,zero,zero
; SSE-NEXT: subss %xmm0, %xmm1
; SSE-NEXT: movaps %xmm1, %xmm0
; SSE-NEXT: callq __truncsfhf2 at PLT
; SSE-NEXT: callq __extendhfsf2 at PLT
-; SSE-NEXT: movss %xmm0, (%rsp) # 4-byte Spill
-; SSE-NEXT: addss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 4-byte Folded Reload
+; SSE-NEXT: movss %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Spill
+; SSE-NEXT: addss (%rsp), %xmm0 # 4-byte Folded Reload
; SSE-NEXT: callq __truncsfhf2 at PLT
; SSE-NEXT: callq __extendhfsf2 at PLT
-; SSE-NEXT: subss (%rsp), %xmm0 # 4-byte Folded Reload
+; SSE-NEXT: subss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 4-byte Folded Reload
; SSE-NEXT: callq __truncsfhf2 at PLT
; SSE-NEXT: callq __extendhfsf2 at PLT
-; SSE-NEXT: movss %xmm0, (%rsp) # 4-byte Spill
-; SSE-NEXT: pinsrw $0, {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
-; SSE-NEXT: callq __extendhfsf2 at PLT
-; SSE-NEXT: mulss (%rsp), %xmm0 # 4-byte Folded Reload
+; SSE-NEXT: mulss {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
; SSE-NEXT: callq __truncsfhf2 at PLT
; SSE-NEXT: callq __extendhfsf2 at PLT
-; SSE-NEXT: subss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 4-byte Folded Reload
+; SSE-NEXT: subss (%rsp), %xmm0 # 4-byte Folded Reload
; SSE-NEXT: callq __truncsfhf2 at PLT
; SSE-NEXT: popq %rax
; SSE-NEXT: retq
@@ -100,32 +81,29 @@ define half @complex_canonicalize_fmul_half(half %a, half %b) nounwind {
; AVX-LABEL: complex_canonicalize_fmul_half:
; AVX: # %bb.0: # %entry
; AVX-NEXT: pushq %rax
-; AVX-NEXT: vmovss %xmm1, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Spill
+; AVX-NEXT: vmovss %xmm1, (%rsp) # 4-byte Spill
; AVX-NEXT: callq __extendhfsf2 at PLT
-; AVX-NEXT: vmovss %xmm0, (%rsp) # 4-byte Spill
-; AVX-NEXT: vmovss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 4-byte Reload
+; AVX-NEXT: vmovss %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Spill
+; AVX-NEXT: vmovss (%rsp), %xmm0 # 4-byte Reload
; AVX-NEXT: # xmm0 = mem[0],zero,zero,zero
; AVX-NEXT: callq __extendhfsf2 at PLT
-; AVX-NEXT: vmovss %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Spill
-; AVX-NEXT: vmovss (%rsp), %xmm1 # 4-byte Reload
+; AVX-NEXT: vmovss %xmm0, (%rsp) # 4-byte Spill
+; AVX-NEXT: vmovss {{[-0-9]+}}(%r{{[sb]}}p), %xmm1 # 4-byte Reload
; AVX-NEXT: # xmm1 = mem[0],zero,zero,zero
; AVX-NEXT: vsubss %xmm0, %xmm1, %xmm0
; AVX-NEXT: callq __truncsfhf2 at PLT
; AVX-NEXT: callq __extendhfsf2 at PLT
-; AVX-NEXT: vmovss %xmm0, (%rsp) # 4-byte Spill
-; AVX-NEXT: vaddss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0, %xmm0 # 4-byte Folded Reload
+; AVX-NEXT: vmovss %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Spill
+; AVX-NEXT: vaddss (%rsp), %xmm0, %xmm0 # 4-byte Folded Reload
; AVX-NEXT: callq __truncsfhf2 at PLT
; AVX-NEXT: callq __extendhfsf2 at PLT
-; AVX-NEXT: vsubss (%rsp), %xmm0, %xmm0 # 4-byte Folded Reload
+; AVX-NEXT: vsubss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0, %xmm0 # 4-byte Folded Reload
; AVX-NEXT: callq __truncsfhf2 at PLT
; AVX-NEXT: callq __extendhfsf2 at PLT
-; AVX-NEXT: vmovss %xmm0, (%rsp) # 4-byte Spill
-; AVX-NEXT: vpinsrw $0, {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0, %xmm0
-; AVX-NEXT: callq __extendhfsf2 at PLT
-; AVX-NEXT: vmulss (%rsp), %xmm0, %xmm0 # 4-byte Folded Reload
+; AVX-NEXT: vmulss {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0, %xmm0
; AVX-NEXT: callq __truncsfhf2 at PLT
; AVX-NEXT: callq __extendhfsf2 at PLT
-; AVX-NEXT: vsubss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0, %xmm0 # 4-byte Folded Reload
+; AVX-NEXT: vsubss (%rsp), %xmm0, %xmm0 # 4-byte Folded Reload
; AVX-NEXT: callq __truncsfhf2 at PLT
; AVX-NEXT: popq %rax
; AVX-NEXT: retq
@@ -142,14 +120,8 @@ define half @complex_canonicalize_fmul_half(half %a, half %b) nounwind {
; AVX512-NEXT: vcvtph2ps %xmm2, %xmm2
; AVX512-NEXT: vsubss %xmm0, %xmm2, %xmm0
; AVX512-NEXT: vcvtps2ph $4, %xmm0, %xmm0
-; AVX512-NEXT: vpmovzxwq {{.*#+}} xmm0 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero
; AVX512-NEXT: vcvtph2ps %xmm0, %xmm0
-; AVX512-NEXT: movzwl {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %eax
-; AVX512-NEXT: vmovd %eax, %xmm2
-; AVX512-NEXT: vcvtph2ps %xmm2, %xmm2
-; AVX512-NEXT: vmulss %xmm2, %xmm0, %xmm0
-; AVX512-NEXT: vxorps %xmm2, %xmm2, %xmm2
-; AVX512-NEXT: vmovss {{.*#+}} xmm0 = xmm0[0],xmm2[1,2,3]
+; AVX512-NEXT: vmulss {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0, %xmm0
; AVX512-NEXT: vcvtps2ph $4, %xmm0, %xmm0
; AVX512-NEXT: vcvtph2ps %xmm0, %xmm0
; AVX512-NEXT: vsubss %xmm1, %xmm0, %xmm0
@@ -169,80 +141,63 @@ define void @v_test_canonicalize_v2half(<2 x half> addrspace(1)* %out) nounwind
; SSE-LABEL: v_test_canonicalize_v2half:
; SSE: # %bb.0: # %entry
; SSE-NEXT: pushq %rbx
-; SSE-NEXT: subq $48, %rsp
+; SSE-NEXT: subq $32, %rsp
; SSE-NEXT: movq %rdi, %rbx
; SSE-NEXT: pinsrw $0, 2(%rdi), %xmm0
; SSE-NEXT: movdqa %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
; SSE-NEXT: pinsrw $0, (%rdi), %xmm0
; SSE-NEXT: callq __extendhfsf2 at PLT
-; SSE-NEXT: movd %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Folded Spill
-; SSE-NEXT: pinsrw $0, {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
-; SSE-NEXT: callq __extendhfsf2 at PLT
-; SSE-NEXT: movd %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Folded Spill
-; SSE-NEXT: movss {{[-0-9]+}}(%r{{[sb]}}p), %xmm1 # 4-byte Reload
-; SSE-NEXT: # xmm1 = mem[0],zero,zero,zero
-; SSE-NEXT: mulss %xmm0, %xmm1
-; SSE-NEXT: movaps %xmm1, %xmm0
+; SSE-NEXT: mulss {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
; SSE-NEXT: callq __truncsfhf2 at PLT
-; SSE-NEXT: movaps %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; SSE-NEXT: movaps %xmm0, (%rsp) # 16-byte Spill
; SSE-NEXT: movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
; SSE-NEXT: callq __extendhfsf2 at PLT
-; SSE-NEXT: mulss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 4-byte Folded Reload
+; SSE-NEXT: mulss {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
; SSE-NEXT: callq __truncsfhf2 at PLT
; SSE-NEXT: pextrw $0, %xmm0, %eax
; SSE-NEXT: movw %ax, 2(%rbx)
-; SSE-NEXT: movdqa {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; SSE-NEXT: movdqa (%rsp), %xmm0 # 16-byte Reload
; SSE-NEXT: pextrw $0, %xmm0, %eax
; SSE-NEXT: movw %ax, (%rbx)
-; SSE-NEXT: addq $48, %rsp
+; SSE-NEXT: addq $32, %rsp
; SSE-NEXT: popq %rbx
; SSE-NEXT: retq
;
; AVX-LABEL: v_test_canonicalize_v2half:
; AVX: # %bb.0: # %entry
; AVX-NEXT: pushq %rbx
-; AVX-NEXT: subq $48, %rsp
+; AVX-NEXT: subq $32, %rsp
; AVX-NEXT: movq %rdi, %rbx
; AVX-NEXT: vpinsrw $0, 2(%rdi), %xmm0, %xmm0
; AVX-NEXT: vmovdqa %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
; AVX-NEXT: vpinsrw $0, (%rdi), %xmm0, %xmm0
; AVX-NEXT: callq __extendhfsf2 at PLT
-; AVX-NEXT: vmovd %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Folded Spill
-; AVX-NEXT: vpinsrw $0, {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0, %xmm0
-; AVX-NEXT: callq __extendhfsf2 at PLT
-; AVX-NEXT: vmovd %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Folded Spill
-; AVX-NEXT: vmulss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0, %xmm0 # 4-byte Folded Reload
+; AVX-NEXT: vmulss {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0, %xmm0
; AVX-NEXT: callq __truncsfhf2 at PLT
-; AVX-NEXT: vmovaps %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; AVX-NEXT: vmovaps %xmm0, (%rsp) # 16-byte Spill
; AVX-NEXT: vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
; AVX-NEXT: callq __extendhfsf2 at PLT
-; AVX-NEXT: vmulss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0, %xmm0 # 4-byte Folded Reload
+; AVX-NEXT: vmulss {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0, %xmm0
; AVX-NEXT: callq __truncsfhf2 at PLT
; AVX-NEXT: vpextrw $0, %xmm0, 2(%rbx)
-; AVX-NEXT: vmovdqa {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
+; AVX-NEXT: vmovdqa (%rsp), %xmm0 # 16-byte Reload
; AVX-NEXT: vpextrw $0, %xmm0, (%rbx)
-; AVX-NEXT: addq $48, %rsp
+; AVX-NEXT: addq $32, %rsp
; AVX-NEXT: popq %rbx
; AVX-NEXT: retq
;
; AVX512-LABEL: v_test_canonicalize_v2half:
; AVX512: # %bb.0: # %entry
; AVX512-NEXT: vmovd {{.*#+}} xmm0 = mem[0],zero,zero,zero
-; AVX512-NEXT: movzwl {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %eax
-; AVX512-NEXT: vmovd %eax, %xmm1
-; AVX512-NEXT: vcvtph2ps %xmm1, %xmm1
-; AVX512-NEXT: vpshufb {{.*#+}} xmm2 = xmm0[2,3],zero,zero,zero,zero,zero,zero,xmm0[u,u,u,u,u,u,u,u]
-; AVX512-NEXT: vcvtph2ps %xmm2, %xmm2
-; AVX512-NEXT: vmulss %xmm1, %xmm2, %xmm2
-; AVX512-NEXT: vxorps %xmm3, %xmm3, %xmm3
-; AVX512-NEXT: vmovss {{.*#+}} xmm2 = xmm2[0],xmm3[1,2,3]
-; AVX512-NEXT: vcvtps2ph $4, %xmm2, %xmm2
-; AVX512-NEXT: vpmovzxwq {{.*#+}} xmm0 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero
+; AVX512-NEXT: vcvtph2ps %xmm0, %xmm1
+; AVX512-NEXT: vmovss {{.*#+}} xmm2 = [1.0E+0,0.0E+0,0.0E+0,0.0E+0]
+; AVX512-NEXT: vmulss %xmm2, %xmm1, %xmm1
+; AVX512-NEXT: vcvtps2ph $4, %xmm1, %xmm1
+; AVX512-NEXT: vpsrld $16, %xmm0, %xmm0
; AVX512-NEXT: vcvtph2ps %xmm0, %xmm0
-; AVX512-NEXT: vmulss %xmm1, %xmm0, %xmm0
-; AVX512-NEXT: vmovss {{.*#+}} xmm0 = xmm0[0],xmm3[1,2,3]
+; AVX512-NEXT: vmulss %xmm2, %xmm0, %xmm0
; AVX512-NEXT: vcvtps2ph $4, %xmm0, %xmm0
-; AVX512-NEXT: vpunpcklwd {{.*#+}} xmm0 = xmm0[0],xmm2[0],xmm0[1],xmm2[1],xmm0[2],xmm2[2],xmm0[3],xmm2[3]
+; AVX512-NEXT: vpunpcklwd {{.*#+}} xmm0 = xmm1[0],xmm0[0],xmm1[1],xmm0[1],xmm1[2],xmm0[2],xmm1[3],xmm0[3]
; AVX512-NEXT: vmovd %xmm0, (%rdi)
; AVX512-NEXT: retq
entry:
More information about the llvm-commits
mailing list