[llvm] [X86] Convert FP compares split by a nested logic op to FP logic (PR #226222)

[Tim Besard via llvm-commits]

================
@@ -52658,9 +52658,60 @@ static unsigned convertIntLogicToFPLogicOpcode(unsigned Opcode) {
   return FPOpcode;
 }
 
+/// Return true if \p V is a tree of single-use vector logic ops over vector
+/// compares of scalar FP values, like the ones convertIntLogicToFPLogic
+/// creates. \p V becomes an operand of new vector logic, from whose extract of
+/// element 0 SimplifyDemandedVectorElts should still reach every compare within
+/// the recursion limit: otherwise the X86ISD::CMPM mask compares are not
+/// scalarized and are widened to 512 bits without AVX512VL.
+static bool isConvertedFPLogic(SDValue V, unsigned Depth) {
+  if (Depth >= SelectionDAG::MaxRecursionDepth)
+    return false;
+  if (V.getOpcode() == ISD::SETCC)
+    return V.getOperand(0).getOpcode() == ISD::SCALAR_TO_VECTOR &&
+           V.getOperand(0).getValueType().isFloatingPoint();
+  if (!ISD::isBitwiseLogicOp(V.getOpcode()) || !V.hasOneUse())
+    return false;
+  return isConvertedFPLogic(V.getOperand(0), Depth + 1) &&
+         isConvertedFPLogic(V.getOperand(1), Depth + 1);
+}
+
+/// If the single-use i1 value \p Op is a scalar FP compare that can be done as
+/// a vector compare, or element 0 of FP compares that were already converted to
+/// vector logic, return the vXi1 type it can be computed in. Otherwise return
+/// an invalid type.
+static MVT getFPLogicBoolVecVT(SDValue Op, const X86Subtarget &Subtarget) {
+  using namespace SDPatternMatch;
+  SDValue Vec, LHS;
+  ISD::CondCode CC;
+  if (!Op.hasOneUse())
+    return MVT();
+  if (sd_match(Op, m_ExtractElt(m_Value(Vec), m_Zero())))
+    return isConvertedFPLogic(Vec, /*Depth=*/1) ? Vec.getSimpleValueType()
+                                                : MVT();
+  if (!sd_match(Op, m_SetCC(m_Value(LHS), m_Value(), m_CondCode(CC))))
+    return MVT();
+
+  // v8f16 compares are only legal with AVX512VL.
+  EVT FPVT = LHS.getValueType();
+  if (!((Subtarget.hasSSE1() && FPVT == MVT::f32) ||
+        (Subtarget.hasSSE2() && FPVT == MVT::f64) ||
+        (Subtarget.hasFP16() && Subtarget.hasVLX() && FPVT == MVT::f16)))
----------------
maleadt wrote:

See https://github.com/llvm/llvm-project/pull/226222#discussion_r4122987409. Without it, the v8f16 compares get scalarized back to vucomish+setcc and go through kmovd/kandw, which is what main already does for plain pairs. With the check they stay as setcc/andb.

https://github.com/llvm/llvm-project/pull/226222