[llvm] [AIX BB] Fix IS_FPCLASS expansion crash for mask fcInf|fcNan on targets without SETUEQ (PR #213298)

[via llvm-commits]

llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-llvm-selectiondag

Author: Daniel Chen (DanielCChen)

<details>
<summary>Changes</summary>

**Problem**
PR #<!-- -->212565 caused AIX buildbot to fail when compiling the compiler-rt builtins library for powerpc64-ibm-aix with -m32 crashes with:

```
fatal error: error in backend: Cannot select: t9: i1 = is_fpclass nofpexcept t8, TargetConstant:i32<519>
```

**Root cause**
The crash involves a chain of four steps:

- Step 1 — IS_FPCLASS registered as Custom for all PPC targets.

7e1aba74 moved the setOperationAction(ISD::IS_FPCLASS, MVT::f32/f64, Custom) registration to an unconditional block, so all PPC targets — including powerpc64-ibm-aix -m32 with -mcpu=pwr7 — now mark IS_FPCLASS as Custom.

- Step 2 — SimplifySetCC injects a new IS_FPCLASS(519) node post-legalize.

Because IS_FPCLASS is Custom, SelectionDAGBuilder emits SETUEQ(fabs(x), +inf) as a SETCC node directly (skipping the early expandIS_FPCLASS call). During LegalizeDAG, the illegal SETUEQ node is processed by SimplifySetCC, which sees IS_FPCLASS is LegalOrCustom and transforms SETUEQ(fabs(x), +inf) into a new IS_FPCLASS(x, fcInf|fcNan=519) node — injected post-legalize.

- Step 3 — LowerIS_FPCLASS cannot handle mask 519.

The new IS_FPCLASS(519) node goes through LowerOperation → LowerIS_FPCLASS. On non-P9 targets, LowerIS_FPCLASS only handles fcNan (3) and ~fcNan (1020) via fcmpu/xscmpudp. For all other masks including 519 it returns SDValue(), causing LegalizeDAG to fall through to ExpandNode → expandIS_FPCLASS.

- Step 4 — expandIS_FPCLASS cannot expand mask 519 post-legalize on PPC.

Inside expandIS_FPCLASS, the existing handler for fcInf|fcNan uses UnorderedCmpOpcode which is SETUEQ. On PPC, SETUEQ is marked Expand (setCondCodeAction(ISD::SETUEQ, MVT::f64, Expand)), so isCondCodeLegalOrCustom(SETUEQ) returns false and that path is skipped. The function then falls to the integer bitcast path, which on pwr7 (no Direct Move) emits BITCAST(f64 → i64) — also not selectable. The IS_FPCLASS node reaches instruction selection unexpanded, causing the crash.

**Fix**
Add a fallback expansion for FPTestMask == (fcInf|fcNan) in expandIS_FPCLASS, inside the existing nofpexcept block, using SETUGE instead of SETUEQ:

```
!isfinite(x)  ≡  fabs(x) u>= +inf   (SETUGE)
```

SETUGE on scalar f64 is legal on PPC (unlike SETUEQ), so this expansion succeeds where the existing one fails. The new block is placed immediately after the existing fcInf|fcNan block so it is only reached when SETUEQ is unavailable on the target. All conditions (isCondCodeLegalOrCustom(SETUGE), isOperationLegalOrCustom(FABS), isOperationLegal(ConstantFP)) mirror the guards in the existing block above and are each necessary.



---
Full diff: https://github.com/llvm/llvm-project/pull/213298.diff


1 Files Affected:

- (modified) llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp (+19) 


``````````diff
diff --git a/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp b/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
index 6f47bf1b81c57..66072d2f7393a 100644
--- a/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
@@ -10470,6 +10470,25 @@ SDValue TargetLowering::expandIS_FPCLASS(EVT ResultVT, SDValue Op,
                           IsOrderedInf ? OrderedCmpOpcode : UnorderedCmpOpcode);
     }
 
+    // !isfinite(x) ==> fabs(x) u>= +inf
+    // Handles the case where the existing fcInf|fcNan block above fails
+    // because UnorderedCmpOpcode (SETUEQ) is not legal on the target (e.g.
+    // PPC), but SETUGE is.  When FPTestMask == fcInf|fcNan,
+    // invertFPClassTestIfSimpler never inverts it (519 has fewer bits than
+    // its complement 504), so IsInvertedFP is always false here.
+    if (FPTestMask == (fcInf | fcNan) &&
+        isCondCodeLegalOrCustom(ISD::SETUGE,
+                                OperandVT.getScalarType().getSimpleVT()) &&
+        isOperationLegalOrCustom(ISD::FABS, OperandVT.getScalarType()) &&
+        (isOperationLegal(ISD::ConstantFP, OperandVT.getScalarType()) ||
+         (OperandVT.isVector() &&
+          isOperationLegalOrCustom(ISD::BUILD_VECTOR, OperandVT)))) {
+      SDValue Abs = DAG.getNode(ISD::FABS, DL, OperandVT, Op);
+      SDValue Inf =
+          DAG.getConstantFP(APFloat::getInf(Semantics), DL, OperandVT);
+      return DAG.getSetCC(DL, ResultVT, Abs, Inf, ISD::SETUGE);
+    }
+
     if ((OrderedFPTestMask == fcPosInf || OrderedFPTestMask == fcNegInf) &&
         isCondCodeLegalOrCustom(IsOrdered ? OrderedCmpOpcode
                                           : UnorderedCmpOpcode,

``````````

</details>


https://github.com/llvm/llvm-project/pull/213298