[llvm] [X86][ISel] Fix VPTERNLOG matching ensuring the InnerOp is logicOp (PR #166591)

[via llvm-commits]

llvmbot wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-backend-x86

Author: Yi-Chi Lee (yichi170)

<details>
<summary>Changes</summary>

This patch fixes a crash in `tryVPTERNLOG` when trying to peel out the outer not in cases like `~(A | B | C)`.

Previously, `InnerOp` was taken directly from `Op->getOperand(0)` before verifying that it was a logical operation. As a result, the code could later access `InnerOp->getOperand(0)` or `InnerOp->getOperand(1)` even when `InnerOp` was something like a bitcast, causing an error.

This patch applies `getFoldableLogicOp` to `InnerOp`, ensuring that `InnerOp` is a valid logic operation before it is dereferenced.

A new test is added to reproduce the same crash mentioned in #<!-- -->164863 by @<!-- -->RKSimon .

---
Full diff: https://github.com/llvm/llvm-project/pull/166591.diff


2 Files Affected:

- (modified) llvm/lib/Target/X86/X86ISelDAGToDAG.cpp (+2-2) 
- (renamed) llvm/test/CodeGen/X86/vpternlog.ll (+12) 


``````````diff
diff --git a/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp b/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp
index d4418c8563780..6c16fcfb282e8 100644
--- a/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp
+++ b/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp
@@ -4728,9 +4728,9 @@ bool X86DAGToDAGISel::tryVPTERNLOG(SDNode *N) {
   auto tryPeelOuterNotWrappingLogic = [&](SDNode *Op) {
     if (Op->getOpcode() == ISD::XOR && Op->hasOneUse() &&
         ISD::isBuildVectorAllOnes(Op->getOperand(1).getNode())) {
-      SDValue InnerOp = Op->getOperand(0);
+      SDValue InnerOp = getFoldableLogicOp(Op->getOperand(0));
 
-      if (!getFoldableLogicOp(InnerOp))
+      if (!InnerOp)
         return SDValue();
 
       N0 = InnerOp.getOperand(0);
diff --git a/llvm/test/CodeGen/X86/issue163738.ll b/llvm/test/CodeGen/X86/vpternlog.ll
similarity index 59%
rename from llvm/test/CodeGen/X86/issue163738.ll
rename to llvm/test/CodeGen/X86/vpternlog.ll
index 61fe043a970dd..bd7478d3a82d5 100644
--- a/llvm/test/CodeGen/X86/issue163738.ll
+++ b/llvm/test/CodeGen/X86/vpternlog.ll
@@ -11,3 +11,15 @@ define <8 x i64> @foo(<8 x i64> %a, <8 x i64> %b, <8 x i64> %c) {
   %and3 = xor <8 x i64> %and3.demorgan, splat (i64 -1)
   ret <8 x i64> %and3
 }
+
+define <8 x i64> @xorbitcast(<64 x i8> %a, <64 x i8> %b, <64 x i8> %c) {
+; CHECK-LABEL: xorbitcast:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vpternlogq {{.*#+}} zmm0 = ~(zmm0 | zmm2 | zmm1)
+; CHECK-NEXT:    retq
+  %or1 = or <64 x i8> %a, %b
+  %or2 = or <64 x i8> %or1, %c
+  %cast = bitcast <64 x i8> %or2 to <8 x i64>
+  %xor = xor <8 x i64> %cast, splat (i64 -1)
+  ret <8 x i64> %xor
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/166591