[llvm] [X86][ISel] Improve VPTERNLOG matching for negated logic trees (PR #164863)

[Simon Pilgrim via llvm-commits]

================
@@ -4740,13 +4737,47 @@ bool X86DAGToDAGISel::tryVPTERNLOG(SDNode *N) {
     return SDValue();
   };
 
-  SDValue A, FoldableOp;
-  if ((FoldableOp = getFoldableLogicOp(N1))) {
-    A = N0;
-  } else if ((FoldableOp = getFoldableLogicOp(N0))) {
-    A = N1;
-  } else
-    return false;
+  SDValue N0, N1, A, FoldableOp;
+
+  // Identify and (optionally) peel an outer NOT that wraps a pure logic tree
+  auto tryPeelOuterNotWrappingLogic = [&](SDNode *Op) {
+    if (Op->getOpcode() == ISD::XOR && Op->hasOneUse() &&
+        ISD::isBuildVectorAllOnes(Op->getOperand(1).getNode())) {
+      SDValue InnerOp = Op->getOperand(0);
+
+      if (!getFoldableLogicOp(InnerOp))
+        return SDValue();
----------------
RKSimon wrote:

yes please - I've managed to get a test case to crash here, but you might have ideas for a better one:
```ll
define <32 x i8> @crashme(<32 x i8> %a0) {
  %cmp = icmp ne <32 x i8> %a0, zeroinitializer
  %sext = sext <32 x i1> %cmp to <32 x i8>
  %xor = xor <32 x i8> %a0, %sext
  %shuffle = shufflevector <32 x i8> %xor, <32 x i8> zeroinitializer, <32 x i32> <i32 3, i32 1, i32 3, i32 2, i32 3, i32 2, i32 0, i32 1, i32 0, i32 1, i32 1, i32 2, i32 3, i32 3, i32 1, i32 3, i32 1, i32 3, i32 2, i32 2, i32 0, i32 3, i32 1, i32 1, i32 3, i32 0, i32 0, i32 3, i32 3, i32 0, i32 0, i32 0>
  ret <32 x i8> %shuffle
}
```

https://github.com/llvm/llvm-project/pull/164863