[PATCH] D34503: AVX-512: Fixed a crash during legalization of <3 x i8> type

[Elena Demikhovsky via Phabricator via llvm-commits]

delena created this revision.

The compiler fails with assertion during legalization of SETCC for <3 x i8> operands.
The result is extended to <4 x i8> and then truncated <4 x i1>.  It does not happen on AVX2, because the final result of SETCC is <4 x i32>.


Repository:
  rL LLVM

https://reviews.llvm.org/D34503

Files:
  lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp
  test/CodeGen/X86/avx512-vec3-crash.ll


Index: test/CodeGen/X86/avx512-vec3-crash.ll
===================================================================
--- test/CodeGen/X86/avx512-vec3-crash.ll
+++ test/CodeGen/X86/avx512-vec3-crash.ll
@@ -0,0 +1,9 @@
+; RUN: llc -mcpu=skx %s  -o /dev/null
+
+; This test crashed during type legalization of SETCC result type.
+define <3 x i8 > @foo(<3 x i8>%x, <3 x i8>%a, <3 x i8>%b) {
+  %cmp.i = icmp slt <3 x i8> %x, %a
+  %res = sext <3 x i1> %cmp.i to <3 x i8>
+  ret <3 x i8> %res
+}
+
Index: lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp
===================================================================
--- lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp
+++ lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp
@@ -615,9 +615,11 @@
   SDValue SetCC = DAG.getNode(N->getOpcode(), dl, SVT, LHS, RHS,
                               N->getOperand(2));
 
-  assert(NVT.bitsLE(SVT) && "Integer type overpromoted?");
   // Convert to the expected type.
-  return DAG.getNode(ISD::TRUNCATE, dl, NVT, SetCC);
+  if (NVT.bitsLE(SVT))
+    return DAG.getNode(ISD::TRUNCATE, dl, NVT, SetCC);
+
+  return DAG.getNode(ISD::SIGN_EXTEND, dl, NVT, SetCC);
 }
 
 SDValue DAGTypeLegalizer::PromoteIntRes_SHL(SDNode *N) {


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D34503.103537.patch
Type: text/x-patch
Size: 1213 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20170622/9530ded3/attachment.bin>