[PATCH] D38546: [ConstantFolding] Avoid assert when folding ptrtoint of vectorized GEP

[Bjorn Pettersson via Phabricator via llvm-commits]

bjope created this revision.

Got asserts in llvm::CastInst::getCastOpcode saying:
`DestBits == SrcBits && "Illegal cast to vector (wrong type or size)"' failed.

Problem seemed to be that llvm::ConstantFoldCastInstruction did
not handle ptrtoint cast of a getelementptr returning a vector
correctly. I assume such situations are quite rare, since the
GEP needs to be considered as a constant value (base pointer
being null).
The solution used here is to simply avoid the constant fold
of ptrtoint when the value is a vector. It is not supported,
and by bailing out we do not fail on assertions later on.

Change-Id: Ic8fb38e2014cdf2cf4027b10b9535c32fe044a33


https://reviews.llvm.org/D38546

Files:
  lib/IR/ConstantFold.cpp
  test/Analysis/ConstantFolding/cast-vector.ll


Index: test/Analysis/ConstantFolding/cast-vector.ll
===================================================================
--- /dev/null
+++ test/Analysis/ConstantFolding/cast-vector.ll
@@ -0,0 +1,17 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -instsimplify -S | FileCheck %s
+
+; The GEP is a Constant value, but the ptrtoint operand is not an isa<ConstantVector>.
+; This used to crash due to not supporting vectors in llvm::ConstantFoldCastInstruction
+; when handling the ptrtoint.
+define <2 x i16> @test1() {
+; CHECK-LABEL: @test1(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    ret <2 x i16> ptrtoint (<2 x i32*> getelementptr ([10 x i32], [10 x i32]* null, <2 x i64> zeroinitializer, <2 x i64> <i64 5, i64 7>) to <2 x i16>)
+;
+entry:
+  %gep = getelementptr inbounds [10 x i32], [10 x i32]* null, i16 0, <2 x i16> <i16 5, i16 7>
+  %vec = ptrtoint <2 x i32*> %gep to <2 x i16>
+  ret <2 x i16> %vec
+}
+
Index: lib/IR/ConstantFold.cpp
===================================================================
--- lib/IR/ConstantFold.cpp
+++ lib/IR/ConstantFold.cpp
@@ -620,6 +620,8 @@
       return ConstantPointerNull::get(cast<PointerType>(DestTy));
     return nullptr;                   // Other pointer types cannot be casted
   case Instruction::PtrToInt:   // always treated as unsigned
+    if (V->getType()->isVectorTy())
+      return nullptr;
     // Is it a null pointer value?
     if (V->isNullValue())
       return ConstantInt::get(DestTy, 0);


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D38546.117678.patch
Type: text/x-patch
Size: 1506 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20171004/24f54fe4/attachment.bin>