[llvm] r280488 - [instsimplify] Fix incorrect folding of an ordered fcmp with a vector of all NaN.

[Andrea Di Biagio via llvm-commits]

Author: adibiagio
Date: Fri Sep  2 09:47:43 2016
New Revision: 280488

URL: http://llvm.org/viewvc/llvm-project?rev=280488&view=rev
Log:
[instsimplify] Fix incorrect folding of an ordered fcmp with a vector of all NaN.

This patch fixes a crash caused by an incorrect folding of an ordered comparison
between a packed floating point vector and a splat vector of NaN.

An ordered comparison between a vector and a constant vector of NaN, should
always be folded into a constant vector where each element is i1 false.

Since revision 266175, SimplifyFCmpInst folds the ordered fcmp into a scalar
'false'. Later on, this would cause an assertion failure, since the value type
of the folded value doesn't match the expected value type of the uses of the
original instruction: "Assertion failed: New->getType() == getType() &&
"replaceAllUses of value with new value of different type!".

This patch fixes the issue and adds a test case to the already existing test
InstSimplify/floating-point-compares.ll.

Differential Revision: https://reviews.llvm.org/D24143

Modified:
    llvm/trunk/lib/Analysis/InstructionSimplify.cpp
    llvm/trunk/test/Transforms/InstSimplify/floating-point-compare.ll

Modified: llvm/trunk/lib/Analysis/InstructionSimplify.cpp
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Analysis/InstructionSimplify.cpp?rev=280488&r1=280487&r2=280488&view=diff
==============================================================================
--- llvm/trunk/lib/Analysis/InstructionSimplify.cpp (original)
+++ llvm/trunk/lib/Analysis/InstructionSimplify.cpp Fri Sep  2 09:47:43 2016
@@ -3235,7 +3235,7 @@ static Value *SimplifyFCmpInst(unsigned
     // If the constant is a nan, see if we can fold the comparison based on it.
     if (CFP->getValueAPF().isNaN()) {
       if (FCmpInst::isOrdered(Pred)) // True "if ordered and foo"
-        return ConstantInt::getFalse(CFP->getContext());
+        return ConstantInt::get(GetCompareTy(LHS), 0);
       assert(FCmpInst::isUnordered(Pred) &&
              "Comparison must be either ordered or unordered!");
       // True if unordered.

Modified: llvm/trunk/test/Transforms/InstSimplify/floating-point-compare.ll
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/InstSimplify/floating-point-compare.ll?rev=280488&r1=280487&r2=280488&view=diff
==============================================================================
--- llvm/trunk/test/Transforms/InstSimplify/floating-point-compare.ll (original)
+++ llvm/trunk/test/Transforms/InstSimplify/floating-point-compare.ll Fri Sep  2 09:47:43 2016
@@ -123,3 +123,11 @@ define i1 @nonans2(double %in1, double %
   %cmp = fcmp nnan ord double %in1, %in2
   ret i1 %cmp
 }
+
+define <2 x i1> @orderedCompareWithNaNVector(<2 x double> %A) {
+; CHECK-LABEL: @orderedCompareWithNaNVector(
+; CHECK: ret <2 x i1> zeroinitializer
+  %cmp = fcmp olt <2 x double> %A, <double 0xFFFFFFFFFFFFFFFF, double 0xFFFFFFFFFFFFFFFF>
+  ret <2 x i1> %cmp
+}
+