[PATCH] D24143: Fix incorrect folding of an ordered fcmp with a vector of all NaN.

[Andrea Di Biagio via llvm-commits]

andreadb created this revision.
andreadb added reviewers: majnemer, spatel.
andreadb added a subscriber: llvm-commits.

This patch fixes a crash caused by an incorrect folding of an ordered comparison between a packed floating point vector and a splat vector of NaN.

An ordered comparison between a vector and a constant vector of NaN, should always be folded into a constant vector where each element is `i1 false`.
Since revision 266175, SimplifyFCmpInst folds the ordered fcmp into a scalar i1 'false'. Later on, this would cause an assertion failure, since the value type of the folded value doesn't match the expected value type of the uses of the original instruction:
"Assertion failed: New->getType() == getType() && "replaceAllUses of value with new value of different type!".

This patch fixes the issue and adds a test case to the already existing test InstSimplify/floating-point-compares.ll.

Please let me know if okay to commit.

Thanks,
Andrea

https://reviews.llvm.org/D24143

Files:
  lib/Analysis/InstructionSimplify.cpp
  test/Transforms/InstSimplify/floating-point-compare.ll

Index: test/Transforms/InstSimplify/floating-point-compare.ll
===================================================================
--- test/Transforms/InstSimplify/floating-point-compare.ll
+++ test/Transforms/InstSimplify/floating-point-compare.ll
@@ -123,3 +123,11 @@
   %cmp = fcmp nnan ord double %in1, %in2
   ret i1 %cmp
 }
+
+define <2 x i1> @orderedCompareWithNaNVector(<2 x double> %A) {
+; CHECK-LABEL: @orderedCompareWithNaNVector(
+; CHECK: ret <2 x i1> zeroinitializer
+  %cmp = fcmp olt <2 x double> %A, <double 0xFFFFFFFFFFFFFFFF, double 0xFFFFFFFFFFFFFFFF>
+  ret <2 x i1> %cmp
+}
+
Index: lib/Analysis/InstructionSimplify.cpp
===================================================================
--- lib/Analysis/InstructionSimplify.cpp
+++ lib/Analysis/InstructionSimplify.cpp
@@ -3235,7 +3235,7 @@
     // If the constant is a nan, see if we can fold the comparison based on it.
     if (CFP->getValueAPF().isNaN()) {
       if (FCmpInst::isOrdered(Pred)) // True "if ordered and foo"
-        return ConstantInt::getFalse(CFP->getContext());
+        return ConstantInt::get(GetCompareTy(LHS), 0);
       assert(FCmpInst::isUnordered(Pred) &&
              "Comparison must be either ordered or unordered!");
       // True if unordered.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D24143.70024.patch
Type: text/x-patch
Size: 1255 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20160901/9b8619ce/attachment.bin>