[llvm] [InstCombine] canonicalize sign bit checks (PR #122962)

[via llvm-commits]

================
@@ -7025,6 +7027,66 @@ static Instruction *canonicalizeICmpBool(ICmpInst &I,
   }
 }
 
+// (icmp X, Y) --> (icmp slt/sgt X, 0/-1) iff Y is outside the signed range of X
+static ICmpInst *canonicalizeSignBitCheck(ICmpInst::Predicate Pred, Value *X,
+                                          const ConstantRange &XRange,
+                                          const ConstantRange &YRange) {
+  if (XRange.isSignWrappedSet())
+    return nullptr;
+  unsigned BitWidth = XRange.getBitWidth();
+  APInt SMin = APInt::getSignedMinValue(BitWidth);
+  APInt Zero = APInt::getZero(BitWidth);
+  auto NegResult =
+      XRange.intersectWith(ConstantRange(SMin, Zero), ConstantRange::Signed)
+          .icmpOrInverse(Pred, YRange);
+  if (!NegResult)
+    return nullptr;
+  auto PosResult =
+      XRange.intersectWith(ConstantRange(Zero, SMin), ConstantRange::Signed)
+          .icmpOrInverse(Pred, YRange);
+  if (!PosResult)
+    return nullptr;
+  assert(NegResult != PosResult &&
+         "Known result should been simplified already.");
+  Type *Ty = X->getType();
+  if (*NegResult)
+    return new ICmpInst(ICmpInst::ICMP_SLT, X, ConstantInt::getNullValue(Ty));
+  return new ICmpInst(ICmpInst::ICMP_SGT, X, ConstantInt::getAllOnesValue(Ty));
+}
----------------
goldsteinn wrote:

I think is depends heavily on the use. I can see the case where if you are branching on the `cmp` then maximally constraining it in one direction leaves us with the least useful region on the other side. But in other use-cases whether only the `true` (or `false`) value matters we would almost certaintly want maximal constraint. I.e `(select (icmp ult X, 4), f(X), C)` is almost always better `(select (icmp sge X, 0), f(X), C)`.

I guess I can see sign-compare is a general middle-ground. Although I think this fold is liable to cause regressions in cases that it relaxes the constraints on a `cmp` that dominates some important expressions.

Truthfully not sure what the correct approach here is. I guess in some ideal case I would think some cost-function to decide which direction of the `cmp` is liable to benefit most from being maximally constrained, altough that might be overkill, especially in InstCombine.

https://github.com/llvm/llvm-project/pull/122962