[llvm] [InstCombine] canonicalize sign bit checks (PR #122962)

[Jacob Young via llvm-commits]

================
@@ -7025,6 +7027,66 @@ static Instruction *canonicalizeICmpBool(ICmpInst &I,
   }
 }
 
+// (icmp X, Y) --> (icmp slt/sgt X, 0/-1) iff Y is outside the signed range of X
+static ICmpInst *canonicalizeSignBitCheck(ICmpInst::Predicate Pred, Value *X,
+                                          const ConstantRange &XRange,
+                                          const ConstantRange &YRange) {
+  if (XRange.isSignWrappedSet())
+    return nullptr;
+  unsigned BitWidth = XRange.getBitWidth();
+  APInt SMin = APInt::getSignedMinValue(BitWidth);
+  APInt Zero = APInt::getZero(BitWidth);
+  auto NegResult =
+      XRange.intersectWith(ConstantRange(SMin, Zero), ConstantRange::Signed)
+          .icmpOrInverse(Pred, YRange);
+  if (!NegResult)
+    return nullptr;
+  auto PosResult =
+      XRange.intersectWith(ConstantRange(Zero, SMin), ConstantRange::Signed)
+          .icmpOrInverse(Pred, YRange);
+  if (!PosResult)
+    return nullptr;
+  assert(NegResult != PosResult &&
+         "Known result should been simplified already.");
+  Type *Ty = X->getType();
+  if (*NegResult)
+    return new ICmpInst(ICmpInst::ICMP_SLT, X, ConstantInt::getNullValue(Ty));
+  return new ICmpInst(ICmpInst::ICMP_SGT, X, ConstantInt::getAllOnesValue(Ty));
+}
----------------
jacobly0 wrote:

Yep, that totally makes sense now, and I don't really have a better solution to propose at the moment.

Do you have a suggestion for how to proceed with my original `add (sdiv X, C), sext i1 (icmp ugt (srem X, C), smin)` => `ashr X, log2(C)` use case in the short term, and in a way that won't need to be reverted?  Is my original PR that only applies to `srem` seem restricted enough enough to avoid major regressions (especially given the relative rareness of `srem`), or should I revert back to something like my initial code which only applied where the full original `foldICmpSRemConstant` is applicable, just slightly improved based on what I have learned from these generalizations?

https://github.com/llvm/llvm-project/pull/122962