[llvm] [ValueTracking] Improve KnownBits for signed min-max clamping (PR #120576)

[via llvm-commits]

adam-bzowski wrote:

This is a reworking of the solution to the problem investigated in this PR: https://github.com/llvm/llvm-project/pull/119577. Its original goal was reformulated to the problem of type narrowing in the presence of a signed min-max clamp. A signed min-max clamp is the sequence of smin and smax intrinsics, which constrain a signed value into the range: smin <= value <= smax. The patch improves the calculation of KnownBits for a value subjected to the signed clamping.

When calculating KnownBits, whenever smin or smax is encountered, the patch checks if it is a part of the smin-smax clamp. If it is the case, it narrows the KnownBits accordingly. To see why such an improvement was needed, consider a simple clamp of the form

y = smax(smin(x, 0xFF), 0)

for a signed variable x. Without the patch, KnownBits cannot infer anything about smin(x, 100) and thus it only treats y as non-negative (leading bit = 0). With the patch, KnownBits correctly identifies all the leading bits but the 8 lowest ones as vanishing.

The submission essentially consists of 6 additional lines of code (definition of unionWithMinMaxIntrinsicClamp and 2 calls). The remaining change is moving the definition of isSignedMinMaxIntrinsicClamp before its use.

https://github.com/llvm/llvm-project/pull/120576