[llvm-dev] Integer -> Floating point -> Integer cast optimizations

[Carlos Liam via llvm-dev]

I brought this up in IRC and was told to consult someone who knows more about floating point numbers; I propose an optimization as follows.

Let's say we have an int x, and we cast it to a float and back. Floats have 8 exponent bits and 23 mantissa bits.

If x matches the condition `countTrailingZeros(abs(x)) > (log2(abs(x)) - 23)`, then we can remove the float casts.

So, if we can establish that abs(x) is <= 2**23, we can remove the casts. LLVM does not currently perform that optimization on this C code:

int floatcast(int x) {
    if (abs(x) <= 16777216) { // abs(x) is definitely <= 2**23 and fits into our mantissa cleanly
        float flt = (float)x;
        return (int)flt;
    }
    return x;
}

Things get more interesting when you bring in higher integers and leading zeros. Floating point can't exactly represent integers that don't fit neatly into the mantissa; they have to round to a multiple of some power of 2. For example, integers between 2**23 and 2**24 round to a multiple of 2**1 - meaning that the result has *at least* 1 trailing zero. Integers between 2**24 and 2**25 round to a multiple of 2**2 - with the result having at least 2 trailing zeros. Et cetera. If we can prove that the input to these casts fits in between one of those ranges *and* has at least the correct number of leading zeros, we can eliminate the casts. LLVM does not currently perform this optimization on this C code:

int floatcast(int x) {
    if (16777217 <= abs(x) && abs(x) <= 33554432) { // abs(x) is definitely between 2**23 and 2**24
        float flt = (float)(x / abs(x) * (abs(x) & (UINT32_MAX ^ 2))); // what's being casted to float definitely has at least one trailing zero in its absolute value
        return (int)flt;
    }
    return x;
}


 - CL