[PATCH] D50465: [InstCombine] Optimize redundant 'signed truncation check pattern'.

[Roman Lebedev via Phabricator via llvm-commits]

lebedev.ri created this revision.
lebedev.ri added reviewers: spatel, craig.topper.

This comes with `Implicit Conversion Sanitizer - integer sign change` (https://reviews.llvm.org/D50250):

  signed char test(unsigned int x) { return x; }

`clang++ -fsanitize=implicit-conversion -S -emit-llvm -o - /tmp/test.cpp -O3`

- Old: F6904292: old.ll <https://reviews.llvm.org/F6904292>
- With this patch: F6904294: new.ll <https://reviews.llvm.org/F6904294>

General pattern:

  X & Y

Where `Y` is checking that all the high bits (covered by a mask `4294967168`)
are uniform, i.e.  `%arg & 4294967168`  can be either  `4294967168`  or  `0`
Pattern can be one of:

  %t = add        i32 %arg,    128
  %r = icmp   ult i32 %t,      256

Or

  %t0 = shl       i32 %arg,    24
  %t1 = ashr      i32 %t0,     24
  %r  = icmp  eq  i32 %t1,     %arg

Or

  %t0 = trunc     i32 %arg  to i8
  %t1 = sext      i8  %t0   to i32
  %r  = icmp  eq  i32 %t1,     %arg

This pattern is a signed truncation check.

And `X` is checking that some bit in that same mask is zero.
I.e. can be one of:

  %r = icmp sgt i32   %arg,    -1

Or

  %t = and      i32   %arg,    2147483648
  %r = icmp eq  i32   %t,      0

Since we are checking that all the bits in that mask are the same,
and a particular bit is zero, what we are really checking is that all the
masked bits are zero.
So this should be transformed to:

  %r = icmp ult i32 %arg, 128

The transform itself ended up being rather horrible, even though i omitted some cases.
Surely there is some infrastructure that can help clean this up that i missed?

Partial alive proofs:
https://rise4fun.com/Alive/br5


Repository:
  rL LLVM

https://reviews.llvm.org/D50465

Files:
  lib/Transforms/InstCombine/InstCombineAndOrXor.cpp
  test/Transforms/InstCombine/signed-truncation-check.ll

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D50465.159752.patch
Type: text/x-patch
Size: 11208 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20180808/abe85dbd/attachment.bin>