[PATCH] D59378: [InstCombine] Prevent icmp transform that can cause inf loop if part of min/max

[Teresa Johnson via Phabricator via llvm-commits]

tejohnson created this revision.
tejohnson added a reviewer: spatel.
Herald added a subscriber: jdoerfert.
Herald added a project: LLVM.

I found an infinite loop due to repeatedly canonicalizing a min/max
cmp/select sequence in canonicalizeMinMaxWithConstant due to a
competing transformation on that icmp due to a dominating cmp.

There is already code in the ICmp folder to try to detect and block
ICmp transformations leading to such infinite loops. Prior fixes along
these lines have extracted and moved the offending ICmp transforms below
the check (e.g. r293345 and r315895). Rather than take that approach, I
moved the detection of the possible min/max sequence earlier, and pass a
flag into foldICmpWithDominatingICmp.

Added a test to minmax-fold.ll that will infinitely loop without this
patch. I also added a test (not subject to the min/max) of this
particular ICmp transform to icmp-dom.ll, since it does not appear that
one exists (when I block that transformation completely, nothing
fails!).

Note that I could not figure out how to trigger the infinite loop with
the NE transformation just below the EQ transformation I am guarding.
Which doesn't mean it can't happen, but guarding that one too resulted
in some test case failures (as we blocked some expected
transformations). In general, I'm concerned that this possibility exists
with other ICmp transformations, and that they are being worked around
one by one as they are encountered in the wild, but I don't have
the expertise here to do a broader assessment/fix.


Repository:
  rL LLVM

https://reviews.llvm.org/D59378

Files:
  lib/Transforms/InstCombine/InstCombineCompares.cpp
  lib/Transforms/InstCombine/InstCombineInternal.h
  test/Transforms/InstCombine/icmp-dom.ll
  test/Transforms/InstCombine/minmax-fold.ll

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D59378.190693.patch
Type: text/x-patch
Size: 7084 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20190314/8350c6ec/attachment.bin>