[PATCH] D46866: [EarlyCSE] Avoid a poorly defined instruction comparison

[Jeremy Morse via Phabricator via llvm-commits]

jmorse created this revision.
jmorse added reviewers: reames, anna, gberry.
Herald added a subscriber: llvm-commits.

EarlyCSE currently attempts to eliminate equivalent comparisons like this:

  fcmp ugt %a, %b
  fcmp ult %b, %a

where the operands are swapped, by using getSwappedPredicate() to find what the equivalent predicate would be. However, it does not take account of comparisons between identical operands, like "fcmp ugt %a, %a". For identical operands getSwappedPredicate() is poorly defined as there is no other predicate that computes the same result. The net effect is that some instruction pairs hash differently, but compare equally, using hasher DenseMapInfo<SimpleValue>::getHashValue and equality DenseMapInfo<SimpleValue>::isEqual. For example:

  fcmp ugt %a, %a
  fcmp ult %a, %a

have different predicates as inputs to the hash, but then compare equal because one of the predicates gets flipped in the equality function.

DenseMap doesn't expect inconsistent hash/equality, which has tripped assertion failures in online fuzzers [0] as least once. However because EarlyCSE (legitimately) uses pointer values in its hash, that doesn't happen often, instead it usually causes nondeterministic behaviour. Running the test in this patch results in a different output to the auto-generated one about 50% of the time on Ubuntu 18.04 and LLVM compiled with clang / libstdc++.

The functional change in this patch filters out the edge case of identical operands. The regression test reproduces the problem with reasonable frequency, however it's not clear how useful it is in general: because the broken behaviour depends on pointer values, it's hard to test.

[0] https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=7892


Repository:
  rL LLVM

https://reviews.llvm.org/D46866

Files:
  lib/Transforms/Scalar/EarlyCSE.cpp
  test/Transforms/EarlyCSE/sameoperand-cmp.ll

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D46866.146767.patch
Type: text/x-patch
Size: 34677 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20180515/6a438280/attachment-0001.bin>