[PATCH] D108938: [SelectionDAG] Fix miscompile bugs related to smul.fix.sat with scale zero

[Bjorn Pettersson via Phabricator via llvm-commits]

bjope created this revision.
bjope added reviewers: lebedev.ri, nikic, leonardchan.
Herald added subscribers: ecnelises, pengfei, hiraditya.
bjope requested review of this revision.
Herald added a project: LLVM.

When expanding a SMULFIXSAT ISD node (usually originating from
a smul.fix.sat intrinsic) we've applied some optimizations for
the special case when the scale is zero. The idea has been that
it would be cheaper to use an SMULO instruction (if legal) to
perform the multiplication and at the same time detect any overflow.
And in case of overflow we could use some SELECT:s to replace the
result with the saturated min/max value. The only tricky part
is to know if we overflowed on the min or max value, i.e. if the
product is positive or negative. Unfortunately the implementation
has been incorrect as it has looked at the product returned by the
SMULO to determine the sign of the product. In case of overflow that
product is truncated and won't give us the correct sign bit.

This patch is adding an extra XOR of the multiplication operands,
which is used to determine the sign of the non truncated product.

This patch fixes PR51677.


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D108938

Files:
  llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp
  llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
  llvm/test/CodeGen/X86/smul_fix_sat.ll
  llvm/test/CodeGen/X86/smul_fix_sat_constants.ll

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D108938.369524.patch
Type: text/x-patch
Size: 16020 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20210830/a3b41031/attachment.bin>