[llvm-bugs] [Bug 47973] New: Undefined behavior in int division with INT_MIN param

[via llvm-bugs]

https://bugs.llvm.org/show_bug.cgi?id=47973

            Bug ID: 47973
           Summary: Undefined behavior in int division with INT_MIN param
           Product: compiler-rt
           Version: 11.0
          Hardware: All
                OS: All
            Status: NEW
          Severity: normal
          Priority: P
         Component: builtins
          Assignee: unassignedbugs at nondot.org
          Reporter: ayshekuran at gmail.com
                CC: llvm-bugs at lists.llvm.org

In the implementation for software integer division a/b (and mod a%b) in
int_div_impl.inc, the absolute values of a and b are calculated like so:

a = (a ^ s_a) - s_a;

(The variable s_a is -1 or 0 depending on the signedness of a.)

In the case where a (or b) is INT_MIN, this can mean that we calculate INT_MAX
+ 1 in signed integer operations. Calculating these as unsigned integers would
avoid this undefined behaviour.

In a similar fashion, the return statement can result in unsigned underflow
because it substracts s_a which gets promoted to unsigned UINT_MAX due to the
COMPUTE_UDIV being unsigned. Rewriting it as + (-s_a) would remove the
undefined behaviour.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-bugs/attachments/20201026/9713a3f2/attachment.html>