[llvm-bugs] [Bug 37304] New: calloc() overflow test eliminated by optimization

[via llvm-bugs]

https://bugs.llvm.org/show_bug.cgi?id=37304

            Bug ID: 37304
           Summary: calloc() overflow test eliminated by optimization
           Product: clang
           Version: 6.0
          Hardware: All
                OS: All
            Status: NEW
          Severity: normal
          Priority: P
         Component: -New Bugs
          Assignee: unassignedclangbugs at nondot.org
          Reporter: wellons at nullprogram.com
                CC: llvm-bugs at lists.llvm.org

Minimal example:

    #include <stdint.h>
    #include <stdlib.h>
    int test_calloc(void) {
        return !!calloc(SIZE_MAX, 2);
    }

With -O1:

    define i32 @test_calloc() local_unnamed_addr #0 {
        ret i32 1
    }

The calloc() is eliminated because the returned pointer is unused. It's
replaced by a simulated successful return despite the allocation being
impossible, at least on the targeted platforms. In a more complicated scenario,
a later part of the function — such an uneliminated malloc() of the same
allocation size — may rely on calloc()'s overflow test for correct results.

A contrived example of this:

    void *test_overflow(size_t n, size_t m) {
        return calloc(n, m) ? malloc(n * m) : 0;
    }

Compiles to:

    define noalias i8* @test_overflow(i64, i64) local_unnamed_addr #0 {
        %3 = mul i64 %1, %0
        %4 = tail call noalias i8* @malloc(i64 %3) #2
        ret i8* %4
    }

An eliminated calloc() should be replaced by an overflow test.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-bugs/attachments/20180501/432e6b92/attachment.html>