[llvm-dev] Testing compiler reliability using Csmith

[Philip Reames via llvm-dev]

There's also a couple other active fuzzing efforts running concurrently:

  * OSSFuzz is continuously running a set of codegen and individual pass
    fuzzers.  A bit more information can be found here:
    http://llvm.org/docs/FuzzingLLVM.html#continuously-running-and-finding-bugs
  * We (Azul Systems) have a continuously running Java fuzzer which
    exercises LLVM through our Falcon compiler, and regularly finds
    regressions and occasional deep long lurking issues.  We don't have
    a public bug tracker for this, but a sizable portion of our upstream
    bug fixing activity is driven by the output of this tool.

Philip

On 12/7/18 11:04 AM, Zhendong Su via llvm-dev wrote:
> Thanks, Vedant.  Yes, we have done a lot of testing of Clang/LLVM (and 
> GCC) in the past several years (more details at 
> https://people.inf.ethz.ch/suz/emi/index.html):
>
> [GCC/LLVM bugs: *1,602* (total) / *1,007* (fixed)]
> [Reports: GCC (link1 <http://tinyurl.com/ojzdt78>, link2 
> <http://tinyurl.com/oxlkmjc>, link3 <http://tinyurl.com/jd3jnl3>, 
> link4 <http://tinyurl.com/zs2826l>, link5 
> <https://tinyurl.com/yb7okzqd>), LLVM (link1 
> <http://tinyurl.com/qj8x5ry>, link2 <http://tinyurl.com/nc9celr>, 
> link3 <http://tinyurl.com/qy58smb>, link4 
> <http://tinyurl.com/z7y8rwo>, link5 <https://tinyurl.com/y7wdf5b4>)]
>
> Best,
> Zhendong
>
> --------
> Zhendong Su
> Professor, Department of Computer Science, ETH Zurich
> Office: CAB E 64.1, Tel: +41 44 633 77 72
> https://people.inf.ethz.ch/suz/
> --------
>
> On Fri, Dec 7, 2018 at 7:42 PM Vedant Kumar <vsk at apple.com 
> <mailto:vsk at apple.com>> wrote:
>
>     Hello,
>
>     + Zhendong, who’s done a lot of work on automated testing of llvm.
>
>>     On Dec 6, 2018, at 10:15 PM, Radu Ometita via llvm-dev
>>     <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>>
>>     Hello everyone!
>>
>>     We are working on writing a paper about testing the reliability
>>     of C compilers by using Csmith (a random C99 program generator).
>>
>>     A previous testing effort, using Csmith, found 202 LLVM bugs,
>>     which represented 2% of all reported bugs at that time (PDF:
>>     https://www.flux.utah.edu/download?uid=114): . However, after
>>     this paper was published we are unaware of any further testing
>>     using Csmith, and we would like to ask you if you are aware of
>>     any such efforts or further results.
>
>     Just speaking for myself here, I use Csmith as part of my
>     pre-commit testing.
>
>     vedant
>
>>
>>     Best regards,
>>     Radu Ometita,
>>     Functional compilers engineer @IOHK
>>     _______________________________________________
>>     LLVM Developers mailing list
>>     llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>>     http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20181207/b4da2884/attachment.html>