[llvm-dev] Testing compiler reliability using Csmith

[Matt P. Dziubinski via llvm-dev]

You may also be interested in the following resources on compilers 
correctness (articles, software, and talks -- from the general topics to 
the ones specifically focused on testing, validation, and verification):

https://github.com/MattPD/cpplinks/blob/master/compilers.correctness.md

Best regards,

Matt P. Dziubinski

On 12/7/2018 20:19, Philip Reames via llvm-dev wrote:

> There's also a couple other active fuzzing efforts running concurrently:
>
>   * OSSFuzz is continuously running a set of codegen and individual
>     pass fuzzers.  A bit more information can be found here:
>     http://llvm.org/docs/FuzzingLLVM.html#continuously-running-and-finding-bugs
>   * We (Azul Systems) have a continuously running Java fuzzer which
>     exercises LLVM through our Falcon compiler, and regularly finds
>     regressions and occasional deep long lurking issues.  We don't
>     have a public bug tracker for this, but a sizable portion of our
>     upstream bug fixing activity is driven by the output of this tool.
>
> Philip
>
> On 12/7/18 11:04 AM, Zhendong Su via llvm-dev wrote:
>> Thanks, Vedant.  Yes, we have done a lot of testing of Clang/LLVM 
>> (and GCC) in the past several years (more details at 
>> https://people.inf.ethz.ch/suz/emi/index.html):
>>
>> [GCC/LLVM bugs: *1,602* (total) / *1,007* (fixed)]
>> [Reports: GCC (link1 <http://tinyurl.com/ojzdt78>, link2 
>> <http://tinyurl.com/oxlkmjc>, link3 <http://tinyurl.com/jd3jnl3>, 
>> link4 <http://tinyurl.com/zs2826l>, link5 
>> <https://tinyurl.com/yb7okzqd>), LLVM (link1 
>> <http://tinyurl.com/qj8x5ry>, link2 <http://tinyurl.com/nc9celr>, 
>> link3 <http://tinyurl.com/qy58smb>, link4 
>> <http://tinyurl.com/z7y8rwo>, link5 <https://tinyurl.com/y7wdf5b4>)]
>>
>> Best,
>> Zhendong
>>
>> --------
>> Zhendong Su
>> Professor, Department of Computer Science, ETH Zurich
>> Office: CAB E 64.1, Tel: +41 44 633 77 72
>> https://people.inf.ethz.ch/suz/
>> --------
>>
>> On Fri, Dec 7, 2018 at 7:42 PM Vedant Kumar <vsk at apple.com 
>> <mailto:vsk at apple.com>> wrote:
>>
>>     Hello,
>>
>>     + Zhendong, who’s done a lot of work on automated testing of llvm.
>>
>>>     On Dec 6, 2018, at 10:15 PM, Radu Ometita via llvm-dev
>>>     <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>>>
>>>     Hello everyone!
>>>
>>>     We are working on writing a paper about testing the reliability
>>>     of C compilers by using Csmith (a random C99 program generator).
>>>
>>>     A previous testing effort, using Csmith, found 202 LLVM bugs,
>>>     which represented 2% of all reported bugs at that time (PDF:
>>>     https://www.flux.utah.edu/download?uid=114): . However, after
>>>     this paper was published we are unaware of any further testing
>>>     using Csmith, and we would like to ask you if you are aware of
>>>     any such efforts or further results.
>>
>>     Just speaking for myself here, I use Csmith as part of my
>>     pre-commit testing.
>>
>>     vedant
>>
>>>
>>>     Best regards,
>>>     Radu Ometita,
>>>     Functional compilers engineer @IOHK
>>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20181212/52a14ac5/attachment.html>