[llvm-dev] Testing compiler reliability using Csmith

[Matt P. Dziubinski via llvm-dev]

Hi Johannes!

Updated; thank you!

Best,

Matt

On 12/24/2018 14:10, Doerfert, Johannes Rudolf wrote:
> I just saw this related paper today on Hacker News:
>
> https://news.ycombinator.com/item?id=18748193
>
>
> On 12/12, Matt P. Dziubinski via llvm-dev wrote:
>> You may also be interested in the following resources on compilers
>> correctness (articles, software, and talks -- from the general topics to the
>> ones specifically focused on testing, validation, and verification):
>>
>> https://github.com/MattPD/cpplinks/blob/master/compilers.correctness.md
>>
>> Best regards,
>>
>> Matt P. Dziubinski
>>
>> On 12/7/2018 20:19, Philip Reames via llvm-dev wrote:
>>
>>> There's also a couple other active fuzzing efforts running concurrently:
>>>
>>>    * OSSFuzz is continuously running a set of codegen and individual
>>>      pass fuzzers.  A bit more information can be found here:
>>>      http://llvm.org/docs/FuzzingLLVM.html#continuously-running-and-finding-bugs
>>>    * We (Azul Systems) have a continuously running Java fuzzer which
>>>      exercises LLVM through our Falcon compiler, and regularly finds
>>>      regressions and occasional deep long lurking issues.  We don't
>>>      have a public bug tracker for this, but a sizable portion of our
>>>      upstream bug fixing activity is driven by the output of this tool.
>>>
>>> Philip
>>>
>>> On 12/7/18 11:04 AM, Zhendong Su via llvm-dev wrote:
>>>> Thanks, Vedant.  Yes, we have done a lot of testing of Clang/LLVM
>>>> (and GCC) in the past several years (more details at
>>>> https://people.inf.ethz.ch/suz/emi/index.html):
>>>>
>>>> [GCC/LLVM bugs: *1,602* (total) / *1,007* (fixed)]
>>>> [Reports: GCC (link1 <http://tinyurl.com/ojzdt78>, link2
>>>> <http://tinyurl.com/oxlkmjc>, link3 <http://tinyurl.com/jd3jnl3>,
>>>> link4 <http://tinyurl.com/zs2826l>, link5
>>>> <https://tinyurl.com/yb7okzqd>), LLVM (link1
>>>> <http://tinyurl.com/qj8x5ry>, link2 <http://tinyurl.com/nc9celr>,
>>>> link3 <http://tinyurl.com/qy58smb>, link4
>>>> <http://tinyurl.com/z7y8rwo>, link5 <https://tinyurl.com/y7wdf5b4>)]
>>>>
>>>> Best,
>>>> Zhendong
>>>>
>>>> --------
>>>> Zhendong Su
>>>> Professor, Department of Computer Science, ETH Zurich
>>>> Office: CAB E 64.1, Tel: +41 44 633 77 72
>>>> https://people.inf.ethz.ch/suz/
>>>> --------
>>>>
>>>> On Fri, Dec 7, 2018 at 7:42 PM Vedant Kumar <vsk at apple.com
>>>> <mailto:vsk at apple.com>> wrote:
>>>>
>>>>      Hello,
>>>>
>>>>      + Zhendong, who’s done a lot of work on automated testing of llvm.
>>>>
>>>>>      On Dec 6, 2018, at 10:15 PM, Radu Ometita via llvm-dev
>>>>>      <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>>>>>
>>>>>      Hello everyone!
>>>>>
>>>>>      We are working on writing a paper about testing the reliability
>>>>>      of C compilers by using Csmith (a random C99 program generator).
>>>>>
>>>>>      A previous testing effort, using Csmith, found 202 LLVM bugs,
>>>>>      which represented 2% of all reported bugs at that time (PDF:
>>>>>      https://www.flux.utah.edu/download?uid=114): . However, after
>>>>>      this paper was published we are unaware of any further testing
>>>>>      using Csmith, and we would like to ask you if you are aware of
>>>>>      any such efforts or further results.
>>>>      Just speaking for myself here, I use Csmith as part of my
>>>>      pre-commit testing.
>>>>
>>>>      vedant
>>>>
>>>>>      Best regards,
>>>>>      Radu Ometita,
>>>>>      Functional compilers engineer @IOHK
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>