[llvm-dev] Testing compiler reliability using Csmith

Mon Dec 24 05:10:48 PST 2018

I just saw this related paper today on Hacker News:

https://news.ycombinator.com/item?id=18748193

On 12/12, Matt P. Dziubinski via llvm-dev wrote:
> You may also be interested in the following resources on compilers
> correctness (articles, software, and talks -- from the general topics to the
> ones specifically focused on testing, validation, and verification):
> 
> https://github.com/MattPD/cpplinks/blob/master/compilers.correctness.md
> 
> Best regards,
> 
> Matt P. Dziubinski
> 
> On 12/7/2018 20:19, Philip Reames via llvm-dev wrote:
> 
> > There's also a couple other active fuzzing efforts running concurrently:
> > 
> >   * OSSFuzz is continuously running a set of codegen and individual
> >     pass fuzzers.  A bit more information can be found here:
> >     http://llvm.org/docs/FuzzingLLVM.html#continuously-running-and-finding-bugs
> >   * We (Azul Systems) have a continuously running Java fuzzer which
> >     exercises LLVM through our Falcon compiler, and regularly finds
> >     regressions and occasional deep long lurking issues.  We don't
> >     have a public bug tracker for this, but a sizable portion of our
> >     upstream bug fixing activity is driven by the output of this tool.
> > 
> > Philip
> > 
> > On 12/7/18 11:04 AM, Zhendong Su via llvm-dev wrote:
> > > Thanks, Vedant.  Yes, we have done a lot of testing of Clang/LLVM
> > > (and GCC) in the past several years (more details at
> > > https://people.inf.ethz.ch/suz/emi/index.html):
> > > 
> > > [GCC/LLVM bugs: *1,602* (total) / *1,007* (fixed)]
> > > [Reports: GCC (link1 <http://tinyurl.com/ojzdt78>, link2
> > > <http://tinyurl.com/oxlkmjc>, link3 <http://tinyurl.com/jd3jnl3>,
> > > link4 <http://tinyurl.com/zs2826l>, link5
> > > <https://tinyurl.com/yb7okzqd>), LLVM (link1
> > > <http://tinyurl.com/qj8x5ry>, link2 <http://tinyurl.com/nc9celr>,
> > > link3 <http://tinyurl.com/qy58smb>, link4
> > > <http://tinyurl.com/z7y8rwo>, link5 <https://tinyurl.com/y7wdf5b4>)]
> > > 
> > > Best,
> > > Zhendong
> > > 
> > > --------
> > > Zhendong Su
> > > Professor, Department of Computer Science, ETH Zurich
> > > Office: CAB E 64.1, Tel: +41 44 633 77 72
> > > https://people.inf.ethz.ch/suz/
> > > --------
> > > 
> > > On Fri, Dec 7, 2018 at 7:42 PM Vedant Kumar <vsk at apple.com
> > > <mailto:vsk at apple.com>> wrote:
> > > 
> > >     Hello,
> > > 
> > >     + Zhendong, who’s done a lot of work on automated testing of llvm.
> > > 
> > > >     On Dec 6, 2018, at 10:15 PM, Radu Ometita via llvm-dev
> > > >     <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
> > > > 
> > > >     Hello everyone!
> > > > 
> > > >     We are working on writing a paper about testing the reliability
> > > >     of C compilers by using Csmith (a random C99 program generator).
> > > > 
> > > >     A previous testing effort, using Csmith, found 202 LLVM bugs,
> > > >     which represented 2% of all reported bugs at that time (PDF:
> > > >     https://www.flux.utah.edu/download?uid=114): . However, after
> > > >     this paper was published we are unaware of any further testing
> > > >     using Csmith, and we would like to ask you if you are aware of
> > > >     any such efforts or further results.
> > > 
> > >     Just speaking for myself here, I use Csmith as part of my
> > >     pre-commit testing.
> > > 
> > >     vedant
> > > 
> > > > 
> > > >     Best regards,
> > > >     Radu Ometita,
> > > >     Functional compilers engineer @IOHK
> > > 
> 

> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

-- 

Johannes Doerfert
Researcher

Argonne National Laboratory
Lemont, IL 60439, USA

jdoerfert at anl.gov
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 228 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20181224/46f4cb7f/attachment.sig>