[cfe-dev] Introducing clang-triage: A bot to test clang with fuzzed inputs

[John Regehr]

Sami, this is very cool.  Here are a few suggestions/comments/requests:

Are you adding these into the LLVM bugzilla?

A lot of your crashers are syntactically invalid, but it often is the case 
that crashes on valid code are more interesting than crashes on invalid 
code.  Can you separate out the ones for which you have a valid test case? 
These might just be the ones that GCC can compile.

Your mail mentions C-Reduce crashing.  Please report bugs to us when this 
happens.

One thing we've seen happen a lot is that running C-Reduce on a crasher 
will find new crashes.  Basically, a reducer is a fuzzer too.  But you 
have to rig your C-Reduce interestingness test to look for these.

If you have time, it would be helpful to add Csmith into the mix here.  I 
would be very interested to learn how Csmith and AFL work together.

You should definitely turn on LLVM optimizations when doing this testing, 
unless there's some specific reason why that isn't a good idea.

John



On Sat, 3 Jan 2015, Sami Liedes wrote:

> Hi,
>
> I've set up a bot to test Clang by running svn HEAD of LLVM and Clang
> with a large corpus of test cases that exercises a lot of different
> code paths (from afl-fuzz). Whenever a test case causes clang to
> crash, it records the output and reduces the test case using CReduce
> or, if CReduce crashes, a dumber reducer. Crashes (assertion failures,
> signals) are the only kind of failure detected.
>
> You can see here the kind of output it produces (running on my desktop
> computer for now, so this URL will probably go away at some point):
>
>    http://sli.dy.fi/~sliedes/clang-triage/triage_report.xhtml
>
> Currently the bot only runs the test cases using clang -std=c++11 -O0;
> trying with different language options would probably require more
> afl-fuzzing with the different option set to really be effective.
>
> If someone wants to try it with different language parameters (or even
> for different frontends), or to set it up on some better
> infrastructure, the code and the instructions are here:
>
>    https://github.com/sliedes/clang-triage
>
> The number of test cases that cause a given kind of failure also
> roughly corresponds to how likely afl-fuzz is to hit that failure
> (though the corpus is minimized in the sense that every test case in
> the bot's database should exercise some code path that other test
> cases do not). By default afl-fuzz stops recording new crashes once it
> has found 5000 crashing inputs. Once some of the most common ones have
> been fixed, it would make sense to rerun the fuzzer and add new test
> cases to the bot.
>
> 	Sami
>