[cfe-dev] Introducing clang-triage: A bot to test clang with fuzzed inputs

[Philip Reames]

On 01/15/2015 11:58 PM, Nick Lewycky wrote:
> On 3 January 2015 at 13:55, Sami Liedes <sami.liedes at iki.fi 
> <mailto:sami.liedes at iki.fi>> wrote:
>
>     Hi,
>
>     I've set up a bot to test Clang by running svn HEAD of LLVM and Clang
>     with a large corpus of test cases that exercises a lot of different
>     code paths (from afl-fuzz). Whenever a test case causes clang to
>     crash, it records the output and reduces the test case using CReduce
>     or, if CReduce crashes, a dumber reducer. Crashes (assertion failures,
>     signals) are the only kind of failure detected.
>
>     You can see here the kind of output it produces (running on my desktop
>     computer for now, so this URL will probably go away at some point):
>
>     http://sli.dy.fi/~sliedes/clang-triage/triage_report.xhtml
>     <http://sli.dy.fi/%7Esliedes/clang-triage/triage_report.xhtml>
>
>     Currently the bot only runs the test cases using clang -std=c++11 -O0;
>     trying with different language options would probably require more
>     afl-fuzzing with the different option set to really be effective.
>
>     If someone wants to try it with different language parameters (or even
>     for different frontends), or to set it up on some better
>     infrastructure, the code and the instructions are here:
>
>     https://github.com/sliedes/clang-triage
>
>     The number of test cases that cause a given kind of failure also
>     roughly corresponds to how likely afl-fuzz is to hit that failure
>     (though the corpus is minimized in the sense that every test case in
>     the bot's database should exercise some code path that other test
>     cases do not). By default afl-fuzz stops recording new crashes once it
>     has found 5000 crashing inputs. Once some of the most common ones have
>     been fixed, it would make sense to rerun the fuzzer and add new test
>     cases to the bot.
>
>
> Once we start to get clean under afl (from a seed of an empty file), 
> it would be great to start testing that new commits don't introduce 
> regressions. Here's my idea: afl-fuzz already requires a starting 
> input to act as a seed. If there is a test file change in a commit, 
> use that test as a seed for afl-fuzz to check that revision. What do 
> you think?
I've been playing with something like this in my spare time.  I've 
gotten most of the infrastructure built to do this, but my initial 
couple of runs haven't been real productive.  The problem is that most 
modified test cases are *massive* and afl-fuzz has a really hard time 
with that.  I've got a couple of ideas on how to address that, but I 
haven't gotten back to it yet.  If others are interested, I can push the 
code I've got up on github.  (It's just some basic python scripting.)

Philip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20150116/4cf81c65/attachment.html>