[cfe-dev] Introducing clang-triage: A bot to test clang with fuzzed inputs

[Nick Lewycky]

On 3 January 2015 at 13:55, Sami Liedes <sami.liedes at iki.fi> wrote:

> Hi,
>
> I've set up a bot to test Clang by running svn HEAD of LLVM and Clang
> with a large corpus of test cases that exercises a lot of different
> code paths (from afl-fuzz). Whenever a test case causes clang to
> crash, it records the output and reduces the test case using CReduce
> or, if CReduce crashes, a dumber reducer. Crashes (assertion failures,
> signals) are the only kind of failure detected.
>
> You can see here the kind of output it produces (running on my desktop
> computer for now, so this URL will probably go away at some point):
>
>     http://sli.dy.fi/~sliedes/clang-triage/triage_report.xhtml
>
> Currently the bot only runs the test cases using clang -std=c++11 -O0;
> trying with different language options would probably require more
> afl-fuzzing with the different option set to really be effective.
>
> If someone wants to try it with different language parameters (or even
> for different frontends), or to set it up on some better
> infrastructure, the code and the instructions are here:
>
>     https://github.com/sliedes/clang-triage
>
> The number of test cases that cause a given kind of failure also
> roughly corresponds to how likely afl-fuzz is to hit that failure
> (though the corpus is minimized in the sense that every test case in
> the bot's database should exercise some code path that other test
> cases do not). By default afl-fuzz stops recording new crashes once it
> has found 5000 crashing inputs. Once some of the most common ones have
> been fixed, it would make sense to rerun the fuzzer and add new test
> cases to the bot.
>

Once we start to get clean under afl (from a seed of an empty file), it
would be great to start testing that new commits don't introduce
regressions. Here's my idea: afl-fuzz already requires a starting input to
act as a seed. If there is a test file change in a commit, use that test as
a seed for afl-fuzz to check that revision. What do you think?


>         Sami
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20150115/99086325/attachment.html>