[cfe-dev] Introducing clang-triage: A bot to test clang with fuzzed inputs

[Sami Liedes]

Hi,

I've set up a bot to test Clang by running svn HEAD of LLVM and Clang
with a large corpus of test cases that exercises a lot of different
code paths (from afl-fuzz). Whenever a test case causes clang to
crash, it records the output and reduces the test case using CReduce
or, if CReduce crashes, a dumber reducer. Crashes (assertion failures,
signals) are the only kind of failure detected.

You can see here the kind of output it produces (running on my desktop
computer for now, so this URL will probably go away at some point):

    http://sli.dy.fi/~sliedes/clang-triage/triage_report.xhtml

Currently the bot only runs the test cases using clang -std=c++11 -O0;
trying with different language options would probably require more
afl-fuzzing with the different option set to really be effective.

If someone wants to try it with different language parameters (or even
for different frontends), or to set it up on some better
infrastructure, the code and the instructions are here:

    https://github.com/sliedes/clang-triage

The number of test cases that cause a given kind of failure also
roughly corresponds to how likely afl-fuzz is to hit that failure
(though the corpus is minimized in the sense that every test case in
the bot's database should exercise some code path that other test
cases do not). By default afl-fuzz stops recording new crashes once it
has found 5000 crashing inputs. Once some of the most common ones have
been fixed, it would make sense to rerun the fuzzer and add new test
cases to the bot.

	Sami
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20150103/0e994dcf/attachment.sig>