[LLVMdev] Fuzzing clang test suite to generate crashing inputs

[Sami Liedes]

Hi,

I've been playing with afl-fuzz[1] to fuzz the clang test suite. In
the first 11 hours I have discovered 34 distinct assertion failures in
clang -std=c++11 and at least one segmentation fault (I didn't yet do
anything to tell different SEGVs apart), all on a recent HEAD. I
thought I'd share the methodology and the initial findings with you. I
haven't reported any bugs yet; feel free to do so if you are so
inclined. I might do that later.

For the 35 minimized test cases and their outputs, see [2], or
download a tarball from [3].

The git versions I'm running the tests against are:

  llvm         1610d6e Add missing FP build attribute tests.
  clang        64a12ad Driver: Objective-C should respect -fno-exceptions
  compiler-rt  d6e5390 [ASan, LSan] Improve tracking of thread creation.

Some quick notes:

* These have been generated using afl-fuzz 0.85b using the clang test
  suite as input.

* The test suite has been first minimized using afl-fuzz's
  minimize_corpus.sh.

* Finding the first 35 crashes (34 distinct assertion failures and at
  least one segfault as I didn't yet do anything to tell them apart)
  took about 11 hours on a Core i7-2600.

* afl-fuzz is a directed fuzzing tool. When it discovers an input that
  exercises new edges in the binary, it adds it to the queue as a new
  input for fuzzing. After 11 hours the fuzzer is at most 0.05%
  through its queue. A small cluster would be nice for this. I suspect
  it would find new crashes after weeks of fuzzing on a binary of this
  complexity.

* If you wish to try it yourself, note the following:

  * Increase MAP_SIZE_POW2 in afl-fuzz's config.h.
    * 17 is not sufficient, 18 has looked good for the first 11 hours...

  * I built LLVM with -DLLVM_ENABLE_ASSERTIONS=ON
   -DLLVM_ENABLE_THREADS=OFF -DLLVM_ENABLE_BACKTRACES=OFF
   -DLLVM_ENABLE_CRASH_OVERRIDES=OFF (and of course using afl-clang as
    the compiler).

  * Running clang -cc1 directly with the input and with gives me about
    50-60 executions per second on each of the 8 instances (for a
    total of 400-480/s) instead of 5-6/s.

  * Use something like -ferror-limit 5 -Werror.
    * You might want to experiment with even smaller error limit.

  * Using ASAN might make sense too (afl supports it quite nicely, at
    least on 32-bit).

* I consider it hideous how well afl-fuzz works for a tool which does
  essentially sed-level magic on compiler-generated assembly to
  instrument the program.

	Sami


[1] http://lcamtuf.coredump.cx/afl/
[2] http://sliedes.kapsi.fi/clang-fuzz/
[3] http://sliedes.kapsi.fi/clang-fuzz.tar.gz
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20141206/f147846a/attachment.sig>