<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Dec 5, 2014 at 4:40 PM, Sami Liedes <span dir="ltr"><<a href="mailto:sami.liedes@iki.fi" target="_blank">sami.liedes@iki.fi</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi,<br>
<br>
I've been playing with afl-fuzz[1] to fuzz the clang test suite. In<br>
the first 11 hours I have discovered 34 distinct assertion failures in<br>
clang -std=c++11 and at least one segmentation fault (I didn't yet do<br>
anything to tell different SEGVs apart), all on a recent HEAD. I<br>
thought I'd share the methodology and the initial findings with you. I<br>
haven't reported any bugs yet; feel free to do so if you are so<br>
inclined. I might do that later.<br>
<br>
For the 35 minimized test cases and their outputs, see [2], or<br>
download a tarball from [3].<br>
<br>
The git versions I'm running the tests against are:<br>
<br>
llvm 1610d6e Add missing FP build attribute tests.<br>
clang 64a12ad Driver: Objective-C should respect -fno-exceptions<br>
compiler-rt d6e5390 [ASan, LSan] Improve tracking of thread creation.<br>
<br>
Some quick notes:<br>
<br>
* These have been generated using afl-fuzz 0.85b using the clang test<br>
suite as input.<br>
<br>
* The test suite has been first minimized using afl-fuzz's<br>
minimize_corpus.sh.<br>
<br>
* Finding the first 35 crashes (34 distinct assertion failures and at<br>
least one segfault as I didn't yet do anything to tell them apart)<br>
took about 11 hours on a Core i7-2600.<br>
<br>
* afl-fuzz is a directed fuzzing tool. When it discovers an input that<br>
exercises new edges in the binary, it adds it to the queue as a new<br>
input for fuzzing. After 11 hours the fuzzer is at most 0.05%<br>
through its queue. A small cluster would be nice for this. I suspect<br>
it would find new crashes after weeks of fuzzing on a binary of this<br>
complexity.<br>
<br>
* If you wish to try it yourself, note the following:<br>
<br>
* Increase MAP_SIZE_POW2 in afl-fuzz's config.h.<br>
* 17 is not sufficient, 18 has looked good for the first 11 hours...<br>
<br>
* I built LLVM with -DLLVM_ENABLE_ASSERTIONS=ON<br>
-DLLVM_ENABLE_THREADS=OFF -DLLVM_ENABLE_BACKTRACES=OFF<br>
-DLLVM_ENABLE_CRASH_OVERRIDES=OFF (and of course using afl-clang as<br>
the compiler).<br>
<br>
* Running clang -cc1 directly with the input and with gives me about<br>
50-60 executions per second on each of the 8 instances (for a<br>
total of 400-480/s) instead of 5-6/s.<br>
<br>
* Use something like -ferror-limit 5 -Werror.<br>
* You might want to experiment with even smaller error limit.<br>
<br>
* Using ASAN might make sense too (afl supports it quite nicely, at<br>
least on 32-bit).<br></blockquote><div><br></div><div>Yes, please!!! (And MSAN too)</div><div>-DLLVM_USE_SANITIZER=Address and (in a separate build) -DLLVM_USE_SANITIZER=Memory</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
* I consider it hideous how well afl-fuzz works for a tool which does<br>
essentially sed-level magic on compiler-generated assembly to<br>
instrument the program.<br>
<br>
Sami<br>
<br>
<br>
[1] <a href="http://lcamtuf.coredump.cx/afl/" target="_blank">http://lcamtuf.coredump.cx/afl/</a><br>
[2] <a href="http://sliedes.kapsi.fi/clang-fuzz/" target="_blank">http://sliedes.kapsi.fi/clang-fuzz/</a><br>
[3] <a href="http://sliedes.kapsi.fi/clang-fuzz.tar.gz" target="_blank">http://sliedes.kapsi.fi/clang-fuzz.tar.gz</a><br>
<br>_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:LLVMdev@cs.uiuc.edu">LLVMdev@cs.uiuc.edu</a> <a href="http://llvm.cs.uiuc.edu" target="_blank">http://llvm.cs.uiuc.edu</a><br>
<a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev</a><br>
<br></blockquote></div><br></div></div>