[cfe-dev] Introducing clang-triage: A bot to test clang with fuzzed inputs

[Sami Liedes]

On Fri, Jan 16, 2015 at 10:27:00AM -0800, Philip Reames wrote:
> I've been playing with something like this in my spare time.  I've gotten
> most of the infrastructure built to do this, but my initial couple of runs
> haven't been real productive.  The problem is that most modified test cases

Yeah, it seems to me that it might be doable and with some careful
planning probably quite useful too, but with some caveats:

* You cannot really reuse old coverage results from previous runs of
  afl-fuzz on older clang binaries (and these in turn are used for
  splicing test cases when producing new cases), so probably this
  would involve taking, say, all the old cases that have reached new
  paths and rerunning afl-fuzz with the new clang for all those.

   * Currently afl-fuzz runs a timing step for all cases that cover
     new paths. I think that step takes somewhere near 1 second. You
     might get away by reusing timings from old clang or something, or
     just disabling the timing logic.

   * Alternatively you could just use the new test case as a single
     input, but then you will be throwing away all the splicing
     possibilities.

* This still involves "throwing away" old results in any case, namely
  the negative results of "this mutation did not cause a new path to
  be executed". In a new clang binary, it might just be that it would
  have covered new paths. Hence, doing this for new clang revisions is
  unlikely to be a substitute for deeper testing by running a more
  comprehensive test run every now and then.

> are *massive* and afl-fuzz has a really hard time with that.  I've got a
> couple of ideas on how to address that, but I haven't gotten back to it yet.
> If others are interested, I can push the code I've got up on github.  (It's
> just some basic python scripting.)

I've been thinking a bit about also minimizing the test cases instead
of only minimizing their number; those could then be a reasonable
starting point as a corpus for rerun on a newer clang binary. I think
it would also in a sense be cool to have minimized test cases where
each exercises some unique path in clang. I have no code though.

Currently, minimizing a corpus (with a script that comes with
afl-fuzz) works roughly like this:

  ALL = the set of edges covered by the entire corpus
  REDUCED = set()
  while REDUCED != ALL:
      e = the edge in ALL-REDUCED that is rarest
      c = the smallest test case that exercises e
      choose c for the minimized corpus
      REDUCED |= edges exercised by c

If this is modified to further minimize the test case c (say, by
creduce or equivalent means) subject to the condition that the
resulting case still needs to exercise edge e, the result should be a
larger corpus of much smaller test cases.

An even simpler idea that I haven't done so far would be to strip
comments from the test cases before feeding them to afl-fuzz. They're
unlikely to be *that* interesting. Apparently afl-fuzz does some
minimization of its own, but I don't think it's very aggressive.

	Sami
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20150117/22b1eb9c/attachment.sig>