[PATCH] D42414: [llvm-opt-fuzzer] Avoid adding incorrect inputs to the fuzzer corpus

[Igor Laevsky via Phabricator via llvm-commits]

igor-laevsky created this revision.
igor-laevsky added reviewers: kcc, bogner.

In general during continuous fuzzing we want to avoid adding invalid inputs into the corpus because they will test the wrong thing (error handling in the bitcode reader). Instead it would be better to concentrate on the actual fuzzing target by only producing valid inputs.

In the perfect world llvm mutator will always produce correct llvm ir and everything would work flawlessly. However there are number of cases when mutator fails to guarantee that. We catch them by running module verification after mutation. In theory this should be sufficient to prevent exposing incorrect inputs to the libFuzzer. However I noticed that still occasionally incorrect inputs would flow into the fuzzer corpus.

Problem lurks with some rare invariants which are only checked by the llvm reader. This means that verification after mutation will not catch them. Ideal solution would be to first fix all those issues in the verifier and then fix the mutator to not produce such mutations. However it's unclear how much of those are there and debugging each of them might prove to be complicated.

So until those are fixed I think it's reasonable to add explicit save/reload step as part of the after-mutation verification. This should produce cleaner continuous runs with clear indications of the mutator problems. On my machine this decreases exec/s by 10-30% but  it seems like reasonable cost to pay for the correct runs.


https://reviews.llvm.org/D42414

Files:
  tools/llvm-opt-fuzzer/llvm-opt-fuzzer.cpp


Index: tools/llvm-opt-fuzzer/llvm-opt-fuzzer.cpp
===================================================================
--- tools/llvm-opt-fuzzer/llvm-opt-fuzzer.cpp
+++ tools/llvm-opt-fuzzer/llvm-opt-fuzzer.cpp
@@ -65,15 +65,37 @@
 
   Mutator->mutateModule(*M, Seed, Size, MaxSize);
 
-#ifndef NDEBUG
   if (verifyModule(*M, &errs())) {
     errs() << "mutation result doesn't pass verification\n";
     M->dump();
-    abort();
+    // Avoid adding incorrect test cases to the corpus.
+    return 0;
+  }
+  
+  std::string Buf;
+  {
+    raw_string_ostream OS(Buf);
+    WriteBitcodeToFile(M.get(), OS);
+  }
+  if (Buf.size() > MaxSize)
+    return 0;
+  
+  // There are some invariants which are not checked by the verifier in favor
+  // of having them checked by the parser. They may be considered as bugs in the
+  // verifier and should be fixed there. However until all of those are covered
+  // we want to check for them explicitly. Otherwise we will add incorrect input
+  // to the corpus and this is going to confuse the fuzzer which will start 
+  // exploration of the bitcode reader error handling code.
+  auto NewM = parseModule(
+      reinterpret_cast<const uint8_t*>(Buf.data()), Buf.size(), Context);
+  if (!NewM || verifyModule(*NewM, &errs())) {
+    errs() << "mutator failed to re-read the module\n";
+    M->dump();
+    return 0;
   }
-#endif
 
-  return writeModule(*M, Data, MaxSize);
+  memcpy(Data, Buf.data(), Buf.size());
+  return Buf.size();
 }
 
 extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D42414.131028.patch
Type: text/x-patch
Size: 1560 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20180123/5532af62/attachment.bin>