[llvm] r227252 - Add a Fuzzer library
Aaron Ballman
aaron at aaronballman.com
Thu Jan 29 08:28:58 PST 2015
On Thu, Jan 29, 2015 at 11:27 AM, Kostya Serebryany <kcc at google.com> wrote:
>
>
> On Thu, Jan 29, 2015 at 8:18 AM, Aaron Ballman <aaron at aaronballman.com>
> wrote:
>>
>> On Thu, Jan 29, 2015 at 11:11 AM, Kostya Serebryany <kcc at google.com>
>> wrote:
>> >
>> >
>> > On Thu, Jan 29, 2015 at 7:28 AM, Aaron Ballman <aaron at aaronballman.com>
>> > wrote:
>> >>
>> >> Actually, this is completely broken on Windows and should be reverted
>> >> until it is fixed up. Comments below.
>> >
>> >
>> >
>> >
>> >>
>> >>
>> >> On Thu, Jan 29, 2015 at 10:11 AM, Aaron Ballman
>> >> <aaron at aaronballman.com>
>> >> wrote:
>> >> > This commit puts the fuzzer targets on the top-level for MSVC
>> >> > solutions when they should be in a separate folder instead. Picture
>> >> > attached is worth 1000 words. ;-)
>> >> >
>> >> > ~Aaron
>> >> >
>> >> > On Tue, Jan 27, 2015 at 5:08 PM, Kostya Serebryany <kcc at google.com>
>> >> > wrote:
>> >> >> Author: kcc
>> >> >> Date: Tue Jan 27 16:08:41 2015
>> >> >> New Revision: 227252
>> >> >>
>> >> >> URL: http://llvm.org/viewvc/llvm-project?rev=227252&view=rev
>> >> >> Log:
>> >> >> Add a Fuzzer library
>> >> >>
>> >> >> Summary:
>> >> >> A simple genetic in-process coverage-guided fuzz testing library.
>> >> >>
>> >> >> I've used this fuzzer to test clang-format
>> >> >> (it found 12+ bugs, thanks djasper@ for the fixes!)
>> >> >> and it may also help us test other parts of LLVM.
>> >> >> So why not keep it in the LLVM repository?
>> >> >>
>> >> >> I plan to add the cmake build rules later (in a separate patch, if
>> >> >> that's ok)
>> >> >> and also add a clang-format-fuzzer target.
>> >> >>
>> >> >> See README.txt for details.
>> >> >>
>> >> >> Test Plan: Tests will follow separately.
>> >> >>
>> >> >> Reviewers: djasper, chandlerc, rnk
>> >> >>
>> >> >> Reviewed By: rnk
>> >> >>
>> >> >> Subscribers: majnemer, ygribov, dblaikie, llvm-commits
>> >> >>
>> >> >> Differential Revision: http://reviews.llvm.org/D7184
>> >> >>
>> >> >> Added:
>> >> >> llvm/trunk/lib/Fuzzer/
>> >> >> llvm/trunk/lib/Fuzzer/CMakeLists.txt
>> >> >> llvm/trunk/lib/Fuzzer/FuzzerCrossOver.cpp
>> >> >> llvm/trunk/lib/Fuzzer/FuzzerFlags.def
>> >> >> llvm/trunk/lib/Fuzzer/FuzzerIO.cpp
>> >> >> llvm/trunk/lib/Fuzzer/FuzzerInternal.h
>> >> >> llvm/trunk/lib/Fuzzer/FuzzerLoop.cpp
>> >> >> llvm/trunk/lib/Fuzzer/FuzzerMain.cpp
>> >> >> llvm/trunk/lib/Fuzzer/FuzzerMutate.cpp
>> >> >> llvm/trunk/lib/Fuzzer/FuzzerUtil.cpp
>> >> >> llvm/trunk/lib/Fuzzer/README.txt
>> >> >> llvm/trunk/lib/Fuzzer/test/
>> >> >> llvm/trunk/lib/Fuzzer/test/ExactTest.cpp
>> >> >> llvm/trunk/lib/Fuzzer/test/InfiniteTest.cpp
>> >> >> llvm/trunk/lib/Fuzzer/test/NullDerefTest.cpp
>> >> >> llvm/trunk/lib/Fuzzer/test/SimpleTest.cpp
>> >> >> llvm/trunk/lib/Fuzzer/test/TestFuzzerCrossOver.cpp
>> >> >> llvm/trunk/lib/Fuzzer/test/TimeoutTest.cpp
>> >> >> Modified:
>> >> >> llvm/trunk/lib/CMakeLists.txt
>> >> >>
>> >> >> Modified: llvm/trunk/lib/CMakeLists.txt
>> >> >> URL:
>> >> >>
>> >> >> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CMakeLists.txt?rev=227252&r1=227251&r2=227252&view=diff
>> >> >>
>> >> >>
>> >> >> ==============================================================================
>> >> >> --- llvm/trunk/lib/CMakeLists.txt (original)
>> >> >> +++ llvm/trunk/lib/CMakeLists.txt Tue Jan 27 16:08:41 2015
>> >> >> @@ -17,3 +17,4 @@ add_subdirectory(Target)
>> >> >> add_subdirectory(AsmParser)
>> >> >> add_subdirectory(LineEditor)
>> >> >> add_subdirectory(ProfileData)
>> >> >> +add_subdirectory(Fuzzer)
>> >> >>
>> >> >> Added: llvm/trunk/lib/Fuzzer/CMakeLists.txt
>> >> >> URL:
>> >> >>
>> >> >> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Fuzzer/CMakeLists.txt?rev=227252&view=auto
>> >> >>
>> >> >>
>> >> >> ==============================================================================
>> >> >> --- llvm/trunk/lib/Fuzzer/CMakeLists.txt (added)
>> >> >> +++ llvm/trunk/lib/Fuzzer/CMakeLists.txt Tue Jan 27 16:08:41 2015
>> >> >> @@ -0,0 +1,9 @@
>> >> >> +add_library(LLVMFuzzer STATIC
>> >> >> + EXCLUDE_FROM_ALL # Do not build if you are not building fuzzers.
>> >> >> + FuzzerCrossOver.cpp
>> >> >> + FuzzerIO.cpp
>> >> >> + FuzzerLoop.cpp
>> >> >> + FuzzerMain.cpp
>> >> >> + FuzzerMutate.cpp
>> >> >> + FuzzerUtil.cpp
>> >> >> + )
>> >> >>
>> >> >> Added: llvm/trunk/lib/Fuzzer/FuzzerCrossOver.cpp
>> >> >> URL:
>> >> >>
>> >> >> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Fuzzer/FuzzerCrossOver.cpp?rev=227252&view=auto
>> >> >>
>> >> >>
>> >> >> ==============================================================================
>> >> >> --- llvm/trunk/lib/Fuzzer/FuzzerCrossOver.cpp (added)
>> >> >> +++ llvm/trunk/lib/Fuzzer/FuzzerCrossOver.cpp Tue Jan 27 16:08:41
>> >> >> 2015
>> >> >> @@ -0,0 +1,46 @@
>> >> >> +//===- FuzzerCrossOver.cpp - Cross over two test inputs
>> >> >> -------------------===//
>> >> >> +//
>> >> >> +// The LLVM Compiler Infrastructure
>> >> >> +//
>> >> >> +// This file is distributed under the University of Illinois Open
>> >> >> Source
>> >> >> +// License. See LICENSE.TXT for details.
>> >> >> +//
>> >> >>
>> >> >>
>> >> >> +//===----------------------------------------------------------------------===//
>> >> >> +// Cross over test inputs.
>> >> >>
>> >> >>
>> >> >> +//===----------------------------------------------------------------------===//
>> >> >> +
>> >> >> +#include "FuzzerInternal.h"
>> >> >> +
>> >> >> +namespace fuzzer {
>> >> >> +
>> >> >> +// Cross A and B, store the result (ap to MaxLen bytes) in U.
>> >> >> +void CrossOver(const Unit &A, const Unit &B, Unit *U, size_t
>> >> >> MaxLen) {
>> >> >> + size_t Size = rand() % MaxLen + 1;
>> >> >> + U->clear();
>> >> >> + const Unit *V = &A;
>> >> >> + size_t PosA = 0;
>> >> >> + size_t PosB = 0;
>> >> >> + size_t *Pos = &PosA;
>> >> >> + while (U->size() < Size && (PosA < A.size() || PosB < B.size()))
>> >> >> {
>> >> >> + // Merge a part of V into U.
>> >> >> + size_t SizeLeftU = Size - U->size();
>> >> >> + if (*Pos < V->size()) {
>> >> >> + size_t SizeLeftV = V->size() - *Pos;
>> >> >> + size_t MaxExtraSize = std::min(SizeLeftU, SizeLeftV);
>> >> >> + size_t ExtraSize = rand() % MaxExtraSize + 1;
>> >> >> + U->insert(U->end(), V->begin() + *Pos, V->begin() + *Pos +
>> >> >> ExtraSize);
>> >> >> + (*Pos) += ExtraSize;
>> >> >> + }
>> >> >> +
>> >> >> + // Use the other Unit on the next iteration.
>> >> >> + if (Pos == &PosA) {
>> >> >> + Pos = &PosB;
>> >> >> + V = &B;
>> >> >> + } else {
>> >> >> + Pos = &PosA;
>> >> >> + V = &A;
>> >> >> + }
>> >> >> + }
>> >> >> +}
>> >> >> +
>> >> >> +} // namespace fuzzer
>> >> >>
>> >> >> Added: llvm/trunk/lib/Fuzzer/FuzzerFlags.def
>> >> >> URL:
>> >> >>
>> >> >> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Fuzzer/FuzzerFlags.def?rev=227252&view=auto
>> >> >>
>> >> >>
>> >> >> ==============================================================================
>> >> >> --- llvm/trunk/lib/Fuzzer/FuzzerFlags.def (added)
>> >> >> +++ llvm/trunk/lib/Fuzzer/FuzzerFlags.def Tue Jan 27 16:08:41 2015
>> >> >> @@ -0,0 +1,24 @@
>> >> >> +//===- FuzzerFlags.def - Run-time flags -------------------------*-
>> >> >> C++ -* ===//
>> >> >> +//
>> >> >> +// The LLVM Compiler Infrastructure
>> >> >> +//
>> >> >> +// This file is distributed under the University of Illinois Open
>> >> >> Source
>> >> >> +// License. See LICENSE.TXT for details.
>> >> >> +//
>> >> >>
>> >> >>
>> >> >> +//===----------------------------------------------------------------------===//
>> >> >> +// Flags. FUZZER_FLAG macro should be defined at the point of
>> >> >> inclusion.
>> >> >> +// We are not using any flag parsing library for better portability
>> >> >> and
>> >> >> +// independence.
>> >> >>
>> >> >>
>> >> >> +//===----------------------------------------------------------------------===//
>> >> >> +FUZZER_FLAG(int, verbosity, 1, "Verbosity level.")
>> >> >> +FUZZER_FLAG(int, seed, 0, "Random seed. If 0, seed is generated.")
>> >> >> +FUZZER_FLAG(int, iterations, -1,
>> >> >> + "Number of iterations of the fuzzer (-1 for infinite
>> >> >> runs).")
>> >> >> +FUZZER_FLAG(int, max_len, 64, "Maximal length of the test input.")
>> >> >> +FUZZER_FLAG(int, cross_over, 1, "If 1, cross over inputs.")
>> >> >> +FUZZER_FLAG(int, mutate_depth, 10,
>> >> >> + "Apply this number of consecutive mutations to each
>> >> >> input.")
>> >> >> +FUZZER_FLAG(int, exit_on_first, 0,
>> >> >> + "If 1, exit after the first new interesting input is
>> >> >> found.")
>> >> >> +FUZZER_FLAG(int, timeout, -1, "Timeout in seconds (if positive).")
>> >> >> +FUZZER_FLAG(int, help, 0, "Print help.")
>> >> >>
>> >> >> Added: llvm/trunk/lib/Fuzzer/FuzzerIO.cpp
>> >> >> URL:
>> >> >>
>> >> >> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Fuzzer/FuzzerIO.cpp?rev=227252&view=auto
>> >> >>
>> >> >>
>> >> >> ==============================================================================
>> >> >> --- llvm/trunk/lib/Fuzzer/FuzzerIO.cpp (added)
>> >> >> +++ llvm/trunk/lib/Fuzzer/FuzzerIO.cpp Tue Jan 27 16:08:41 2015
>> >> >> @@ -0,0 +1,44 @@
>> >> >> +//===- FuzzerIO.cpp - IO utils.
>> >> >> -------------------------------------------===//
>> >> >> +//
>> >> >> +// The LLVM Compiler Infrastructure
>> >> >> +//
>> >> >> +// This file is distributed under the University of Illinois Open
>> >> >> Source
>> >> >> +// License. See LICENSE.TXT for details.
>> >> >> +//
>> >> >>
>> >> >>
>> >> >> +//===----------------------------------------------------------------------===//
>> >> >> +// IO functions.
>> >> >>
>> >> >>
>> >> >> +//===----------------------------------------------------------------------===//
>> >> >> +#include "FuzzerInternal.h"
>> >> >> +#include <fstream>
>> >> >> +#include <dirent.h>
>> >>
>> >> dirent.h does not exist on Windows with MSVC.
>> >>
>> >> >> +namespace fuzzer {
>> >> >> +
>> >> >> +std::vector<std::string> ListFilesInDir(const std::string &Dir) {
>> >> >> + std::vector<std::string> V;
>> >> >> + DIR *D = opendir(Dir.c_str());
>> >> >> + if (!D) return V;
>> >> >> + while (auto E = readdir(D)) {
>> >> >> + if (E->d_type == DT_REG || E->d_type == DT_LNK)
>> >> >> + V.push_back(E->d_name);
>> >> >> + }
>> >> >> + closedir(D);
>> >> >> + return V;
>> >>
>> >> We have filesystem APIs to do this, don't we?
>> >>
>> >
>> > As the README.txt explains, this library can not use any of the great
>> > LLVM
>> > support.
>> > It has to remain independent. I amy even have to remove the use of STL
>> > from
>> > it later.
>> >
>> >>
>> >> >> +}
>> >> >> +
>> >> >> +Unit FileToVector(const std::string &Path) {
>> >> >> + std::ifstream T(Path);
>> >> >> + return Unit((std::istreambuf_iterator<char>(T)),
>> >> >> + std::istreambuf_iterator<char>());
>> >> >> +}
>> >> >> +
>> >> >> +void WriteToFile(const Unit &U, const std::string &Path) {
>> >> >> + std::ofstream OF(Path);
>> >> >> + OF.write((const char*)U.data(), U.size());
>> >> >> +}
>> >> >> +
>> >> >> +void ReadDirToVectorOfUnits(const char *Path, std::vector<Unit> *V)
>> >> >> {
>> >> >> + for (auto &X : ListFilesInDir(Path))
>> >> >> + V->push_back(FileToVector(std::string(Path) + "/" + X));
>> >> >> +}
>> >> >> +
>> >> >> +} // namespace fuzzer
>> >> >>
>> >> >> Added: llvm/trunk/lib/Fuzzer/FuzzerInternal.h
>> >> >> URL:
>> >> >>
>> >> >> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Fuzzer/FuzzerInternal.h?rev=227252&view=auto
>> >> >>
>> >> >>
>> >> >> ==============================================================================
>> >> >> --- llvm/trunk/lib/Fuzzer/FuzzerInternal.h (added)
>> >> >> +++ llvm/trunk/lib/Fuzzer/FuzzerInternal.h Tue Jan 27 16:08:41 2015
>> >> >> @@ -0,0 +1,77 @@
>> >> >> +//===- FuzzerInternal.h - Internal header for the Fuzzer --------*-
>> >> >> C++ -* ===//
>> >> >> +//
>> >> >> +// The LLVM Compiler Infrastructure
>> >> >> +//
>> >> >> +// This file is distributed under the University of Illinois Open
>> >> >> Source
>> >> >> +// License. See LICENSE.TXT for details.
>> >> >> +//
>> >> >>
>> >> >>
>> >> >> +//===----------------------------------------------------------------------===//
>> >> >> +// Define the main class fuzzer::Fuzzer and most functions.
>> >> >>
>> >> >>
>> >> >> +//===----------------------------------------------------------------------===//
>> >> >> +#include <cassert>
>> >> >> +#include <chrono>
>> >> >> +#include <cstddef>
>> >> >> +#include <cstdlib>
>> >> >> +#include <string>
>> >> >> +#include <vector>
>> >> >> +
>> >> >> +namespace fuzzer {
>> >> >> +typedef std::vector<uint8_t> Unit;
>> >> >> +using namespace std::chrono;
>> >> >> +
>> >> >> +Unit ReadFile(const char *Path);
>> >> >> +std::vector<std::string> ListFilesInDir(const std::string &Dir);
>> >> >> +void ReadDirToVectorOfUnits(const char *Path, std::vector<Unit>
>> >> >> *V);
>> >> >> +void WriteToFile(const Unit &U, const std::string &Path);
>> >> >> +
>> >> >> +void Mutate(Unit *U, size_t MaxLen);
>> >> >> +
>> >> >> +void CrossOver(const Unit &A, const Unit &B, Unit *U, size_t
>> >> >> MaxLen);
>> >> >> +
>> >> >> +void Print(const Unit &U, const char *PrintAfter = "");
>> >> >> +void PrintASCII(const Unit &U, const char *PrintAfter = "");
>> >> >> +std::string Hash(const Unit &U);
>> >> >> +void SetTimer(int Seconds);
>> >> >> +
>> >> >> +class Fuzzer {
>> >> >> + public:
>> >> >> + struct FuzzingOptions {
>> >> >> + int Verbosity = 1;
>> >> >> + int MaxLen = 0;
>> >> >> + bool DoCrossOver = true;
>> >> >> + bool MutateDepth = 10;
>> >> >> + bool ExitOnFirst = false;
>> >> >> + std::string OutputCorpus;
>> >> >> + };
>> >> >> + Fuzzer(FuzzingOptions Options) : Options(Options) {
>> >> >> + SetDeathCallback();
>> >> >> + }
>> >> >> + void AddToCorpus(const Unit &U) { Corpus.push_back(U); }
>> >> >> + size_t Loop(size_t NumIterations);
>> >> >> + void ShuffleAndMinimize();
>> >> >> + size_t CorpusSize() const { return Corpus.size(); }
>> >> >> + void ReadDir(const std::string &Path) {
>> >> >> + ReadDirToVectorOfUnits(Path.c_str(), &Corpus);
>> >> >> + }
>> >> >> +
>> >> >> + static void AlarmCallback();
>> >> >> +
>> >> >> + private:
>> >> >> + size_t MutateAndTestOne(Unit *U);
>> >> >> + size_t RunOne(const Unit &U);
>> >> >> + void WriteToOutputCorpus(const Unit &U);
>> >> >> + static void WriteToCrash(const Unit &U, const char *Prefix);
>> >> >> +
>> >> >> + void SetDeathCallback();
>> >> >> + static void DeathCallback();
>> >> >> + static Unit CurrentUnit;
>> >> >> +
>> >> >> + size_t TotalNumberOfRuns = 0;
>> >> >> +
>> >> >> + std::vector<Unit> Corpus;
>> >> >> + FuzzingOptions Options;
>> >> >> + system_clock::time_point ProcessStartTime = system_clock::now();
>> >> >> + static system_clock::time_point UnitStartTime;
>> >> >> +};
>> >> >> +
>> >> >> +}; // namespace fuzzer
>> >> >>
>> >> >> Added: llvm/trunk/lib/Fuzzer/FuzzerLoop.cpp
>> >> >> URL:
>> >> >>
>> >> >> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Fuzzer/FuzzerLoop.cpp?rev=227252&view=auto
>> >> >>
>> >> >>
>> >> >> ==============================================================================
>> >> >> --- llvm/trunk/lib/Fuzzer/FuzzerLoop.cpp (added)
>> >> >> +++ llvm/trunk/lib/Fuzzer/FuzzerLoop.cpp Tue Jan 27 16:08:41 2015
>> >> >> @@ -0,0 +1,161 @@
>> >> >> +//===- FuzzerLoop.cpp - Fuzzer's main loop
>> >> >> --------------------------------===//
>> >> >> +//
>> >> >> +// The LLVM Compiler Infrastructure
>> >> >> +//
>> >> >> +// This file is distributed under the University of Illinois Open
>> >> >> Source
>> >> >> +// License. See LICENSE.TXT for details.
>> >> >> +//
>> >> >>
>> >> >>
>> >> >> +//===----------------------------------------------------------------------===//
>> >> >> +// Fuzzer's main loop.
>> >> >>
>> >> >>
>> >> >> +//===----------------------------------------------------------------------===//
>> >> >> +
>> >> >> +#include "FuzzerInternal.h"
>> >> >> +#include <sanitizer/asan_interface.h>
>> >>
>> >> Please do not assume everyone has asan checked out. Also, this is
>> >> using a system include path on Windows instead of a project include
>> >> path.
>> >
>> >
>> >
>> > I do not. This is what EXCLUDE_FROM_ALL was for.
>> > Apparently it did not help.
>> > Suggestions?
>>
>> I'm not certain how EXCLUDE_FROM_ALL would help for this #include?
>
>
> Not sure what you mean here.
> My understanding of EXCLUDE_FROM_ALL is that a target will not be built
> unless it is explicitly requested from the build system,
> at which point the requester makes sure there is proper sanitizer support.
> This library will not function w/o sanitizers, so there is no way to have it
> available in all builds.
Ah, I understand your logic now. You want CMake to only include the
entire fuzzer library when sanitizers are present, and otherwise not
build it. So then it would be safe for this #include. That makes far
more sense to me. :-)
~Aaron
More information about the llvm-commits
mailing list