[compiler-rt] r334058 - [libFuzzer] initial implementation of -data_flow_trace. It parses the data flow trace and prints the summary, but doesn't use the information in any other way yet
via llvm-commits
llvm-commits at lists.llvm.org
Tue Jun 5 23:57:39 PDT 2018
Hi Kostya,
This commit is causing a build failure on several of the linux bots. Can you take a look?
http://lab.llvm.org:8011/builders/clang-x86_64-debian-fast/builds/9903
http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-debian-fast/builds/9096
132.501 [1813/18/2813] Building CXX object projects/compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.x86_64.dir/FuzzerDataFlowTrace.cpp.o
FAILED: projects/compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.x86_64.dir/FuzzerDataFlowTrace.cpp.o
/usr/bin/ccache clang++ -D_DEBUG -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Iprojects/compiler-rt/lib/fuzzer -I/home/llvmbb/llvm-build-dir/llvm-clang-lld-x86_64-debian-fast/llvm.src/projects/compiler-rt/lib/fuzzer -Iinclude -I/home/llvmbb/llvm-build-dir/llvm-clang-lld-x86_64-debian-fast/llvm.src/include -std=c++11 -Wdocumentation -Wno-documentation-deprecated-sync -fPIC -fvisibility-inlines-hidden -Werror=date-time -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wcovered-switch-default -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -Wall -std=c++11 -Wno-unused-parameter -O3 -UNDEBUG -std=c++11 -Wdocumentation -Wno-documentation-deprecated-sync -fPIC -fvisibility-inlines-hidden -Werror=date-time -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wcovered-switch-default -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -Wall -std=c++11 -Wno-unused-parameter -m64 -fPIC -fno-builtin -fno-exceptions -fomit-frame-pointer -funwind-tables -fno-stack-protector -fno-sanitize=safe-stack -fvisibility=hidden -fno-lto -O3 -gline-tables-only -Wno-gnu -Wno-variadic-macros -Wno-c99-extensions -Wno-non-virtual-dtor -MD -MT projects/compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.x86_64.dir/FuzzerDataFlowTrace.cpp.o -MF projects/compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.x86_64.dir/FuzzerDataFlowTrace.cpp.o.d -o projects/compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.x86_64.dir/FuzzerDataFlowTrace.cpp.o -c /home/llvmbb/llvm-build-dir/llvm-clang-lld-x86_64-debian-fast/llvm.src/projects/compiler-rt/lib/fuzzer/FuzzerDataFlowTrace.cpp
In file included from /home/llvmbb/llvm-build-dir/llvm-clang-lld-x86_64-debian-fast/llvm.src/projects/compiler-rt/lib/fuzzer/FuzzerDataFlowTrace.cpp:12:
In file included from /home/llvmbb/llvm-build-dir/llvm-clang-lld-x86_64-debian-fast/llvm.src/projects/compiler-rt/lib/fuzzer/FuzzerDataFlowTrace.h:32:
In file included from /home/llvmbb/llvm-build-dir/llvm-clang-lld-x86_64-debian-fast/llvm.src/projects/compiler-rt/lib/fuzzer/FuzzerDefs.h:20:
In file included from /usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/vector:65:
/usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/stl_bvector.h:469:9: error: no matching constructor for initialization of 'std::_Bvector_base<fuzzer::fuzzer_allocator<bool> >::_Bvector_impl'
: _M_impl(__a) { }
^ ~~~
/usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/stl_bvector.h:595:7: note: in instantiation of member function 'std::_Bvector_base<fuzzer::fuzzer_allocator<bool> >::_Bvector_base' requested here
: _Base(__a)
^
/usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/stl_bvector.h:590:7: note: in instantiation of member function 'std::vector<bool, fuzzer::fuzzer_allocator<bool> >::vector' requested here
: vector(__n, false, __a)
^
/home/llvmbb/llvm-build-dir/llvm-clang-lld-x86_64-debian-fast/llvm.src/projects/compiler-rt/lib/fuzzer/FuzzerDataFlowTrace.cpp:70:22: note: in instantiation of member function 'std::vector<bool, fuzzer::fuzzer_allocator<bool> >::vector' requested here
Vector<bool> V(Len);
^
/usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/stl_bvector.h:419:14: note: candidate constructor (the implicit copy constructor) not viable: no known conversion from 'const allocator_type' (aka 'const fuzzer::fuzzer_allocator<bool>') to 'const std::_Bvector_base<fuzzer::fuzzer_allocator<bool> >::_Bvector_impl' for 1st argument
struct _Bvector_impl
^
/usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/stl_bvector.h:419:14: note: candidate constructor (the implicit move constructor) not viable: no known conversion from 'const allocator_type' (aka 'const fuzzer::fuzzer_allocator<bool>') to 'std::_Bvector_base<fuzzer::fuzzer_allocator<bool> >::_Bvector_impl' for 1st argument
struct _Bvector_impl
^
/usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/stl_bvector.h:430:2: note: candidate constructor not viable: no known conversion from 'const fuzzer_allocator<bool>' to 'const fuzzer_allocator<unsigned long>' for 1st argument
_Bvector_impl(const _Bit_alloc_type& __a)
^
/usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/stl_bvector.h:435:2: note: candidate constructor not viable: no known conversion from 'const fuzzer_allocator<bool>' to 'fuzzer_allocator<unsigned long>' for 1st argument
_Bvector_impl(_Bit_alloc_type&& __a)
^
/usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/stl_bvector.h:426:2: note: candidate constructor not viable: requires 0 arguments, but 1 was provided
_Bvector_impl()
^
1 error generated.
Douglas Yung
> -----Original Message-----
> From: llvm-commits [mailto:llvm-commits-bounces at lists.llvm.org] On
> Behalf Of Kostya Serebryany via llvm-commits
> Sent: Tuesday, June 05, 2018 18:23
> To: llvm-commits at lists.llvm.org
> Subject: [compiler-rt] r334058 - [libFuzzer] initial implementation of
> -data_flow_trace. It parses the data flow trace and prints the summary,
> but doesn't use the information in any other way yet
>
> Author: kcc
> Date: Tue Jun 5 18:23:29 2018
> New Revision: 334058
>
> URL: http://llvm.org/viewvc/llvm-project?rev=334058&view=rev
> Log:
> [libFuzzer] initial implementation of -data_flow_trace. It parses the
> data flow trace and prints the summary, but doesn't use the information
> in any other way yet
>
> Added:
> compiler-rt/trunk/lib/fuzzer/FuzzerDataFlowTrace.cpp
> compiler-rt/trunk/lib/fuzzer/FuzzerDataFlowTrace.h
> Modified:
> compiler-rt/trunk/lib/fuzzer/CMakeLists.txt
> compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp
> compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def
> compiler-rt/trunk/lib/fuzzer/FuzzerIO.cpp
> compiler-rt/trunk/lib/fuzzer/FuzzerIO.h
> compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h
> compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp
> compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h
> compiler-rt/trunk/lib/fuzzer/dataflow/DataFlow.cpp
> compiler-rt/trunk/lib/fuzzer/scripts/collect_data_flow.py
> compiler-rt/trunk/lib/fuzzer/tests/FuzzerUnittest.cpp
> compiler-rt/trunk/test/fuzzer/ThreeFunctionsTest.cpp
> compiler-rt/trunk/test/fuzzer/dataflow.test
>
> Modified: compiler-rt/trunk/lib/fuzzer/CMakeLists.txt
> URL: http://llvm.org/viewvc/llvm-project/compiler-
> rt/trunk/lib/fuzzer/CMakeLists.txt?rev=334058&r1=334057&r2=334058&view=
> diff
> =======================================================================
> =======
> --- compiler-rt/trunk/lib/fuzzer/CMakeLists.txt (original)
> +++ compiler-rt/trunk/lib/fuzzer/CMakeLists.txt Tue Jun 5 18:23:29
> 2018
> @@ -1,5 +1,6 @@
> set(LIBFUZZER_SOURCES
> FuzzerCrossOver.cpp
> + FuzzerDataFlowTrace.cpp
> FuzzerDriver.cpp
> FuzzerExtFunctionsDlsym.cpp
> FuzzerExtFunctionsDlsymWin.cpp
>
> Added: compiler-rt/trunk/lib/fuzzer/FuzzerDataFlowTrace.cpp
> URL: http://llvm.org/viewvc/llvm-project/compiler-
> rt/trunk/lib/fuzzer/FuzzerDataFlowTrace.cpp?rev=334058&view=auto
> =======================================================================
> =======
> --- compiler-rt/trunk/lib/fuzzer/FuzzerDataFlowTrace.cpp (added)
> +++ compiler-rt/trunk/lib/fuzzer/FuzzerDataFlowTrace.cpp Tue Jun 5
> 18:23:29 2018
> @@ -0,0 +1,90 @@
> +//===- FuzzerDataFlowTrace.cpp - DataFlowTrace ---*-
> C++ -* ===//
> +//
> +// The LLVM Compiler Infrastructure
> +//
> +// This file is distributed under the University of Illinois Open
> Source
> +// License. See LICENSE.TXT for details.
> +//
> +//===-----------------------------------------------------------------
> -----===//
> +// fuzzer::DataFlowTrace
> +//===-----------------------------------------------------------------
> -----===//
> +
> +#include "FuzzerDataFlowTrace.h"
> +#include "FuzzerIO.h"
> +
> +#include <cstdlib>
> +#include <fstream>
> +#include <string>
> +#include <vector>
> +
> +namespace fuzzer {
> +
> +void DataFlowTrace::Init(const std::string &DirPath,
> + const std::string &FocusFunction) {
> + if (DirPath.empty()) return;
> + const char *kFunctionsTxt = "functions.txt";
> + Printf("INFO: DataFlowTrace: reading from '%s'\n", DirPath.c_str());
> + Vector<SizedFile> Files;
> + GetSizedFilesFromDir(DirPath, &Files);
> + std::string L;
> +
> + // Read functions.txt
> + std::ifstream IF(DirPlusFile(DirPath, kFunctionsTxt));
> + size_t FocusFuncIdx = SIZE_MAX;
> + size_t NumFunctions = 0;
> + while (std::getline(IF, L, '\n')) {
> + NumFunctions++;
> + if (FocusFunction == L)
> + FocusFuncIdx = NumFunctions - 1;
> + }
> + if (!NumFunctions || FocusFuncIdx == SIZE_MAX || Files.size() <= 1)
> + return;
> + // Read traces.
> + size_t NumTraceFiles = 0;
> + size_t NumTracesWithFocusFunction = 0;
> + for (auto &SF : Files) {
> + auto Name = Basename(SF.File);
> + if (Name == kFunctionsTxt) continue;
> + auto ParseError = [&](const char *Err) {
> + Printf("DataFlowTrace: parse error: %s\n File: %s\n Line:
> %s\n", Err,
> + Name.c_str(), L.c_str());
> + };
> + NumTraceFiles++;
> + // Printf("=== %s\n", Name.c_str());
> + std::ifstream IF(SF.File);
> + while (std::getline(IF, L, '\n')) {
> + size_t SpacePos = L.find(' ');
> + if (SpacePos == std::string::npos)
> + return ParseError("no space in the trace line");
> + if (L.empty() || L[0] != 'F')
> + return ParseError("the trace line doesn't start with 'F'");
> + size_t N = std::atol(L.c_str() + 1);
> + if (N >= NumFunctions)
> + return ParseError("N is greater than the number of
> functions");
> + if (N == FocusFuncIdx) {
> + NumTracesWithFocusFunction++;
> + const char *Beg = L.c_str() + SpacePos + 1;
> + const char *End = L.c_str() + L.size();
> + assert(Beg < End);
> + size_t Len = End - Beg;
> + Vector<bool> V(Len);
> + for (size_t I = 0; I < Len; I++) {
> + if (Beg[I] != '0' && Beg[I] != '1')
> + ParseError("the trace should contain only 0 or 1");
> + V[I] = Beg[I] == '1';
> + }
> + // Print just a few small traces.
> + if (NumTracesWithFocusFunction <= 3 && Len <= 16)
> + Printf("%s => |%s|\n", Name.c_str(), L.c_str() + SpacePos +
> 1);
> + break; // No need to parse the following lines.
> + }
> + }
> + }
> + assert(NumTraceFiles == Files.size() - 1);
> + Printf("INFO: DataFlowTrace: %zd trace files, %zd functions, "
> + "%zd traces with focus function\n",
> + NumTraceFiles, NumFunctions, NumTracesWithFocusFunction);
> +}
> +
> +} // namespace fuzzer
> +
>
> Added: compiler-rt/trunk/lib/fuzzer/FuzzerDataFlowTrace.h
> URL: http://llvm.org/viewvc/llvm-project/compiler-
> rt/trunk/lib/fuzzer/FuzzerDataFlowTrace.h?rev=334058&view=auto
> =======================================================================
> =======
> --- compiler-rt/trunk/lib/fuzzer/FuzzerDataFlowTrace.h (added)
> +++ compiler-rt/trunk/lib/fuzzer/FuzzerDataFlowTrace.h Tue Jun 5
> 18:23:29 2018
> @@ -0,0 +1,40 @@
> +//===- FuzzerDataFlowTrace.h - Internal header for the Fuzzer ---*-
> C++ -* ===//
> +//
> +// The LLVM Compiler Infrastructure
> +//
> +// This file is distributed under the University of Illinois Open
> Source
> +// License. See LICENSE.TXT for details.
> +//
> +//===-----------------------------------------------------------------
> -----===//
> +// fuzzer::DataFlowTrace; reads and handles a data-flow trace.
> +//
> +// A data flow trace is generated by e.g. dataflow/DataFlow.cpp
> +// and is stored on disk in a separate directory.
> +//
> +// The trace dir contains a file 'functions.txt' which lists function
> names,
> +// oner per line, e.g.
> +// ==> functions.txt <==
> +// Func2
> +// LLVMFuzzerTestOneInput
> +// Func1
> +//
> +// All other files in the dir are the traces, see
> dataflow/DataFlow.cpp.
> +// The name of the file is sha1 of the input used to generate the
> trace.
> +//
> +// Current status:
> +// the data is parsed and the summary is printed, but the data is
> not yet
> +// used in any other way.
> +//===-----------------------------------------------------------------
> -----===//
> +
> +#ifndef LLVM_FUZZER_DATA_FLOW_TRACE
> +#define LLVM_FUZZER_DATA_FLOW_TRACE
> +
> +#include "FuzzerDefs.h"
> +
> +namespace fuzzer {
> +struct DataFlowTrace {
> + void Init(const std::string &DirPath, const std::string
> &FocusFunction);
> +};
> +} // namespace fuzzer
> +
> +#endif // LLVM_FUZZER_DATA_FLOW_TRACE
>
> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp
> URL: http://llvm.org/viewvc/llvm-project/compiler-
> rt/trunk/lib/fuzzer/FuzzerDriver.cpp?rev=334058&r1=334057&r2=334058&vie
> w=diff
> =======================================================================
> =======
> --- compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp (original)
> +++ compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp Tue Jun 5 18:23:29
> 2018
> @@ -623,6 +623,8 @@ int FuzzerDriver(int *argc, char ***argv
> Options.ExitOnItem = Flags.exit_on_item;
> if (Flags.focus_function)
> Options.FocusFunction = Flags.focus_function;
> + if (Flags.data_flow_trace)
> + Options.DataFlowTrace = Flags.data_flow_trace;
>
> unsigned Seed = Flags.seed;
> // Initialize Seed.
>
> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def
> URL: http://llvm.org/viewvc/llvm-project/compiler-
> rt/trunk/lib/fuzzer/FuzzerFlags.def?rev=334058&r1=334057&r2=334058&view
> =diff
> =======================================================================
> =======
> --- compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def (original)
> +++ compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def Tue Jun 5 18:23:29
> 2018
> @@ -153,3 +153,5 @@ FUZZER_DEPRECATED_FLAG(use_equivalence_s
> FUZZER_FLAG_INT(analyze_dict, 0, "Experimental")
> FUZZER_DEPRECATED_FLAG(use_clang_coverage)
> FUZZER_FLAG_INT(use_feature_frequency, 0, "Experimental/internal")
> +
> +FUZZER_FLAG_STRING(data_flow_trace, "Experimental: use the data flow
> trace")
>
> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerIO.cpp
> URL: http://llvm.org/viewvc/llvm-project/compiler-
> rt/trunk/lib/fuzzer/FuzzerIO.cpp?rev=334058&r1=334057&r2=334058&view=di
> ff
> =======================================================================
> =======
> --- compiler-rt/trunk/lib/fuzzer/FuzzerIO.cpp (original)
> +++ compiler-rt/trunk/lib/fuzzer/FuzzerIO.cpp Tue Jun 5 18:23:29 2018
> @@ -100,6 +100,14 @@ std::string DirPlusFile(const std::strin
> return DirPath + GetSeparator() + FileName;
> }
>
> +std::string Basename(const std::string &Path, char Separator) {
> + size_t Pos = Path.rfind(Separator);
> + if (Pos == std::string::npos)
> + return Path;
> + assert(Pos < Path.size());
> + return Path.substr(Pos + 1);
> +}
> +
> void DupAndCloseStderr() {
> int OutputFd = DuplicateFile(2);
> if (OutputFd > 0) {
>
> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerIO.h
> URL: http://llvm.org/viewvc/llvm-project/compiler-
> rt/trunk/lib/fuzzer/FuzzerIO.h?rev=334058&r1=334057&r2=334058&view=diff
> =======================================================================
> =======
> --- compiler-rt/trunk/lib/fuzzer/FuzzerIO.h (original)
> +++ compiler-rt/trunk/lib/fuzzer/FuzzerIO.h Tue Jun 5 18:23:29 2018
> @@ -67,6 +67,8 @@ struct SizedFile {
> void GetSizedFilesFromDir(const std::string &Dir, Vector<SizedFile>
> *V);
>
> char GetSeparator();
> +// Similar to the basename utility: returns the file name w/o the dir
> prefix.
> +std::string Basename(const std::string &Path, char Separator =
> GetSeparator());
>
> FILE* OpenFile(int Fd, const char *Mode);
>
>
> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h
> URL: http://llvm.org/viewvc/llvm-project/compiler-
> rt/trunk/lib/fuzzer/FuzzerInternal.h?rev=334058&r1=334057&r2=334058&vie
> w=diff
> =======================================================================
> =======
> --- compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h (original)
> +++ compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h Tue Jun 5 18:23:29
> 2018
> @@ -12,6 +12,7 @@
> #ifndef LLVM_FUZZER_INTERNAL_H
> #define LLVM_FUZZER_INTERNAL_H
>
> +#include "FuzzerDataFlowTrace.h"
> #include "FuzzerDefs.h"
> #include "FuzzerExtFunctions.h"
> #include "FuzzerInterface.h"
> @@ -134,6 +135,7 @@ private:
> InputCorpus &Corpus;
> MutationDispatcher &MD;
> FuzzingOptions Options;
> + DataFlowTrace DFT;
>
> system_clock::time_point ProcessStartTime = system_clock::now();
> system_clock::time_point UnitStartTime, UnitStopTime;
>
> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp
> URL: http://llvm.org/viewvc/llvm-project/compiler-
> rt/trunk/lib/fuzzer/FuzzerLoop.cpp?rev=334058&r1=334057&r2=334058&view=
> diff
> =======================================================================
> =======
> --- compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp (original)
> +++ compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp Tue Jun 5 18:23:29
> 2018
> @@ -160,6 +160,7 @@ Fuzzer::Fuzzer(UserCallback CB, InputCor
> CurrentUnitSize = 0;
> memset(BaseSha1, 0, sizeof(BaseSha1));
> TPC.SetFocusFunction(Options.FocusFunction);
> + DFT.Init(Options.DataFlowTrace, Options.FocusFunction);
> }
>
> Fuzzer::~Fuzzer() {}
>
> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h
> URL: http://llvm.org/viewvc/llvm-project/compiler-
> rt/trunk/lib/fuzzer/FuzzerOptions.h?rev=334058&r1=334057&r2=334058&view
> =diff
> =======================================================================
> =======
> --- compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h (original)
> +++ compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h Tue Jun 5 18:23:29
> 2018
> @@ -46,6 +46,7 @@ struct FuzzingOptions {
> std::string ExitOnSrcPos;
> std::string ExitOnItem;
> std::string FocusFunction;
> + std::string DataFlowTrace;
> bool SaveArtifacts = true;
> bool PrintNEW = true; // Print a status line when new units are
> found;
> bool PrintNewCovPcs = false;
>
> Modified: compiler-rt/trunk/lib/fuzzer/dataflow/DataFlow.cpp
> URL: http://llvm.org/viewvc/llvm-project/compiler-
> rt/trunk/lib/fuzzer/dataflow/DataFlow.cpp?rev=334058&r1=334057&r2=33405
> 8&view=diff
> =======================================================================
> =======
> --- compiler-rt/trunk/lib/fuzzer/dataflow/DataFlow.cpp (original)
> +++ compiler-rt/trunk/lib/fuzzer/dataflow/DataFlow.cpp Tue Jun 5
> 18:23:29 2018
> @@ -69,6 +69,7 @@ static const uintptr_t *FuncsBeg;
> static __thread size_t CurrentFunc;
> static dfsan_label *FuncLabels; // Array of NumFuncs elements.
> static char *PrintableStringForLabel; // InputLen + 2 bytes.
> +static bool LabelSeen[1 << 8 * sizeof(dfsan_label)];
>
> // Prints all instrumented functions.
> static int PrintFunctions() {
> @@ -89,7 +90,11 @@ static int PrintFunctions() {
> return 0;
> }
>
> -static void SetBytesForLabel(dfsan_label L, char *Bytes) {
> +extern "C"
> +void SetBytesForLabel(dfsan_label L, char *Bytes) {
> + if (LabelSeen[L])
> + return;
> + LabelSeen[L] = true;
> assert(L);
> if (L <= InputLen + 1) {
> Bytes[L - 1] = '1';
> @@ -103,6 +108,7 @@ static void SetBytesForLabel(dfsan_label
> static char *GetPrintableStringForLabel(dfsan_label L) {
> memset(PrintableStringForLabel, '0', InputLen + 1);
> PrintableStringForLabel[InputLen + 1] = 0;
> + memset(LabelSeen, 0, sizeof(LabelSeen));
> SetBytesForLabel(L, PrintableStringForLabel);
> return PrintableStringForLabel;
> }
>
> Modified: compiler-rt/trunk/lib/fuzzer/scripts/collect_data_flow.py
> URL: http://llvm.org/viewvc/llvm-project/compiler-
> rt/trunk/lib/fuzzer/scripts/collect_data_flow.py?rev=334058&r1=334057&r
> 2=334058&view=diff
> =======================================================================
> =======
> --- compiler-rt/trunk/lib/fuzzer/scripts/collect_data_flow.py
> (original)
> +++ compiler-rt/trunk/lib/fuzzer/scripts/collect_data_flow.py Tue Jun
> 5 18:23:29 2018
> @@ -11,9 +11,15 @@
> # the complete trace for all input bytes (running it on all bytes at
> once
> # may fail if DFSan runs out of labels).
> # Usage:
> -# collect_data_flow.py BINARY INPUT [RESULT]
> +#
> +# # Collect dataflow for one input, store it in OUTPUT (default is
> stdout)
> +# collect_data_flow.py BINARY INPUT [OUTPUT]
> +#
> +# # Collect dataflow for all inputs in CORPUS_DIR, store them in
> OUTPUT_DIR
> +# collect_data_flow.py BINARY CORPUS_DIR OUTPUT_DIR
> #===------------------------------------------------------------------
> ------===#
> import atexit
> +import hashlib
> import sys
> import os
> import subprocess
> @@ -26,9 +32,26 @@ def cleanup(d):
> print "removing: ", d
> shutil.rmtree(d)
>
> +def collect_dataflow_for_corpus(self, exe, corpus_dir, output_dir):
> + print "Collecting dataflow for corpus:", corpus_dir, \
> + "output_dir:", output_dir
> + assert not os.path.exists(output_dir)
> + os.mkdir(output_dir)
> + for root, dirs, files in os.walk(corpus_dir):
> + for f in files:
> + path = os.path.join(root, f)
> + sha1 = hashlib.sha1(open(path).read()).hexdigest()
> + output = os.path.join(output_dir, sha1)
> + subprocess.call([self, exe, path, output])
> + functions_txt = open(os.path.join(output_dir, "functions.txt"), "w")
> + subprocess.call([exe], stdout=functions_txt)
> +
> +
> def main(argv):
> exe = argv[1]
> inp = argv[2]
> + if os.path.isdir(inp):
> + return collect_dataflow_for_corpus(argv[0], exe, inp, argv[3])
> size = os.path.getsize(inp)
> q = [[0, size]]
> tmpdir = tempfile.mkdtemp(prefix="libfuzzer-tmp-")
>
> Modified: compiler-rt/trunk/lib/fuzzer/tests/FuzzerUnittest.cpp
> URL: http://llvm.org/viewvc/llvm-project/compiler-
> rt/trunk/lib/fuzzer/tests/FuzzerUnittest.cpp?rev=334058&r1=334057&r2=33
> 4058&view=diff
> =======================================================================
> =======
> --- compiler-rt/trunk/lib/fuzzer/tests/FuzzerUnittest.cpp (original)
> +++ compiler-rt/trunk/lib/fuzzer/tests/FuzzerUnittest.cpp Tue Jun 5
> 18:23:29 2018
> @@ -28,6 +28,14 @@ extern "C" int LLVMFuzzerTestOneInput(co
> abort();
> }
>
> +TEST(Fuzzer, Basename) {
> + EXPECT_EQ(Basename("foo/bar"), "bar");
> + EXPECT_EQ(Basename("bar"), "bar");
> + EXPECT_EQ(Basename("/bar"), "bar");
> + EXPECT_EQ(Basename("foo/x"), "x");
> + EXPECT_EQ(Basename("foo/"), "");
> +}
> +
> TEST(Fuzzer, CrossOver) {
> std::unique_ptr<ExternalFunctions> t(new ExternalFunctions());
> fuzzer::EF = t.get();
>
> Modified: compiler-rt/trunk/test/fuzzer/ThreeFunctionsTest.cpp
> URL: http://llvm.org/viewvc/llvm-project/compiler-
> rt/trunk/test/fuzzer/ThreeFunctionsTest.cpp?rev=334058&r1=334057&r2=334
> 058&view=diff
> =======================================================================
> =======
> --- compiler-rt/trunk/test/fuzzer/ThreeFunctionsTest.cpp (original)
> +++ compiler-rt/trunk/test/fuzzer/ThreeFunctionsTest.cpp Tue Jun 5
> 18:23:29 2018
> @@ -8,12 +8,14 @@
> #include <cstdlib>
> #include <cstdio>
>
> +extern "C"
> __attribute__((noinline))
> -static bool Func1(const uint8_t *Data, size_t Size) {
> +bool Func1(const uint8_t *Data, size_t Size) {
> // assumes Size >= 5, doesn't check it.
> return Data[4] == 'M';
> }
>
> +extern "C"
> __attribute__((noinline))
> bool Func2(const uint8_t *Data, size_t Size) {
> return Size >= 6 && Data[5] == 'E';
>
> Modified: compiler-rt/trunk/test/fuzzer/dataflow.test
> URL: http://llvm.org/viewvc/llvm-project/compiler-
> rt/trunk/test/fuzzer/dataflow.test?rev=334058&r1=334057&r2=334058&view=
> diff
> =======================================================================
> =======
> --- compiler-rt/trunk/test/fuzzer/dataflow.test (original)
> +++ compiler-rt/trunk/test/fuzzer/dataflow.test Tue Jun 5 18:23:29
> 2018
> @@ -5,6 +5,7 @@ REQUIRES: linux
> RUN: %no_fuzzer_cpp_compiler -c -fno-sanitize=all -fsanitize=dataflow
> %S/../../lib/fuzzer/dataflow/DataFlow.cpp -o %t-DataFlow.o
> RUN: %no_fuzzer_cpp_compiler -fno-sanitize=all -fsanitize=dataflow
> -fsanitize-coverage=trace-pc-guard,pc-table,func,trace-cmp
> %S/ThreeFunctionsTest.cpp %t-DataFlow.o -o %t-ThreeFunctionsTestDF
> RUN: %no_fuzzer_cpp_compiler -fno-sanitize=all -fsanitize=dataflow
> -fsanitize-coverage=trace-pc-guard,pc-table,func,trace-cmp
> %S/ExplodeDFSanLabelsTest.cpp %t-DataFlow.o -o %t-
> ExplodeDFSanLabelsTestDF
> +RUN: %cpp_compiler %S/ThreeFunctionsTest.cpp -o %t-ThreeFunctionsTest
>
> # Dump the function list.
> RUN: %t-ThreeFunctionsTestDF 2>&1 | FileCheck %s --check-
> prefix=FUNC_LIST
> @@ -70,3 +71,13 @@ RUN: %t-ExplodeDFSanLabelsTestDF 2 4 %t
> RUN: %t-ExplodeDFSanLabelsTestDF 4 6 %t/IN/1234567890123456
> # Or we can use collect_data_flow
> RUN: %libfuzzer_src/scripts/collect_data_flow.py %t-
> ExplodeDFSanLabelsTestDF %t/IN/1234567890123456
> +
> +# Test that we can run collect_data_flow on the entire corpus dir
> +RUN: rm -rf %t/OUT
> +RUN: %libfuzzer_src/scripts/collect_data_flow.py %t-
> ThreeFunctionsTestDF %t/IN %t/OUT
> +RUN: %t-ThreeFunctionsTest -data_flow_trace=%t/OUT -runs=0 -
> focus_function=Func2 2>&1 | FileCheck %s --check-
> prefix=USE_DATA_FLOW_TRACE
> +USE_DATA_FLOW_TRACE: INFO: Focus function is set to 'Func2'
> +USE_DATA_FLOW_TRACE: INFO: DataFlowTrace: reading from {{.*}}/OUT
> +USE_DATA_FLOW_TRACE-DAG: a8eefe2fd5d6b32028f355fafa3e739a6bf5edc =>
> |000001|
> +USE_DATA_FLOW_TRACE-DGA: d28cb407e8e1a702c72d25473f0553d3ec172262 =>
> |0000011|
> +USE_DATA_FLOW_TRACE: INFO: DataFlowTrace: 6 trace files, 3 functions,
> 2 traces with focus function
>
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
More information about the llvm-commits
mailing list