[compiler-rt] r334058 - [libFuzzer] initial implementation of -data_flow_trace. It parses the data flow trace and prints the summary, but doesn't use the information in any other way yet
Vitaly Buka via llvm-commits
llvm-commits at lists.llvm.org
Wed Jun 6 13:39:43 PDT 2018
Looks like "Vector<bool> V(Len);" is not used and can be removed.
On Wed, Jun 6, 2018 at 8:07 AM Kostya Serebryany via llvm-commits <
llvm-commits at lists.llvm.org> wrote:
> thanks!
> I'll fix this in ~ 3 hours (my VPN seems to be broken, can't do it now)
>
> On Tue, Jun 5, 2018 at 11:57 PM <douglas.yung at sony.com> wrote:
>
>> Hi Kostya,
>>
>> This commit is causing a build failure on several of the linux bots. Can
>> you take a look?
>>
>> http://lab.llvm.org:8011/builders/clang-x86_64-debian-fast/builds/9903
>>
>> http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-debian-fast/builds/9096
>>
>> 132.501 [1813/18/2813] Building CXX object
>> projects/compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.x86_64.dir/FuzzerDataFlowTrace.cpp.o
>> FAILED:
>> projects/compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.x86_64.dir/FuzzerDataFlowTrace.cpp.o
>>
>> /usr/bin/ccache clang++ -D_DEBUG -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS
>> -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS
>> -Iprojects/compiler-rt/lib/fuzzer
>> -I/home/llvmbb/llvm-build-dir/llvm-clang-lld-x86_64-debian-fast/llvm.src/projects/compiler-rt/lib/fuzzer
>> -Iinclude
>> -I/home/llvmbb/llvm-build-dir/llvm-clang-lld-x86_64-debian-fast/llvm.src/include
>> -std=c++11 -Wdocumentation -Wno-documentation-deprecated-sync -fPIC
>> -fvisibility-inlines-hidden -Werror=date-time -std=c++11 -Wall -Wextra
>> -Wno-unused-parameter -Wwrite-strings -Wcast-qual
>> -Wmissing-field-initializers -pedantic -Wno-long-long
>> -Wcovered-switch-default -Wnon-virtual-dtor -Wdelete-non-virtual-dtor
>> -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections
>> -Wall -std=c++11 -Wno-unused-parameter -O3 -UNDEBUG -std=c++11
>> -Wdocumentation -Wno-documentation-deprecated-sync -fPIC
>> -fvisibility-inlines-hidden -Werror=date-time -std=c++11 -Wall -Wextra
>> -Wno-unused-parameter -Wwrite-strings -Wcast-qual
>> -Wmissing-field-initializers -pedantic -Wno-long-long
>> -Wcovered-switch-default -Wnon-virtual-dtor -Wdelete-non-virtual-dtor
>> -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections
>> -Wall -std=c++11 -Wno-unused-parameter -m64 -fPIC -fno-builtin
>> -fno-exceptions -fomit-frame-pointer -funwind-tables -fno-stack-protector
>> -fno-sanitize=safe-stack -fvisibility=hidden -fno-lto -O3
>> -gline-tables-only -Wno-gnu -Wno-variadic-macros -Wno-c99-extensions
>> -Wno-non-virtual-dtor -MD -MT
>> projects/compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.x86_64.dir/FuzzerDataFlowTrace.cpp.o
>> -MF
>> projects/compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.x86_64.dir/FuzzerDataFlowTrace.cpp.o.d
>> -o
>> projects/compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.x86_64.dir/FuzzerDataFlowTrace.cpp.o
>> -c
>> /home/llvmbb/llvm-build-dir/llvm-clang-lld-x86_64-debian-fast/llvm.src/projects/compiler-rt/lib/fuzzer/FuzzerDataFlowTrace.cpp
>> In file included from
>> /home/llvmbb/llvm-build-dir/llvm-clang-lld-x86_64-debian-fast/llvm.src/projects/compiler-rt/lib/fuzzer/FuzzerDataFlowTrace.cpp:12:
>> In file included from
>> /home/llvmbb/llvm-build-dir/llvm-clang-lld-x86_64-debian-fast/llvm.src/projects/compiler-rt/lib/fuzzer/FuzzerDataFlowTrace.h:32:
>> In file included from
>> /home/llvmbb/llvm-build-dir/llvm-clang-lld-x86_64-debian-fast/llvm.src/projects/compiler-rt/lib/fuzzer/FuzzerDefs.h:20:
>> In file included from
>> /usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/vector:65:
>> /usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/stl_bvector.h:469:9:
>> error: no matching constructor for initialization of
>> 'std::_Bvector_base<fuzzer::fuzzer_allocator<bool> >::_Bvector_impl'
>> : _M_impl(__a) { }
>> ^ ~~~
>> /usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/stl_bvector.h:595:7:
>> note: in instantiation of member function
>> 'std::_Bvector_base<fuzzer::fuzzer_allocator<bool> >::_Bvector_base'
>> requested here
>> : _Base(__a)
>> ^
>> /usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/stl_bvector.h:590:7:
>> note: in instantiation of member function 'std::vector<bool,
>> fuzzer::fuzzer_allocator<bool> >::vector' requested here
>> : vector(__n, false, __a)
>> ^
>> /home/llvmbb/llvm-build-dir/llvm-clang-lld-x86_64-debian-fast/llvm.src/projects/compiler-rt/lib/fuzzer/FuzzerDataFlowTrace.cpp:70:22:
>> note: in instantiation of member function 'std::vector<bool,
>> fuzzer::fuzzer_allocator<bool> >::vector' requested here
>> Vector<bool> V(Len);
>> ^
>> /usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/stl_bvector.h:419:14:
>> note: candidate constructor (the implicit copy constructor) not viable: no
>> known conversion from 'const allocator_type' (aka 'const
>> fuzzer::fuzzer_allocator<bool>') to 'const
>> std::_Bvector_base<fuzzer::fuzzer_allocator<bool> >::_Bvector_impl' for 1st
>> argument
>> struct _Bvector_impl
>> ^
>> /usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/stl_bvector.h:419:14:
>> note: candidate constructor (the implicit move constructor) not viable: no
>> known conversion from 'const allocator_type' (aka 'const
>> fuzzer::fuzzer_allocator<bool>') to
>> 'std::_Bvector_base<fuzzer::fuzzer_allocator<bool> >::_Bvector_impl' for
>> 1st argument
>> struct _Bvector_impl
>> ^
>> /usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/stl_bvector.h:430:2:
>> note: candidate constructor not viable: no known conversion from 'const
>> fuzzer_allocator<bool>' to 'const fuzzer_allocator<unsigned long>' for 1st
>> argument
>> _Bvector_impl(const _Bit_alloc_type& __a)
>> ^
>> /usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/stl_bvector.h:435:2:
>> note: candidate constructor not viable: no known conversion from 'const
>> fuzzer_allocator<bool>' to 'fuzzer_allocator<unsigned long>' for 1st
>> argument
>> _Bvector_impl(_Bit_alloc_type&& __a)
>> ^
>> /usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/stl_bvector.h:426:2:
>> note: candidate constructor not viable: requires 0 arguments, but 1 was
>> provided
>> _Bvector_impl()
>> ^
>> 1 error generated.
>>
>> Douglas Yung
>>
>> > -----Original Message-----
>> > From: llvm-commits [mailto:llvm-commits-bounces at lists.llvm.org] On
>> > Behalf Of Kostya Serebryany via llvm-commits
>> > Sent: Tuesday, June 05, 2018 18:23
>> > To: llvm-commits at lists.llvm.org
>> > Subject: [compiler-rt] r334058 - [libFuzzer] initial implementation of
>> > -data_flow_trace. It parses the data flow trace and prints the summary,
>> > but doesn't use the information in any other way yet
>> >
>> > Author: kcc
>> > Date: Tue Jun 5 18:23:29 2018
>> > New Revision: 334058
>> >
>> > URL: http://llvm.org/viewvc/llvm-project?rev=334058&view=rev
>> > Log:
>> > [libFuzzer] initial implementation of -data_flow_trace. It parses the
>> > data flow trace and prints the summary, but doesn't use the information
>> > in any other way yet
>> >
>> > Added:
>> > compiler-rt/trunk/lib/fuzzer/FuzzerDataFlowTrace.cpp
>> > compiler-rt/trunk/lib/fuzzer/FuzzerDataFlowTrace.h
>> > Modified:
>> > compiler-rt/trunk/lib/fuzzer/CMakeLists.txt
>> > compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp
>> > compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def
>> > compiler-rt/trunk/lib/fuzzer/FuzzerIO.cpp
>> > compiler-rt/trunk/lib/fuzzer/FuzzerIO.h
>> > compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h
>> > compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp
>> > compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h
>> > compiler-rt/trunk/lib/fuzzer/dataflow/DataFlow.cpp
>> > compiler-rt/trunk/lib/fuzzer/scripts/collect_data_flow.py
>> > compiler-rt/trunk/lib/fuzzer/tests/FuzzerUnittest.cpp
>> > compiler-rt/trunk/test/fuzzer/ThreeFunctionsTest.cpp
>> > compiler-rt/trunk/test/fuzzer/dataflow.test
>> >
>> > Modified: compiler-rt/trunk/lib/fuzzer/CMakeLists.txt
>> > URL: http://llvm.org/viewvc/llvm-project/compiler-
>> > rt/trunk/lib/fuzzer/CMakeLists.txt?rev=334058&r1=334057&r2=334058&view=
>> > diff
>> > =======================================================================
>> > =======
>> > --- compiler-rt/trunk/lib/fuzzer/CMakeLists.txt (original)
>> > +++ compiler-rt/trunk/lib/fuzzer/CMakeLists.txt Tue Jun 5 18:23:29
>> > 2018
>> > @@ -1,5 +1,6 @@
>> > set(LIBFUZZER_SOURCES
>> > FuzzerCrossOver.cpp
>> > + FuzzerDataFlowTrace.cpp
>> > FuzzerDriver.cpp
>> > FuzzerExtFunctionsDlsym.cpp
>> > FuzzerExtFunctionsDlsymWin.cpp
>> >
>> > Added: compiler-rt/trunk/lib/fuzzer/FuzzerDataFlowTrace.cpp
>> > URL: http://llvm.org/viewvc/llvm-project/compiler-
>> > rt/trunk/lib/fuzzer/FuzzerDataFlowTrace.cpp?rev=334058&view=auto
>> > =======================================================================
>> > =======
>> > --- compiler-rt/trunk/lib/fuzzer/FuzzerDataFlowTrace.cpp (added)
>> > +++ compiler-rt/trunk/lib/fuzzer/FuzzerDataFlowTrace.cpp Tue Jun 5
>> > 18:23:29 2018
>> > @@ -0,0 +1,90 @@
>> > +//===- FuzzerDataFlowTrace.cpp - DataFlowTrace ---*-
>> > C++ -* ===//
>> > +//
>> > +// The LLVM Compiler Infrastructure
>> > +//
>> > +// This file is distributed under the University of Illinois Open
>> > Source
>> > +// License. See LICENSE.TXT for details.
>> > +//
>> > +//===-----------------------------------------------------------------
>> > -----===//
>> > +// fuzzer::DataFlowTrace
>> > +//===-----------------------------------------------------------------
>> > -----===//
>> > +
>> > +#include "FuzzerDataFlowTrace.h"
>> > +#include "FuzzerIO.h"
>> > +
>> > +#include <cstdlib>
>> > +#include <fstream>
>> > +#include <string>
>> > +#include <vector>
>> > +
>> > +namespace fuzzer {
>> > +
>> > +void DataFlowTrace::Init(const std::string &DirPath,
>> > + const std::string &FocusFunction) {
>> > + if (DirPath.empty()) return;
>> > + const char *kFunctionsTxt = "functions.txt";
>> > + Printf("INFO: DataFlowTrace: reading from '%s'\n", DirPath.c_str());
>> > + Vector<SizedFile> Files;
>> > + GetSizedFilesFromDir(DirPath, &Files);
>> > + std::string L;
>> > +
>> > + // Read functions.txt
>> > + std::ifstream IF(DirPlusFile(DirPath, kFunctionsTxt));
>> > + size_t FocusFuncIdx = SIZE_MAX;
>> > + size_t NumFunctions = 0;
>> > + while (std::getline(IF, L, '\n')) {
>> > + NumFunctions++;
>> > + if (FocusFunction == L)
>> > + FocusFuncIdx = NumFunctions - 1;
>> > + }
>> > + if (!NumFunctions || FocusFuncIdx == SIZE_MAX || Files.size() <= 1)
>> > + return;
>> > + // Read traces.
>> > + size_t NumTraceFiles = 0;
>> > + size_t NumTracesWithFocusFunction = 0;
>> > + for (auto &SF : Files) {
>> > + auto Name = Basename(SF.File);
>> > + if (Name == kFunctionsTxt) continue;
>> > + auto ParseError = [&](const char *Err) {
>> > + Printf("DataFlowTrace: parse error: %s\n File: %s\n Line:
>> > %s\n", Err,
>> > + Name.c_str(), L.c_str());
>> > + };
>> > + NumTraceFiles++;
>> > + // Printf("=== %s\n", Name.c_str());
>> > + std::ifstream IF(SF.File);
>> > + while (std::getline(IF, L, '\n')) {
>> > + size_t SpacePos = L.find(' ');
>> > + if (SpacePos == std::string::npos)
>> > + return ParseError("no space in the trace line");
>> > + if (L.empty() || L[0] != 'F')
>> > + return ParseError("the trace line doesn't start with 'F'");
>> > + size_t N = std::atol(L.c_str() + 1);
>> > + if (N >= NumFunctions)
>> > + return ParseError("N is greater than the number of
>> > functions");
>> > + if (N == FocusFuncIdx) {
>> > + NumTracesWithFocusFunction++;
>> > + const char *Beg = L.c_str() + SpacePos + 1;
>> > + const char *End = L.c_str() + L.size();
>> > + assert(Beg < End);
>> > + size_t Len = End - Beg;
>> > + Vector<bool> V(Len);
>> > + for (size_t I = 0; I < Len; I++) {
>> > + if (Beg[I] != '0' && Beg[I] != '1')
>> > + ParseError("the trace should contain only 0 or 1");
>> > + V[I] = Beg[I] == '1';
>> > + }
>> > + // Print just a few small traces.
>> > + if (NumTracesWithFocusFunction <= 3 && Len <= 16)
>> > + Printf("%s => |%s|\n", Name.c_str(), L.c_str() + SpacePos +
>> > 1);
>> > + break; // No need to parse the following lines.
>> > + }
>> > + }
>> > + }
>> > + assert(NumTraceFiles == Files.size() - 1);
>> > + Printf("INFO: DataFlowTrace: %zd trace files, %zd functions, "
>> > + "%zd traces with focus function\n",
>> > + NumTraceFiles, NumFunctions, NumTracesWithFocusFunction);
>> > +}
>> > +
>> > +} // namespace fuzzer
>> > +
>> >
>> > Added: compiler-rt/trunk/lib/fuzzer/FuzzerDataFlowTrace.h
>> > URL: http://llvm.org/viewvc/llvm-project/compiler-
>> > rt/trunk/lib/fuzzer/FuzzerDataFlowTrace.h?rev=334058&view=auto
>> > =======================================================================
>> > =======
>> > --- compiler-rt/trunk/lib/fuzzer/FuzzerDataFlowTrace.h (added)
>> > +++ compiler-rt/trunk/lib/fuzzer/FuzzerDataFlowTrace.h Tue Jun 5
>> > 18:23:29 2018
>> > @@ -0,0 +1,40 @@
>> > +//===- FuzzerDataFlowTrace.h - Internal header for the Fuzzer ---*-
>> > C++ -* ===//
>> > +//
>> > +// The LLVM Compiler Infrastructure
>> > +//
>> > +// This file is distributed under the University of Illinois Open
>> > Source
>> > +// License. See LICENSE.TXT for details.
>> > +//
>> > +//===-----------------------------------------------------------------
>> > -----===//
>> > +// fuzzer::DataFlowTrace; reads and handles a data-flow trace.
>> > +//
>> > +// A data flow trace is generated by e.g. dataflow/DataFlow.cpp
>> > +// and is stored on disk in a separate directory.
>> > +//
>> > +// The trace dir contains a file 'functions.txt' which lists function
>> > names,
>> > +// oner per line, e.g.
>> > +// ==> functions.txt <==
>> > +// Func2
>> > +// LLVMFuzzerTestOneInput
>> > +// Func1
>> > +//
>> > +// All other files in the dir are the traces, see
>> > dataflow/DataFlow.cpp.
>> > +// The name of the file is sha1 of the input used to generate the
>> > trace.
>> > +//
>> > +// Current status:
>> > +// the data is parsed and the summary is printed, but the data is
>> > not yet
>> > +// used in any other way.
>> > +//===-----------------------------------------------------------------
>> > -----===//
>> > +
>> > +#ifndef LLVM_FUZZER_DATA_FLOW_TRACE
>> > +#define LLVM_FUZZER_DATA_FLOW_TRACE
>> > +
>> > +#include "FuzzerDefs.h"
>> > +
>> > +namespace fuzzer {
>> > +struct DataFlowTrace {
>> > + void Init(const std::string &DirPath, const std::string
>> > &FocusFunction);
>> > +};
>> > +} // namespace fuzzer
>> > +
>> > +#endif // LLVM_FUZZER_DATA_FLOW_TRACE
>> >
>> > Modified: compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp
>> > URL: http://llvm.org/viewvc/llvm-project/compiler-
>> > rt/trunk/lib/fuzzer/FuzzerDriver.cpp?rev=334058&r1=334057&r2=334058&vie
>> > w=diff
>> > =======================================================================
>> > =======
>> > --- compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp (original)
>> > +++ compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp Tue Jun 5 18:23:29
>> > 2018
>> > @@ -623,6 +623,8 @@ int FuzzerDriver(int *argc, char ***argv
>> > Options.ExitOnItem = Flags.exit_on_item;
>> > if (Flags.focus_function)
>> > Options.FocusFunction = Flags.focus_function;
>> > + if (Flags.data_flow_trace)
>> > + Options.DataFlowTrace = Flags.data_flow_trace;
>> >
>> > unsigned Seed = Flags.seed;
>> > // Initialize Seed.
>> >
>> > Modified: compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def
>> > URL: http://llvm.org/viewvc/llvm-project/compiler-
>> > rt/trunk/lib/fuzzer/FuzzerFlags.def?rev=334058&r1=334057&r2=334058&view
>> > =diff
>> > =======================================================================
>> > =======
>> > --- compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def (original)
>> > +++ compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def Tue Jun 5 18:23:29
>> > 2018
>> > @@ -153,3 +153,5 @@ FUZZER_DEPRECATED_FLAG(use_equivalence_s
>> > FUZZER_FLAG_INT(analyze_dict, 0, "Experimental")
>> > FUZZER_DEPRECATED_FLAG(use_clang_coverage)
>> > FUZZER_FLAG_INT(use_feature_frequency, 0, "Experimental/internal")
>> > +
>> > +FUZZER_FLAG_STRING(data_flow_trace, "Experimental: use the data flow
>> > trace")
>> >
>> > Modified: compiler-rt/trunk/lib/fuzzer/FuzzerIO.cpp
>> > URL: http://llvm.org/viewvc/llvm-project/compiler-
>> > rt/trunk/lib/fuzzer/FuzzerIO.cpp?rev=334058&r1=334057&r2=334058&view=di
>> > ff
>> > =======================================================================
>> > =======
>> > --- compiler-rt/trunk/lib/fuzzer/FuzzerIO.cpp (original)
>> > +++ compiler-rt/trunk/lib/fuzzer/FuzzerIO.cpp Tue Jun 5 18:23:29 2018
>> > @@ -100,6 +100,14 @@ std::string DirPlusFile(const std::strin
>> > return DirPath + GetSeparator() + FileName;
>> > }
>> >
>> > +std::string Basename(const std::string &Path, char Separator) {
>> > + size_t Pos = Path.rfind(Separator);
>> > + if (Pos == std::string::npos)
>> > + return Path;
>> > + assert(Pos < Path.size());
>> > + return Path.substr(Pos + 1);
>> > +}
>> > +
>> > void DupAndCloseStderr() {
>> > int OutputFd = DuplicateFile(2);
>> > if (OutputFd > 0) {
>> >
>> > Modified: compiler-rt/trunk/lib/fuzzer/FuzzerIO.h
>> > URL: http://llvm.org/viewvc/llvm-project/compiler-
>> > rt/trunk/lib/fuzzer/FuzzerIO.h?rev=334058&r1=334057&r2=334058&view=diff
>> > =======================================================================
>> > =======
>> > --- compiler-rt/trunk/lib/fuzzer/FuzzerIO.h (original)
>> > +++ compiler-rt/trunk/lib/fuzzer/FuzzerIO.h Tue Jun 5 18:23:29 2018
>> > @@ -67,6 +67,8 @@ struct SizedFile {
>> > void GetSizedFilesFromDir(const std::string &Dir, Vector<SizedFile>
>> > *V);
>> >
>> > char GetSeparator();
>> > +// Similar to the basename utility: returns the file name w/o the dir
>> > prefix.
>> > +std::string Basename(const std::string &Path, char Separator =
>> > GetSeparator());
>> >
>> > FILE* OpenFile(int Fd, const char *Mode);
>> >
>> >
>> > Modified: compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h
>> > URL: http://llvm.org/viewvc/llvm-project/compiler-
>> > rt/trunk/lib/fuzzer/FuzzerInternal.h?rev=334058&r1=334057&r2=334058&vie
>> > w=diff
>> > =======================================================================
>> > =======
>> > --- compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h (original)
>> > +++ compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h Tue Jun 5 18:23:29
>> > 2018
>> > @@ -12,6 +12,7 @@
>> > #ifndef LLVM_FUZZER_INTERNAL_H
>> > #define LLVM_FUZZER_INTERNAL_H
>> >
>> > +#include "FuzzerDataFlowTrace.h"
>> > #include "FuzzerDefs.h"
>> > #include "FuzzerExtFunctions.h"
>> > #include "FuzzerInterface.h"
>> > @@ -134,6 +135,7 @@ private:
>> > InputCorpus &Corpus;
>> > MutationDispatcher &MD;
>> > FuzzingOptions Options;
>> > + DataFlowTrace DFT;
>> >
>> > system_clock::time_point ProcessStartTime = system_clock::now();
>> > system_clock::time_point UnitStartTime, UnitStopTime;
>> >
>> > Modified: compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp
>> > URL: http://llvm.org/viewvc/llvm-project/compiler-
>> > rt/trunk/lib/fuzzer/FuzzerLoop.cpp?rev=334058&r1=334057&r2=334058&view=
>> > diff
>> > =======================================================================
>> > =======
>> > --- compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp (original)
>> > +++ compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp Tue Jun 5 18:23:29
>> > 2018
>> > @@ -160,6 +160,7 @@ Fuzzer::Fuzzer(UserCallback CB, InputCor
>> > CurrentUnitSize = 0;
>> > memset(BaseSha1, 0, sizeof(BaseSha1));
>> > TPC.SetFocusFunction(Options.FocusFunction);
>> > + DFT.Init(Options.DataFlowTrace, Options.FocusFunction);
>> > }
>> >
>> > Fuzzer::~Fuzzer() {}
>> >
>> > Modified: compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h
>> > URL: http://llvm.org/viewvc/llvm-project/compiler-
>> > rt/trunk/lib/fuzzer/FuzzerOptions.h?rev=334058&r1=334057&r2=334058&view
>> > =diff
>> > =======================================================================
>> > =======
>> > --- compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h (original)
>> > +++ compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h Tue Jun 5 18:23:29
>> > 2018
>> > @@ -46,6 +46,7 @@ struct FuzzingOptions {
>> > std::string ExitOnSrcPos;
>> > std::string ExitOnItem;
>> > std::string FocusFunction;
>> > + std::string DataFlowTrace;
>> > bool SaveArtifacts = true;
>> > bool PrintNEW = true; // Print a status line when new units are
>> > found;
>> > bool PrintNewCovPcs = false;
>> >
>> > Modified: compiler-rt/trunk/lib/fuzzer/dataflow/DataFlow.cpp
>> > URL: http://llvm.org/viewvc/llvm-project/compiler-
>> > rt/trunk/lib/fuzzer/dataflow/DataFlow.cpp?rev=334058&r1=334057&r2=33405
>> > 8&view=diff
>> > =======================================================================
>> > =======
>> > --- compiler-rt/trunk/lib/fuzzer/dataflow/DataFlow.cpp (original)
>> > +++ compiler-rt/trunk/lib/fuzzer/dataflow/DataFlow.cpp Tue Jun 5
>> > 18:23:29 2018
>> > @@ -69,6 +69,7 @@ static const uintptr_t *FuncsBeg;
>> > static __thread size_t CurrentFunc;
>> > static dfsan_label *FuncLabels; // Array of NumFuncs elements.
>> > static char *PrintableStringForLabel; // InputLen + 2 bytes.
>> > +static bool LabelSeen[1 << 8 * sizeof(dfsan_label)];
>> >
>> > // Prints all instrumented functions.
>> > static int PrintFunctions() {
>> > @@ -89,7 +90,11 @@ static int PrintFunctions() {
>> > return 0;
>> > }
>> >
>> > -static void SetBytesForLabel(dfsan_label L, char *Bytes) {
>> > +extern "C"
>> > +void SetBytesForLabel(dfsan_label L, char *Bytes) {
>> > + if (LabelSeen[L])
>> > + return;
>> > + LabelSeen[L] = true;
>> > assert(L);
>> > if (L <= InputLen + 1) {
>> > Bytes[L - 1] = '1';
>> > @@ -103,6 +108,7 @@ static void SetBytesForLabel(dfsan_label
>> > static char *GetPrintableStringForLabel(dfsan_label L) {
>> > memset(PrintableStringForLabel, '0', InputLen + 1);
>> > PrintableStringForLabel[InputLen + 1] = 0;
>> > + memset(LabelSeen, 0, sizeof(LabelSeen));
>> > SetBytesForLabel(L, PrintableStringForLabel);
>> > return PrintableStringForLabel;
>> > }
>> >
>> > Modified: compiler-rt/trunk/lib/fuzzer/scripts/collect_data_flow.py
>> > URL: http://llvm.org/viewvc/llvm-project/compiler-
>> > rt/trunk/lib/fuzzer/scripts/collect_data_flow.py?rev=334058&r1=334057&r
>> > 2=334058&view=diff
>> > =======================================================================
>> > =======
>> > --- compiler-rt/trunk/lib/fuzzer/scripts/collect_data_flow.py
>> > (original)
>> > +++ compiler-rt/trunk/lib/fuzzer/scripts/collect_data_flow.py Tue Jun
>> > 5 18:23:29 2018
>> > @@ -11,9 +11,15 @@
>> > # the complete trace for all input bytes (running it on all bytes at
>> > once
>> > # may fail if DFSan runs out of labels).
>> > # Usage:
>> > -# collect_data_flow.py BINARY INPUT [RESULT]
>> > +#
>> > +# # Collect dataflow for one input, store it in OUTPUT (default is
>> > stdout)
>> > +# collect_data_flow.py BINARY INPUT [OUTPUT]
>> > +#
>> > +# # Collect dataflow for all inputs in CORPUS_DIR, store them in
>> > OUTPUT_DIR
>> > +# collect_data_flow.py BINARY CORPUS_DIR OUTPUT_DIR
>> > #===------------------------------------------------------------------
>> > ------===#
>> > import atexit
>> > +import hashlib
>> > import sys
>> > import os
>> > import subprocess
>> > @@ -26,9 +32,26 @@ def cleanup(d):
>> > print "removing: ", d
>> > shutil.rmtree(d)
>> >
>> > +def collect_dataflow_for_corpus(self, exe, corpus_dir, output_dir):
>> > + print "Collecting dataflow for corpus:", corpus_dir, \
>> > + "output_dir:", output_dir
>> > + assert not os.path.exists(output_dir)
>> > + os.mkdir(output_dir)
>> > + for root, dirs, files in os.walk(corpus_dir):
>> > + for f in files:
>> > + path = os.path.join(root, f)
>> > + sha1 = hashlib.sha1(open(path).read()).hexdigest()
>> > + output = os.path.join(output_dir, sha1)
>> > + subprocess.call([self, exe, path, output])
>> > + functions_txt = open(os.path.join(output_dir, "functions.txt"), "w")
>> > + subprocess.call([exe], stdout=functions_txt)
>> > +
>> > +
>> > def main(argv):
>> > exe = argv[1]
>> > inp = argv[2]
>> > + if os.path.isdir(inp):
>> > + return collect_dataflow_for_corpus(argv[0], exe, inp, argv[3])
>> > size = os.path.getsize(inp)
>> > q = [[0, size]]
>> > tmpdir = tempfile.mkdtemp(prefix="libfuzzer-tmp-")
>> >
>> > Modified: compiler-rt/trunk/lib/fuzzer/tests/FuzzerUnittest.cpp
>> > URL: http://llvm.org/viewvc/llvm-project/compiler-
>> > rt/trunk/lib/fuzzer/tests/FuzzerUnittest.cpp?rev=334058&r1=334057&r2=33
>> > 4058&view=diff
>> > =======================================================================
>> > =======
>> > --- compiler-rt/trunk/lib/fuzzer/tests/FuzzerUnittest.cpp (original)
>> > +++ compiler-rt/trunk/lib/fuzzer/tests/FuzzerUnittest.cpp Tue Jun 5
>> > 18:23:29 2018
>> > @@ -28,6 +28,14 @@ extern "C" int LLVMFuzzerTestOneInput(co
>> > abort();
>> > }
>> >
>> > +TEST(Fuzzer, Basename) {
>> > + EXPECT_EQ(Basename("foo/bar"), "bar");
>> > + EXPECT_EQ(Basename("bar"), "bar");
>> > + EXPECT_EQ(Basename("/bar"), "bar");
>> > + EXPECT_EQ(Basename("foo/x"), "x");
>> > + EXPECT_EQ(Basename("foo/"), "");
>> > +}
>> > +
>> > TEST(Fuzzer, CrossOver) {
>> > std::unique_ptr<ExternalFunctions> t(new ExternalFunctions());
>> > fuzzer::EF = t.get();
>> >
>> > Modified: compiler-rt/trunk/test/fuzzer/ThreeFunctionsTest.cpp
>> > URL: http://llvm.org/viewvc/llvm-project/compiler-
>> > rt/trunk/test/fuzzer/ThreeFunctionsTest.cpp?rev=334058&r1=334057&r2=334
>> > 058&view=diff
>> > =======================================================================
>> > =======
>> > --- compiler-rt/trunk/test/fuzzer/ThreeFunctionsTest.cpp (original)
>> > +++ compiler-rt/trunk/test/fuzzer/ThreeFunctionsTest.cpp Tue Jun 5
>> > 18:23:29 2018
>> > @@ -8,12 +8,14 @@
>> > #include <cstdlib>
>> > #include <cstdio>
>> >
>> > +extern "C"
>> > __attribute__((noinline))
>> > -static bool Func1(const uint8_t *Data, size_t Size) {
>> > +bool Func1(const uint8_t *Data, size_t Size) {
>> > // assumes Size >= 5, doesn't check it.
>> > return Data[4] == 'M';
>> > }
>> >
>> > +extern "C"
>> > __attribute__((noinline))
>> > bool Func2(const uint8_t *Data, size_t Size) {
>> > return Size >= 6 && Data[5] == 'E';
>> >
>> > Modified: compiler-rt/trunk/test/fuzzer/dataflow.test
>> > URL: http://llvm.org/viewvc/llvm-project/compiler-
>> > rt/trunk/test/fuzzer/dataflow.test?rev=334058&r1=334057&r2=334058&view=
>> > diff
>> > =======================================================================
>> > =======
>> > --- compiler-rt/trunk/test/fuzzer/dataflow.test (original)
>> > +++ compiler-rt/trunk/test/fuzzer/dataflow.test Tue Jun 5 18:23:29
>> > 2018
>> > @@ -5,6 +5,7 @@ REQUIRES: linux
>> > RUN: %no_fuzzer_cpp_compiler -c -fno-sanitize=all -fsanitize=dataflow
>> > %S/../../lib/fuzzer/dataflow/DataFlow.cpp -o %t-DataFlow.o
>> > RUN: %no_fuzzer_cpp_compiler -fno-sanitize=all -fsanitize=dataflow
>> > -fsanitize-coverage=trace-pc-guard,pc-table,func,trace-cmp
>> > %S/ThreeFunctionsTest.cpp %t-DataFlow.o -o %t-ThreeFunctionsTestDF
>> > RUN: %no_fuzzer_cpp_compiler -fno-sanitize=all -fsanitize=dataflow
>> > -fsanitize-coverage=trace-pc-guard,pc-table,func,trace-cmp
>> > %S/ExplodeDFSanLabelsTest.cpp %t-DataFlow.o -o %t-
>> > ExplodeDFSanLabelsTestDF
>> > +RUN: %cpp_compiler %S/ThreeFunctionsTest.cpp -o %t-ThreeFunctionsTest
>> >
>> > # Dump the function list.
>> > RUN: %t-ThreeFunctionsTestDF 2>&1 | FileCheck %s --check-
>> > prefix=FUNC_LIST
>> > @@ -70,3 +71,13 @@ RUN: %t-ExplodeDFSanLabelsTestDF 2 4 %t
>> > RUN: %t-ExplodeDFSanLabelsTestDF 4 6 %t/IN/1234567890123456
>> > # Or we can use collect_data_flow
>> > RUN: %libfuzzer_src/scripts/collect_data_flow.py %t-
>> > ExplodeDFSanLabelsTestDF %t/IN/1234567890123456
>> > +
>> > +# Test that we can run collect_data_flow on the entire corpus dir
>> > +RUN: rm -rf %t/OUT
>> > +RUN: %libfuzzer_src/scripts/collect_data_flow.py %t-
>> > ThreeFunctionsTestDF %t/IN %t/OUT
>> > +RUN: %t-ThreeFunctionsTest -data_flow_trace=%t/OUT -runs=0 -
>> > focus_function=Func2 2>&1 | FileCheck %s --check-
>> > prefix=USE_DATA_FLOW_TRACE
>> > +USE_DATA_FLOW_TRACE: INFO: Focus function is set to 'Func2'
>> > +USE_DATA_FLOW_TRACE: INFO: DataFlowTrace: reading from {{.*}}/OUT
>> > +USE_DATA_FLOW_TRACE-DAG: a8eefe2fd5d6b32028f355fafa3e739a6bf5edc =>
>> > |000001|
>> > +USE_DATA_FLOW_TRACE-DGA: d28cb407e8e1a702c72d25473f0553d3ec172262 =>
>> > |0000011|
>> > +USE_DATA_FLOW_TRACE: INFO: DataFlowTrace: 6 trace files, 3 functions,
>> > 2 traces with focus function
>> >
>> >
>> > _______________________________________________
>> > llvm-commits mailing list
>> > llvm-commits at lists.llvm.org
>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20180606/74a0f988/attachment.html>
More information about the llvm-commits
mailing list