<div dir="ltr">thanks!<div>I'll fix this in ~ 3 hours (my VPN seems to be broken, can't do it now)</div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Jun 5, 2018 at 11:57 PM <<a href="mailto:douglas.yung@sony.com">douglas.yung@sony.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Kostya,<br>
<br>
This commit is causing a build failure on several of the linux bots. Can you take a look?<br>
<br>
<a href="http://lab.llvm.org:8011/builders/clang-x86_64-debian-fast/builds/9903" rel="noreferrer" target="_blank">http://lab.llvm.org:8011/builders/clang-x86_64-debian-fast/builds/9903</a><br>
<a href="http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-debian-fast/builds/9096" rel="noreferrer" target="_blank">http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-debian-fast/builds/9096</a><br>
<br>
132.501 [1813/18/2813] Building CXX object projects/compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.x86_64.dir/FuzzerDataFlowTrace.cpp.o<br>
FAILED: projects/compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.x86_64.dir/FuzzerDataFlowTrace.cpp.o <br>
/usr/bin/ccache clang++ -D_DEBUG -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Iprojects/compiler-rt/lib/fuzzer -I/home/llvmbb/llvm-build-dir/llvm-clang-lld-x86_64-debian-fast/llvm.src/projects/compiler-rt/lib/fuzzer -Iinclude -I/home/llvmbb/llvm-build-dir/llvm-clang-lld-x86_64-debian-fast/llvm.src/include -std=c++11 -Wdocumentation -Wno-documentation-deprecated-sync -fPIC -fvisibility-inlines-hidden -Werror=date-time -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wcovered-switch-default -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -Wall -std=c++11 -Wno-unused-parameter -O3 -UNDEBUG -std=c++11 -Wdocumentation -Wno-documentation-deprecated-sync -fPIC -fvisibility-inlines-hidden -Werror=date-time -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wcovered-switch-default -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -Wall -std=c++11 -Wno-unused-parameter -m64 -fPIC -fno-builtin -fno-exceptions -fomit-frame-pointer -funwind-tables -fno-stack-protector -fno-sanitize=safe-stack -fvisibility=hidden -fno-lto -O3 -gline-tables-only -Wno-gnu -Wno-variadic-macros -Wno-c99-extensions -Wno-non-virtual-dtor -MD -MT projects/compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.x86_64.dir/FuzzerDataFlowTrace.cpp.o -MF projects/compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.x86_64.dir/FuzzerDataFlowTrace.cpp.o.d -o projects/compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.x86_64.dir/FuzzerDataFlowTrace.cpp.o -c /home/llvmbb/llvm-build-dir/llvm-clang-lld-x86_64-debian-fast/llvm.src/projects/compiler-rt/lib/fuzzer/FuzzerDataFlowTrace.cpp<br>
In file included from /home/llvmbb/llvm-build-dir/llvm-clang-lld-x86_64-debian-fast/llvm.src/projects/compiler-rt/lib/fuzzer/FuzzerDataFlowTrace.cpp:12:<br>
In file included from /home/llvmbb/llvm-build-dir/llvm-clang-lld-x86_64-debian-fast/llvm.src/projects/compiler-rt/lib/fuzzer/FuzzerDataFlowTrace.h:32:<br>
In file included from /home/llvmbb/llvm-build-dir/llvm-clang-lld-x86_64-debian-fast/llvm.src/projects/compiler-rt/lib/fuzzer/FuzzerDefs.h:20:<br>
In file included from /usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/vector:65:<br>
/usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/stl_bvector.h:469:9: error: no matching constructor for initialization of 'std::_Bvector_base<fuzzer::fuzzer_allocator<bool> >::_Bvector_impl'<br>
: _M_impl(__a) { }<br>
^ ~~~<br>
/usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/stl_bvector.h:595:7: note: in instantiation of member function 'std::_Bvector_base<fuzzer::fuzzer_allocator<bool> >::_Bvector_base' requested here<br>
: _Base(__a)<br>
^<br>
/usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/stl_bvector.h:590:7: note: in instantiation of member function 'std::vector<bool, fuzzer::fuzzer_allocator<bool> >::vector' requested here<br>
: vector(__n, false, __a)<br>
^<br>
/home/llvmbb/llvm-build-dir/llvm-clang-lld-x86_64-debian-fast/llvm.src/projects/compiler-rt/lib/fuzzer/FuzzerDataFlowTrace.cpp:70:22: note: in instantiation of member function 'std::vector<bool, fuzzer::fuzzer_allocator<bool> >::vector' requested here<br>
Vector<bool> V(Len);<br>
^<br>
/usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/stl_bvector.h:419:14: note: candidate constructor (the implicit copy constructor) not viable: no known conversion from 'const allocator_type' (aka 'const fuzzer::fuzzer_allocator<bool>') to 'const std::_Bvector_base<fuzzer::fuzzer_allocator<bool> >::_Bvector_impl' for 1st argument<br>
struct _Bvector_impl<br>
^<br>
/usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/stl_bvector.h:419:14: note: candidate constructor (the implicit move constructor) not viable: no known conversion from 'const allocator_type' (aka 'const fuzzer::fuzzer_allocator<bool>') to 'std::_Bvector_base<fuzzer::fuzzer_allocator<bool> >::_Bvector_impl' for 1st argument<br>
struct _Bvector_impl<br>
^<br>
/usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/stl_bvector.h:430:2: note: candidate constructor not viable: no known conversion from 'const fuzzer_allocator<bool>' to 'const fuzzer_allocator<unsigned long>' for 1st argument<br>
_Bvector_impl(const _Bit_alloc_type& __a)<br>
^<br>
/usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/stl_bvector.h:435:2: note: candidate constructor not viable: no known conversion from 'const fuzzer_allocator<bool>' to 'fuzzer_allocator<unsigned long>' for 1st argument<br>
_Bvector_impl(_Bit_alloc_type&& __a)<br>
^<br>
/usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/stl_bvector.h:426:2: note: candidate constructor not viable: requires 0 arguments, but 1 was provided<br>
_Bvector_impl()<br>
^<br>
1 error generated.<br>
<br>
Douglas Yung<br>
<br>
> -----Original Message-----<br>
> From: llvm-commits [mailto:<a href="mailto:llvm-commits-bounces@lists.llvm.org" target="_blank">llvm-commits-bounces@lists.llvm.org</a>] On<br>
> Behalf Of Kostya Serebryany via llvm-commits<br>
> Sent: Tuesday, June 05, 2018 18:23<br>
> To: <a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a><br>
> Subject: [compiler-rt] r334058 - [libFuzzer] initial implementation of<br>
> -data_flow_trace. It parses the data flow trace and prints the summary,<br>
> but doesn't use the information in any other way yet<br>
> <br>
> Author: kcc<br>
> Date: Tue Jun 5 18:23:29 2018<br>
> New Revision: 334058<br>
> <br>
> URL: <a href="http://llvm.org/viewvc/llvm-project?rev=334058&view=rev" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project?rev=334058&view=rev</a><br>
> Log:<br>
> [libFuzzer] initial implementation of -data_flow_trace. It parses the<br>
> data flow trace and prints the summary, but doesn't use the information<br>
> in any other way yet<br>
> <br>
> Added:<br>
> compiler-rt/trunk/lib/fuzzer/FuzzerDataFlowTrace.cpp<br>
> compiler-rt/trunk/lib/fuzzer/FuzzerDataFlowTrace.h<br>
> Modified:<br>
> compiler-rt/trunk/lib/fuzzer/CMakeLists.txt<br>
> compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp<br>
> compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def<br>
> compiler-rt/trunk/lib/fuzzer/FuzzerIO.cpp<br>
> compiler-rt/trunk/lib/fuzzer/FuzzerIO.h<br>
> compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h<br>
> compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp<br>
> compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h<br>
> compiler-rt/trunk/lib/fuzzer/dataflow/DataFlow.cpp<br>
> compiler-rt/trunk/lib/fuzzer/scripts/collect_data_flow.py<br>
> compiler-rt/trunk/lib/fuzzer/tests/FuzzerUnittest.cpp<br>
> compiler-rt/trunk/test/fuzzer/ThreeFunctionsTest.cpp<br>
> compiler-rt/trunk/test/fuzzer/dataflow.test<br>
> <br>
> Modified: compiler-rt/trunk/lib/fuzzer/CMakeLists.txt<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-</a><br>
> rt/trunk/lib/fuzzer/CMakeLists.txt?rev=334058&r1=334057&r2=334058&view=<br>
> diff<br>
> =======================================================================<br>
> =======<br>
> --- compiler-rt/trunk/lib/fuzzer/CMakeLists.txt (original)<br>
> +++ compiler-rt/trunk/lib/fuzzer/CMakeLists.txt Tue Jun 5 18:23:29<br>
> 2018<br>
> @@ -1,5 +1,6 @@<br>
> set(LIBFUZZER_SOURCES<br>
> FuzzerCrossOver.cpp<br>
> + FuzzerDataFlowTrace.cpp<br>
> FuzzerDriver.cpp<br>
> FuzzerExtFunctionsDlsym.cpp<br>
> FuzzerExtFunctionsDlsymWin.cpp<br>
> <br>
> Added: compiler-rt/trunk/lib/fuzzer/FuzzerDataFlowTrace.cpp<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-</a><br>
> rt/trunk/lib/fuzzer/FuzzerDataFlowTrace.cpp?rev=334058&view=auto<br>
> =======================================================================<br>
> =======<br>
> --- compiler-rt/trunk/lib/fuzzer/FuzzerDataFlowTrace.cpp (added)<br>
> +++ compiler-rt/trunk/lib/fuzzer/FuzzerDataFlowTrace.cpp Tue Jun 5<br>
> 18:23:29 2018<br>
> @@ -0,0 +1,90 @@<br>
> +//===- FuzzerDataFlowTrace.cpp - DataFlowTrace ---*-<br>
> C++ -* ===//<br>
> +//<br>
> +// The LLVM Compiler Infrastructure<br>
> +//<br>
> +// This file is distributed under the University of Illinois Open<br>
> Source<br>
> +// License. See LICENSE.TXT for details.<br>
> +//<br>
> +//===-----------------------------------------------------------------<br>
> -----===//<br>
> +// fuzzer::DataFlowTrace<br>
> +//===-----------------------------------------------------------------<br>
> -----===//<br>
> +<br>
> +#include "FuzzerDataFlowTrace.h"<br>
> +#include "FuzzerIO.h"<br>
> +<br>
> +#include <cstdlib><br>
> +#include <fstream><br>
> +#include <string><br>
> +#include <vector><br>
> +<br>
> +namespace fuzzer {<br>
> +<br>
> +void DataFlowTrace::Init(const std::string &DirPath,<br>
> + const std::string &FocusFunction) {<br>
> + if (DirPath.empty()) return;<br>
> + const char *kFunctionsTxt = "functions.txt";<br>
> + Printf("INFO: DataFlowTrace: reading from '%s'\n", DirPath.c_str());<br>
> + Vector<SizedFile> Files;<br>
> + GetSizedFilesFromDir(DirPath, &Files);<br>
> + std::string L;<br>
> +<br>
> + // Read functions.txt<br>
> + std::ifstream IF(DirPlusFile(DirPath, kFunctionsTxt));<br>
> + size_t FocusFuncIdx = SIZE_MAX;<br>
> + size_t NumFunctions = 0;<br>
> + while (std::getline(IF, L, '\n')) {<br>
> + NumFunctions++;<br>
> + if (FocusFunction == L)<br>
> + FocusFuncIdx = NumFunctions - 1;<br>
> + }<br>
> + if (!NumFunctions || FocusFuncIdx == SIZE_MAX || Files.size() <= 1)<br>
> + return;<br>
> + // Read traces.<br>
> + size_t NumTraceFiles = 0;<br>
> + size_t NumTracesWithFocusFunction = 0;<br>
> + for (auto &SF : Files) {<br>
> + auto Name = Basename(SF.File);<br>
> + if (Name == kFunctionsTxt) continue;<br>
> + auto ParseError = [&](const char *Err) {<br>
> + Printf("DataFlowTrace: parse error: %s\n File: %s\n Line:<br>
> %s\n", Err,<br>
> + Name.c_str(), L.c_str());<br>
> + };<br>
> + NumTraceFiles++;<br>
> + // Printf("=== %s\n", Name.c_str());<br>
> + std::ifstream IF(SF.File);<br>
> + while (std::getline(IF, L, '\n')) {<br>
> + size_t SpacePos = L.find(' ');<br>
> + if (SpacePos == std::string::npos)<br>
> + return ParseError("no space in the trace line");<br>
> + if (L.empty() || L[0] != 'F')<br>
> + return ParseError("the trace line doesn't start with 'F'");<br>
> + size_t N = std::atol(L.c_str() + 1);<br>
> + if (N >= NumFunctions)<br>
> + return ParseError("N is greater than the number of<br>
> functions");<br>
> + if (N == FocusFuncIdx) {<br>
> + NumTracesWithFocusFunction++;<br>
> + const char *Beg = L.c_str() + SpacePos + 1;<br>
> + const char *End = L.c_str() + L.size();<br>
> + assert(Beg < End);<br>
> + size_t Len = End - Beg;<br>
> + Vector<bool> V(Len);<br>
> + for (size_t I = 0; I < Len; I++) {<br>
> + if (Beg[I] != '0' && Beg[I] != '1')<br>
> + ParseError("the trace should contain only 0 or 1");<br>
> + V[I] = Beg[I] == '1';<br>
> + }<br>
> + // Print just a few small traces.<br>
> + if (NumTracesWithFocusFunction <= 3 && Len <= 16)<br>
> + Printf("%s => |%s|\n", Name.c_str(), L.c_str() + SpacePos +<br>
> 1);<br>
> + break; // No need to parse the following lines.<br>
> + }<br>
> + }<br>
> + }<br>
> + assert(NumTraceFiles == Files.size() - 1);<br>
> + Printf("INFO: DataFlowTrace: %zd trace files, %zd functions, "<br>
> + "%zd traces with focus function\n",<br>
> + NumTraceFiles, NumFunctions, NumTracesWithFocusFunction);<br>
> +}<br>
> +<br>
> +} // namespace fuzzer<br>
> +<br>
> <br>
> Added: compiler-rt/trunk/lib/fuzzer/FuzzerDataFlowTrace.h<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-</a><br>
> rt/trunk/lib/fuzzer/FuzzerDataFlowTrace.h?rev=334058&view=auto<br>
> =======================================================================<br>
> =======<br>
> --- compiler-rt/trunk/lib/fuzzer/FuzzerDataFlowTrace.h (added)<br>
> +++ compiler-rt/trunk/lib/fuzzer/FuzzerDataFlowTrace.h Tue Jun 5<br>
> 18:23:29 2018<br>
> @@ -0,0 +1,40 @@<br>
> +//===- FuzzerDataFlowTrace.h - Internal header for the Fuzzer ---*-<br>
> C++ -* ===//<br>
> +//<br>
> +// The LLVM Compiler Infrastructure<br>
> +//<br>
> +// This file is distributed under the University of Illinois Open<br>
> Source<br>
> +// License. See LICENSE.TXT for details.<br>
> +//<br>
> +//===-----------------------------------------------------------------<br>
> -----===//<br>
> +// fuzzer::DataFlowTrace; reads and handles a data-flow trace.<br>
> +//<br>
> +// A data flow trace is generated by e.g. dataflow/DataFlow.cpp<br>
> +// and is stored on disk in a separate directory.<br>
> +//<br>
> +// The trace dir contains a file 'functions.txt' which lists function<br>
> names,<br>
> +// oner per line, e.g.<br>
> +// ==> functions.txt <==<br>
> +// Func2<br>
> +// LLVMFuzzerTestOneInput<br>
> +// Func1<br>
> +//<br>
> +// All other files in the dir are the traces, see<br>
> dataflow/DataFlow.cpp.<br>
> +// The name of the file is sha1 of the input used to generate the<br>
> trace.<br>
> +//<br>
> +// Current status:<br>
> +// the data is parsed and the summary is printed, but the data is<br>
> not yet<br>
> +// used in any other way.<br>
> +//===-----------------------------------------------------------------<br>
> -----===//<br>
> +<br>
> +#ifndef LLVM_FUZZER_DATA_FLOW_TRACE<br>
> +#define LLVM_FUZZER_DATA_FLOW_TRACE<br>
> +<br>
> +#include "FuzzerDefs.h"<br>
> +<br>
> +namespace fuzzer {<br>
> +struct DataFlowTrace {<br>
> + void Init(const std::string &DirPath, const std::string<br>
> &FocusFunction);<br>
> +};<br>
> +} // namespace fuzzer<br>
> +<br>
> +#endif // LLVM_FUZZER_DATA_FLOW_TRACE<br>
> <br>
> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-</a><br>
> rt/trunk/lib/fuzzer/FuzzerDriver.cpp?rev=334058&r1=334057&r2=334058&vie<br>
> w=diff<br>
> =======================================================================<br>
> =======<br>
> --- compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp (original)<br>
> +++ compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp Tue Jun 5 18:23:29<br>
> 2018<br>
> @@ -623,6 +623,8 @@ int FuzzerDriver(int *argc, char ***argv<br>
> Options.ExitOnItem = Flags.exit_on_item;<br>
> if (Flags.focus_function)<br>
> Options.FocusFunction = Flags.focus_function;<br>
> + if (Flags.data_flow_trace)<br>
> + Options.DataFlowTrace = Flags.data_flow_trace;<br>
> <br>
> unsigned Seed = Flags.seed;<br>
> // Initialize Seed.<br>
> <br>
> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-</a><br>
> rt/trunk/lib/fuzzer/FuzzerFlags.def?rev=334058&r1=334057&r2=334058&view<br>
> =diff<br>
> =======================================================================<br>
> =======<br>
> --- compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def (original)<br>
> +++ compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def Tue Jun 5 18:23:29<br>
> 2018<br>
> @@ -153,3 +153,5 @@ FUZZER_DEPRECATED_FLAG(use_equivalence_s<br>
> FUZZER_FLAG_INT(analyze_dict, 0, "Experimental")<br>
> FUZZER_DEPRECATED_FLAG(use_clang_coverage)<br>
> FUZZER_FLAG_INT(use_feature_frequency, 0, "Experimental/internal")<br>
> +<br>
> +FUZZER_FLAG_STRING(data_flow_trace, "Experimental: use the data flow<br>
> trace")<br>
> <br>
> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerIO.cpp<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-</a><br>
> rt/trunk/lib/fuzzer/FuzzerIO.cpp?rev=334058&r1=334057&r2=334058&view=di<br>
> ff<br>
> =======================================================================<br>
> =======<br>
> --- compiler-rt/trunk/lib/fuzzer/FuzzerIO.cpp (original)<br>
> +++ compiler-rt/trunk/lib/fuzzer/FuzzerIO.cpp Tue Jun 5 18:23:29 2018<br>
> @@ -100,6 +100,14 @@ std::string DirPlusFile(const std::strin<br>
> return DirPath + GetSeparator() + FileName;<br>
> }<br>
> <br>
> +std::string Basename(const std::string &Path, char Separator) {<br>
> + size_t Pos = Path.rfind(Separator);<br>
> + if (Pos == std::string::npos)<br>
> + return Path;<br>
> + assert(Pos < Path.size());<br>
> + return Path.substr(Pos + 1);<br>
> +}<br>
> +<br>
> void DupAndCloseStderr() {<br>
> int OutputFd = DuplicateFile(2);<br>
> if (OutputFd > 0) {<br>
> <br>
> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerIO.h<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-</a><br>
> rt/trunk/lib/fuzzer/FuzzerIO.h?rev=334058&r1=334057&r2=334058&view=diff<br>
> =======================================================================<br>
> =======<br>
> --- compiler-rt/trunk/lib/fuzzer/FuzzerIO.h (original)<br>
> +++ compiler-rt/trunk/lib/fuzzer/FuzzerIO.h Tue Jun 5 18:23:29 2018<br>
> @@ -67,6 +67,8 @@ struct SizedFile {<br>
> void GetSizedFilesFromDir(const std::string &Dir, Vector<SizedFile><br>
> *V);<br>
> <br>
> char GetSeparator();<br>
> +// Similar to the basename utility: returns the file name w/o the dir<br>
> prefix.<br>
> +std::string Basename(const std::string &Path, char Separator =<br>
> GetSeparator());<br>
> <br>
> FILE* OpenFile(int Fd, const char *Mode);<br>
> <br>
> <br>
> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-</a><br>
> rt/trunk/lib/fuzzer/FuzzerInternal.h?rev=334058&r1=334057&r2=334058&vie<br>
> w=diff<br>
> =======================================================================<br>
> =======<br>
> --- compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h (original)<br>
> +++ compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h Tue Jun 5 18:23:29<br>
> 2018<br>
> @@ -12,6 +12,7 @@<br>
> #ifndef LLVM_FUZZER_INTERNAL_H<br>
> #define LLVM_FUZZER_INTERNAL_H<br>
> <br>
> +#include "FuzzerDataFlowTrace.h"<br>
> #include "FuzzerDefs.h"<br>
> #include "FuzzerExtFunctions.h"<br>
> #include "FuzzerInterface.h"<br>
> @@ -134,6 +135,7 @@ private:<br>
> InputCorpus &Corpus;<br>
> MutationDispatcher &MD;<br>
> FuzzingOptions Options;<br>
> + DataFlowTrace DFT;<br>
> <br>
> system_clock::time_point ProcessStartTime = system_clock::now();<br>
> system_clock::time_point UnitStartTime, UnitStopTime;<br>
> <br>
> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-</a><br>
> rt/trunk/lib/fuzzer/FuzzerLoop.cpp?rev=334058&r1=334057&r2=334058&view=<br>
> diff<br>
> =======================================================================<br>
> =======<br>
> --- compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp (original)<br>
> +++ compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp Tue Jun 5 18:23:29<br>
> 2018<br>
> @@ -160,6 +160,7 @@ Fuzzer::Fuzzer(UserCallback CB, InputCor<br>
> CurrentUnitSize = 0;<br>
> memset(BaseSha1, 0, sizeof(BaseSha1));<br>
> TPC.SetFocusFunction(Options.FocusFunction);<br>
> + DFT.Init(Options.DataFlowTrace, Options.FocusFunction);<br>
> }<br>
> <br>
> Fuzzer::~Fuzzer() {}<br>
> <br>
> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-</a><br>
> rt/trunk/lib/fuzzer/FuzzerOptions.h?rev=334058&r1=334057&r2=334058&view<br>
> =diff<br>
> =======================================================================<br>
> =======<br>
> --- compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h (original)<br>
> +++ compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h Tue Jun 5 18:23:29<br>
> 2018<br>
> @@ -46,6 +46,7 @@ struct FuzzingOptions {<br>
> std::string ExitOnSrcPos;<br>
> std::string ExitOnItem;<br>
> std::string FocusFunction;<br>
> + std::string DataFlowTrace;<br>
> bool SaveArtifacts = true;<br>
> bool PrintNEW = true; // Print a status line when new units are<br>
> found;<br>
> bool PrintNewCovPcs = false;<br>
> <br>
> Modified: compiler-rt/trunk/lib/fuzzer/dataflow/DataFlow.cpp<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-</a><br>
> rt/trunk/lib/fuzzer/dataflow/DataFlow.cpp?rev=334058&r1=334057&r2=33405<br>
> 8&view=diff<br>
> =======================================================================<br>
> =======<br>
> --- compiler-rt/trunk/lib/fuzzer/dataflow/DataFlow.cpp (original)<br>
> +++ compiler-rt/trunk/lib/fuzzer/dataflow/DataFlow.cpp Tue Jun 5<br>
> 18:23:29 2018<br>
> @@ -69,6 +69,7 @@ static const uintptr_t *FuncsBeg;<br>
> static __thread size_t CurrentFunc;<br>
> static dfsan_label *FuncLabels; // Array of NumFuncs elements.<br>
> static char *PrintableStringForLabel; // InputLen + 2 bytes.<br>
> +static bool LabelSeen[1 << 8 * sizeof(dfsan_label)];<br>
> <br>
> // Prints all instrumented functions.<br>
> static int PrintFunctions() {<br>
> @@ -89,7 +90,11 @@ static int PrintFunctions() {<br>
> return 0;<br>
> }<br>
> <br>
> -static void SetBytesForLabel(dfsan_label L, char *Bytes) {<br>
> +extern "C"<br>
> +void SetBytesForLabel(dfsan_label L, char *Bytes) {<br>
> + if (LabelSeen[L])<br>
> + return;<br>
> + LabelSeen[L] = true;<br>
> assert(L);<br>
> if (L <= InputLen + 1) {<br>
> Bytes[L - 1] = '1';<br>
> @@ -103,6 +108,7 @@ static void SetBytesForLabel(dfsan_label<br>
> static char *GetPrintableStringForLabel(dfsan_label L) {<br>
> memset(PrintableStringForLabel, '0', InputLen + 1);<br>
> PrintableStringForLabel[InputLen + 1] = 0;<br>
> + memset(LabelSeen, 0, sizeof(LabelSeen));<br>
> SetBytesForLabel(L, PrintableStringForLabel);<br>
> return PrintableStringForLabel;<br>
> }<br>
> <br>
> Modified: compiler-rt/trunk/lib/fuzzer/scripts/collect_data_flow.py<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-</a><br>
> rt/trunk/lib/fuzzer/scripts/collect_data_flow.py?rev=334058&r1=334057&r<br>
> 2=334058&view=diff<br>
> =======================================================================<br>
> =======<br>
> --- compiler-rt/trunk/lib/fuzzer/scripts/collect_data_flow.py<br>
> (original)<br>
> +++ compiler-rt/trunk/lib/fuzzer/scripts/collect_data_flow.py Tue Jun<br>
> 5 18:23:29 2018<br>
> @@ -11,9 +11,15 @@<br>
> # the complete trace for all input bytes (running it on all bytes at<br>
> once<br>
> # may fail if DFSan runs out of labels).<br>
> # Usage:<br>
> -# collect_data_flow.py BINARY INPUT [RESULT]<br>
> +#<br>
> +# # Collect dataflow for one input, store it in OUTPUT (default is<br>
> stdout)<br>
> +# collect_data_flow.py BINARY INPUT [OUTPUT]<br>
> +#<br>
> +# # Collect dataflow for all inputs in CORPUS_DIR, store them in<br>
> OUTPUT_DIR<br>
> +# collect_data_flow.py BINARY CORPUS_DIR OUTPUT_DIR<br>
> #===------------------------------------------------------------------<br>
> ------===#<br>
> import atexit<br>
> +import hashlib<br>
> import sys<br>
> import os<br>
> import subprocess<br>
> @@ -26,9 +32,26 @@ def cleanup(d):<br>
> print "removing: ", d<br>
> shutil.rmtree(d)<br>
> <br>
> +def collect_dataflow_for_corpus(self, exe, corpus_dir, output_dir):<br>
> + print "Collecting dataflow for corpus:", corpus_dir, \<br>
> + "output_dir:", output_dir<br>
> + assert not os.path.exists(output_dir)<br>
> + os.mkdir(output_dir)<br>
> + for root, dirs, files in os.walk(corpus_dir):<br>
> + for f in files:<br>
> + path = os.path.join(root, f)<br>
> + sha1 = hashlib.sha1(open(path).read()).hexdigest()<br>
> + output = os.path.join(output_dir, sha1)<br>
> + subprocess.call([self, exe, path, output])<br>
> + functions_txt = open(os.path.join(output_dir, "functions.txt"), "w")<br>
> + subprocess.call([exe], stdout=functions_txt)<br>
> +<br>
> +<br>
> def main(argv):<br>
> exe = argv[1]<br>
> inp = argv[2]<br>
> + if os.path.isdir(inp):<br>
> + return collect_dataflow_for_corpus(argv[0], exe, inp, argv[3])<br>
> size = os.path.getsize(inp)<br>
> q = [[0, size]]<br>
> tmpdir = tempfile.mkdtemp(prefix="libfuzzer-tmp-")<br>
> <br>
> Modified: compiler-rt/trunk/lib/fuzzer/tests/FuzzerUnittest.cpp<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-</a><br>
> rt/trunk/lib/fuzzer/tests/FuzzerUnittest.cpp?rev=334058&r1=334057&r2=33<br>
> 4058&view=diff<br>
> =======================================================================<br>
> =======<br>
> --- compiler-rt/trunk/lib/fuzzer/tests/FuzzerUnittest.cpp (original)<br>
> +++ compiler-rt/trunk/lib/fuzzer/tests/FuzzerUnittest.cpp Tue Jun 5<br>
> 18:23:29 2018<br>
> @@ -28,6 +28,14 @@ extern "C" int LLVMFuzzerTestOneInput(co<br>
> abort();<br>
> }<br>
> <br>
> +TEST(Fuzzer, Basename) {<br>
> + EXPECT_EQ(Basename("foo/bar"), "bar");<br>
> + EXPECT_EQ(Basename("bar"), "bar");<br>
> + EXPECT_EQ(Basename("/bar"), "bar");<br>
> + EXPECT_EQ(Basename("foo/x"), "x");<br>
> + EXPECT_EQ(Basename("foo/"), "");<br>
> +}<br>
> +<br>
> TEST(Fuzzer, CrossOver) {<br>
> std::unique_ptr<ExternalFunctions> t(new ExternalFunctions());<br>
> fuzzer::EF = t.get();<br>
> <br>
> Modified: compiler-rt/trunk/test/fuzzer/ThreeFunctionsTest.cpp<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-</a><br>
> rt/trunk/test/fuzzer/ThreeFunctionsTest.cpp?rev=334058&r1=334057&r2=334<br>
> 058&view=diff<br>
> =======================================================================<br>
> =======<br>
> --- compiler-rt/trunk/test/fuzzer/ThreeFunctionsTest.cpp (original)<br>
> +++ compiler-rt/trunk/test/fuzzer/ThreeFunctionsTest.cpp Tue Jun 5<br>
> 18:23:29 2018<br>
> @@ -8,12 +8,14 @@<br>
> #include <cstdlib><br>
> #include <cstdio><br>
> <br>
> +extern "C"<br>
> __attribute__((noinline))<br>
> -static bool Func1(const uint8_t *Data, size_t Size) {<br>
> +bool Func1(const uint8_t *Data, size_t Size) {<br>
> // assumes Size >= 5, doesn't check it.<br>
> return Data[4] == 'M';<br>
> }<br>
> <br>
> +extern "C"<br>
> __attribute__((noinline))<br>
> bool Func2(const uint8_t *Data, size_t Size) {<br>
> return Size >= 6 && Data[5] == 'E';<br>
> <br>
> Modified: compiler-rt/trunk/test/fuzzer/dataflow.test<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-</a><br>
> rt/trunk/test/fuzzer/dataflow.test?rev=334058&r1=334057&r2=334058&view=<br>
> diff<br>
> =======================================================================<br>
> =======<br>
> --- compiler-rt/trunk/test/fuzzer/dataflow.test (original)<br>
> +++ compiler-rt/trunk/test/fuzzer/dataflow.test Tue Jun 5 18:23:29<br>
> 2018<br>
> @@ -5,6 +5,7 @@ REQUIRES: linux<br>
> RUN: %no_fuzzer_cpp_compiler -c -fno-sanitize=all -fsanitize=dataflow<br>
> %S/../../lib/fuzzer/dataflow/DataFlow.cpp -o %t-DataFlow.o<br>
> RUN: %no_fuzzer_cpp_compiler -fno-sanitize=all -fsanitize=dataflow<br>
> -fsanitize-coverage=trace-pc-guard,pc-table,func,trace-cmp<br>
> %S/ThreeFunctionsTest.cpp %t-DataFlow.o -o %t-ThreeFunctionsTestDF<br>
> RUN: %no_fuzzer_cpp_compiler -fno-sanitize=all -fsanitize=dataflow<br>
> -fsanitize-coverage=trace-pc-guard,pc-table,func,trace-cmp<br>
> %S/ExplodeDFSanLabelsTest.cpp %t-DataFlow.o -o %t-<br>
> ExplodeDFSanLabelsTestDF<br>
> +RUN: %cpp_compiler %S/ThreeFunctionsTest.cpp -o %t-ThreeFunctionsTest<br>
> <br>
> # Dump the function list.<br>
> RUN: %t-ThreeFunctionsTestDF 2>&1 | FileCheck %s --check-<br>
> prefix=FUNC_LIST<br>
> @@ -70,3 +71,13 @@ RUN: %t-ExplodeDFSanLabelsTestDF 2 4 %t<br>
> RUN: %t-ExplodeDFSanLabelsTestDF 4 6 %t/IN/1234567890123456<br>
> # Or we can use collect_data_flow<br>
> RUN: %libfuzzer_src/scripts/collect_data_flow.py %t-<br>
> ExplodeDFSanLabelsTestDF %t/IN/1234567890123456<br>
> +<br>
> +# Test that we can run collect_data_flow on the entire corpus dir<br>
> +RUN: rm -rf %t/OUT<br>
> +RUN: %libfuzzer_src/scripts/collect_data_flow.py %t-<br>
> ThreeFunctionsTestDF %t/IN %t/OUT<br>
> +RUN: %t-ThreeFunctionsTest -data_flow_trace=%t/OUT -runs=0 -<br>
> focus_function=Func2 2>&1 | FileCheck %s --check-<br>
> prefix=USE_DATA_FLOW_TRACE<br>
> +USE_DATA_FLOW_TRACE: INFO: Focus function is set to 'Func2'<br>
> +USE_DATA_FLOW_TRACE: INFO: DataFlowTrace: reading from {{.*}}/OUT<br>
> +USE_DATA_FLOW_TRACE-DAG: a8eefe2fd5d6b32028f355fafa3e739a6bf5edc =><br>
> |000001|<br>
> +USE_DATA_FLOW_TRACE-DGA: d28cb407e8e1a702c72d25473f0553d3ec172262 =><br>
> |0000011|<br>
> +USE_DATA_FLOW_TRACE: INFO: DataFlowTrace: 6 trace files, 3 functions,<br>
> 2 traces with focus function<br>
> <br>
> <br>
> _______________________________________________<br>
> llvm-commits mailing list<br>
> <a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a><br>
> <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits</a><br>
</blockquote></div>