<div dir="ltr">Yes, definitely! I'm passing the feedback to Kevin (CC'ed). Thanks for the suggestion!<div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Jul 16, 2018 at 12:47 PM Kostya Serebryany <<a href="mailto:kcc@google.com" target="_blank">kcc@google.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">may I ask for a follow up refactoring (no functional change)? <div>The code already had quite a bit of boiler plate to iterate over the 8bit counters in two nested loops, </div><div>now it has two more instances of that -- bad. </div><div><br></div><div>I would ask to change the code so that it has a single template function to visit the counters,</div><div>and several calls to it with different (labda) callbacks. </div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Jul 16, 2018 at 9:06 AM Max Moroz via llvm-commits <<a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Author: dor1s<br>
Date: Mon Jul 16 09:01:31 2018<br>
New Revision: 337187<br>
<br>
URL: <a href="http://llvm.org/viewvc/llvm-project?rev=337187&view=rev" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project?rev=337187&view=rev</a><br>
Log:<br>
[libFuzzer] Implement stat::stability_rate based on the percentage of unstable edges.<br>
<br>
Summary:<br>
Created a -print_unstable_stats flag.<br>
When -print_unstable_stats=1, we run it 2 more times on interesting inputs poisoning unstable edges in an array.<br>
On program termination, we run PrintUnstableStats() which will print a line with a stability percentage like AFL does.<br>
<br>
Patch by Kyungtak Woo (@kevinwkt).<br>
<br>
Reviewers: metzman, Dor1s, kcc, morehouse<br>
<br>
Reviewed By: metzman, Dor1s, morehouse<br>
<br>
Subscribers: delcypher, llvm-commits, #sanitizers, kcc, morehouse, Dor1s<br>
<br>
Differential Revision: <a href="https://reviews.llvm.org/D49212" rel="noreferrer" target="_blank">https://reviews.llvm.org/D49212</a><br>
<br>
Added:<br>
compiler-rt/trunk/test/fuzzer/PrintUnstableStatsTest.cpp<br>
compiler-rt/trunk/test/fuzzer/print_unstable_stats.test<br>
Modified:<br>
compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp<br>
compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def<br>
compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h<br>
compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp<br>
compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h<br>
compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.cpp<br>
compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.h<br>
<br>
Modified: compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp?rev=337187&r1=337186&r2=337187&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp?rev=337187&r1=337186&r2=337187&view=diff</a><br>
==============================================================================<br>
--- compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp (original)<br>
+++ compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp Mon Jul 16 09:01:31 2018<br>
@@ -617,6 +617,7 @@ int FuzzerDriver(int *argc, char ***argv<br>
Options.PrintFinalStats = Flags.print_final_stats;<br>
Options.PrintCorpusStats = Flags.print_corpus_stats;<br>
Options.PrintCoverage = Flags.print_coverage;<br>
+ Options.PrintUnstableStats = Flags.print_unstable_stats;<br>
Options.DumpCoverage = Flags.dump_coverage;<br>
if (Flags.exit_on_src_pos)<br>
Options.ExitOnSrcPos = Flags.exit_on_src_pos;<br>
<br>
Modified: compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def?rev=337187&r1=337186&r2=337187&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def?rev=337187&r1=337186&r2=337187&view=diff</a><br>
==============================================================================<br>
--- compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def (original)<br>
+++ compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def Mon Jul 16 09:01:31 2018<br>
@@ -110,6 +110,8 @@ FUZZER_FLAG_INT(print_coverage, 0, "If 1<br>
FUZZER_FLAG_INT(dump_coverage, 0, "Deprecated."<br>
" If 1, dump coverage information as a"<br>
" .sancov file at exit.")<br>
+FUZZER_FLAG_INT(print_unstable_stats, 0, "Experimental."<br>
+ " If 1, print unstable statistics at exit.")<br>
FUZZER_FLAG_INT(handle_segv, 1, "If 1, try to intercept SIGSEGV.")<br>
FUZZER_FLAG_INT(handle_bus, 1, "If 1, try to intercept SIGBUS.")<br>
FUZZER_FLAG_INT(handle_abrt, 1, "If 1, try to intercept SIGABRT.")<br>
<br>
Modified: compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h?rev=337187&r1=337186&r2=337187&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h?rev=337187&r1=337186&r2=337187&view=diff</a><br>
==============================================================================<br>
--- compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h (original)<br>
+++ compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h Mon Jul 16 09:01:31 2018<br>
@@ -67,6 +67,7 @@ public:<br>
static void StaticGracefulExitCallback();<br>
<br>
void ExecuteCallback(const uint8_t *Data, size_t Size);<br>
+ void CheckForUnstableCounters(const uint8_t *Data, size_t Size);<br>
bool RunOne(const uint8_t *Data, size_t Size, bool MayDeleteFile = false,<br>
InputInfo *II = nullptr, bool *FoundUniqFeatures = nullptr);<br>
<br>
<br>
Modified: compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp?rev=337187&r1=337186&r2=337187&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp?rev=337187&r1=337186&r2=337187&view=diff</a><br>
==============================================================================<br>
--- compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp (original)<br>
+++ compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp Mon Jul 16 09:01:31 2018<br>
@@ -352,6 +352,8 @@ void Fuzzer::PrintStats(const char *Wher<br>
void Fuzzer::PrintFinalStats() {<br>
if (Options.PrintCoverage)<br>
TPC.PrintCoverage();<br>
+ if (Options.PrintUnstableStats)<br>
+ TPC.PrintUnstableStats();<br>
if (Options.DumpCoverage)<br>
TPC.DumpCoverage();<br>
if (Options.PrintCorpusStats)<br>
@@ -444,6 +446,29 @@ void Fuzzer::PrintPulseAndReportSlowInpu<br>
}<br>
}<br>
<br>
+void Fuzzer::CheckForUnstableCounters(const uint8_t *Data, size_t Size) {<br>
+ auto CBSetupAndRun = [&]() {<br>
+ ScopedEnableMsanInterceptorChecks S;<br>
+ UnitStartTime = system_clock::now();<br>
+ TPC.ResetMaps();<br>
+ RunningCB = true;<br>
+ CB(Data, Size);<br>
+ RunningCB = false;<br>
+ UnitStopTime = system_clock::now();<br>
+ };<br>
+<br>
+ // Copy original run counters into our unstable counters<br>
+ TPC.InitializeUnstableCounters();<br>
+<br>
+ // First Rerun<br>
+ CBSetupAndRun();<br>
+ TPC.UpdateUnstableCounters();<br>
+<br>
+ // Second Rerun<br>
+ CBSetupAndRun();<br>
+ TPC.UpdateUnstableCounters();<br>
+}<br>
+<br>
bool Fuzzer::RunOne(const uint8_t *Data, size_t Size, bool MayDeleteFile,<br>
InputInfo *II, bool *FoundUniqFeatures) {<br>
if (!Size)<br>
@@ -466,6 +491,12 @@ bool Fuzzer::RunOne(const uint8_t *Data,<br>
*FoundUniqFeatures = FoundUniqFeaturesOfII;<br>
PrintPulseAndReportSlowInput(Data, Size);<br>
size_t NumNewFeatures = Corpus.NumFeatureUpdates() - NumUpdatesBefore;<br>
+<br>
+ // If print_unstable_stats, execute the same input two more times to detect<br>
+ // unstable edges.<br>
+ if (NumNewFeatures && Options.PrintUnstableStats)<br>
+ CheckForUnstableCounters(Data, Size);<br>
+<br>
if (NumNewFeatures) {<br>
TPC.UpdateObservedPCs();<br>
Corpus.AddToCorpus({Data, Data + Size}, NumNewFeatures, MayDeleteFile,<br>
<br>
Modified: compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h?rev=337187&r1=337186&r2=337187&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h?rev=337187&r1=337186&r2=337187&view=diff</a><br>
==============================================================================<br>
--- compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h (original)<br>
+++ compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h Mon Jul 16 09:01:31 2018<br>
@@ -54,6 +54,7 @@ struct FuzzingOptions {<br>
bool PrintFinalStats = false;<br>
bool PrintCorpusStats = false;<br>
bool PrintCoverage = false;<br>
+ bool PrintUnstableStats = false;<br>
bool DumpCoverage = false;<br>
bool DetectLeaks = true;<br>
int PurgeAllocatorIntervalSec = 1;<br>
<br>
Modified: compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.cpp?rev=337187&r1=337186&r2=337187&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.cpp?rev=337187&r1=337186&r2=337187&view=diff</a><br>
==============================================================================<br>
--- compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.cpp (original)<br>
+++ compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.cpp Mon Jul 16 09:01:31 2018<br>
@@ -59,6 +59,37 @@ size_t TracePC::GetTotalPCCoverage() {<br>
return Res;<br>
}<br>
<br>
+// Initializes unstable counters by copying Inline8bitCounters to unstable<br>
+// counters.<br>
+void TracePC::InitializeUnstableCounters() {<br>
+ if (NumInline8bitCounters && NumInline8bitCounters == NumPCsInPCTables) {<br>
+ size_t UnstableIdx = 0;<br>
+ for (size_t i = 0; i < NumModulesWithInline8bitCounters; i++) {<br>
+ uint8_t *Beg = ModuleCounters[i].Start;<br>
+ size_t Size = ModuleCounters[i].Stop - Beg;<br>
+ assert(Size == (size_t)(ModulePCTable[i].Stop - ModulePCTable[i].Start));<br>
+ for (size_t j = 0; j < Size; j++, UnstableIdx++)<br>
+ if (UnstableCounters[UnstableIdx] != kUnstableCounter)<br>
+ UnstableCounters[UnstableIdx] = Beg[j];<br>
+ }<br>
+ }<br>
+}<br>
+<br>
+// Compares the current counters with counters from previous runs<br>
+// and records differences as unstable edges.<br>
+void TracePC::UpdateUnstableCounters() {<br>
+ if (NumInline8bitCounters && NumInline8bitCounters == NumPCsInPCTables) {<br>
+ size_t UnstableIdx = 0;<br>
+ for (size_t i = 0; i < NumModulesWithInline8bitCounters; i++) {<br>
+ uint8_t *Beg = ModuleCounters[i].Start;<br>
+ size_t Size = ModuleCounters[i].Stop - Beg;<br>
+ assert(Size == (size_t)(ModulePCTable[i].Stop - ModulePCTable[i].Start));<br>
+ for (size_t j = 0; j < Size; j++, UnstableIdx++)<br>
+ if (Beg[j] != UnstableCounters[UnstableIdx])<br>
+ UnstableCounters[UnstableIdx] = kUnstableCounter;<br>
+ }<br>
+ }<br>
+}<br>
<br>
void TracePC::HandleInline8bitCountersInit(uint8_t *Start, uint8_t *Stop) {<br>
if (Start == Stop) return;<br>
@@ -310,6 +341,15 @@ void TracePC::DumpCoverage() {<br>
}<br>
}<br>
<br>
+void TracePC::PrintUnstableStats() {<br>
+ size_t count = 0;<br>
+ for (size_t i = 0; i < NumInline8bitCounters; i++)<br>
+ if (UnstableCounters[i] == kUnstableCounter)<br>
+ count++;<br>
+ Printf("stat::stability_rate: %.2f\n",<br>
+ 100 - static_cast<float>(count * 100) / NumInline8bitCounters);<br>
+}<br>
+<br>
// Value profile.<br>
// We keep track of various values that affect control flow.<br>
// These values are inserted into a bit-set-based hash map.<br>
<br>
Modified: compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.h?rev=337187&r1=337186&r2=337187&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.h?rev=337187&r1=337186&r2=337187&view=diff</a><br>
==============================================================================<br>
--- compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.h (original)<br>
+++ compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.h Mon Jul 16 09:01:31 2018<br>
@@ -103,6 +103,7 @@ class TracePC {<br>
<br>
void PrintCoverage();<br>
void DumpCoverage();<br>
+ void PrintUnstableStats();<br>
<br>
template<class CallBack><br>
void IterateCoveredFunctions(CallBack CB);<br>
@@ -135,7 +136,17 @@ class TracePC {<br>
void SetFocusFunction(const std::string &FuncName);<br>
bool ObservedFocusFunction();<br>
<br>
+ void InitializeUnstableCounters();<br>
+ void UpdateUnstableCounters();<br>
+<br>
private:<br>
+ // Value used to represent unstable edge.<br>
+ static constexpr int16_t kUnstableCounter = -1;<br>
+<br>
+ // Uses 16-bit signed type to be able to accommodate any possible value from<br>
+ // uint8_t counter and -1 constant as well.<br>
+ int16_t UnstableCounters[kNumPCs];<br>
+<br>
bool UseCounters = false;<br>
uint32_t UseValueProfileMask = false;<br>
bool DoPrintNewPCs = false;<br>
<br>
Added: compiler-rt/trunk/test/fuzzer/PrintUnstableStatsTest.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/test/fuzzer/PrintUnstableStatsTest.cpp?rev=337187&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/test/fuzzer/PrintUnstableStatsTest.cpp?rev=337187&view=auto</a><br>
==============================================================================<br>
--- compiler-rt/trunk/test/fuzzer/PrintUnstableStatsTest.cpp (added)<br>
+++ compiler-rt/trunk/test/fuzzer/PrintUnstableStatsTest.cpp Mon Jul 16 09:01:31 2018<br>
@@ -0,0 +1,69 @@<br>
+#include <assert.h><br>
+#include <cstdint><br>
+#include <cstdio><br>
+#include <cstdlib><br>
+<br>
+int x = 0;<br>
+bool skip0 = false;<br>
+bool skip1 = false;<br>
+bool skip2 = false;<br>
+<br>
+__attribute__((noinline)) void det0() { x++; }<br>
+__attribute__((noinline)) void det1() { x++; }<br>
+__attribute__((noinline)) void det2() { x++; }<br>
+__attribute__((noinline)) void det3() { x++; }<br>
+__attribute__((noinline)) void det4() { x++; }<br>
+<br>
+__attribute__((noinline)) void ini0() { x++; }<br>
+__attribute__((noinline)) void ini1() { x++; }<br>
+__attribute__((noinline)) void ini2() { x++; }<br>
+<br>
+__attribute__((noinline)) void t0() { x++; }<br>
+__attribute__((noinline)) void t1() { x++; }<br>
+__attribute__((noinline)) void t2() { x++; }<br>
+__attribute__((noinline)) void t3() { x++; }<br>
+__attribute__((noinline)) void t4() { x++; }<br>
+<br>
+extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {<br>
+ if (Size == 1 && Data[0] == 'A' && !skip0) {<br>
+ skip0 = true;<br>
+ ini0();<br>
+ }<br>
+ if (Size == 1 && Data[0] == 'B' && !skip1) {<br>
+ skip1 = true;<br>
+ ini1();<br>
+ }<br>
+ if (Size == 1 && Data[0] == 'C' && !skip2) {<br>
+ skip2 = true;<br>
+ ini2();<br>
+ }<br>
+<br>
+ det0();<br>
+ det1();<br>
+ int a = rand();<br>
+ det2();<br>
+<br>
+ switch (a % 5) {<br>
+ case 0:<br>
+ t0();<br>
+ break;<br>
+ case 1:<br>
+ t1();<br>
+ break;<br>
+ case 2:<br>
+ t2();<br>
+ break;<br>
+ case 3:<br>
+ t3();<br>
+ break;<br>
+ case 4:<br>
+ t4();<br>
+ break;<br>
+ default:<br>
+ assert(false);<br>
+ }<br>
+<br>
+ det3();<br>
+ det4();<br>
+ return 0;<br>
+}<br>
<br>
Added: compiler-rt/trunk/test/fuzzer/print_unstable_stats.test<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/test/fuzzer/print_unstable_stats.test?rev=337187&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/test/fuzzer/print_unstable_stats.test?rev=337187&view=auto</a><br>
==============================================================================<br>
--- compiler-rt/trunk/test/fuzzer/print_unstable_stats.test (added)<br>
+++ compiler-rt/trunk/test/fuzzer/print_unstable_stats.test Mon Jul 16 09:01:31 2018<br>
@@ -0,0 +1,3 @@<br>
+RUN: %cpp_compiler %S/PrintUnstableStatsTest.cpp -o %t-PrintUnstableStatsTest<br>
+RUN: %run %t-PrintUnstableStatsTest -print_unstable_stats=1 -runs=100000 2>&1 | FileCheck %s --check-prefix=LONG<br>
+LONG: stat::stability_rate: 27.59<br>
<br>
<br>
_______________________________________________<br>
llvm-commits mailing list<br>
<a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits</a><br>
</blockquote></div>
</blockquote></div>