[compiler-rt] r337187 - [libFuzzer] Implement stat::stability_rate based on the percentage of unstable edges.
Max Moroz via llvm-commits
llvm-commits at lists.llvm.org
Mon Jul 16 12:57:19 PDT 2018
Yes, definitely! I'm passing the feedback to Kevin (CC'ed). Thanks for the
suggestion!
On Mon, Jul 16, 2018 at 12:47 PM Kostya Serebryany <kcc at google.com> wrote:
> may I ask for a follow up refactoring (no functional change)?
> The code already had quite a bit of boiler plate to iterate over the 8bit
> counters in two nested loops,
> now it has two more instances of that -- bad.
>
> I would ask to change the code so that it has a single template function
> to visit the counters,
> and several calls to it with different (labda) callbacks.
>
> On Mon, Jul 16, 2018 at 9:06 AM Max Moroz via llvm-commits <
> llvm-commits at lists.llvm.org> wrote:
>
>> Author: dor1s
>> Date: Mon Jul 16 09:01:31 2018
>> New Revision: 337187
>>
>> URL: http://llvm.org/viewvc/llvm-project?rev=337187&view=rev
>> Log:
>> [libFuzzer] Implement stat::stability_rate based on the percentage of
>> unstable edges.
>>
>> Summary:
>> Created a -print_unstable_stats flag.
>> When -print_unstable_stats=1, we run it 2 more times on interesting
>> inputs poisoning unstable edges in an array.
>> On program termination, we run PrintUnstableStats() which will print a
>> line with a stability percentage like AFL does.
>>
>> Patch by Kyungtak Woo (@kevinwkt).
>>
>> Reviewers: metzman, Dor1s, kcc, morehouse
>>
>> Reviewed By: metzman, Dor1s, morehouse
>>
>> Subscribers: delcypher, llvm-commits, #sanitizers, kcc, morehouse, Dor1s
>>
>> Differential Revision: https://reviews.llvm.org/D49212
>>
>> Added:
>> compiler-rt/trunk/test/fuzzer/PrintUnstableStatsTest.cpp
>> compiler-rt/trunk/test/fuzzer/print_unstable_stats.test
>> Modified:
>> compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp
>> compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def
>> compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h
>> compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp
>> compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h
>> compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.cpp
>> compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.h
>>
>> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp
>> URL:
>> http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp?rev=337187&r1=337186&r2=337187&view=diff
>>
>> ==============================================================================
>> --- compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp (original)
>> +++ compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp Mon Jul 16 09:01:31 2018
>> @@ -617,6 +617,7 @@ int FuzzerDriver(int *argc, char ***argv
>> Options.PrintFinalStats = Flags.print_final_stats;
>> Options.PrintCorpusStats = Flags.print_corpus_stats;
>> Options.PrintCoverage = Flags.print_coverage;
>> + Options.PrintUnstableStats = Flags.print_unstable_stats;
>> Options.DumpCoverage = Flags.dump_coverage;
>> if (Flags.exit_on_src_pos)
>> Options.ExitOnSrcPos = Flags.exit_on_src_pos;
>>
>> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def
>> URL:
>> http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def?rev=337187&r1=337186&r2=337187&view=diff
>>
>> ==============================================================================
>> --- compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def (original)
>> +++ compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def Mon Jul 16 09:01:31 2018
>> @@ -110,6 +110,8 @@ FUZZER_FLAG_INT(print_coverage, 0, "If 1
>> FUZZER_FLAG_INT(dump_coverage, 0, "Deprecated."
>> " If 1, dump coverage information as a"
>> " .sancov file at exit.")
>> +FUZZER_FLAG_INT(print_unstable_stats, 0, "Experimental."
>> + " If 1, print unstable statistics at
>> exit.")
>> FUZZER_FLAG_INT(handle_segv, 1, "If 1, try to intercept SIGSEGV.")
>> FUZZER_FLAG_INT(handle_bus, 1, "If 1, try to intercept SIGBUS.")
>> FUZZER_FLAG_INT(handle_abrt, 1, "If 1, try to intercept SIGABRT.")
>>
>> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h
>> URL:
>> http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h?rev=337187&r1=337186&r2=337187&view=diff
>>
>> ==============================================================================
>> --- compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h (original)
>> +++ compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h Mon Jul 16 09:01:31 2018
>> @@ -67,6 +67,7 @@ public:
>> static void StaticGracefulExitCallback();
>>
>> void ExecuteCallback(const uint8_t *Data, size_t Size);
>> + void CheckForUnstableCounters(const uint8_t *Data, size_t Size);
>> bool RunOne(const uint8_t *Data, size_t Size, bool MayDeleteFile =
>> false,
>> InputInfo *II = nullptr, bool *FoundUniqFeatures =
>> nullptr);
>>
>>
>> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp
>> URL:
>> http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp?rev=337187&r1=337186&r2=337187&view=diff
>>
>> ==============================================================================
>> --- compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp (original)
>> +++ compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp Mon Jul 16 09:01:31 2018
>> @@ -352,6 +352,8 @@ void Fuzzer::PrintStats(const char *Wher
>> void Fuzzer::PrintFinalStats() {
>> if (Options.PrintCoverage)
>> TPC.PrintCoverage();
>> + if (Options.PrintUnstableStats)
>> + TPC.PrintUnstableStats();
>> if (Options.DumpCoverage)
>> TPC.DumpCoverage();
>> if (Options.PrintCorpusStats)
>> @@ -444,6 +446,29 @@ void Fuzzer::PrintPulseAndReportSlowInpu
>> }
>> }
>>
>> +void Fuzzer::CheckForUnstableCounters(const uint8_t *Data, size_t Size) {
>> + auto CBSetupAndRun = [&]() {
>> + ScopedEnableMsanInterceptorChecks S;
>> + UnitStartTime = system_clock::now();
>> + TPC.ResetMaps();
>> + RunningCB = true;
>> + CB(Data, Size);
>> + RunningCB = false;
>> + UnitStopTime = system_clock::now();
>> + };
>> +
>> + // Copy original run counters into our unstable counters
>> + TPC.InitializeUnstableCounters();
>> +
>> + // First Rerun
>> + CBSetupAndRun();
>> + TPC.UpdateUnstableCounters();
>> +
>> + // Second Rerun
>> + CBSetupAndRun();
>> + TPC.UpdateUnstableCounters();
>> +}
>> +
>> bool Fuzzer::RunOne(const uint8_t *Data, size_t Size, bool MayDeleteFile,
>> InputInfo *II, bool *FoundUniqFeatures) {
>> if (!Size)
>> @@ -466,6 +491,12 @@ bool Fuzzer::RunOne(const uint8_t *Data,
>> *FoundUniqFeatures = FoundUniqFeaturesOfII;
>> PrintPulseAndReportSlowInput(Data, Size);
>> size_t NumNewFeatures = Corpus.NumFeatureUpdates() - NumUpdatesBefore;
>> +
>> + // If print_unstable_stats, execute the same input two more times to
>> detect
>> + // unstable edges.
>> + if (NumNewFeatures && Options.PrintUnstableStats)
>> + CheckForUnstableCounters(Data, Size);
>> +
>> if (NumNewFeatures) {
>> TPC.UpdateObservedPCs();
>> Corpus.AddToCorpus({Data, Data + Size}, NumNewFeatures,
>> MayDeleteFile,
>>
>> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h
>> URL:
>> http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h?rev=337187&r1=337186&r2=337187&view=diff
>>
>> ==============================================================================
>> --- compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h (original)
>> +++ compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h Mon Jul 16 09:01:31 2018
>> @@ -54,6 +54,7 @@ struct FuzzingOptions {
>> bool PrintFinalStats = false;
>> bool PrintCorpusStats = false;
>> bool PrintCoverage = false;
>> + bool PrintUnstableStats = false;
>> bool DumpCoverage = false;
>> bool DetectLeaks = true;
>> int PurgeAllocatorIntervalSec = 1;
>>
>> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.cpp
>> URL:
>> http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.cpp?rev=337187&r1=337186&r2=337187&view=diff
>>
>> ==============================================================================
>> --- compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.cpp (original)
>> +++ compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.cpp Mon Jul 16 09:01:31
>> 2018
>> @@ -59,6 +59,37 @@ size_t TracePC::GetTotalPCCoverage() {
>> return Res;
>> }
>>
>> +// Initializes unstable counters by copying Inline8bitCounters to
>> unstable
>> +// counters.
>> +void TracePC::InitializeUnstableCounters() {
>> + if (NumInline8bitCounters && NumInline8bitCounters ==
>> NumPCsInPCTables) {
>> + size_t UnstableIdx = 0;
>> + for (size_t i = 0; i < NumModulesWithInline8bitCounters; i++) {
>> + uint8_t *Beg = ModuleCounters[i].Start;
>> + size_t Size = ModuleCounters[i].Stop - Beg;
>> + assert(Size == (size_t)(ModulePCTable[i].Stop -
>> ModulePCTable[i].Start));
>> + for (size_t j = 0; j < Size; j++, UnstableIdx++)
>> + if (UnstableCounters[UnstableIdx] != kUnstableCounter)
>> + UnstableCounters[UnstableIdx] = Beg[j];
>> + }
>> + }
>> +}
>> +
>> +// Compares the current counters with counters from previous runs
>> +// and records differences as unstable edges.
>> +void TracePC::UpdateUnstableCounters() {
>> + if (NumInline8bitCounters && NumInline8bitCounters ==
>> NumPCsInPCTables) {
>> + size_t UnstableIdx = 0;
>> + for (size_t i = 0; i < NumModulesWithInline8bitCounters; i++) {
>> + uint8_t *Beg = ModuleCounters[i].Start;
>> + size_t Size = ModuleCounters[i].Stop - Beg;
>> + assert(Size == (size_t)(ModulePCTable[i].Stop -
>> ModulePCTable[i].Start));
>> + for (size_t j = 0; j < Size; j++, UnstableIdx++)
>> + if (Beg[j] != UnstableCounters[UnstableIdx])
>> + UnstableCounters[UnstableIdx] = kUnstableCounter;
>> + }
>> + }
>> +}
>>
>> void TracePC::HandleInline8bitCountersInit(uint8_t *Start, uint8_t
>> *Stop) {
>> if (Start == Stop) return;
>> @@ -310,6 +341,15 @@ void TracePC::DumpCoverage() {
>> }
>> }
>>
>> +void TracePC::PrintUnstableStats() {
>> + size_t count = 0;
>> + for (size_t i = 0; i < NumInline8bitCounters; i++)
>> + if (UnstableCounters[i] == kUnstableCounter)
>> + count++;
>> + Printf("stat::stability_rate: %.2f\n",
>> + 100 - static_cast<float>(count * 100) / NumInline8bitCounters);
>> +}
>> +
>> // Value profile.
>> // We keep track of various values that affect control flow.
>> // These values are inserted into a bit-set-based hash map.
>>
>> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.h
>> URL:
>> http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.h?rev=337187&r1=337186&r2=337187&view=diff
>>
>> ==============================================================================
>> --- compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.h (original)
>> +++ compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.h Mon Jul 16 09:01:31 2018
>> @@ -103,6 +103,7 @@ class TracePC {
>>
>> void PrintCoverage();
>> void DumpCoverage();
>> + void PrintUnstableStats();
>>
>> template<class CallBack>
>> void IterateCoveredFunctions(CallBack CB);
>> @@ -135,7 +136,17 @@ class TracePC {
>> void SetFocusFunction(const std::string &FuncName);
>> bool ObservedFocusFunction();
>>
>> + void InitializeUnstableCounters();
>> + void UpdateUnstableCounters();
>> +
>> private:
>> + // Value used to represent unstable edge.
>> + static constexpr int16_t kUnstableCounter = -1;
>> +
>> + // Uses 16-bit signed type to be able to accommodate any possible
>> value from
>> + // uint8_t counter and -1 constant as well.
>> + int16_t UnstableCounters[kNumPCs];
>> +
>> bool UseCounters = false;
>> uint32_t UseValueProfileMask = false;
>> bool DoPrintNewPCs = false;
>>
>> Added: compiler-rt/trunk/test/fuzzer/PrintUnstableStatsTest.cpp
>> URL:
>> http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/test/fuzzer/PrintUnstableStatsTest.cpp?rev=337187&view=auto
>>
>> ==============================================================================
>> --- compiler-rt/trunk/test/fuzzer/PrintUnstableStatsTest.cpp (added)
>> +++ compiler-rt/trunk/test/fuzzer/PrintUnstableStatsTest.cpp Mon Jul 16
>> 09:01:31 2018
>> @@ -0,0 +1,69 @@
>> +#include <assert.h>
>> +#include <cstdint>
>> +#include <cstdio>
>> +#include <cstdlib>
>> +
>> +int x = 0;
>> +bool skip0 = false;
>> +bool skip1 = false;
>> +bool skip2 = false;
>> +
>> +__attribute__((noinline)) void det0() { x++; }
>> +__attribute__((noinline)) void det1() { x++; }
>> +__attribute__((noinline)) void det2() { x++; }
>> +__attribute__((noinline)) void det3() { x++; }
>> +__attribute__((noinline)) void det4() { x++; }
>> +
>> +__attribute__((noinline)) void ini0() { x++; }
>> +__attribute__((noinline)) void ini1() { x++; }
>> +__attribute__((noinline)) void ini2() { x++; }
>> +
>> +__attribute__((noinline)) void t0() { x++; }
>> +__attribute__((noinline)) void t1() { x++; }
>> +__attribute__((noinline)) void t2() { x++; }
>> +__attribute__((noinline)) void t3() { x++; }
>> +__attribute__((noinline)) void t4() { x++; }
>> +
>> +extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
>> + if (Size == 1 && Data[0] == 'A' && !skip0) {
>> + skip0 = true;
>> + ini0();
>> + }
>> + if (Size == 1 && Data[0] == 'B' && !skip1) {
>> + skip1 = true;
>> + ini1();
>> + }
>> + if (Size == 1 && Data[0] == 'C' && !skip2) {
>> + skip2 = true;
>> + ini2();
>> + }
>> +
>> + det0();
>> + det1();
>> + int a = rand();
>> + det2();
>> +
>> + switch (a % 5) {
>> + case 0:
>> + t0();
>> + break;
>> + case 1:
>> + t1();
>> + break;
>> + case 2:
>> + t2();
>> + break;
>> + case 3:
>> + t3();
>> + break;
>> + case 4:
>> + t4();
>> + break;
>> + default:
>> + assert(false);
>> + }
>> +
>> + det3();
>> + det4();
>> + return 0;
>> +}
>>
>> Added: compiler-rt/trunk/test/fuzzer/print_unstable_stats.test
>> URL:
>> http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/test/fuzzer/print_unstable_stats.test?rev=337187&view=auto
>>
>> ==============================================================================
>> --- compiler-rt/trunk/test/fuzzer/print_unstable_stats.test (added)
>> +++ compiler-rt/trunk/test/fuzzer/print_unstable_stats.test Mon Jul 16
>> 09:01:31 2018
>> @@ -0,0 +1,3 @@
>> +RUN: %cpp_compiler %S/PrintUnstableStatsTest.cpp -o
>> %t-PrintUnstableStatsTest
>> +RUN: %run %t-PrintUnstableStatsTest -print_unstable_stats=1 -runs=100000
>> 2>&1 | FileCheck %s --check-prefix=LONG
>> +LONG: stat::stability_rate: 27.59
>>
>>
>> _______________________________________________
>> llvm-commits mailing list
>> llvm-commits at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20180716/922caf36/attachment.html>
More information about the llvm-commits
mailing list