[compiler-rt] r316382 - [libFuzzer] Periodically purge allocator's quarantine to prolong fuzzing sessions.

Kostya Serebryany via llvm-commits llvm-commits at lists.llvm.org
Wed Oct 25 16:25:20 PDT 2017 

r316627 bumps the number of runs, let's see if it helps.

On Wed, Oct 25, 2017 at 4:20 PM, Steven Wu <stevenwu at apple.com> wrote:

> I tried -purge_allocator_interval=-1 already and it doesn't really have
> visible effects. It still takes about 800000 iterations. But reverting the
> commit will dramatically reduce the number.
>
> I couldn't reproduce any of the above behavior on a macOS 10.13 machine
> but the bot is running 10.11 and I don't think it is going to be upgraded
> soon.
>
> Here is 100 runs with random seed on the node:
> 27678, 86464, 292063, 223933, 270628, 369673, 69198, 104383, 360948,
> 493949, 201033, 69598, 38278, 369223, 1447353, 141653, 287133, 102902,
> 42483, 232098, 322363, 221209, 1351098, 334748, 133003, 154933, 283288,
> 53163, 142843, 684879, 231868, 129793, 65793, 289628, 661468, 39453,
> 1060768, 56888, 597893, 420758, 83254, 423834, 254628, 259539, 353788,
> 311658, 223364, 902998, 264956, 187758, 342088, 225504, 104525, 258820,
> 428398, 15638, 872103, 383763, 34288, 744653, 603998, 67733, 283263,
> 127868, 29828, 250905, 752238, 66483, 756183, 180353, 269019, 222198,
> 26859, 1542408, 77888, 300393, 370358, 382433, 173995, 186238, 218598,
> 215368, 166723, 427288, 147498, 255508, 215403, 181738, 524928, 74126,
> 48463, 278483, 533148, 27683, 200058, 127196, 405728, 203908, 487173, 104873
>
> Let me know what else I can help.
>
> Steven
>
>
> On Oct 25, 2017, at 4:00 PM, Kostya Serebryany <kcc at google.com> wrote:
>
>
>
> On Wed, Oct 25, 2017 at 3:11 PM, Steven Wu via llvm-commits <llvm-commits@
> lists.llvm.org> wrote:
>
>> Hi Alex
>>
>> This commit seems to break the greendragon libFuzzer bots in a weird way.
>> http://green.lab.llvm.org/green/job/libFuzzer/8140/
>>
>> Personally, I have to rollback to macOS 10.11 to reproduce the issue.
>> After your commit, it takes much more runs to trigger the exit
>> on fuzzer-customcrossover.test. I am not an expert in fuzzer so I don't
>> understand how this commit actually trigger the issue.
>>
>> Before this commit:
>>
>> $ fuzzer-customcrossover.test.tmp-CustomCrossOverTest -seed=1
>> -runs=100000
>>
>>
> Weird.
> On linux this test finishes much sooner regardless the seed.
> Can you run 1000 times without -seed=1 (i.e. with a random seed) and tell
> us the iteration numbers?
>
>
>
>> ...
>> #80269 REDUCE cov: 38 ft: 64 corp: 17/10063b exec/s: 26756 rss: 306Mb L:
>> 961/4035 MS: 2 ChangeBinInt-EraseBytes-
>> #97458 NEW    cov: 39 ft: 65 corp: 18/10064b exec/s: 32486 rss: 361Mb L:
>> 1/4035 MS: 1 ChangeByte-
>> BINGO; Found the target, exiting
>> ==71874== ERROR: libFuzzer: fuzz target exited
>>     #0 0x1094e8ee0 in __sanitizer_print_stack_trace asan_stack.cc:38
>>     #1 0x1081f3c3b in fuzzer::Fuzzer::StaticExitCallback()
>> FuzzerLoop.cpp:214
>>     #2 0x7fff960c7450 in __cxa_finalize_ranges
>> (libsystem_c.dylib:x86_64+0x5f450)
>>     #3 0x7fff960c7766 in exit (libsystem_c.dylib:x86_64+0x5f766)
>>     #4 0x1081e3b2b in LLVMFuzzerTestOneInput CustomCrossOverTest.cpp:34
>>     #5 0x1081f4def in fuzzer::Fuzzer::ExecuteCallback(unsigned char
>> const*, unsigned long) FuzzerLoop.cpp:471
>>     #6 0x1081f479b in fuzzer::Fuzzer::RunOne(unsigned char const*,
>> unsigned long, bool, fuzzer::InputInfo*) FuzzerLoop.cpp:399
>>     #7 0x1081f6641 in fuzzer::Fuzzer::MutateAndTestOne()
>> FuzzerLoop.cpp:602
>>     #8 0x1081f7277 in fuzzer::Fuzzer::Loop(std::__1:
>> :vector<std::__1::basic_string<char, std::__1::char_traits<char>,
>> std::__1::allocator<char> >, fuzzer::fuzzer_allocator<std::__1::basic_string<char,
>> std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)
>> FuzzerLoop.cpp:710
>>     #9 0x1081eeece in fuzzer::FuzzerDriver(int*, char***, int
>> (*)(unsigned char const*, unsigned long)) FuzzerDriver.cpp:738
>>     #10 0x108214cd2 in main FuzzerMain.cpp:20
>>     #11 0x7fff8c78d5ac in start (libdyld.dylib:x86_64+0x35ac)
>>
>> SUMMARY: libFuzzer: fuzz target exited
>>
>>
>> After the commit, I have to use -run=1000000 to trigger:
>>
>> $ fuzzer-customcrossover.test.tmp-CustomCrossOverTest -seed=1
>> -runs=1000000
>> ...
>> #819960 REDUCE cov: 38 ft: 64 corp: 17/10063b exec/s: 35650 rss: 407Mb
>> L: 4035/4035 MS: 3 EraseBytes-CopyPart-CopyPart-
>> #837409 NEW    cov: 39 ft: 65 corp: 18/10064b exec/s: 34892 rss: 407Mb
>> L: 1/4035 MS: 2 ChangeBit-ChangeBit-
>> BINGO; Found the target, exiting
>> ==72288== ERROR: libFuzzer: fuzz target exited
>>     #0 0x1025d6ee0 in __sanitizer_print_stack_trace asan_stack.cc:38
>>     #1 0x1012dea2b in fuzzer::Fuzzer::StaticExitCallback()
>> FuzzerLoop.cpp:214
>>     #2 0x7fff960c7450 in __cxa_finalize_ranges
>> (libsystem_c.dylib:x86_64+0x5f450)
>>     #3 0x7fff960c7766 in exit (libsystem_c.dylib:x86_64+0x5f766)
>>     #4 0x1012ce8cb in LLVMFuzzerTestOneInput CustomCrossOverTest.cpp:34
>>     #5 0x1012dfbdf in fuzzer::Fuzzer::ExecuteCallback(unsigned char
>> const*, unsigned long) FuzzerLoop.cpp:471
>>     #6 0x1012df58b in fuzzer::Fuzzer::RunOne(unsigned char const*,
>> unsigned long, bool, fuzzer::InputInfo*) FuzzerLoop.cpp:399
>>     #7 0x1012e1431 in fuzzer::Fuzzer::MutateAndTestOne()
>> FuzzerLoop.cpp:602
>>     #8 0x1012e2225 in fuzzer::Fuzzer::Loop(std::__1:
>> :vector<std::__1::basic_string<char, std::__1::char_traits<char>,
>> std::__1::allocator<char> >, fuzzer::fuzzer_allocator<std::__1::basic_string<char,
>> std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)
>> FuzzerLoop.cpp:725
>>     #9 0x1012d9c8e in fuzzer::FuzzerDriver(int*, char***, int
>> (*)(unsigned char const*, unsigned long)) FuzzerDriver.cpp:739
>>     #10 0x1012ffbd2 in main FuzzerMain.cpp:20
>>     #11 0x7fff8c78d5ac in start (libdyld.dylib:x86_64+0x35ac)
>>
>> SUMMARY: libFuzzer: fuzz target exited
>>
>>
>> The easiest fix might be to update the test to run a bit longer but I am
>> not sure if that is the right fix without understanding what is the
>> underlying reason:
>> diff --git a/test/fuzzer/fuzzer-customcrossover.test
>> b/test/fuzzer/fuzzer-customcrossover.test
>> index c32079f45..4b87c2cd7 100644
>> --- a/test/fuzzer/fuzzer-customcrossover.test
>> +++ b/test/fuzzer/fuzzer-customcrossover.test
>> @@ -1,6 +1,6 @@
>>  RUN: %cpp_compiler %S/CustomCrossOverTest.cpp -o %t-CustomCrossOverTest
>>
>> -RUN: not %t-CustomCrossOverTest -seed=1 -runs=100000                2>&1
>> | FileCheck %s --check-prefix=CHECK_CO
>> +RUN: not %t-CustomCrossOverTest -seed=1 -runs=1000000               2>&1
>> | FileCheck %s --check-prefix=CHECK_CO
>>  Disable cross_over, verify that we can't find the target w/o it.
>>  RUN:     %t-CustomCrossOverTest -seed=1 -runs=1000000 -cross_over=0 2>&1
>> | FileCheck %s --check-prefix=CHECK_NO_CO
>>
>> Thanks
>>
>> Steven
>>
>>
>> On Oct 23, 2017, at 3:04 PM, Alex Shlyapnikov via llvm-commits <
>> llvm-commits at lists.llvm.org> wrote:
>>
>> Author: alekseyshl
>> Date: Mon Oct 23 15:04:30 2017
>> New Revision: 316382
>>
>> URL: http://llvm.org/viewvc/llvm-project?rev=316382&view=rev
>> Log:
>> [libFuzzer] Periodically purge allocator's quarantine to prolong fuzzing
>> sessions.
>>
>> Summary:
>> Fuzzing targets that allocate/deallocate a lot of memory tend to consume
>> a lot of RSS when ASan quarantine is enabled. Purging quarantine between
>> iterations and returning memory to OS keeps RSS down and should not
>> reduce the quarantine effectiveness provided the fuzz target does not
>> preserve state between iterations (in this case this feature can be
>> turned off).
>>
>> Based on D39153.
>>
>> Reviewers: vitalybuka
>>
>> Subscribers: llvm-commits
>>
>> Differential Revision: https://reviews.llvm.org/D39155
>>
>> Modified:
>>    compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp
>>    compiler-rt/trunk/lib/fuzzer/FuzzerExtFunctions.def
>>    compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def
>>    compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h
>>    compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp
>>    compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h
>>
>> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp
>> URL: http://llvm.org/viewvc/llvm-project/compiler-rt/
>> trunk/lib/fuzzer/FuzzerDriver.cpp?rev=316382&r1=316381&r2=
>> 316382&view=diff
>> ============================================================
>> ==================
>> --- compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp (original)
>> +++ compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp Mon Oct 23 15:04:30
>> 2017
>> @@ -578,6 +578,7 @@ int FuzzerDriver(int *argc, char ***argv
>>   Options.ReloadIntervalSec = Flags.reload;
>>   Options.OnlyASCII = Flags.only_ascii;
>>   Options.DetectLeaks = Flags.detect_leaks;
>> +  Options.PurgeAllocatorIntervalSec = Flags.purge_allocator_interval;
>>   Options.TraceMalloc = Flags.trace_malloc;
>>   Options.RssLimitMb = Flags.rss_limit_mb;
>>   if (Flags.runs >= 0)
>>
>> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerExtFunctions.def
>> URL: http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/
>> FuzzerExtFunctions.def?rev=316382&r1=316381&r2=316382&view=diff
>> ============================================================
>> ==================
>> --- compiler-rt/trunk/lib/fuzzer/FuzzerExtFunctions.def (original)
>> +++ compiler-rt/trunk/lib/fuzzer/FuzzerExtFunctions.def Mon Oct 23
>> 15:04:30 2017
>> @@ -33,6 +33,7 @@ EXT_FUNC(__sanitizer_install_malloc_and_
>>          (void (*malloc_hook)(const volatile void *, size_t),
>>           void (*free_hook)(const volatile void *)),
>>          false);
>> +EXT_FUNC(__sanitizer_purge_allocator, void, (), false);
>> EXT_FUNC(__sanitizer_print_memory_profile, int, (size_t, size_t), false);
>> EXT_FUNC(__sanitizer_print_stack_trace, void, (), true);
>> EXT_FUNC(__sanitizer_symbolize_pc, void,
>>
>> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def
>> URL: http://llvm.org/viewvc/llvm-project/compiler-rt/
>> trunk/lib/fuzzer/FuzzerFlags.def?rev=316382&r1=316381&r2=316382&view=diff
>> ============================================================
>> ==================
>> --- compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def (original)
>> +++ compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def Mon Oct 23 15:04:30 2017
>> @@ -114,6 +114,10 @@ FUZZER_FLAG_INT(close_fd_mask, 0, "If 1,
>>     "Be careful, this will also close e.g. asan's stderr/stdout.")
>> FUZZER_FLAG_INT(detect_leaks, 1, "If 1, and if LeakSanitizer is enabled "
>>     "try to detect memory leaks during fuzzing (i.e. not only at shut
>> down).")
>> +FUZZER_FLAG_INT(purge_allocator_interval, 1, "Purge allocator caches
>> and "
>> +    "quarantines every <N> seconds. When rss_limit_mb is specified (>0),
>> "
>> +    "purging starts when RSS exceeds 50% of rss_limit_mb. Pass "
>> +    "purge_allocator_interval=-1 to disable this functionality.")
>> FUZZER_FLAG_INT(trace_malloc, 0, "If >= 1 will print all mallocs/frees. "
>>     "If >= 2 will also print stack traces.")
>> FUZZER_FLAG_INT(rss_limit_mb, 2048, "If non-zero, the fuzzer will exit
>> upon"
>>
>> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h
>> URL: http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/
>> FuzzerInternal.h?rev=316382&r1=316381&r2=316382&view=diff
>> ============================================================
>> ==================
>> --- compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h (original)
>> +++ compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h Mon Oct 23 15:04:30
>> 2017
>> @@ -96,6 +96,7 @@ private:
>>   void CrashOnOverwrittenData();
>>   void InterruptCallback();
>>   void MutateAndTestOne();
>> +  void PurgeAllocator();
>>   void ReportNewCoverage(InputInfo *II, const Unit &U);
>>   void PrintPulseAndReportSlowInput(const uint8_t *Data, size_t Size);
>>   void WriteToOutputCorpus(const Unit &U);
>> @@ -124,6 +125,8 @@ private:
>>   bool HasMoreMallocsThanFrees = false;
>>   size_t NumberOfLeakDetectionAttempts = 0;
>>
>> +  system_clock::time_point LastAllocatorPurgeAttemptTime =
>> system_clock::now();
>> +
>>   UserCallback CB;
>>   InputCorpus &Corpus;
>>   MutationDispatcher &MD;
>>
>> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp
>> URL: http://llvm.org/viewvc/llvm-project/compiler-rt/
>> trunk/lib/fuzzer/FuzzerLoop.cpp?rev=316382&r1=316381&r2=316382&view=diff
>> ============================================================
>> ==================
>> --- compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp (original)
>> +++ compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp Mon Oct 23 15:04:30 2017
>> @@ -587,7 +587,7 @@ void Fuzzer::MutateAndTestOne() {
>>     size_t NewSize = 0;
>>     NewSize = MD.Mutate(CurrentUnitData, Size, CurrentMaxMutationLen);
>>     assert(NewSize > 0 && "Mutator returned empty unit");
>> -    assert(NewSize <= CurrentMaxMutationLen && "Mutator return
>> overisized unit");
>> +    assert(NewSize <= CurrentMaxMutationLen && "Mutator return oversized
>> unit");
>>     Size = NewSize;
>>     II.NumExecutedMutations++;
>>     if (RunOne(CurrentUnitData, Size, /*MayDeleteFile=*/true, &II))
>> @@ -598,6 +598,25 @@ void Fuzzer::MutateAndTestOne() {
>>   }
>> }
>>
>> +void Fuzzer::PurgeAllocator() {
>> +  if (Options.PurgeAllocatorIntervalSec < 0 ||
>> +      !EF->__sanitizer_purge_allocator) {
>> +    return;
>> +  }
>> +  if (duration_cast<seconds>(system_clock::now() -
>> +                             LastAllocatorPurgeAttemptTime).count() <
>> +      Options.PurgeAllocatorIntervalSec) {
>> +    return;
>> +  }
>> +
>> +  if (Options.RssLimitMb <= 0 ||
>> +      GetPeakRSSMb() > static_cast<size_t>(Options.RssLimitMb) / 2) {
>> +    EF->__sanitizer_purge_allocator();
>> +  }
>> +
>> +  LastAllocatorPurgeAttemptTime = system_clock::now();
>> +}
>> +
>> void Fuzzer::ReadAndExecuteSeedCorpora(const Vector<std::string>
>> &CorpusDirs) {
>>   const size_t kMaxSaneLen = 1 << 20;
>>   const size_t kMinDefaultLen = 4096;
>> @@ -699,6 +718,8 @@ void Fuzzer::Loop(const Vector<std::stri
>>
>>     // Perform several mutations and runs.
>>     MutateAndTestOne();
>> +
>> +    PurgeAllocator();
>>   }
>>
>>   PrintStats("DONE  ", "\n");
>>
>> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h
>> URL: http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/
>> FuzzerOptions.h?rev=316382&r1=316381&r2=316382&view=diff
>> ============================================================
>> ==================
>> --- compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h (original)
>> +++ compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h Mon Oct 23 15:04:30 2017
>> @@ -54,6 +54,7 @@ struct FuzzingOptions {
>>   bool DumpCoverage = false;
>>   bool UseClangCoverage = false;
>>   bool DetectLeaks = true;
>> +  int PurgeAllocatorIntervalSec = 1;
>>   int UseFeatureFrequency = false;
>>   int  TraceMalloc = 0;
>>   bool HandleAbrt = false;
>>
>>
>> _______________________________________________
>> llvm-commits mailing list
>> llvm-commits at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>>
>>
>>
>> _______________________________________________
>> llvm-commits mailing list
>> llvm-commits at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20171025/c758db80/attachment.html>

More information about the llvm-commits mailing list