[compiler-rt] r316382 - [libFuzzer] Periodically purge allocator's quarantine to prolong fuzzing sessions.
Aleksey Shlyapnikov via llvm-commits
llvm-commits at lists.llvm.org
Wed Oct 25 15:58:51 PDT 2017
Thank you for report, Steven!
I'm not sure why it causes fuzzer to take longer now, looking into it.
Would you mind helping me with an experiment? Try turning this feature off
by passing -purge_allocator_interval=-1 to the fuzzer and see if it fixes
the problem.
On Wed, Oct 25, 2017 at 3:11 PM, Steven Wu <stevenwu at apple.com> wrote:
> Hi Alex
>
> This commit seems to break the greendragon libFuzzer bots in a weird way.
> http://green.lab.llvm.org/green/job/libFuzzer/8140/
>
> Personally, I have to rollback to macOS 10.11 to reproduce the issue.
> After your commit, it takes much more runs to trigger the exit
> on fuzzer-customcrossover.test. I am not an expert in fuzzer so I don't
> understand how this commit actually trigger the issue.
>
> Before this commit:
>
> $ fuzzer-customcrossover.test.tmp-CustomCrossOverTest -seed=1 -runs=100000
> ...
> #80269 REDUCE cov: 38 ft: 64 corp: 17/10063b exec/s: 26756 rss: 306Mb L:
> 961/4035 MS: 2 ChangeBinInt-EraseBytes-
> #97458 NEW cov: 39 ft: 65 corp: 18/10064b exec/s: 32486 rss: 361Mb L:
> 1/4035 MS: 1 ChangeByte-
> BINGO; Found the target, exiting
> ==71874== ERROR: libFuzzer: fuzz target exited
> #0 0x1094e8ee0 in __sanitizer_print_stack_trace asan_stack.cc:38
> #1 0x1081f3c3b in fuzzer::Fuzzer::StaticExitCallback()
> FuzzerLoop.cpp:214
> #2 0x7fff960c7450 in __cxa_finalize_ranges (libsystem_c.dylib:x86_64+
> 0x5f450)
> #3 0x7fff960c7766 in exit (libsystem_c.dylib:x86_64+0x5f766)
> #4 0x1081e3b2b in LLVMFuzzerTestOneInput CustomCrossOverTest.cpp:34
> #5 0x1081f4def in fuzzer::Fuzzer::ExecuteCallback(unsigned char
> const*, unsigned long) FuzzerLoop.cpp:471
> #6 0x1081f479b in fuzzer::Fuzzer::RunOne(unsigned char const*,
> unsigned long, bool, fuzzer::InputInfo*) FuzzerLoop.cpp:399
> #7 0x1081f6641 in fuzzer::Fuzzer::MutateAndTestOne()
> FuzzerLoop.cpp:602
> #8 0x1081f7277 in fuzzer::Fuzzer::Loop(std::__1:
> :vector<std::__1::basic_string<char, std::__1::char_traits<char>,
> std::__1::allocator<char> >, fuzzer::fuzzer_allocator<std::__1::basic_string<char,
> std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)
> FuzzerLoop.cpp:710
> #9 0x1081eeece in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned
> char const*, unsigned long)) FuzzerDriver.cpp:738
> #10 0x108214cd2 in main FuzzerMain.cpp:20
> #11 0x7fff8c78d5ac in start (libdyld.dylib:x86_64+0x35ac)
>
> SUMMARY: libFuzzer: fuzz target exited
>
>
> After the commit, I have to use -run=1000000 to trigger:
>
> $ fuzzer-customcrossover.test.tmp-CustomCrossOverTest -seed=1
> -runs=1000000
> ...
> #819960 REDUCE cov: 38 ft: 64 corp: 17/10063b exec/s: 35650 rss: 407Mb L:
> 4035/4035 MS: 3 EraseBytes-CopyPart-CopyPart-
> #837409 NEW cov: 39 ft: 65 corp: 18/10064b exec/s: 34892 rss: 407Mb L:
> 1/4035 MS: 2 ChangeBit-ChangeBit-
> BINGO; Found the target, exiting
> ==72288== ERROR: libFuzzer: fuzz target exited
> #0 0x1025d6ee0 in __sanitizer_print_stack_trace asan_stack.cc:38
> #1 0x1012dea2b in fuzzer::Fuzzer::StaticExitCallback()
> FuzzerLoop.cpp:214
> #2 0x7fff960c7450 in __cxa_finalize_ranges (libsystem_c.dylib:x86_64+
> 0x5f450)
> #3 0x7fff960c7766 in exit (libsystem_c.dylib:x86_64+0x5f766)
> #4 0x1012ce8cb in LLVMFuzzerTestOneInput CustomCrossOverTest.cpp:34
> #5 0x1012dfbdf in fuzzer::Fuzzer::ExecuteCallback(unsigned char
> const*, unsigned long) FuzzerLoop.cpp:471
> #6 0x1012df58b in fuzzer::Fuzzer::RunOne(unsigned char const*,
> unsigned long, bool, fuzzer::InputInfo*) FuzzerLoop.cpp:399
> #7 0x1012e1431 in fuzzer::Fuzzer::MutateAndTestOne()
> FuzzerLoop.cpp:602
> #8 0x1012e2225 in fuzzer::Fuzzer::Loop(std::__1:
> :vector<std::__1::basic_string<char, std::__1::char_traits<char>,
> std::__1::allocator<char> >, fuzzer::fuzzer_allocator<std::__1::basic_string<char,
> std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)
> FuzzerLoop.cpp:725
> #9 0x1012d9c8e in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned
> char const*, unsigned long)) FuzzerDriver.cpp:739
> #10 0x1012ffbd2 in main FuzzerMain.cpp:20
> #11 0x7fff8c78d5ac in start (libdyld.dylib:x86_64+0x35ac)
>
> SUMMARY: libFuzzer: fuzz target exited
>
>
> The easiest fix might be to update the test to run a bit longer but I am
> not sure if that is the right fix without understanding what is the
> underlying reason:
> diff --git a/test/fuzzer/fuzzer-customcrossover.test b/test/fuzzer/fuzzer-
> customcrossover.test
> index c32079f45..4b87c2cd7 100644
> --- a/test/fuzzer/fuzzer-customcrossover.test
> +++ b/test/fuzzer/fuzzer-customcrossover.test
> @@ -1,6 +1,6 @@
> RUN: %cpp_compiler %S/CustomCrossOverTest.cpp -o %t-CustomCrossOverTest
>
> -RUN: not %t-CustomCrossOverTest -seed=1 -runs=100000 2>&1
> | FileCheck %s --check-prefix=CHECK_CO
> +RUN: not %t-CustomCrossOverTest -seed=1 -runs=1000000 2>&1
> | FileCheck %s --check-prefix=CHECK_CO
> Disable cross_over, verify that we can't find the target w/o it.
> RUN: %t-CustomCrossOverTest -seed=1 -runs=1000000 -cross_over=0 2>&1
> | FileCheck %s --check-prefix=CHECK_NO_CO
>
> Thanks
>
> Steven
>
>
> On Oct 23, 2017, at 3:04 PM, Alex Shlyapnikov via llvm-commits <
> llvm-commits at lists.llvm.org> wrote:
>
> Author: alekseyshl
> Date: Mon Oct 23 15:04:30 2017
> New Revision: 316382
>
> URL: http://llvm.org/viewvc/llvm-project?rev=316382&view=rev
> Log:
> [libFuzzer] Periodically purge allocator's quarantine to prolong fuzzing
> sessions.
>
> Summary:
> Fuzzing targets that allocate/deallocate a lot of memory tend to consume
> a lot of RSS when ASan quarantine is enabled. Purging quarantine between
> iterations and returning memory to OS keeps RSS down and should not
> reduce the quarantine effectiveness provided the fuzz target does not
> preserve state between iterations (in this case this feature can be turned
> off).
>
> Based on D39153.
>
> Reviewers: vitalybuka
>
> Subscribers: llvm-commits
>
> Differential Revision: https://reviews.llvm.org/D39155
>
> Modified:
> compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp
> compiler-rt/trunk/lib/fuzzer/FuzzerExtFunctions.def
> compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def
> compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h
> compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp
> compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h
>
> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp
> URL: http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/
> fuzzer/FuzzerDriver.cpp?rev=316382&r1=316381&r2=316382&view=diff
> ============================================================
> ==================
> --- compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp (original)
> +++ compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp Mon Oct 23 15:04:30 2017
> @@ -578,6 +578,7 @@ int FuzzerDriver(int *argc, char ***argv
> Options.ReloadIntervalSec = Flags.reload;
> Options.OnlyASCII = Flags.only_ascii;
> Options.DetectLeaks = Flags.detect_leaks;
> + Options.PurgeAllocatorIntervalSec = Flags.purge_allocator_interval;
> Options.TraceMalloc = Flags.trace_malloc;
> Options.RssLimitMb = Flags.rss_limit_mb;
> if (Flags.runs >= 0)
>
> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerExtFunctions.def
> URL: http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/
> fuzzer/FuzzerExtFunctions.def?rev=316382&r1=316381&r2=316382&view=diff
> ============================================================
> ==================
> --- compiler-rt/trunk/lib/fuzzer/FuzzerExtFunctions.def (original)
> +++ compiler-rt/trunk/lib/fuzzer/FuzzerExtFunctions.def Mon Oct 23
> 15:04:30 2017
> @@ -33,6 +33,7 @@ EXT_FUNC(__sanitizer_install_malloc_and_
> (void (*malloc_hook)(const volatile void *, size_t),
> void (*free_hook)(const volatile void *)),
> false);
> +EXT_FUNC(__sanitizer_purge_allocator, void, (), false);
> EXT_FUNC(__sanitizer_print_memory_profile, int, (size_t, size_t), false);
> EXT_FUNC(__sanitizer_print_stack_trace, void, (), true);
> EXT_FUNC(__sanitizer_symbolize_pc, void,
>
> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def
> URL: http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/
> fuzzer/FuzzerFlags.def?rev=316382&r1=316381&r2=316382&view=diff
> ============================================================
> ==================
> --- compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def (original)
> +++ compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def Mon Oct 23 15:04:30 2017
> @@ -114,6 +114,10 @@ FUZZER_FLAG_INT(close_fd_mask, 0, "If 1,
> "Be careful, this will also close e.g. asan's stderr/stdout.")
> FUZZER_FLAG_INT(detect_leaks, 1, "If 1, and if LeakSanitizer is enabled "
> "try to detect memory leaks during fuzzing (i.e. not only at shut
> down).")
> +FUZZER_FLAG_INT(purge_allocator_interval, 1, "Purge allocator caches and
> "
> + "quarantines every <N> seconds. When rss_limit_mb is specified (>0), "
> + "purging starts when RSS exceeds 50% of rss_limit_mb. Pass "
> + "purge_allocator_interval=-1 to disable this functionality.")
> FUZZER_FLAG_INT(trace_malloc, 0, "If >= 1 will print all mallocs/frees. "
> "If >= 2 will also print stack traces.")
> FUZZER_FLAG_INT(rss_limit_mb, 2048, "If non-zero, the fuzzer will exit
> upon"
>
> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h
> URL: http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/
> fuzzer/FuzzerInternal.h?rev=316382&r1=316381&r2=316382&view=diff
> ============================================================
> ==================
> --- compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h (original)
> +++ compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h Mon Oct 23 15:04:30 2017
> @@ -96,6 +96,7 @@ private:
> void CrashOnOverwrittenData();
> void InterruptCallback();
> void MutateAndTestOne();
> + void PurgeAllocator();
> void ReportNewCoverage(InputInfo *II, const Unit &U);
> void PrintPulseAndReportSlowInput(const uint8_t *Data, size_t Size);
> void WriteToOutputCorpus(const Unit &U);
> @@ -124,6 +125,8 @@ private:
> bool HasMoreMallocsThanFrees = false;
> size_t NumberOfLeakDetectionAttempts = 0;
>
> + system_clock::time_point LastAllocatorPurgeAttemptTime =
> system_clock::now();
> +
> UserCallback CB;
> InputCorpus &Corpus;
> MutationDispatcher &MD;
>
> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp
> URL: http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/
> fuzzer/FuzzerLoop.cpp?rev=316382&r1=316381&r2=316382&view=diff
> ============================================================
> ==================
> --- compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp (original)
> +++ compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp Mon Oct 23 15:04:30 2017
> @@ -587,7 +587,7 @@ void Fuzzer::MutateAndTestOne() {
> size_t NewSize = 0;
> NewSize = MD.Mutate(CurrentUnitData, Size, CurrentMaxMutationLen);
> assert(NewSize > 0 && "Mutator returned empty unit");
> - assert(NewSize <= CurrentMaxMutationLen && "Mutator return overisized
> unit");
> + assert(NewSize <= CurrentMaxMutationLen && "Mutator return oversized
> unit");
> Size = NewSize;
> II.NumExecutedMutations++;
> if (RunOne(CurrentUnitData, Size, /*MayDeleteFile=*/true, &II))
> @@ -598,6 +598,25 @@ void Fuzzer::MutateAndTestOne() {
> }
> }
>
> +void Fuzzer::PurgeAllocator() {
> + if (Options.PurgeAllocatorIntervalSec < 0 ||
> + !EF->__sanitizer_purge_allocator) {
> + return;
> + }
> + if (duration_cast<seconds>(system_clock::now() -
> + LastAllocatorPurgeAttemptTime).count() <
> + Options.PurgeAllocatorIntervalSec) {
> + return;
> + }
> +
> + if (Options.RssLimitMb <= 0 ||
> + GetPeakRSSMb() > static_cast<size_t>(Options.RssLimitMb) / 2) {
> + EF->__sanitizer_purge_allocator();
> + }
> +
> + LastAllocatorPurgeAttemptTime = system_clock::now();
> +}
> +
> void Fuzzer::ReadAndExecuteSeedCorpora(const Vector<std::string>
> &CorpusDirs) {
> const size_t kMaxSaneLen = 1 << 20;
> const size_t kMinDefaultLen = 4096;
> @@ -699,6 +718,8 @@ void Fuzzer::Loop(const Vector<std::stri
>
> // Perform several mutations and runs.
> MutateAndTestOne();
> +
> + PurgeAllocator();
> }
>
> PrintStats("DONE ", "\n");
>
> Modified: compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h
> URL: http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/
> fuzzer/FuzzerOptions.h?rev=316382&r1=316381&r2=316382&view=diff
> ============================================================
> ==================
> --- compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h (original)
> +++ compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h Mon Oct 23 15:04:30 2017
> @@ -54,6 +54,7 @@ struct FuzzingOptions {
> bool DumpCoverage = false;
> bool UseClangCoverage = false;
> bool DetectLeaks = true;
> + int PurgeAllocatorIntervalSec = 1;
> int UseFeatureFrequency = false;
> int TraceMalloc = 0;
> bool HandleAbrt = false;
>
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20171025/3e0385ec/attachment.html>
More information about the llvm-commits
mailing list