<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Alex<div class=""><br class=""></div><div class="">This commit seems to break the greendragon libFuzzer bots in a weird way. </div><div class=""><a href="http://green.lab.llvm.org/green/job/libFuzzer/8140/" class="">http://green.lab.llvm.org/green/job/libFuzzer/8140/</a></div><div class=""><br class=""></div><div class="">Personally, I have to rollback to macOS 10.11 to reproduce the issue. After your commit, it takes much more runs to trigger the exit on fuzzer-customcrossover.test. I am not an expert in fuzzer so I don't understand how this commit actually trigger the issue.</div><div class=""><br class=""></div><div class="">Before this commit:</div><div class=""></div><blockquote type="cite" class=""><div class="">$ fuzzer-customcrossover.test.tmp-CustomCrossOverTest -seed=1 -runs=100000</div><div class="">...</div><div class=""><div class="">#80269<span class="Apple-tab-span" style="white-space:pre"> </span>REDUCE cov: 38 ft: 64 corp: 17/10063b exec/s: 26756 rss: 306Mb L: 961/4035 MS: 2 ChangeBinInt-EraseBytes-</div><div class="">#97458<span class="Apple-tab-span" style="white-space:pre"> </span>NEW cov: 39 ft: 65 corp: 18/10064b exec/s: 32486 rss: 361Mb L: 1/4035 MS: 1 ChangeByte-</div><div class="">BINGO; Found the target, exiting</div><div class="">==71874== ERROR: libFuzzer: fuzz target exited</div><div class=""> #0 0x1094e8ee0 in __sanitizer_print_stack_trace <a href="http://asan_stack.cc:38" class="">asan_stack.cc:38</a></div><div class=""> #1 0x1081f3c3b in fuzzer::Fuzzer::StaticExitCallback() FuzzerLoop.cpp:214</div><div class=""> #2 0x7fff960c7450 in __cxa_finalize_ranges (libsystem_c.dylib:x86_64+0x5f450)</div><div class=""> #3 0x7fff960c7766 in exit (libsystem_c.dylib:x86_64+0x5f766)</div><div class=""> #4 0x1081e3b2b in LLVMFuzzerTestOneInput CustomCrossOverTest.cpp:34</div><div class=""> #5 0x1081f4def in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) FuzzerLoop.cpp:471</div><div class=""> #6 0x1081f479b in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*) FuzzerLoop.cpp:399</div><div class=""> #7 0x1081f6641 in fuzzer::Fuzzer::MutateAndTestOne() FuzzerLoop.cpp:602</div><div class=""> #8 0x1081f7277 in fuzzer::Fuzzer::Loop(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, fuzzer::fuzzer_allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) FuzzerLoop.cpp:710</div><div class=""> #9 0x1081eeece in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) FuzzerDriver.cpp:738</div><div class=""> #10 0x108214cd2 in main FuzzerMain.cpp:20</div><div class=""> #11 0x7fff8c78d5ac in start (libdyld.dylib:x86_64+0x35ac)</div><div class=""><br class=""></div><div class="">SUMMARY: libFuzzer: fuzz target exited</div></div></blockquote><div class=""><div class=""><br class=""></div><div class="">After the commit, I have to use -run=1000000 to trigger:</div><div class=""></div><blockquote type="cite" class=""><div class="">$ fuzzer-customcrossover.test.tmp-CustomCrossOverTest -seed=1 -runs=1000000</div><div class="">...</div><div class="">#819960<span class="Apple-tab-span" style="white-space:pre"> </span>REDUCE cov: 38 ft: 64 corp: 17/10063b exec/s: 35650 rss: 407Mb L: 4035/4035 MS: 3 EraseBytes-CopyPart-CopyPart-<br class="">#837409<span class="Apple-tab-span" style="white-space:pre"> </span>NEW cov: 39 ft: 65 corp: 18/10064b exec/s: 34892 rss: 407Mb L: 1/4035 MS: 2 ChangeBit-ChangeBit-<br class="">BINGO; Found the target, exiting<br class="">==72288== ERROR: libFuzzer: fuzz target exited<br class=""> #0 0x1025d6ee0 in __sanitizer_print_stack_trace <a href="http://asan_stack.cc:38" class="">asan_stack.cc:38</a><br class=""> #1 0x1012dea2b in fuzzer::Fuzzer::StaticExitCallback() FuzzerLoop.cpp:214<br class=""> #2 0x7fff960c7450 in __cxa_finalize_ranges (libsystem_c.dylib:x86_64+0x5f450)<br class=""> #3 0x7fff960c7766 in exit (libsystem_c.dylib:x86_64+0x5f766)<br class=""> #4 0x1012ce8cb in LLVMFuzzerTestOneInput CustomCrossOverTest.cpp:34<br class=""> #5 0x1012dfbdf in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) FuzzerLoop.cpp:471<br class=""> #6 0x1012df58b in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*) FuzzerLoop.cpp:399<br class=""> #7 0x1012e1431 in fuzzer::Fuzzer::MutateAndTestOne() FuzzerLoop.cpp:602<br class=""> #8 0x1012e2225 in fuzzer::Fuzzer::Loop(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, fuzzer::fuzzer_allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) FuzzerLoop.cpp:725<br class=""> #9 0x1012d9c8e in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) FuzzerDriver.cpp:739<br class=""> #10 0x1012ffbd2 in main FuzzerMain.cpp:20<br class=""> #11 0x7fff8c78d5ac in start (libdyld.dylib:x86_64+0x35ac)<br class=""><br class="">SUMMARY: libFuzzer: fuzz target exited</div></blockquote><div class=""><br class=""></div><div class="">The easiest fix might be to update the test to run a bit longer but I am not sure if that is the right fix without understanding what is the underlying reason:</div><div class=""><div class="">diff --git a/test/fuzzer/fuzzer-customcrossover.test b/test/fuzzer/fuzzer-customcrossover.test</div><div class="">index c32079f45..4b87c2cd7 100644</div><div class="">--- a/test/fuzzer/fuzzer-customcrossover.test</div><div class="">+++ b/test/fuzzer/fuzzer-customcrossover.test</div><div class="">@@ -1,6 +1,6 @@</div><div class=""> RUN: %cpp_compiler %S/CustomCrossOverTest.cpp -o %t-CustomCrossOverTest</div><div class=""><br class=""></div><div class="">-RUN: not %t-CustomCrossOverTest -seed=1 -runs=100000 2>&1 | FileCheck %s --check-prefix=CHECK_CO</div><div class="">+RUN: not %t-CustomCrossOverTest -seed=1 -runs=1000000 2>&1 | FileCheck %s --check-prefix=CHECK_CO</div><div class=""> Disable cross_over, verify that we can't find the target w/o it.</div><div class=""> RUN: %t-CustomCrossOverTest -seed=1 -runs=1000000 -cross_over=0 2>&1 | FileCheck %s --check-prefix=CHECK_NO_CO</div></div><div class=""><br class=""></div><div class="">Thanks</div><div class=""><br class=""></div><div class="">Steven</div><div class=""><br class=""></div><div><br class=""><blockquote type="cite" class=""><div class="">On Oct 23, 2017, at 3:04 PM, Alex Shlyapnikov via llvm-commits <<a href="mailto:llvm-commits@lists.llvm.org" class="">llvm-commits@lists.llvm.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">Author: alekseyshl<br class="">Date: Mon Oct 23 15:04:30 2017<br class="">New Revision: 316382<br class=""><br class="">URL: <a href="http://llvm.org/viewvc/llvm-project?rev=316382&view=rev" class="">http://llvm.org/viewvc/llvm-project?rev=316382&view=rev</a><br class="">Log:<br class="">[libFuzzer] Periodically purge allocator's quarantine to prolong fuzzing sessions.<br class=""><br class="">Summary:<br class="">Fuzzing targets that allocate/deallocate a lot of memory tend to consume<br class="">a lot of RSS when ASan quarantine is enabled. Purging quarantine between<br class="">iterations and returning memory to OS keeps RSS down and should not<br class="">reduce the quarantine effectiveness provided the fuzz target does not<br class="">preserve state between iterations (in this case this feature can be turned off).<br class=""><br class="">Based on D39153.<br class=""><br class="">Reviewers: vitalybuka<br class=""><br class="">Subscribers: llvm-commits<br class=""><br class="">Differential Revision: <a href="https://reviews.llvm.org/D39155" class="">https://reviews.llvm.org/D39155</a><br class=""><br class="">Modified:<br class=""> compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp<br class=""> compiler-rt/trunk/lib/fuzzer/FuzzerExtFunctions.def<br class=""> compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def<br class=""> compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h<br class=""> compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp<br class=""> compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h<br class=""><br class="">Modified: compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp<br class="">URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp?rev=316382&r1=316381&r2=316382&view=diff" class="">http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp?rev=316382&r1=316381&r2=316382&view=diff</a><br class="">==============================================================================<br class="">--- compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp (original)<br class="">+++ compiler-rt/trunk/lib/fuzzer/FuzzerDriver.cpp Mon Oct 23 15:04:30 2017<br class="">@@ -578,6 +578,7 @@ int FuzzerDriver(int *argc, char ***argv<br class=""> Options.ReloadIntervalSec = Flags.reload;<br class=""> Options.OnlyASCII = Flags.only_ascii;<br class=""> Options.DetectLeaks = Flags.detect_leaks;<br class="">+ Options.PurgeAllocatorIntervalSec = Flags.purge_allocator_interval;<br class=""> Options.TraceMalloc = Flags.trace_malloc;<br class=""> Options.RssLimitMb = Flags.rss_limit_mb;<br class=""> if (Flags.runs >= 0)<br class=""><br class="">Modified: compiler-rt/trunk/lib/fuzzer/FuzzerExtFunctions.def<br class="">URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerExtFunctions.def?rev=316382&r1=316381&r2=316382&view=diff" class="">http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerExtFunctions.def?rev=316382&r1=316381&r2=316382&view=diff</a><br class="">==============================================================================<br class="">--- compiler-rt/trunk/lib/fuzzer/FuzzerExtFunctions.def (original)<br class="">+++ compiler-rt/trunk/lib/fuzzer/FuzzerExtFunctions.def Mon Oct 23 15:04:30 2017<br class="">@@ -33,6 +33,7 @@ EXT_FUNC(__sanitizer_install_malloc_and_<br class=""> (void (*malloc_hook)(const volatile void *, size_t),<br class=""> void (*free_hook)(const volatile void *)),<br class=""> false);<br class="">+EXT_FUNC(__sanitizer_purge_allocator, void, (), false);<br class=""> EXT_FUNC(__sanitizer_print_memory_profile, int, (size_t, size_t), false);<br class=""> EXT_FUNC(__sanitizer_print_stack_trace, void, (), true);<br class=""> EXT_FUNC(__sanitizer_symbolize_pc, void,<br class=""><br class="">Modified: compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def<br class="">URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def?rev=316382&r1=316381&r2=316382&view=diff" class="">http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def?rev=316382&r1=316381&r2=316382&view=diff</a><br class="">==============================================================================<br class="">--- compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def (original)<br class="">+++ compiler-rt/trunk/lib/fuzzer/FuzzerFlags.def Mon Oct 23 15:04:30 2017<br class="">@@ -114,6 +114,10 @@ FUZZER_FLAG_INT(close_fd_mask, 0, "If 1,<br class=""> "Be careful, this will also close e.g. asan's stderr/stdout.")<br class=""> FUZZER_FLAG_INT(detect_leaks, 1, "If 1, and if LeakSanitizer is enabled "<br class=""> "try to detect memory leaks during fuzzing (i.e. not only at shut down).")<br class="">+FUZZER_FLAG_INT(purge_allocator_interval, 1, "Purge allocator caches and "<br class="">+ "quarantines every <N> seconds. When rss_limit_mb is specified (>0), "<br class="">+ "purging starts when RSS exceeds 50% of rss_limit_mb. Pass "<br class="">+ "purge_allocator_interval=-1 to disable this functionality.")<br class=""> FUZZER_FLAG_INT(trace_malloc, 0, "If >= 1 will print all mallocs/frees. "<br class=""> "If >= 2 will also print stack traces.")<br class=""> FUZZER_FLAG_INT(rss_limit_mb, 2048, "If non-zero, the fuzzer will exit upon"<br class=""><br class="">Modified: compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h<br class="">URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h?rev=316382&r1=316381&r2=316382&view=diff" class="">http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h?rev=316382&r1=316381&r2=316382&view=diff</a><br class="">==============================================================================<br class="">--- compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h (original)<br class="">+++ compiler-rt/trunk/lib/fuzzer/FuzzerInternal.h Mon Oct 23 15:04:30 2017<br class="">@@ -96,6 +96,7 @@ private:<br class=""> void CrashOnOverwrittenData();<br class=""> void InterruptCallback();<br class=""> void MutateAndTestOne();<br class="">+ void PurgeAllocator();<br class=""> void ReportNewCoverage(InputInfo *II, const Unit &U);<br class=""> void PrintPulseAndReportSlowInput(const uint8_t *Data, size_t Size);<br class=""> void WriteToOutputCorpus(const Unit &U);<br class="">@@ -124,6 +125,8 @@ private:<br class=""> bool HasMoreMallocsThanFrees = false;<br class=""> size_t NumberOfLeakDetectionAttempts = 0;<br class=""><br class="">+ system_clock::time_point LastAllocatorPurgeAttemptTime = system_clock::now();<br class="">+<br class=""> UserCallback CB;<br class=""> InputCorpus &Corpus;<br class=""> MutationDispatcher &MD;<br class=""><br class="">Modified: compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp<br class="">URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp?rev=316382&r1=316381&r2=316382&view=diff" class="">http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp?rev=316382&r1=316381&r2=316382&view=diff</a><br class="">==============================================================================<br class="">--- compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp (original)<br class="">+++ compiler-rt/trunk/lib/fuzzer/FuzzerLoop.cpp Mon Oct 23 15:04:30 2017<br class="">@@ -587,7 +587,7 @@ void Fuzzer::MutateAndTestOne() {<br class=""> size_t NewSize = 0;<br class=""> NewSize = MD.Mutate(CurrentUnitData, Size, CurrentMaxMutationLen);<br class=""> assert(NewSize > 0 && "Mutator returned empty unit");<br class="">- assert(NewSize <= CurrentMaxMutationLen && "Mutator return overisized unit");<br class="">+ assert(NewSize <= CurrentMaxMutationLen && "Mutator return oversized unit");<br class=""> Size = NewSize;<br class=""> II.NumExecutedMutations++;<br class=""> if (RunOne(CurrentUnitData, Size, /*MayDeleteFile=*/true, &II))<br class="">@@ -598,6 +598,25 @@ void Fuzzer::MutateAndTestOne() {<br class=""> }<br class=""> }<br class=""><br class="">+void Fuzzer::PurgeAllocator() {<br class="">+ if (Options.PurgeAllocatorIntervalSec < 0 ||<br class="">+ !EF->__sanitizer_purge_allocator) {<br class="">+ return;<br class="">+ }<br class="">+ if (duration_cast<seconds>(system_clock::now() -<br class="">+ LastAllocatorPurgeAttemptTime).count() <<br class="">+ Options.PurgeAllocatorIntervalSec) {<br class="">+ return;<br class="">+ }<br class="">+<br class="">+ if (Options.RssLimitMb <= 0 ||<br class="">+ GetPeakRSSMb() > static_cast<size_t>(Options.RssLimitMb) / 2) {<br class="">+ EF->__sanitizer_purge_allocator();<br class="">+ }<br class="">+<br class="">+ LastAllocatorPurgeAttemptTime = system_clock::now();<br class="">+}<br class="">+<br class=""> void Fuzzer::ReadAndExecuteSeedCorpora(const Vector<std::string> &CorpusDirs) {<br class=""> const size_t kMaxSaneLen = 1 << 20;<br class=""> const size_t kMinDefaultLen = 4096;<br class="">@@ -699,6 +718,8 @@ void Fuzzer::Loop(const Vector<std::stri<br class=""><br class=""> // Perform several mutations and runs.<br class=""> MutateAndTestOne();<br class="">+<br class="">+ PurgeAllocator();<br class=""> }<br class=""><br class=""> PrintStats("DONE ", "\n");<br class=""><br class="">Modified: compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h<br class="">URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h?rev=316382&r1=316381&r2=316382&view=diff" class="">http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h?rev=316382&r1=316381&r2=316382&view=diff</a><br class="">==============================================================================<br class="">--- compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h (original)<br class="">+++ compiler-rt/trunk/lib/fuzzer/FuzzerOptions.h Mon Oct 23 15:04:30 2017<br class="">@@ -54,6 +54,7 @@ struct FuzzingOptions {<br class=""> bool DumpCoverage = false;<br class=""> bool UseClangCoverage = false;<br class=""> bool DetectLeaks = true;<br class="">+ int PurgeAllocatorIntervalSec = 1;<br class=""> int UseFeatureFrequency = false;<br class=""> int TraceMalloc = 0;<br class=""> bool HandleAbrt = false;<br class=""><br class=""><br class="">_______________________________________________<br class="">llvm-commits mailing list<br class=""><a href="mailto:llvm-commits@lists.llvm.org" class="">llvm-commits@lists.llvm.org</a><br class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits<br class=""></div></div></blockquote></div><br class=""></div></body></html>