[llvm-dev] libfuzzer questions

Tue Aug 11 16:58:31 PDT 2015

On Mon, Aug 10, 2015 at 8:08 PM, Kostya Serebryany <kcc at google.com> wrote:

>
>
> On Mon, Aug 10, 2015 at 5:53 PM, Brian Cain via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>>
>> First off, thanks -- this is a pretty great library and it feels like I'm
>> learning a lot.
>>
>
> Thanks!
>
>
>> I'm getting some more experience with libfuzzer and finding that I have a
>> couple of questions:
>>
>
>
>>
>> - How does libfuzzer decide to write a new test file?  What distinguishes
>> this one from all the other cases for which new test inputs were not
>> written?  Must be something about the path taken through the code?
>>
>
> Exactly.
> It uses http://clang.llvm.org/docs/SanitizerCoverage.html to figure out
> if any new edge in the control flow graph has been discovered with the
> given input.
>
>

So if I'm seeing tens of thousands of distinct test files, that represents
tens of thousands of distinct edges?  Does the CFG span functions/methods
or are they scoped more sanely?

>
>> - Can I use afl-cmin or is there something similar for libFuzzer?
>>
>
> I've never tried that. I'd expect you can.
> libFuzzer and afl both use plain files to store the corpus.
>
>
I think afl-cmin uses some afl-specific behavior.

> I find that sometimes I get an enormous amount of tests and it becomes
>> unmanageable.
>>
>
> libFuzzer has an option to minimize the corpus.
> It's not perfect, but very simple.
> -------------
>  save_minimized_corpus               0 If 1, the minimized corpus is
> saved into the first input directory
> -------------
>
>>
Ohh, ok.  I think I misunderstood this to trying to minimize the size of
the test case while still reproducing a crash.  Similar to how afl-tmin
works, I was thinking.  I'll give this a try.

Should I only use this option periodically or can I run it this way all the
time?  Do we end up spending more execution time minimizing the corpus?
Will it delete redundant test cases, including ones that were there before
this test run started?

>
>> - sometimes my process being tested appears to deadlock.  A common
>> feature seems to be that AlarmCallback is allocating memory and as a
>> consequence the ASan code is pending on a lock.  I'll speculate that this
>> is because the alarm expired while the lock was already held.  Is this
>> expected?  I can share specific call stacks if it helps.  I can just extend
>> the timeout but I think it's probably appropriate.
>>
>
> Yes, please give more details.
>
>

Traces attached.  Not sure if the mailing list will preserve the
attachments, though.

>
>> - AFL has a curses based display where a bunch of different stats are
>> shown.  I'll be honest, I don't know how to read those yet. ;)  But I'd
>> like to find some way to determine whether I'm seeing diminishing returns
>> with libfuzzer.  Is there a good strategy?
>>
>
> libFuzzer just dumps stats to stderr.
> As long as you periodically see lines like
> #325 NEW    cov 11985 bits 14108 units 113 exec/s 325 ...
> you are good.
>
> Once you stop getting those, you may start playing with the flags.
> (e.g. increase the max_len).
> Unlike AFL which knows it all, libFuzzer still relies on a bit of user
> help. :)
>
>
Ok, that's good advice.

-- 
-Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150811/0323f018/attachment.html>
-------------- next part --------------
#0  atomic_exchange<__sanitizer::atomic_uint32_t> (mo=__sanitizer::memory_order_acquire, v=2, a=0x640000001290)
#1  __sanitizer::BlockingMutex::Lock (this=this at entry=0x640000001290) at /home/brian/tmp/testing/llvm_src/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_linux.cc:471
#2  0x0000000000447899 in GenericScopedLock (mu=mu at entry=0x640000001290, this=<synthetic pointer>)
#3  __sanitizer::SizeClassAllocator64<105553116266496ul, 4398046511104ul, 0ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>, __asan::AsanMapUnmapCallback>::PopulateFreeList (
#4  0x0000000000447ff8 in __sanitizer::SizeClassAllocator64<105553116266496ul, 4398046511104ul, 0ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>, __asan::AsanMapUnmapCallback>::AllocateBatch (this=this at entry=0x192db80 <__asan::instance>, stat=stat at entry=0x7f76e1bf2c18, c=c at entry=0x7f76e1bd80c8, class_id=class_id at entry=33)
#5  0x0000000000448065 in __sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator64<105553116266496ul, 4398046511104ul, 0ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>, __asan::AsanMapUnmapCallback> >::Refill (this=0x7f76e1bd80c8, allocator=allocator at entry=0x192db80 <__asan::instance>, class_id=class_id at entry=33)
#6  0x000000000044747e in Allocate (class_id=33, allocator=0x192db80 <__asan::instance>, this=0x7f76e1bd80c8)
#7  Allocate (check_rss_limit=true, cleared=false, alignment=8, size=<optimized out>, cache=0x7f76e1bd80c8, this=0x192db80 <__asan::instance>)
#8  Allocate (can_fill=true, alloc_type=__asan::FROM_MALLOC, stack=0x7fffd956a360, alignment=8, size=<optimized out>, this=0x192db80 <__asan::instance>)
#9  __asan::asan_malloc (size=size at entry=4096, stack=stack at entry=0x7fffd956a360) at /home/brian/tmp/testing/llvm_src/llvm/projects/compiler-rt/lib/asan/asan_allocator.cc:718
#10 0x00000000004da852 in __interceptor_malloc (size=4096) at /home/brian/tmp/testing/llvm_src/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:41
#11 0x00007f76e14dfc65 in operator new(unsigned long) () from /home/brian/tmp/testing/testing_install//lib/libc++.so.1
#12 0x00007f76e14dfd05 in operator new[](unsigned long) () from /home/brian/tmp/testing/testing_install//lib/libc++.so.1
#13 0x000000000054bd70 in std::__1::basic_filebuf<char, std::__1::char_traits<char> >::setbuf (this=0x7fffd956b268, __s=0x0, __n=4096)
#14 0x00000000005562be in std::__1::basic_filebuf<char, std::__1::char_traits<char> >::basic_filebuf (this=0x7fffd956b268)
#15 0x00000000005409e5 in basic_ofstream (this=0x7fffd956b260, __s=..., __mode=16) at /home/brian/tmp/testing/testing_install/bin/../include/c++/v1/fstream:1166
#16 fuzzer::WriteToFile (U=..., Path=...) at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerIO.cpp:69
#17 0x0000000000579636 in fuzzer::Fuzzer::WriteToCrash (this=0x7fffd956f440, U=..., Prefix=0x1435ec0 <.str.11> "timeout-")
#18 0x000000000057a89a in fuzzer::Fuzzer::AlarmCallback (this=0x7fffd956f440) at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerLoop.cpp:73
#19 0x0000000000579b97 in fuzzer::Fuzzer::StaticAlarmCallback () at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerLoop.cpp:57
#20 0x0000000000572355 in fuzzer::AlarmHandler () at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerUtil.cpp:49
#21 <signal handler called>
#22 __sanitizer::internal_mmap (addr=0x621000f20000, length=65536, prot=prot at entry=3, flags=flags at entry=50, fd=fd at entry=-1, offset=offset at entry=0)
#23 0x00000000004edbf8 in __sanitizer::MmapFixedOrDie (fixed_addr=fixed_addr at entry=107820874858496, size=size at entry=65536)
#24 0x00000000004479b1 in MapWithCallback (this=0x192db80 <__asan::instance>, size=65536, beg=107820874858496)
#25 __sanitizer::SizeClassAllocator64<105553116266496ul, 4398046511104ul, 0ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>, __asan::AsanMapUnmapCallback>::PopulateFreeList (
#26 0x0000000000447ff8 in __sanitizer::SizeClassAllocator64<105553116266496ul, 4398046511104ul, 0ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>, __asan::AsanMapUnmapCallback>::AllocateBatch (this=this at entry=0x192db80 <__asan::instance>, stat=stat at entry=0x7f76e1bf2c18, c=c at entry=0x7f76e1bd80c8, class_id=class_id at entry=33)
#27 0x0000000000448065 in __sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator64<105553116266496ul, 4398046511104ul, 0ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>, __asan::AsanMapUnmapCallback> >::Refill (this=0x7f76e1bd80c8, allocator=allocator at entry=0x192db80 <__asan::instance>, class_id=class_id at entry=33)
#28 0x000000000044747e in Allocate (class_id=33, allocator=0x192db80 <__asan::instance>, this=0x7f76e1bd80c8)
#29 Allocate (check_rss_limit=true, cleared=false, alignment=8, size=<optimized out>, cache=0x7f76e1bd80c8, this=0x192db80 <__asan::instance>)
#30 Allocate (can_fill=true, alloc_type=__asan::FROM_MALLOC, stack=0x7fffd956c5c0, alignment=8, size=<optimized out>, this=0x192db80 <__asan::instance>)
#31 __asan::asan_malloc (size=size at entry=4096, stack=stack at entry=0x7fffd956c5c0) at /home/brian/tmp/testing/llvm_src/llvm/projects/compiler-rt/lib/asan/asan_allocator.cc:718
#32 0x00000000004da852 in __interceptor_malloc (size=4096) at /home/brian/tmp/testing/llvm_src/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:41
#33 0x00007f76e14dfc65 in operator new(unsigned long) () from /home/brian/tmp/testing/testing_install//lib/libc++.so.1
#34 0x00007f76e14dfd05 in operator new[](unsigned long) () from /home/brian/tmp/testing/testing_install//lib/libc++.so.1
#35 0x000000000054bd70 in std::__1::basic_filebuf<char, std::__1::char_traits<char> >::setbuf (this=0x7fffd956d4d0, __s=0x0, __n=4096)
#36 0x00000000005562be in std::__1::basic_filebuf<char, std::__1::char_traits<char> >::basic_filebuf (this=0x7fffd956d4d0)
#37 0x000000000053ddc0 in basic_ifstream (this=0x7fffd956d4c0, __s=..., __mode=8) at /home/brian/tmp/testing/testing_install/bin/../include/c++/v1/fstream:1014
#38 fuzzer::FileToVector (Path=...) at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerIO.cpp:53
#39 0x00000000005421e4 in fuzzer::ReadDirToVectorOfUnits (Path=0x60300000eef0 "tests/testplist/inputs/", V=0x7fffd956e5e0, Epoch=0x7fffd956f580)
#40 0x000000000057b99e in fuzzer::Fuzzer::RereadOutputCorpus (this=0x7fffd956f440) at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerLoop.cpp:89
#41 0x000000000050b9fc in fuzzer::FuzzerDriver (argc=7, argv=0x7fffd95703e8, USF=...) at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerDriver.cpp:273
#42 0x00000000005095fc in fuzzer::FuzzerDriver (argc=7, argv=0x7fffd95703e8, Callback=0x5d0880 <LLVMFuzzerTestOneInput(unsigned char const*, unsigned long)>)
#43 0x00000000005598cf in main (argc=7, argv=0x7fffd95703e8) at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerMain.cpp:19
-------------- next part --------------
#0  atomic_exchange<__sanitizer::atomic_uint32_t> (mo=__sanitizer::memory_order_acquire, v=2, a=0x640000001290)
#1  __sanitizer::BlockingMutex::Lock (this=this at entry=0x640000001290) at /home/brian/tmp/testing/llvm_src/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_linux.cc:471
#2  0x0000000000447899 in GenericScopedLock (mu=mu at entry=0x640000001290, this=<synthetic pointer>)
#3  __sanitizer::SizeClassAllocator64<105553116266496ul, 4398046511104ul, 0ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>, __asan::AsanMapUnmapCallback>::PopulateFreeList (
#4  0x0000000000447ff8 in __sanitizer::SizeClassAllocator64<105553116266496ul, 4398046511104ul, 0ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>, __asan::AsanMapUnmapCallback>::AllocateBatch (this=this at entry=0x192db40 <__asan::instance>, stat=stat at entry=0x7ffc91f06c18, c=c at entry=0x7ffc91eec0c8, class_id=class_id at entry=33)
#5  0x0000000000448065 in __sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator64<105553116266496ul, 4398046511104ul, 0ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>, __asan::AsanMapUnmapCallback> >::Refill (this=0x7ffc91eec0c8, allocator=allocator at entry=0x192db40 <__asan::instance>, class_id=class_id at entry=33)
#6  0x000000000044747e in Allocate (class_id=33, allocator=0x192db40 <__asan::instance>, this=0x7ffc91eec0c8)
#7  Allocate (check_rss_limit=true, cleared=false, alignment=8, size=<optimized out>, cache=0x7ffc91eec0c8, this=0x192db40 <__asan::instance>)
#8  Allocate (can_fill=true, alloc_type=__asan::FROM_MALLOC, stack=0x7fffd58669a0, alignment=8, size=<optimized out>, this=0x192db40 <__asan::instance>)
#9  __asan::asan_malloc (size=size at entry=4096, stack=stack at entry=0x7fffd58669a0) at /home/brian/tmp/testing/llvm_src/llvm/projects/compiler-rt/lib/asan/asan_allocator.cc:718
#10 0x00000000004da852 in __interceptor_malloc (size=4096) at /home/brian/tmp/testing/llvm_src/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:41
#11 0x00007ffc917f3c65 in operator new(unsigned long) () from /home/brian/tmp/testing/testing_install//lib/libc++.so.1
#12 0x00007ffc917f3d05 in operator new[](unsigned long) () from /home/brian/tmp/testing/testing_install//lib/libc++.so.1
#13 0x000000000054bd70 in std::__1::basic_filebuf<char, std::__1::char_traits<char> >::setbuf (this=0x7fffd58678a8, __s=0x0, __n=4096)
#14 0x00000000005562be in std::__1::basic_filebuf<char, std::__1::char_traits<char> >::basic_filebuf (this=0x7fffd58678a8)
#15 0x00000000005409e5 in basic_ofstream (this=0x7fffd58678a0, __s=..., __mode=16) at /home/brian/tmp/testing/testing_install/bin/../include/c++/v1/fstream:1166
#16 fuzzer::WriteToFile (U=..., Path=...) at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerIO.cpp:69
#17 0x0000000000579636 in fuzzer::Fuzzer::WriteToCrash (this=0x7fffd586baa0, U=..., Prefix=0x1435ec0 <.str.11> "timeout-")
#18 0x000000000057a89a in fuzzer::Fuzzer::AlarmCallback (this=0x7fffd586baa0) at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerLoop.cpp:73
#19 0x0000000000579b97 in fuzzer::Fuzzer::StaticAlarmCallback () at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerLoop.cpp:57
#20 0x0000000000572355 in fuzzer::AlarmHandler () at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerUtil.cpp:49
#21 <signal handler called>
#22 __sanitizer::internal_mmap (addr=0x621001740000, length=65536, prot=prot at entry=3, flags=flags at entry=50, fd=fd at entry=-1, offset=offset at entry=0)
#23 0x00000000004edbf8 in __sanitizer::MmapFixedOrDie (fixed_addr=fixed_addr at entry=107820883378176, size=size at entry=65536)
#24 0x00000000004479b1 in MapWithCallback (this=0x192db40 <__asan::instance>, size=65536, beg=107820883378176)
#25 __sanitizer::SizeClassAllocator64<105553116266496ul, 4398046511104ul, 0ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>, __asan::AsanMapUnmapCallback>::PopulateFreeList (
#26 0x0000000000447ff8 in __sanitizer::SizeClassAllocator64<105553116266496ul, 4398046511104ul, 0ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>, __asan::AsanMapUnmapCallback>::AllocateBatch (this=this at entry=0x192db40 <__asan::instance>, stat=stat at entry=0x7ffc91f06c18, c=c at entry=0x7ffc91eec0c8, class_id=class_id at entry=33)
#27 0x0000000000448065 in __sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator64<105553116266496ul, 4398046511104ul, 0ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>, __asan::AsanMapUnmapCallback> >::Refill (this=0x7ffc91eec0c8, allocator=allocator at entry=0x192db40 <__asan::instance>, class_id=class_id at entry=33)
#28 0x000000000044747e in Allocate (class_id=33, allocator=0x192db40 <__asan::instance>, this=0x7ffc91eec0c8)
#29 Allocate (check_rss_limit=true, cleared=false, alignment=8, size=<optimized out>, cache=0x7ffc91eec0c8, this=0x192db40 <__asan::instance>)
#30 Allocate (can_fill=true, alloc_type=__asan::FROM_MALLOC, stack=0x7fffd5868c20, alignment=8, size=<optimized out>, this=0x192db40 <__asan::instance>)
#31 __asan::asan_malloc (size=size at entry=4096, stack=stack at entry=0x7fffd5868c20) at /home/brian/tmp/testing/llvm_src/llvm/projects/compiler-rt/lib/asan/asan_allocator.cc:718
#32 0x00000000004da852 in __interceptor_malloc (size=4096) at /home/brian/tmp/testing/llvm_src/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:41
#33 0x00007ffc917f3c65 in operator new(unsigned long) () from /home/brian/tmp/testing/testing_install//lib/libc++.so.1
#34 0x00007ffc917f3d05 in operator new[](unsigned long) () from /home/brian/tmp/testing/testing_install//lib/libc++.so.1
#35 0x000000000054bd70 in std::__1::basic_filebuf<char, std::__1::char_traits<char> >::setbuf (this=0x7fffd5869b30, __s=0x0, __n=4096)
#36 0x00000000005562be in std::__1::basic_filebuf<char, std::__1::char_traits<char> >::basic_filebuf (this=0x7fffd5869b30)
#37 0x000000000053ddc0 in basic_ifstream (this=0x7fffd5869b20, __s=..., __mode=8) at /home/brian/tmp/testing/testing_install/bin/../include/c++/v1/fstream:1014
#38 fuzzer::FileToVector (Path=...) at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerIO.cpp:53
#39 0x00000000005421e4 in fuzzer::ReadDirToVectorOfUnits (Path=0x7fffd586bb79 "tests/testbz2/inputs/", V=0x7fffd586ac40, Epoch=0x7fffd586bbe0)
#40 0x000000000057b99e in fuzzer::Fuzzer::RereadOutputCorpus (this=0x7fffd586baa0) at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerLoop.cpp:89
#41 0x000000000050b9fc in fuzzer::FuzzerDriver (argc=7, argv=0x7fffd586ca48, USF=...) at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerDriver.cpp:273
#42 0x00000000005095fc in fuzzer::FuzzerDriver (argc=7, argv=0x7fffd586ca48, Callback=0x5d0880 <LLVMFuzzerTestOneInput(unsigned char const*, unsigned long)>)
#43 0x00000000005598cf in main (argc=7, argv=0x7fffd586ca48) at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerMain.cpp:19
-------------- next part --------------
#0  atomic_exchange<__sanitizer::atomic_uint32_t> (mo=__sanitizer::memory_order_acquire, v=2, a=0x640000001290)
#1  __sanitizer::BlockingMutex::Lock (this=this at entry=0x640000001290) at /home/brian/tmp/testing/llvm_src/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_linux.cc:471
#2  0x0000000000447899 in GenericScopedLock (mu=mu at entry=0x640000001290, this=<synthetic pointer>)
#3  __sanitizer::SizeClassAllocator64<105553116266496ul, 4398046511104ul, 0ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>, __asan::AsanMapUnmapCallback>::PopulateFreeList (
#4  0x0000000000447ff8 in __sanitizer::SizeClassAllocator64<105553116266496ul, 4398046511104ul, 0ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>, __asan::AsanMapUnmapCallback>::AllocateBatch (this=this at entry=0x192db80 <__asan::instance>, stat=stat at entry=0x7fae5eddbc18, c=c at entry=0x7fae5edc10c8, class_id=class_id at entry=33)
#5  0x0000000000448065 in __sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator64<105553116266496ul, 4398046511104ul, 0ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>, __asan::AsanMapUnmapCallback> >::Refill (this=0x7fae5edc10c8, allocator=allocator at entry=0x192db80 <__asan::instance>, class_id=class_id at entry=33)
#6  0x000000000044747e in Allocate (class_id=33, allocator=0x192db80 <__asan::instance>, this=0x7fae5edc10c8)
#7  Allocate (check_rss_limit=true, cleared=false, alignment=8, size=<optimized out>, cache=0x7fae5edc10c8, this=0x192db80 <__asan::instance>)
#8  Allocate (can_fill=true, alloc_type=__asan::FROM_MALLOC, stack=0x7fff6db80e60, alignment=8, size=<optimized out>, this=0x192db80 <__asan::instance>)
#9  __asan::asan_malloc (size=size at entry=4096, stack=stack at entry=0x7fff6db80e60) at /home/brian/tmp/testing/llvm_src/llvm/projects/compiler-rt/lib/asan/asan_allocator.cc:718
#10 0x00000000004da852 in __interceptor_malloc (size=4096) at /home/brian/tmp/testing/llvm_src/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:41
#11 0x00007fae5e6c8c65 in operator new(unsigned long) () from /home/brian/tmp/testing/testing_install//lib/libc++.so.1
#12 0x00007fae5e6c8d05 in operator new[](unsigned long) () from /home/brian/tmp/testing/testing_install//lib/libc++.so.1
#13 0x000000000054bd70 in std::__1::basic_filebuf<char, std::__1::char_traits<char> >::setbuf (this=0x7fff6db81d68, __s=0x0, __n=4096)
#14 0x00000000005562be in std::__1::basic_filebuf<char, std::__1::char_traits<char> >::basic_filebuf (this=0x7fff6db81d68)
#15 0x00000000005409e5 in basic_ofstream (this=0x7fff6db81d60, __s=..., __mode=16) at /home/brian/tmp/testing/testing_install/bin/../include/c++/v1/fstream:1166
#16 fuzzer::WriteToFile (U=..., Path=...) at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerIO.cpp:69
#17 0x0000000000579636 in fuzzer::Fuzzer::WriteToCrash (this=0x7fff6db85f00, U=..., Prefix=0x1435ec0 <.str.11> "timeout-")
#18 0x000000000057a89a in fuzzer::Fuzzer::AlarmCallback (this=0x7fff6db85f00) at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerLoop.cpp:73
#19 0x0000000000579b97 in fuzzer::Fuzzer::StaticAlarmCallback () at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerLoop.cpp:57
#20 0x0000000000572355 in fuzzer::AlarmHandler () at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerUtil.cpp:49
#21 <signal handler called>
#22 memset () at ../sysdeps/x86_64/memset.S:80
#23 0x00000000004479dc in OnMap (size=65536, p=<optimized out>, this=<optimized out>) at /home/brian/tmp/testing/llvm_src/llvm/projects/compiler-rt/lib/asan/asan_allocator.cc:177
#24 MapWithCallback (this=0x192db80 <__asan::instance>, size=65536, beg=<optimized out>)
#25 __sanitizer::SizeClassAllocator64<105553116266496ul, 4398046511104ul, 0ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>, __asan::AsanMapUnmapCallback>::PopulateFreeList (
#26 0x0000000000447ff8 in __sanitizer::SizeClassAllocator64<105553116266496ul, 4398046511104ul, 0ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>, __asan::AsanMapUnmapCallback>::AllocateBatch (this=this at entry=0x192db80 <__asan::instance>, stat=stat at entry=0x7fae5eddbc18, c=c at entry=0x7fae5edc10c8, class_id=class_id at entry=33)
#27 0x0000000000448065 in __sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator64<105553116266496ul, 4398046511104ul, 0ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>, __asan::AsanMapUnmapCallback> >::Refill (this=0x7fae5edc10c8, allocator=allocator at entry=0x192db80 <__asan::instance>, class_id=class_id at entry=33)
#28 0x000000000044747e in Allocate (class_id=33, allocator=0x192db80 <__asan::instance>, this=0x7fae5edc10c8)
#29 Allocate (check_rss_limit=true, cleared=false, alignment=8, size=<optimized out>, cache=0x7fae5edc10c8, this=0x192db80 <__asan::instance>)
#30 Allocate (can_fill=true, alloc_type=__asan::FROM_MALLOC, stack=0x7fff6db83080, alignment=8, size=<optimized out>, this=0x192db80 <__asan::instance>)
#31 __asan::asan_malloc (size=size at entry=4096, stack=stack at entry=0x7fff6db83080) at /home/brian/tmp/testing/llvm_src/llvm/projects/compiler-rt/lib/asan/asan_allocator.cc:718
#32 0x00000000004da852 in __interceptor_malloc (size=4096) at /home/brian/tmp/testing/llvm_src/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:41
#33 0x00007fae5e6c8c65 in operator new(unsigned long) () from /home/brian/tmp/testing/testing_install//lib/libc++.so.1
#34 0x00007fae5e6c8d05 in operator new[](unsigned long) () from /home/brian/tmp/testing/testing_install//lib/libc++.so.1
#35 0x000000000054bd70 in std::__1::basic_filebuf<char, std::__1::char_traits<char> >::setbuf (this=0x7fff6db83f90, __s=0x0, __n=4096)
#36 0x00000000005562be in std::__1::basic_filebuf<char, std::__1::char_traits<char> >::basic_filebuf (this=0x7fff6db83f90)
#37 0x000000000053ddc0 in basic_ifstream (this=0x7fff6db83f80, __s=..., __mode=8) at /home/brian/tmp/testing/testing_install/bin/../include/c++/v1/fstream:1014
#38 fuzzer::FileToVector (Path=...) at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerIO.cpp:53
#39 0x00000000005421e4 in fuzzer::ReadDirToVectorOfUnits (Path=0x60300000eef0 "tests/testplist/inputs/", V=0x7fff6db850a0, Epoch=0x7fff6db86040)
#40 0x000000000057b99e in fuzzer::Fuzzer::RereadOutputCorpus (this=0x7fff6db85f00) at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerLoop.cpp:89
#41 0x000000000050b9fc in fuzzer::FuzzerDriver (argc=7, argv=0x7fff6db86ea8, USF=...) at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerDriver.cpp:273
#42 0x00000000005095fc in fuzzer::FuzzerDriver (argc=7, argv=0x7fff6db86ea8, Callback=0x5d0880 <LLVMFuzzerTestOneInput(unsigned char const*, unsigned long)>)
#43 0x00000000005598cf in main (argc=7, argv=0x7fff6db86ea8) at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerMain.cpp:19
#0  atomic_exchange<__sanitizer::atomic_uint32_t> (mo=__sanitizer::memory_order_acquire, v=2, a=0x640000001290)
#1  __sanitizer::BlockingMutex::Lock (this=this at entry=0x640000001290) at /home/brian/tmp/testing/llvm_src/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_linux.cc:471
#2  0x0000000000447899 in GenericScopedLock (mu=mu at entry=0x640000001290, this=<synthetic pointer>)
#3  __sanitizer::SizeClassAllocator64<105553116266496ul, 4398046511104ul, 0ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>, __asan::AsanMapUnmapCallback>::PopulateFreeList (
#4  0x0000000000447ff8 in __sanitizer::SizeClassAllocator64<105553116266496ul, 4398046511104ul, 0ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>, __asan::AsanMapUnmapCallback>::AllocateBatch (this=this at entry=0x192db80 <__asan::instance>, stat=stat at entry=0x7fae5eddbc18, c=c at entry=0x7fae5edc10c8, class_id=class_id at entry=33)
#5  0x0000000000448065 in __sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator64<105553116266496ul, 4398046511104ul, 0ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>, __asan::AsanMapUnmapCallback> >::Refill (this=0x7fae5edc10c8, allocator=allocator at entry=0x192db80 <__asan::instance>, class_id=class_id at entry=33)
#6  0x000000000044747e in Allocate (class_id=33, allocator=0x192db80 <__asan::instance>, this=0x7fae5edc10c8)
#7  Allocate (check_rss_limit=true, cleared=false, alignment=8, size=<optimized out>, cache=0x7fae5edc10c8, this=0x192db80 <__asan::instance>)
#8  Allocate (can_fill=true, alloc_type=__asan::FROM_MALLOC, stack=0x7fff6db80e60, alignment=8, size=<optimized out>, this=0x192db80 <__asan::instance>)
#9  __asan::asan_malloc (size=size at entry=4096, stack=stack at entry=0x7fff6db80e60) at /home/brian/tmp/testing/llvm_src/llvm/projects/compiler-rt/lib/asan/asan_allocator.cc:718
#10 0x00000000004da852 in __interceptor_malloc (size=4096) at /home/brian/tmp/testing/llvm_src/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:41
#11 0x00007fae5e6c8c65 in operator new(unsigned long) () from /home/brian/tmp/testing/testing_install//lib/libc++.so.1
#12 0x00007fae5e6c8d05 in operator new[](unsigned long) () from /home/brian/tmp/testing/testing_install//lib/libc++.so.1
#13 0x000000000054bd70 in std::__1::basic_filebuf<char, std::__1::char_traits<char> >::setbuf (this=0x7fff6db81d68, __s=0x0, __n=4096)
#14 0x00000000005562be in std::__1::basic_filebuf<char, std::__1::char_traits<char> >::basic_filebuf (this=0x7fff6db81d68)
#15 0x00000000005409e5 in basic_ofstream (this=0x7fff6db81d60, __s=..., __mode=16) at /home/brian/tmp/testing/testing_install/bin/../include/c++/v1/fstream:1166
#16 fuzzer::WriteToFile (U=..., Path=...) at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerIO.cpp:69
#17 0x0000000000579636 in fuzzer::Fuzzer::WriteToCrash (this=0x7fff6db85f00, U=..., Prefix=0x1435ec0 <.str.11> "timeout-")
#18 0x000000000057a89a in fuzzer::Fuzzer::AlarmCallback (this=0x7fff6db85f00) at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerLoop.cpp:73
#19 0x0000000000579b97 in fuzzer::Fuzzer::StaticAlarmCallback () at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerLoop.cpp:57
#20 0x0000000000572355 in fuzzer::AlarmHandler () at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerUtil.cpp:49
#21 <signal handler called>
#22 memset () at ../sysdeps/x86_64/memset.S:80
#23 0x00000000004479dc in OnMap (size=65536, p=<optimized out>, this=<optimized out>) at /home/brian/tmp/testing/llvm_src/llvm/projects/compiler-rt/lib/asan/asan_allocator.cc:177
#24 MapWithCallback (this=0x192db80 <__asan::instance>, size=65536, beg=<optimized out>)
#25 __sanitizer::SizeClassAllocator64<105553116266496ul, 4398046511104ul, 0ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>, __asan::AsanMapUnmapCallback>::PopulateFreeList (
#26 0x0000000000447ff8 in __sanitizer::SizeClassAllocator64<105553116266496ul, 4398046511104ul, 0ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>, __asan::AsanMapUnmapCallback>::AllocateBatch (this=this at entry=0x192db80 <__asan::instance>, stat=stat at entry=0x7fae5eddbc18, c=c at entry=0x7fae5edc10c8, class_id=class_id at entry=33)
#27 0x0000000000448065 in __sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator64<105553116266496ul, 4398046511104ul, 0ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>, __asan::AsanMapUnmapCallback> >::Refill (this=0x7fae5edc10c8, allocator=allocator at entry=0x192db80 <__asan::instance>, class_id=class_id at entry=33)
#28 0x000000000044747e in Allocate (class_id=33, allocator=0x192db80 <__asan::instance>, this=0x7fae5edc10c8)
#29 Allocate (check_rss_limit=true, cleared=false, alignment=8, size=<optimized out>, cache=0x7fae5edc10c8, this=0x192db80 <__asan::instance>)
#30 Allocate (can_fill=true, alloc_type=__asan::FROM_MALLOC, stack=0x7fff6db83080, alignment=8, size=<optimized out>, this=0x192db80 <__asan::instance>)
#31 __asan::asan_malloc (size=size at entry=4096, stack=stack at entry=0x7fff6db83080) at /home/brian/tmp/testing/llvm_src/llvm/projects/compiler-rt/lib/asan/asan_allocator.cc:718
#32 0x00000000004da852 in __interceptor_malloc (size=4096) at /home/brian/tmp/testing/llvm_src/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:41
#33 0x00007fae5e6c8c65 in operator new(unsigned long) () from /home/brian/tmp/testing/testing_install//lib/libc++.so.1
#34 0x00007fae5e6c8d05 in operator new[](unsigned long) () from /home/brian/tmp/testing/testing_install//lib/libc++.so.1
#35 0x000000000054bd70 in std::__1::basic_filebuf<char, std::__1::char_traits<char> >::setbuf (this=0x7fff6db83f90, __s=0x0, __n=4096)
#36 0x00000000005562be in std::__1::basic_filebuf<char, std::__1::char_traits<char> >::basic_filebuf (this=0x7fff6db83f90)
#37 0x000000000053ddc0 in basic_ifstream (this=0x7fff6db83f80, __s=..., __mode=8) at /home/brian/tmp/testing/testing_install/bin/../include/c++/v1/fstream:1014
#38 fuzzer::FileToVector (Path=...) at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerIO.cpp:53
#39 0x00000000005421e4 in fuzzer::ReadDirToVectorOfUnits (Path=0x60300000eef0 "tests/testplist/inputs/", V=0x7fff6db850a0, Epoch=0x7fff6db86040)
#40 0x000000000057b99e in fuzzer::Fuzzer::RereadOutputCorpus (this=0x7fff6db85f00) at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerLoop.cpp:89
#41 0x000000000050b9fc in fuzzer::FuzzerDriver (argc=7, argv=0x7fff6db86ea8, USF=...) at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerDriver.cpp:273
#42 0x00000000005095fc in fuzzer::FuzzerDriver (argc=7, argv=0x7fff6db86ea8, Callback=0x5d0880 <LLVMFuzzerTestOneInput(unsigned char const*, unsigned long)>)
#43 0x00000000005598cf in main (argc=7, argv=0x7fff6db86ea8) at /home/brian/tmp/testing/llvm_src//llvm/lib/Fuzzer/FuzzerMain.cpp:19