[llvm-dev] fuzzer crash (but not the good kind)

Thu Dec 3 12:46:59 PST 2015

On Thu, Dec 3, 2015 at 12:34 PM, Brian Cain <brian.cain at gmail.com> wrote:

> Oh I thought it was really recent (1-2 weeks behind ToT ).  I must have
> done something wrong there -- I'll check.
>
> Yes, it's extraordinarily slow
>
and it may do dlopen (), I'll find out.
>
> No, I don't *believe* it's using other threads but I will check.
>
> I would let it run for 24h before giving up on a crash though.
>

This makes little sense because with such a rare reproducer it's hard to
fix the bug.
So far my only idea is a race between libFuzzer's actions in the main
thread and dlopen in a non-main thread.
And this is not the code I want to keep long term, so I am reluctant to fix
it now. :(
And you have a workaround :)

> On Dec 3, 2015 2:15 PM, "Kostya Serebryany" <kcc at google.com> wrote:
>
>> I am able to build the fuzzer, but not to reproduce the crash.
>>
>> First: you seem to be using a not-the-most-recent libfuzzer source,
>> please update.
>> You will need to change LLVMFuzzerTestOneInput to return 0.
>>
>> Second, for some reason, the fuzzing is very slow, about 1 exec/s.
>> You probably want to fix that too.
>>
>> Third: does this code have threads? dlopen?
>> Running it for a short period does not show either, but probably the
>> fuzzer discovers some such input that triggers dlopen in a non-main thread.
>> I guess the sanitizer coverage has a race in this case.
>>
>> So, just drop 8bit-counters. (This flag is far from optimal, we may
>> actually remove it completely at some point).
>>
>> hth,
>> --kcc
>>
>>
>>
>>
>> On Thu, Dec 3, 2015 at 11:14 AM, Brian Cain <brian.cain at gmail.com> wrote:
>>
>>> Ah, yes -- you need to clone with --recursive.
>>>
>>> I will try the workaround though.
>>> On Dec 3, 2015 1:12 PM, "Kostya Serebryany" <kcc at google.com> wrote:
>>>
>>>>
>>>>
>>>> On Wed, Dec 2, 2015 at 7:17 PM, Brian Cain <brian.cain at gmail.com>
>>>> wrote:
>>>>
>>>>> Kostya,
>>>>>
>>>>> Here's the git repo: https://bitbucket.org/ebadf/fuzzpy
>>>>>
>>>>> I've only tested it on arm7 and x86_64 linux, I expect there's a good
>>>>> chance it may not work on other OSs.
>>>>>
>>>>> If you can build it successfully ("./build.sh", requires clang and
>>>>> clang++ in your path), then you should run the "testemail" case like so:
>>>>>
>>>>>
>>>> Does not build for me out of the box:
>>>>
>>>> ./build.sh: line 70: ./configure: No such file or directory
>>>>
>>>> I wonder if a smaller test possible here.
>>>> Meanwhile, here is a workaround for you.
>>>> Instead of
>>>> SANITIZE_COV_OPTS="-fsanitize-coverage=bb,indirect-calls,8bit-counters"
>>>> try using
>>>> SANITIZE_COV_OPTS="-fsanitize-coverage=edge,indirect-calls"
>>>>
>>>>
>>>> while true; do ITERS=1000 ./run.sh tests/build/testemail
>>>>> tests/testemail/inputs/; done
>>>>>
>>>>> Let me know if you have any challenges building or running the test
>>>>> case.
>>>>>
>>>>>
>>>>> On Tue, Dec 1, 2015 at 7:26 PM, Kostya Serebryany <kcc at google.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Brian,
>>>>>> Yes, looks like a bug in sanitizer coverage, please send the
>>>>>> reproducer.
>>>>>>
>>>>>> On Tue, Dec 1, 2015 at 5:22 PM, Brian Cain <brian.cain at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> Kostya,
>>>>>>>
>>>>>>> I think I've found what looks like a reproducible bug in libFuzzer.
>>>>>>> The code under test is built with ASan and the first ASan CHECK failure
>>>>>>> shows fuzzer in the stack trace.  (see below)
>>>>>>>
>>>>>>> One of the factors that may be unique in my testing is that each
>>>>>>> iteration can take a very long time to execute (tens or hundreds of
>>>>>>> seconds).
>>>>>>>
>>>>>>> Let me know if you need more info, I think it shouldn't take much
>>>>>>> test time to reproduce this.
>>>>>>>
>>>>>>> ================== Job 2 exited with exit code 256 ============
>>>>>>> Flag: verbosity 3
>>>>>>> Flag: use_traces 1
>>>>>>> Flag: timeout 100
>>>>>>> Flag: max_len 16384
>>>>>>> Seed: 3259211893
>>>>>>> PreferSmall: 0
>>>>>>> #0      READ   units: 4975 exec/s: 0
>>>>>>> #1      pulse  cov: 32410 bits: 30791 indir: 714 units: 4975 exec/s:
>>>>>>> 0
>>>>>>> NEW0: 32410 L 13869
>>>>>>> ==31301==AddressSanitizer CHECK failed:
>>>>>>> /home/brian/src/fuzzpy/llvm_src/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_coverage_libcdep.cc:467
>>>>>>> "((n % 16)) == ((0))" (0x1, 0x0)
>>>>>>>     #0 0x11d3b7 in __asan::AsanCheckFailed(char const*, int, char
>>>>>>> const*, unsigned long long, unsigned long long)
>>>>>>> /home/brian/src/fuzzpy/llvm_src/llvm/projects/compiler-rt/lib/asan/asan_rtl.cc:67:3
>>>>>>>     #1 0x122f1f in __sanitizer::CheckFailed(char const*, int, char
>>>>>>> const*, unsigned long long, unsigned long long)
>>>>>>> /home/brian/src/fuzzpy/llvm_src/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_common.cc:159:5
>>>>>>>     #2 0x134317 in
>>>>>>> __sanitizer::CoverageData::Update8bitCounterBitsetAndClearCounters(unsigned
>>>>>>> char*)
>>>>>>> /home/brian/src/fuzzpy/llvm_src/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_coverage_libcdep.cc:467:5
>>>>>>>     #3 0x1b7b53 in fuzzer::Fuzzer::PrepareCoverageBeforeRun()
>>>>>>> /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerLoop.cpp:264:5
>>>>>>>     #4 0x1b501b in fuzzer::Fuzzer::RunOne(std::vector<unsigned char,
>>>>>>> std::allocator<unsigned char> > const&)
>>>>>>> /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerLoop.cpp:212:3
>>>>>>>     #5 0x1b6be3 in fuzzer::Fuzzer::ShuffleAndMinimize()
>>>>>>> /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerLoop.cpp:195:11
>>>>>>>     #6 0x14477b in fuzzer::FuzzerDriver(std::vector<std::string,
>>>>>>> std::allocator<std::string> > const&, fuzzer::UserSuppliedFuzzer&)
>>>>>>> /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerDriver.cpp:303:3
>>>>>>>     #7 0x14183f in fuzzer::FuzzerDriver(int, char**,
>>>>>>> fuzzer::UserSuppliedFuzzer&)
>>>>>>> /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerDriver.cpp:201:10
>>>>>>>     #8 0x141427 in fuzzer::FuzzerDriver(int, char**, int
>>>>>>> (*)(unsigned char const*, unsigned int))
>>>>>>> /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerDriver.cpp:196:10
>>>>>>>     #9 0x1873e3 in main
>>>>>>> /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerMain.cpp:19:10
>>>>>>>     #10 0xb6c86775 in __libc_start_main
>>>>>>> /build/buildd/glibc-2.21/csu/libc-start.c:289
>>>>>>>
>>>>>>> DEATH:
>>>>>>> artifact_prefix='./'; Test unit written to
>>>>>>> ./crash-ec9fa023e9db127e2589d0ab4c506055e4174611
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> -Brian
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> -Brian
>>>>>
>>>>
>>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20151203/7a3b0cbe/attachment.html>