[llvm-dev] -sanitizer-coverage-prune-blocks=true and LibFuzzer

Wed Sep 21 13:25:19 PDT 2016

On Wed, Sep 21, 2016 at 12:58 PM, Mehdi Amini <mehdi.amini at apple.com> wrote:

>
> On Sep 21, 2016, at 12:56 PM, Kostya Serebryany <kcc at google.com> wrote:
>
>
>
> On Wed, Sep 21, 2016 at 12:32 PM, Mehdi Amini <mehdi.amini at apple.com>
> wrote:
>
>>
>> On Sep 21, 2016, at 9:36 AM, Kostya Serebryany via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>> Exciting!
>>
>> (btw, I'd prefer libfuzzer at googlegroups.com for such discussions, please
>> start new topics there)
>>
>>
>> You mean a LLVM library has a separate mailing-list? Why?
>>
>
> Because the topic is very separate.
>
>
> Can you clarify?
>
> I thought is was about the development/debug/evolution/usability of
> http://llvm.org/svn/llvm-project/llvm/trunk/lib/Fuzzer/
>

Yes, and this topic is substantially different from most other topics
discussed on llvm-dev.
We also have separate maliing lists for asan/tsan/msan

--kcc

>
> —
> Mehid
>
>
>
>
>
>
>>
>> —
>> Mehdi
>>
>>
>>
>> I can reproduce this too, but if i either increase FUZZER_TESTING_SECONDS
>> to 600 or change seed=1 to seed=2 the problem is gone.
>> Looks like one of the binaries got simply unlucky with a particular seed.
>> You can observe it like this:
>>
>> for S in 1 2 3 4 5 6; do ./target-asan-8bit-prune-build/fuzzer -seed=$S -runs=10000000 2>&1 | grep DONE &  done
>>
>> #10000000       DONE   cov: 60 bits: 91 indir: 1 units: 59 exec/s: 625000
>> #10000000       DONE   cov: 60 bits: 91 indir: 1 units: 57 exec/s: 588235
>> #10000000       DONE   cov: 253 bits: 901 indir: 12 units: 467 exec/s: 526315
>> #10000000       DONE   cov: 63 bits: 95 indir: 1 units: 64 exec/s: 476190
>> #10000000       DONE   cov: 252 bits: 923 indir: 12 units: 491 exec/s: 454545
>> #10000000       DONE   cov: 253 bits: 880 indir: 12 units: 471 exec/s: 384615
>>
>>
>> Similar things happen with other binaries:
>>
>> for S in 1 2 3 4 5 6; do ./target-asan-8bit-nopru-build/fuzzer -seed=$S -runs=10000000 2>&1 | grep DONE &  done
>>
>> #10000000       DONE   cov: 103 bits: 190 indir: 1 units: 62 exec/s: 526315
>> #10000000       DONE   cov: 443 bits: 1730 indir: 12 units: 529 exec/s: 357142
>> #10000000       DONE   cov: 443 bits: 1695 indir: 12 units: 509 exec/s: 344827
>> #10000000       DONE   cov: 443 bits: 1682 indir: 12 units: 500 exec/s: 333333
>> #10000000       DONE   cov: 444 bits: 1675 indir: 12 units: 501 exec/s: 277777
>> #10000000       DONE   cov: 401 bits: 1443 indir: 12 units: 341 exec/s: 263157
>>
>>
>> I've also tried building with trace-pc-guard (the new thing) and results
>> are similar.
>>
>> name    cov     bits    execs   execs_per_sec   units   actual_cov      actual_bits
>> asan-8bit-nopru 401     1443    19790806        324439  340     401     1441
>> asan-8bit-prune 256     897     26528866        434899  485     447     1651
>> asan-edge-nopru 447     0       35589496        583434  137     447     719
>> asan-edge-prune 256     0       37576436        616007  137     447     719
>> asan-trac-nopru 401     1443    12566606        206009  340     401     1441
>> asan-trac-prune 256     891     16295346        267136  480     447     1640
>>
>>
>> Conclusions:
>> * testing a fuzzing engine is not trivial :(
>> * testing it on a very short run with a single seed may be misleading
>>
>>
>> BTW, I am also looking into more automation of libFuzzer testing.
>> With trace-pc-guard we now have libFuzzer's flag -print_coverage=1 that
>> will print all the covered lines.
>> My hope is that this feature can be used for more detailed analysis of
>> coverage differences.
>>
>> --kcc
>>
>>
>> On Wed, Sep 21, 2016 at 6:00 AM, Jonas Wagner <jonas.wagner at epfl.ch>
>> wrote:
>>
>>> Hello,
>>>
>>> Is this reproducible?
>>>> Fuzzing is a probabilistic business and one or even two runs don't
>>>> prove much.
>>>>
>>>
>>> I've reproduced the behavior on two different machines. Attached is a
>>> script to do so. To use the script,
>>>
>>> - create an empty folder and copy both prune-blocks.sh and
>>> ff-http-parser.sh in there
>>> - ensure clang and clang++ are in your $PATH
>>> - cd /path/to/prune-blocks.sh
>>> - ./prune-blocks.sh
>>>
>>> Let me know how it goes.
>>>
>>>
>>>
>>>> Note that I am going to change all of these coverage options soon.
>>>> The new thing will be http://clang.llvm.org/docs/
>>>> SanitizerCoverage.html#tracing-pcs-with-guards
>>>> It will replace regular (boolean) and 8-bit-counters coverage.
>>>>
>>>
>>> Yay, sounds exciting! I've done a couple experiments to measure the
>>> performance and effect of the different coverage options in the recent
>>> past. If you're interested, I'd be happy to discuss off-list; simply send
>>> me an email.
>>>
>>> Best,
>>> Jonas
>>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160921/7ba952d0/attachment.html>