[llvm-dev] -sanitizer-coverage-prune-blocks=true and LibFuzzer

[Jonas Wagner via llvm-dev]

Hi,

Thanks a lot for the tests and the info!

Looks like one of the binaries got simply unlucky with a particular seed.
>

Ouch... I should have tried :/

Conclusions:
> * testing a fuzzing engine is not trivial :(
> * testing it on a very short run with a single seed may be misleading
>

Yup. I'll try to be more careful with this in the future.

And if I'll have time, I'll try to analyze that coverage plateau a bit. It
would be insightful to know which mutation takes the fuzzer beyond the 60
basic blocks, and which of the coverage counters picks up the change --
even if it's just one specific case for one specific benchmark.

BTW, I am also looking into more automation of libFuzzer testing.
> With trace-pc-guard we now have libFuzzer's flag -print_coverage=1 that
> will print all the covered lines.
> My hope is that this feature can be used for more detailed analysis of
> coverage differences.
>

Cool. I'll give this a try too. I have some ideas about testing and
analyzing coverage... will post them on libfuzzer@ once they are more
mature.

Cheers,
Jonas


On Wed, Sep 21, 2016 at 6:00 AM, Jonas Wagner <jonas.wagner at epfl.ch> wrote:
>
>> Hello,
>>
>> Is this reproducible?
>>> Fuzzing is a probabilistic business and one or even two runs don't prove
>>> much.
>>>
>>
>> I've reproduced the behavior on two different machines. Attached is a
>> script to do so. To use the script,
>>
>> - create an empty folder and copy both prune-blocks.sh and
>> ff-http-parser.sh in there
>> - ensure clang and clang++ are in your $PATH
>> - cd /path/to/prune-blocks.sh
>> - ./prune-blocks.sh
>>
>> Let me know how it goes.
>>
>>
>>
>>> Note that I am going to change all of these coverage options soon.
>>> The new thing will be
>>> http://clang.llvm.org/docs/SanitizerCoverage.html#tracing-pcs-with-guards
>>> It will replace regular (boolean) and 8-bit-counters coverage.
>>>
>>
>> Yay, sounds exciting! I've done a couple experiments to measure the
>> performance and effect of the different coverage options in the recent
>> past. If you're interested, I'd be happy to discuss off-list; simply send
>> me an email.
>>
>> Best,
>> Jonas
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160922/bbb39cee/attachment.html>