[llvm-dev] -sanitizer-coverage-prune-blocks=true and LibFuzzer

[Kostya Serebryany via llvm-dev]

On Tue, Sep 20, 2016 at 8:00 AM, Jonas Wagner via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> Hello LLVM devs,
>
> I'm running lots of experiments with LibFuzzer these days -- it's an
> amazing tool!
>

Thank you!


>
> I've noticed something weird while examining the effect of various
> coverage options: for one of my benchmarks, the fuzzer was achieving a
> higher total coverage before April 2016, when -sanitizer-coverage-prune-blocks
> became true by default (commit
> <https://github.com/llvm-mirror/llvm/commit/f593646d9ade6ab311cd6c55880bf3bdfc4b26a4>
> ).
>
> Here are the numbers from running the fuzzer with different options for a
> fixed amount of time.
>
> #units  #blocks  #bits   options
> 492     447      1666    -fsanitize-coverage=edge,indirect-calls,8bit-counters
> -mllvm -sanitizer-coverage-prune-blocks=false
> 135     447      754     -fsanitize-coverage=edge -mllvm
> -sanitizer-coverage-prune-blocks=false
>  58     103      185     -fsanitize-coverage=edge,
> indirect-calls,8bit-counters
> 135     447      754     -fsanitize-coverage=edge
>
> Here, units is the number of test cases generated, blocks is the number of
> basic blocks covered (as measured by replaying the corpus using the
> -sanitizer-coverage-prune-blocks=false version), and bits is the number
> of bits from the 8bit-counters that were seen.
>
> The outlier is the case where I use 8bit-counters and set
> -sanitizer-coverage-prune-blocks to true. There are two things I don't
> understand: (1) why can the 8bit-counters perform worse than using edge
> coverage only, and (2) why does -sanitizer-coverage-prune-blocks affect
> the result so much?
>
> Any ideas?
>

No idea out of the box, and we have not seen such effect.
But admittedly, our benchmarking is far from perfect.

Is this reproducible?
Fuzzing is a probabilistic business and one or even two runs don't prove
much.



> If needed, I can provide scripts to reproduce this (it's a bit of work to
> clean them up, though).
>

If you can do that, please do!

Note that I am going to change all of these coverage options soon.
The new thing will be
http://clang.llvm.org/docs/SanitizerCoverage.html#tracing-pcs-with-guards
It will replace regular (boolean) and 8-bit-counters coverage.

This will not affect sanitizer-coverage-prune-blocks

--kcc


> But maybe someone already has an idea why this could happen.
>
> Cheers,
> Jonas
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160920/74788371/attachment.html>