<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Sep 20, 2016 at 8:00 AM, Jonas Wagner via llvm-dev <span dir="ltr"><<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hello LLVM devs,<div><br></div><div>I'm running lots of experiments with LibFuzzer these days -- it's an amazing tool!</div></div></blockquote><div><br></div><div>Thank you!</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>I've noticed something weird while examining the effect of various coverage options: for one of my benchmarks, the fuzzer was achieving a higher total coverage before April 2016, when -sanitizer-coverage-<wbr>prune-blocks became true by default (<a href="https://github.com/llvm-mirror/llvm/commit/f593646d9ade6ab311cd6c55880bf3bdfc4b26a4" target="_blank">commit</a>).</div><div><br></div><div>Here are the numbers from running the fuzzer with different options for a fixed amount of time.</div><div><br></div><div><font face="monospace">#units #blocks #bits options</font></div><div><font face="monospace">492 447 1666 <span style="line-height:18px">-fsanitize-coverage=edge,<wbr>indirect-calls,8bit-counters -mllvm -sanitizer-coverage-prune-<wbr>blocks=false</span></font><br></div><div><font face="monospace"><span style="line-height:18px">135 447 754 </span>-fsanitize-coverage=edge -mllvm -sanitizer-coverage-prune-<wbr>blocks=false<br></font></div><div><font face="monospace"> 58 103 185 -fsanitize-coverage=edge,<wbr>indirect-calls,8bit-counters</font><br><div><font face="monospace">135 447 754 -fsanitize-coverage=edge<br></font></div><div><br></div></div><div>Here, units is the number of test cases generated, blocks is the number of basic blocks covered (as measured by replaying the corpus using the -sanitizer-coverage-prune-<wbr>blocks=false version), and bits is the number of bits from the 8bit-counters that were seen.</div><div><br></div><div>The outlier is the case where I use 8bit-counters and set -sanitizer-coverage-prune-<wbr>blocks to true. There are two things I don't understand: (1) why can the 8bit-counters perform worse than using edge coverage only, and (2) why does -sanitizer-coverage-prune-<wbr>blocks affect the result so much?</div><div><br></div><div>Any ideas?</div></div></blockquote><div><br></div><div>No idea out of the box, and we have not seen such effect. </div><div>But admittedly, our benchmarking is far from perfect. </div><div><br></div><div>Is this reproducible? </div><div>Fuzzing is a probabilistic business and one or even two runs don't prove much. </div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>If needed, I can provide scripts to reproduce this (it's a bit of work to clean them up, though). </div></div></blockquote><div><br></div><div>If you can do that, please do! </div><div><br></div><div>Note that I am going to change all of these coverage options soon. </div><div>The new thing will be <a href="http://clang.llvm.org/docs/SanitizerCoverage.html#tracing-pcs-with-guards">http://clang.llvm.org/docs/SanitizerCoverage.html#tracing-pcs-with-guards</a></div><div>It will replace regular (boolean) and 8-bit-counters coverage. </div><br>This will not affect sanitizer-coverage-prune-blocks<div><br></div><div>--kcc </div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>But maybe someone already has an idea why this could happen.</div><div><br></div><div>Cheers,</div><div>Jonas</div></div>
<br>______________________________<wbr>_________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/<wbr>mailman/listinfo/llvm-dev</a><br>
<br></blockquote></div><br></div></div>