<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Sep 21, 2016 at 12:58 PM, Mehdi Amini <span dir="ltr"><<a href="mailto:mehdi.amini@apple.com" target="_blank">mehdi.amini@apple.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><br><div><span class="gmail-"><blockquote type="cite"><div>On Sep 21, 2016, at 12:56 PM, Kostya Serebryany <<a href="mailto:kcc@google.com" target="_blank">kcc@google.com</a>> wrote:</div><br class="gmail-m_-651008630571810729Apple-interchange-newline"><div><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Sep 21, 2016 at 12:32 PM, Mehdi Amini <span dir="ltr"><<a href="mailto:mehdi.amini@apple.com" target="_blank">mehdi.amini@apple.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><br><div><span><blockquote type="cite"><div>On Sep 21, 2016, at 9:36 AM, Kostya Serebryany via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:</div><br class="gmail-m_-651008630571810729m_6798234943043160321Apple-interchange-newline"><div><div dir="ltr">Exciting! <div><br></div><div><div><span>(btw, I'd prefer </span><font><a href="mailto:libfuzzer@googlegroups.com" target="_blank">libfuzzer@googlegroups.<wbr>com</a> for such discussions, please start new topics there)</font></div></div></div></div></blockquote><div><br></div></span><div>You mean a LLVM library has a separate mailing-list? Why?</div></div></div></blockquote><div><br></div><div>Because the topic is very separate. </div></div></div></div></div></blockquote><div><br></div></span><div>Can you clarify?</div><div><br></div><div>I thought is was about the development/debug/evolution/<wbr>usability of <a href="http://llvm.org/svn/llvm-project/llvm/trunk/lib/Fuzzer/" target="_blank">http://llvm.org/svn/llvm-<wbr>project/llvm/trunk/lib/Fuzzer/</a></div></div></div></blockquote><div><br></div><div>Yes, and this topic is substantially different from most other topics discussed on llvm-dev. </div><div>We also have separate maliing lists for asan/tsan/msan</div><div><br></div><div>--kcc </div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><div><div><br></div><div>— </div><div>Mehid</div><div><div class="gmail-h5"><div><br></div><div><br></div><div><br></div><br><blockquote type="cite"><div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><div><div><br></div><div>— </div><span class="gmail-m_-651008630571810729HOEnZb"><font color="#888888"><div>Mehdi</div><div><br></div><br></font></span><blockquote type="cite"><div><div><div class="gmail-m_-651008630571810729h5"><div dir="ltr"><div><br></div><div>I can reproduce this too, but if i either increase <span>FUZZER_TESTING_SECOND<wbr>S to 600 or change seed=1 to seed=2 the problem is gone. </span></div><div><span>Looks like one of the binaries got simply unlucky with a particular seed. </span></div><div><span>You can observe it like this: </span></div><div><pre>for S in 1 2 3 4 5 6; do ./target-asan-8bit-prune-build<wbr>/fuzzer -seed=$S -runs=10000000 2>&1 | grep DONE & done </pre><pre><pre>#10000000 DONE cov: 60 bits: 91 indir: 1 units: 59 exec/s: 625000
#10000000 DONE cov: 60 bits: 91 indir: 1 units: 57 exec/s: 588235
#10000000 DONE cov: 253 bits: 901 indir: 12 units: 467 exec/s: 526315
#10000000 DONE cov: 63 bits: 95 indir: 1 units: 64 exec/s: 476190
#10000000 DONE cov: 252 bits: 923 indir: 12 units: 491 exec/s: 454545
#10000000 DONE cov: 253 bits: 880 indir: 12 units: 471 exec/s: 384615</pre></pre></div><div><pre><br></pre><pre>Similar things happen with other binaries: </pre><pre>for S in 1 2 3 4 5 6; do ./target-asan-8bit-nopru-build<wbr>/fuzzer -seed=$S -runs=10000000 2>&1 | grep DONE & done</pre><pre><pre>#10000000 DONE cov: 103 bits: 190 indir: 1 units: 62 exec/s: 526315
#10000000 DONE cov: 443 bits: 1730 indir: 12 units: 529 exec/s: 357142
#10000000 DONE cov: 443 bits: 1695 indir: 12 units: 509 exec/s: 344827
#10000000 DONE cov: 443 bits: 1682 indir: 12 units: 500 exec/s: 333333
#10000000 DONE cov: 444 bits: 1675 indir: 12 units: 501 exec/s: 277777
#10000000 DONE cov: 401 bits: 1443 indir: 12 units: 341 exec/s: 263157</pre></pre></div><div><span><br></span></div><div><span>I've also tried building with trace-pc-guard (the new thing) and results are similar. </span></div><div><pre><pre>name cov bits execs execs_per_sec units actual_cov actual_bits
asan-8bit-nopru 401 1443 19790806 324439 340 401 1441
asan-8bit-prune 256 897 26528866 434899 485 447 1651
asan-edge-nopru 447 0 35589496 583434 137 447 719
asan-edge-prune 256 0 37576436 616007 137 447 719
asan-trac-nopru 401 1443 12566606 206009 340 401 1441
asan-trac-prune 256 891 16295346 267136 480 447 1640</pre></pre></div><div><br></div><div>Conclusions: </div><div>* testing a fuzzing engine is not trivial :( </div><div>* testing it on a very short run with a single seed may be misleading</div><div><br></div><div><br></div><div>BTW, I am also looking into more automation of libFuzzer testing.</div><div>With trace-pc-guard we now have libFuzzer's flag -print_coverage=1 that will print all the covered lines. </div><div>My hope is that this feature can be used for more detailed analysis of coverage differences. </div><div><br></div><div>--kcc </div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Sep 21, 2016 at 6:00 AM, Jonas Wagner <span dir="ltr"><<a href="mailto:jonas.wagner@epfl.ch" target="_blank">jonas.wagner@epfl.ch</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hello,<div><br></div><div><div class="gmail_quote"><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>Is this reproducible? </div><div>Fuzzing is a probabilistic business and one or even two runs don't prove much. </div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><span style="line-height:1.5"></span></div></div></div></div></blockquote><div><br></div></span><div>I've reproduced the behavior on two different machines. Attached is a script to do so. To use the script,</div><div><br></div><div>- create an empty folder and copy both prune-blocks.sh and ff-http-parser.sh in there</div><div><span style="line-height:1.5">- ensure clang and clang++ are in your $PATH</span></div><div>- cd /path/to/prune-blocks.sh</div><div>- ./prune-blocks.sh</div><div><br></div><div>Let me know how it goes.</div><span><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><span style="line-height:1.5">Note that I am going to change all of these coverage options soon. </span><br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>The new thing will be <a href="http://clang.llvm.org/docs/SanitizerCoverage.html#tracing-pcs-with-guards" target="_blank">http://clang.llvm.org/docs/<wbr>SanitizerCoverage.html#tracing<wbr>-pcs-with-guards</a></div><div>It will replace regular (boolean) and 8-bit-counters coverage. </div></div></div></div></blockquote><div><br></div></span><div>Yay, sounds exciting! I've done a couple experiments to measure the performance and effect of the different coverage options in the recent past. If you're interested, I'd be happy to discuss off-list; simply send me an email.</div><div><br></div><div>Best,</div><div>Jonas</div></div></div></div>
</blockquote></div><br></div></div></div><span>
______________________________<wbr>_________________<br>LLVM Developers mailing list<br><a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br><a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" target="_blank">http://lists.llvm.org/cgi-bin/<wbr>mailman/listinfo/llvm-dev</a><br></span></div></blockquote></div><br></div></blockquote></div><br></div></div>
</div></blockquote></div></div></div><br></div></blockquote></div><br></div></div>