<div dir="ltr"><div>> You may also want to know exactly which edges are not covered. </div><div>> There is a handy python script that does this for you: </div><div><a href="http://clang.llvm.org/docs/SanitizerCoverage.html#how-good-is-the-coverage" target="_blank">> http://clang.llvm.org/docs/SanitizerCoverage.html#how-good-is-the-coverage</a><br></div><div>> (no good visualization yet, just the raw info)</div><div><br></div><div>Yes, it works. Need to relate missing PCs to actual codes and may need to improve<br>LLVMFuzzerTestOneInput implementation somehow.<br><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Sep 17, 2015 at 5:48 PM, Kostya Serebryany <span dir="ltr"><<a href="mailto:kcc@google.com" target="_blank">kcc@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Sep 17, 2015 at 12:01 PM, Y Song <span dir="ltr"><<a href="mailto:ys114321@gmail.com" target="_blank">ys114321@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div>I am able to use the following commands to increase test input length and limit the number of runs<br></div>so that I will get a memory leak report.<br><br>./test_fuzzer -max_len=1024 -runs=1000000 ./corpus<br></div></div></div></div></div></blockquote><div><br></div><div>Exactly! </div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><br></div>Further, use the following command<br>objdump -D test_fuzzer | grep '<__sanitizer_cov>' | wc<br></div></div></div></div></blockquote><div><br></div><div>Yep!</div><div>You may also want to know exactly which edges are not covered. </div><div>There is a handy python script that does this for you: </div><div><a href="http://clang.llvm.org/docs/SanitizerCoverage.html#how-good-is-the-coverage" target="_blank">http://clang.llvm.org/docs/SanitizerCoverage.html#how-good-is-the-coverage</a><br></div><div> (no good visualization yet, just the raw info)</div><div><br></div><div>--kcc </div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div><div><br></div>I am able to get the number of coverage edges, which is 1106. <br><br></div>Considering my last run reaches:<span><br><span><span>#<a href="tel:3495108828" value="+73495108828" target="_blank">3495108828</a> NEW cov: 886 bits: 3754 units: 939 exec/s: 38391 L: 58<br><br></span></span></span></div><span><span>Not bad.<br></span></span></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Sep 17, 2015 at 9:05 AM, Y Song <span dir="ltr"><<a href="mailto:ys114321@gmail.com" target="_blank">ys114321@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div>Hi, Kostya,<br><br></div>I converted 62 existing tests in linux:samples/bpf/test_verifier.c as the seed test cases<br></div>in corpus. Indeed, I got a good initial jump of coverage (edge,8bit-counters) compared to<br></div>non corpus. The change is at<br><span><span><a href="https://github.com/iovisor/bpf-fuzzer" rel="noreferrer" target="_blank">https://github.com/iovisor/bpf-fuzzer</a><br><br></span></span></div><span><span>I am using a virtual machine on my laptop and in the fast two days it may have run<br></span></span></div><span><span>on-off for some times. Last screen shot before I have to reboot the machine is:<br><br>#3284486954 NEW cov: 883 bits: 3739 units: 928 exec/s: 38510 L: 55<br>#3294370138 NEW cov: 883 bits: 3742 units: 929 exec/s: 38495 L: 62<br>#<a href="tel:3309856564" value="+13309856564" target="_blank">3309856564</a> NEW cov: 883 bits: 3743 units: 930 exec/s: 38484 L: 63<br>#3311967702 NEW cov: 883 bits: 3744 units: 931 exec/s: 38483 L: 63<br>#<a href="tel:3312177180" value="+13312177180" target="_blank">3312177180</a> NEW cov: 883 bits: 3745 units: 932 exec/s: 38482 L: 64<br>#<a href="tel:3424548361" value="+73424548361" target="_blank">3424548361</a> NEW cov: 883 bits: 3746 units: 933 exec/s: 38426 L: 61<br>#<a href="tel:3450807353" value="+73450807353" target="_blank">3450807353</a> NEW cov: 883 bits: 3747 units: 934 exec/s: 38412 L: 49<br>#<a href="tel:3460220158" value="+73460220158" target="_blank">3460220158</a> NEW cov: 884 bits: 3748 units: 935 exec/s: 38410 L: 47<br>#<a href="tel:3466257868" value="+13466257868" target="_blank">3466257868</a> NEW cov: 884 bits: 3749 units: 936 exec/s: 38408 L: 47<br>#3486994800 NEW cov: 884 bits: 3750 units: 937 exec/s: 38396 L: 64<br>#<a href="tel:3491234878" value="+73491234878" target="_blank">3491234878</a> NEW cov: 886 bits: 3752 units: 938 exec/s: 38394 L: 58<br>#<a href="tel:3495108828" value="+73495108828" target="_blank">3495108828</a> NEW cov: 886 bits: 3754 units: 939 exec/s: 38391 L: 58<br><br></span></span><div><div><div><div><div>The corpus itself has 1000 test cases including 62 initial ones.<br><br></div><div>I do not know the total edge blocks in the code to assess how much room is left<br></div><div>for coverage testing. A rough annotation with source code to find which parts of<br>source codes are covered will be really helpful. Any support in llvm for this?<br><br>Also memory leak may not be tested in this setup since it does not quit.<br></div><div><br></div><div>Thanks,<br><br></div><div>Yonghong<br></div><div><div><div><div class="gmail_extra"><br><div class="gmail_quote">[ I changed my email address from company one to my personal address<br></div><div class="gmail_quote"> for which I have subscription for various llvm aliases.]<br></div><div class="gmail_quote"><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_quote">---------- Forwarded message ----------<br>From: <b class="gmail_sendername">Kostya Serebryany</b> <span dir="ltr"><<a href="mailto:kcc@google.com" target="_blank">kcc@google.com</a>></span><br>Date: Fri, Sep 11, 2015 at 1:12 PM<br>Subject: Re: [llvm] r247425 - [libFuzzer] mention more trophies<br>To: Yonghong Song <<a href="mailto:yhs@plumgrid.com" target="_blank">yhs@plumgrid.com</a>><br>Cc: Alexei Starovoitov <<a href="mailto:alexei.starovoitov@gmail.com" target="_blank">alexei.starovoitov@gmail.com</a>>, LLVM Commits <<a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a>><br><br><br><div dir="ltr">You may also want to add "8bit-counters" to -fsanitize-coverage=...</div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Sep 11, 2015 at 10:36 AM, Kostya Serebryany <span dir="ltr"><<a href="mailto:kcc@google.com" target="_blank">kcc@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Fri, Sep 11, 2015 at 10:14 AM, Yonghong Song <span dir="ltr"><<a href="mailto:yhs@plumgrid.com" target="_blank">yhs@plumgrid.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">Currently, it starts with an empty corpus. One option may be converting the existing test_verifier test cases into corpus so that fuzzer<div>can start with better initial coverage. </div></div></blockquote><div><br></div></span><div>Yes, I would certainly start from that. </div><div>Please let me know how it goes. </div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>I have not experimented this yet. Any suggestions?</div></div><div class="gmail_extra"><div><div><br><div class="gmail_quote">On Fri, Sep 11, 2015 at 10:09 AM, Kostya Serebryany <span dir="ltr"><<a href="mailto:kcc@google.com" target="_blank">kcc@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><div><div>On Fri, Sep 11, 2015 at 10:07 AM, Alexei Starovoitov <span dir="ltr"><<a href="mailto:alexei.starovoitov@gmail.com" target="_blank">alexei.starovoitov@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><span>On Fri, Sep 11, 2015 at 9:34 AM, Kostya Serebryany via llvm-commits<br>
<<a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a>> wrote:<br>
> +<br>
> +* Linux Kernel's BPF verifier: <a href="https://github.com/iovisor/bpf-fuzzer" rel="noreferrer" target="_blank">https://github.com/iovisor/bpf-fuzzer</a><br>
<br>
</span>yep :)<br>
It found one bug so far, but looks like we need custom<br>
instruction generation. Pure random fuzzing cannot generate<br>
long enough instruction sequences to stress all pieces of verifier.<br></blockquote><div><br></div></div></div><div>Did you fuzz starting from an empty corpus, or you gave it something to start with? </div></div><br></div></div>
</blockquote></div><br><br clear="all"><span class="HOEnZb"><font color="#888888"><span><font color="#888888"><span><font color="#888888"><div><br></div></font></span></font></span></font></span></div></div><span class="HOEnZb"><font color="#888888"><span><font color="#888888"><span><font color="#888888"><span><font color="#888888">-- <br><div><div dir="ltr">Yonghong<div><a href="http://www.plumgrid.com/plumgrid-ignition/" target="_blank">http://www.plumgrid.com/plumgrid-ignition/</a></div></div></div>
</font></span></font></span></font></span></font></span></div><span class="HOEnZb"><font color="#888888"><span><font color="#888888"><span><font color="#888888">
</font></span></font></span></font></span></blockquote></span></div><span class="HOEnZb"><font color="#888888"><span><font color="#888888"><span><font color="#888888"><br></font></span></font></span></font></span></div></div><span class="HOEnZb"><font color="#888888"><span><font color="#888888"><span><font color="#888888">
</font></span></font></span></font></span></blockquote></div><span class="HOEnZb"><font color="#888888"><span><font color="#888888"><span><font color="#888888"><br></font></span></font></span></font></span></div><span class="HOEnZb"><font color="#888888"><span><font color="#888888"><span><font color="#888888">
</font></span></font></span></font></span></div></div></div><span class="HOEnZb"><font color="#888888"><span><font color="#888888"><span><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div><div dir="ltr">Yonghong<div><a href="http://www.plumgrid.com/plumgrid-ignition/" target="_blank">http://www.plumgrid.com/plumgrid-ignition/</a></div></div></div>
</font></span></font></span></font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote></div><span class="HOEnZb"><font color="#888888"><br></font></span></div></div></div></div></div></div></div></div></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote></div><span class="HOEnZb"><font color="#888888"><br></font></span></div>
</div></div></blockquote></div><br></div></div>
</blockquote></div><br></div>