[llvm] r247425 - [libFuzzer] mention more trophies

Fri Sep 18 11:26:09 PDT 2015

> You may also want to know exactly which edges are not covered.
> There is a handy python script that does this for you:
> http://clang.llvm.org/docs/SanitizerCoverage.html#how-good-is-the-coverage
<http://clang.llvm.org/docs/SanitizerCoverage.html#how-good-is-the-coverage>
> (no good visualization yet, just the raw info)

Yes, it works. Need to relate missing PCs to actual codes and may need to
improve
LLVMFuzzerTestOneInput implementation somehow.

On Thu, Sep 17, 2015 at 5:48 PM, Kostya Serebryany <kcc at google.com> wrote:

>
>
> On Thu, Sep 17, 2015 at 12:01 PM, Y Song <ys114321 at gmail.com> wrote:
>
>> I am able to use the following commands to increase test input length and
>> limit the number of runs
>> so that I will get a memory leak report.
>>
>> ./test_fuzzer -max_len=1024 -runs=1000000 ./corpus
>>
>
> Exactly!
>
>
>>
>> Further, use the following command
>> objdump -D test_fuzzer | grep '<__sanitizer_cov>' | wc
>>
>
> Yep!
> You may also want to know exactly which edges are not covered.
> There is a handy python script that does this for you:
> http://clang.llvm.org/docs/SanitizerCoverage.html#how-good-is-the-coverage
>  (no good visualization yet, just the raw info)
>
> --kcc
>
>
>> I am able to get the number of coverage edges, which is 1106.
>>
>> Considering my last run reaches:
>> #3495108828    NEW    cov: 886 bits: 3754 units: 939 exec/s: 38391 L: 58
>>
>> Not bad.
>>
>> On Thu, Sep 17, 2015 at 9:05 AM, Y Song <ys114321 at gmail.com> wrote:
>>
>>> Hi, Kostya,
>>>
>>> I converted 62 existing tests in linux:samples/bpf/test_verifier.c as
>>> the seed test cases
>>> in corpus. Indeed, I got a good initial jump of coverage
>>> (edge,8bit-counters) compared to
>>> non corpus. The change is at
>>> https://github.com/iovisor/bpf-fuzzer
>>>
>>> I am using a virtual machine on my laptop and in the fast two days it
>>> may have run
>>> on-off for some times. Last screen shot before I have to reboot the
>>> machine is:
>>>
>>> #3284486954    NEW    cov: 883 bits: 3739 units: 928 exec/s: 38510 L: 55
>>> #3294370138    NEW    cov: 883 bits: 3742 units: 929 exec/s: 38495 L: 62
>>> #3309856564    NEW    cov: 883 bits: 3743 units: 930 exec/s: 38484 L: 63
>>> #3311967702    NEW    cov: 883 bits: 3744 units: 931 exec/s: 38483 L: 63
>>> #3312177180    NEW    cov: 883 bits: 3745 units: 932 exec/s: 38482 L: 64
>>> #3424548361    NEW    cov: 883 bits: 3746 units: 933 exec/s: 38426 L: 61
>>> #3450807353    NEW    cov: 883 bits: 3747 units: 934 exec/s: 38412 L: 49
>>> #3460220158    NEW    cov: 884 bits: 3748 units: 935 exec/s: 38410 L: 47
>>> #3466257868    NEW    cov: 884 bits: 3749 units: 936 exec/s: 38408 L: 47
>>> #3486994800    NEW    cov: 884 bits: 3750 units: 937 exec/s: 38396 L: 64
>>> #3491234878    NEW    cov: 886 bits: 3752 units: 938 exec/s: 38394 L: 58
>>> #3495108828    NEW    cov: 886 bits: 3754 units: 939 exec/s: 38391 L: 58
>>>
>>> The corpus itself has 1000 test cases including 62 initial ones.
>>>
>>> I do not know the total edge blocks in the code to assess how much room
>>> is left
>>> for coverage testing. A rough annotation with source code to find which
>>> parts of
>>> source codes are covered will be really helpful. Any support in llvm for
>>> this?
>>>
>>> Also memory leak may not be tested in this setup since it does not quit.
>>>
>>> Thanks,
>>>
>>> Yonghong
>>>
>>> [ I changed my email address from company one to my personal address
>>>   for which I have subscription for various llvm aliases.]
>>>
>>>
>>>> ---------- Forwarded message ----------
>>>> From: Kostya Serebryany <kcc at google.com>
>>>> Date: Fri, Sep 11, 2015 at 1:12 PM
>>>> Subject: Re: [llvm] r247425 - [libFuzzer] mention more trophies
>>>> To: Yonghong Song <yhs at plumgrid.com>
>>>> Cc: Alexei Starovoitov <alexei.starovoitov at gmail.com>, LLVM Commits <
>>>> llvm-commits at lists.llvm.org>
>>>>
>>>>
>>>> You may also want to add "8bit-counters" to -fsanitize-coverage=...
>>>>
>>>> On Fri, Sep 11, 2015 at 10:36 AM, Kostya Serebryany <kcc at google.com>
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Fri, Sep 11, 2015 at 10:14 AM, Yonghong Song <yhs at plumgrid.com>
>>>>> wrote:
>>>>>
>>>>>> Currently, it starts with an empty corpus. One option may be
>>>>>> converting the existing test_verifier test cases into corpus so that fuzzer
>>>>>> can start with better initial coverage.
>>>>>>
>>>>>
>>>>> Yes, I would certainly start from that.
>>>>> Please let me know how it goes.
>>>>>
>>>>>
>>>>>> I have not experimented this yet. Any suggestions?
>>>>>>
>>>>>> On Fri, Sep 11, 2015 at 10:09 AM, Kostya Serebryany <kcc at google.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Sep 11, 2015 at 10:07 AM, Alexei Starovoitov <
>>>>>>> alexei.starovoitov at gmail.com> wrote:
>>>>>>>
>>>>>>>> On Fri, Sep 11, 2015 at 9:34 AM, Kostya Serebryany via llvm-commits
>>>>>>>> <llvm-commits at lists.llvm.org> wrote:
>>>>>>>> > +
>>>>>>>> > +* Linux Kernel's BPF verifier:
>>>>>>>> https://github.com/iovisor/bpf-fuzzer
>>>>>>>>
>>>>>>>> yep :)
>>>>>>>> It found one bug so far, but looks like we need custom
>>>>>>>> instruction generation. Pure random fuzzing cannot generate
>>>>>>>> long enough instruction sequences to stress all pieces of verifier.
>>>>>>>>
>>>>>>>
>>>>>>> Did you fuzz starting from an empty corpus, or you gave it something
>>>>>>> to start with?
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Yonghong
>>>>>> http://www.plumgrid.com/plumgrid-ignition/
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Yonghong
>>>> http://www.plumgrid.com/plumgrid-ignition/
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20150918/2c8d691c/attachment.html>