[llvm] r247425 - [libFuzzer] mention more trophies

Fri Sep 18 11:31:06 PDT 2015

On Fri, Sep 18, 2015 at 11:26 AM, Y Song <ys114321 at gmail.com> wrote:

> > You may also want to know exactly which edges are not covered.
> > There is a handy python script that does this for you:
> >
> http://clang.llvm.org/docs/SanitizerCoverage.html#how-good-is-the-coverage
> <http://clang.llvm.org/docs/SanitizerCoverage.html#how-good-is-the-coverage>
> > (no good visualization yet, just the raw info)
>
> Yes, it works. Need to relate missing PCs to actual codes
>

That's done with llvm-symbolizer:
http://clang.llvm.org/docs/SanitizerCoverage.html#postprocessing

--kcc

> and may need to improve
> LLVMFuzzerTestOneInput implementation somehow.
>

>
>
> On Thu, Sep 17, 2015 at 5:48 PM, Kostya Serebryany <kcc at google.com> wrote:
>
>>
>>
>> On Thu, Sep 17, 2015 at 12:01 PM, Y Song <ys114321 at gmail.com> wrote:
>>
>>> I am able to use the following commands to increase test input length
>>> and limit the number of runs
>>> so that I will get a memory leak report.
>>>
>>> ./test_fuzzer -max_len=1024 -runs=1000000 ./corpus
>>>
>>
>> Exactly!
>>
>>
>>>
>>> Further, use the following command
>>> objdump -D test_fuzzer | grep '<__sanitizer_cov>' | wc
>>>
>>
>> Yep!
>> You may also want to know exactly which edges are not covered.
>> There is a handy python script that does this for you:
>> http://clang.llvm.org/docs/SanitizerCoverage.html#how-good-is-the-coverage
>>  (no good visualization yet, just the raw info)
>>
>> --kcc
>>
>>
>>> I am able to get the number of coverage edges, which is 1106.
>>>
>>> Considering my last run reaches:
>>> #3495108828    NEW    cov: 886 bits: 3754 units: 939 exec/s: 38391 L: 58
>>>
>>> Not bad.
>>>
>>> On Thu, Sep 17, 2015 at 9:05 AM, Y Song <ys114321 at gmail.com> wrote:
>>>
>>>> Hi, Kostya,
>>>>
>>>> I converted 62 existing tests in linux:samples/bpf/test_verifier.c as
>>>> the seed test cases
>>>> in corpus. Indeed, I got a good initial jump of coverage
>>>> (edge,8bit-counters) compared to
>>>> non corpus. The change is at
>>>> https://github.com/iovisor/bpf-fuzzer
>>>>
>>>> I am using a virtual machine on my laptop and in the fast two days it
>>>> may have run
>>>> on-off for some times. Last screen shot before I have to reboot the
>>>> machine is:
>>>>
>>>> #3284486954    NEW    cov: 883 bits: 3739 units: 928 exec/s: 38510 L: 55
>>>> #3294370138    NEW    cov: 883 bits: 3742 units: 929 exec/s: 38495 L: 62
>>>> #3309856564    NEW    cov: 883 bits: 3743 units: 930 exec/s: 38484 L:
>>>> 63
>>>> #3311967702    NEW    cov: 883 bits: 3744 units: 931 exec/s: 38483 L: 63
>>>> #3312177180    NEW    cov: 883 bits: 3745 units: 932 exec/s: 38482 L:
>>>> 64
>>>> #3424548361    NEW    cov: 883 bits: 3746 units: 933 exec/s: 38426 L:
>>>> 61
>>>> #3450807353    NEW    cov: 883 bits: 3747 units: 934 exec/s: 38412 L:
>>>> 49
>>>> #3460220158    NEW    cov: 884 bits: 3748 units: 935 exec/s: 38410 L:
>>>> 47
>>>> #3466257868    NEW    cov: 884 bits: 3749 units: 936 exec/s: 38408 L:
>>>> 47
>>>> #3486994800    NEW    cov: 884 bits: 3750 units: 937 exec/s: 38396 L: 64
>>>> #3491234878    NEW    cov: 886 bits: 3752 units: 938 exec/s: 38394 L:
>>>> 58
>>>> #3495108828    NEW    cov: 886 bits: 3754 units: 939 exec/s: 38391 L:
>>>> 58
>>>>
>>>> The corpus itself has 1000 test cases including 62 initial ones.
>>>>
>>>> I do not know the total edge blocks in the code to assess how much room
>>>> is left
>>>> for coverage testing. A rough annotation with source code to find which
>>>> parts of
>>>> source codes are covered will be really helpful. Any support in llvm
>>>> for this?
>>>>
>>>> Also memory leak may not be tested in this setup since it does not quit.
>>>>
>>>> Thanks,
>>>>
>>>> Yonghong
>>>>
>>>> [ I changed my email address from company one to my personal address
>>>>   for which I have subscription for various llvm aliases.]
>>>>
>>>>
>>>>> ---------- Forwarded message ----------
>>>>> From: Kostya Serebryany <kcc at google.com>
>>>>> Date: Fri, Sep 11, 2015 at 1:12 PM
>>>>> Subject: Re: [llvm] r247425 - [libFuzzer] mention more trophies
>>>>> To: Yonghong Song <yhs at plumgrid.com>
>>>>> Cc: Alexei Starovoitov <alexei.starovoitov at gmail.com>, LLVM Commits <
>>>>> llvm-commits at lists.llvm.org>
>>>>>
>>>>>
>>>>> You may also want to add "8bit-counters" to -fsanitize-coverage=...
>>>>>
>>>>> On Fri, Sep 11, 2015 at 10:36 AM, Kostya Serebryany <kcc at google.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Sep 11, 2015 at 10:14 AM, Yonghong Song <yhs at plumgrid.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Currently, it starts with an empty corpus. One option may be
>>>>>>> converting the existing test_verifier test cases into corpus so that fuzzer
>>>>>>> can start with better initial coverage.
>>>>>>>
>>>>>>
>>>>>> Yes, I would certainly start from that.
>>>>>> Please let me know how it goes.
>>>>>>
>>>>>>
>>>>>>> I have not experimented this yet. Any suggestions?
>>>>>>>
>>>>>>> On Fri, Sep 11, 2015 at 10:09 AM, Kostya Serebryany <kcc at google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Sep 11, 2015 at 10:07 AM, Alexei Starovoitov <
>>>>>>>> alexei.starovoitov at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> On Fri, Sep 11, 2015 at 9:34 AM, Kostya Serebryany via llvm-commits
>>>>>>>>> <llvm-commits at lists.llvm.org> wrote:
>>>>>>>>> > +
>>>>>>>>> > +* Linux Kernel's BPF verifier:
>>>>>>>>> https://github.com/iovisor/bpf-fuzzer
>>>>>>>>>
>>>>>>>>> yep :)
>>>>>>>>> It found one bug so far, but looks like we need custom
>>>>>>>>> instruction generation. Pure random fuzzing cannot generate
>>>>>>>>> long enough instruction sequences to stress all pieces of verifier.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Did you fuzz starting from an empty corpus, or you gave it
>>>>>>>> something to start with?
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Yonghong
>>>>>>> http://www.plumgrid.com/plumgrid-ignition/
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Yonghong
>>>>> http://www.plumgrid.com/plumgrid-ignition/
>>>>>
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20150918/011c07aa/attachment.html>