[llvm] r247425 - [libFuzzer] mention more trophies

Fri Sep 18 12:11:08 PDT 2015

On Fri, Sep 18, 2015 at 11:31 AM, Kostya Serebryany <kcc at google.com> wrote:

>
>
> On Fri, Sep 18, 2015 at 11:26 AM, Y Song <ys114321 at gmail.com> wrote:
>
>> > You may also want to know exactly which edges are not covered.
>> > There is a handy python script that does this for you:
>> >
>> http://clang.llvm.org/docs/SanitizerCoverage.html#how-good-is-the-coverage
>> <http://clang.llvm.org/docs/SanitizerCoverage.html#how-good-is-the-coverage>
>> > (no good visualization yet, just the raw info)
>>
>> Yes, it works. Need to relate missing PCs to actual codes
>>
>
> That's done with llvm-symbolizer:
> http://clang.llvm.org/docs/SanitizerCoverage.html#postprocessing
>

Thanks. Now I got the place in the source code:
.....
bpf_check
/home/yhs/work/bpf-fuzzer/bld/kernel/bpf/verifier.c:2158:291

bpf_check
/home/yhs/work/bpf-fuzzer/bld/kernel/bpf/verifier.c:2094:24

bpf_check
/home/yhs/work/bpf-fuzzer/bld/kernel/bpf/verifier.c:2101:6

bpf_check
/home/yhs/work/bpf-fuzzer/bld/kernel/bpf/verifier.c:2120:22
.....

Will improve the test based on the above information.

Yonghong

>
>
> --kcc
>
>
>> and may need to improve
>> LLVMFuzzerTestOneInput implementation somehow.
>>
>
>
>
>
>>
>>
>> On Thu, Sep 17, 2015 at 5:48 PM, Kostya Serebryany <kcc at google.com>
>> wrote:
>>
>>>
>>>
>>> On Thu, Sep 17, 2015 at 12:01 PM, Y Song <ys114321 at gmail.com> wrote:
>>>
>>>> I am able to use the following commands to increase test input length
>>>> and limit the number of runs
>>>> so that I will get a memory leak report.
>>>>
>>>> ./test_fuzzer -max_len=1024 -runs=1000000 ./corpus
>>>>
>>>
>>> Exactly!
>>>
>>>
>>>>
>>>> Further, use the following command
>>>> objdump -D test_fuzzer | grep '<__sanitizer_cov>' | wc
>>>>
>>>
>>> Yep!
>>> You may also want to know exactly which edges are not covered.
>>> There is a handy python script that does this for you:
>>>
>>> http://clang.llvm.org/docs/SanitizerCoverage.html#how-good-is-the-coverage
>>>  (no good visualization yet, just the raw info)
>>>
>>> --kcc
>>>
>>>
>>>> I am able to get the number of coverage edges, which is 1106.
>>>>
>>>> Considering my last run reaches:
>>>> #3495108828    NEW    cov: 886 bits: 3754 units: 939 exec/s: 38391 L:
>>>> 58
>>>>
>>>> Not bad.
>>>>
>>>> On Thu, Sep 17, 2015 at 9:05 AM, Y Song <ys114321 at gmail.com> wrote:
>>>>
>>>>> Hi, Kostya,
>>>>>
>>>>> I converted 62 existing tests in linux:samples/bpf/test_verifier.c as
>>>>> the seed test cases
>>>>> in corpus. Indeed, I got a good initial jump of coverage
>>>>> (edge,8bit-counters) compared to
>>>>> non corpus. The change is at
>>>>> https://github.com/iovisor/bpf-fuzzer
>>>>>
>>>>> I am using a virtual machine on my laptop and in the fast two days it
>>>>> may have run
>>>>> on-off for some times. Last screen shot before I have to reboot the
>>>>> machine is:
>>>>>
>>>>> #3284486954    NEW    cov: 883 bits: 3739 units: 928 exec/s: 38510 L:
>>>>> 55
>>>>> #3294370138    NEW    cov: 883 bits: 3742 units: 929 exec/s: 38495 L:
>>>>> 62
>>>>> #3309856564    NEW    cov: 883 bits: 3743 units: 930 exec/s: 38484 L:
>>>>> 63
>>>>> #3311967702    NEW    cov: 883 bits: 3744 units: 931 exec/s: 38483 L:
>>>>> 63
>>>>> #3312177180    NEW    cov: 883 bits: 3745 units: 932 exec/s: 38482 L:
>>>>> 64
>>>>> #3424548361    NEW    cov: 883 bits: 3746 units: 933 exec/s: 38426 L:
>>>>> 61
>>>>> #3450807353    NEW    cov: 883 bits: 3747 units: 934 exec/s: 38412 L:
>>>>> 49
>>>>> #3460220158    NEW    cov: 884 bits: 3748 units: 935 exec/s: 38410 L:
>>>>> 47
>>>>> #3466257868    NEW    cov: 884 bits: 3749 units: 936 exec/s: 38408 L:
>>>>> 47
>>>>> #3486994800    NEW    cov: 884 bits: 3750 units: 937 exec/s: 38396 L:
>>>>> 64
>>>>> #3491234878    NEW    cov: 886 bits: 3752 units: 938 exec/s: 38394 L:
>>>>> 58
>>>>> #3495108828    NEW    cov: 886 bits: 3754 units: 939 exec/s: 38391 L:
>>>>> 58
>>>>>
>>>>> The corpus itself has 1000 test cases including 62 initial ones.
>>>>>
>>>>> I do not know the total edge blocks in the code to assess how much
>>>>> room is left
>>>>> for coverage testing. A rough annotation with source code to find
>>>>> which parts of
>>>>> source codes are covered will be really helpful. Any support in llvm
>>>>> for this?
>>>>>
>>>>> Also memory leak may not be tested in this setup since it does not
>>>>> quit.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Yonghong
>>>>>
>>>>> [ I changed my email address from company one to my personal address
>>>>>   for which I have subscription for various llvm aliases.]
>>>>>
>>>>>
>>>>>> ---------- Forwarded message ----------
>>>>>> From: Kostya Serebryany <kcc at google.com>
>>>>>> Date: Fri, Sep 11, 2015 at 1:12 PM
>>>>>> Subject: Re: [llvm] r247425 - [libFuzzer] mention more trophies
>>>>>> To: Yonghong Song <yhs at plumgrid.com>
>>>>>> Cc: Alexei Starovoitov <alexei.starovoitov at gmail.com>, LLVM Commits <
>>>>>> llvm-commits at lists.llvm.org>
>>>>>>
>>>>>>
>>>>>> You may also want to add "8bit-counters" to -fsanitize-coverage=...
>>>>>>
>>>>>> On Fri, Sep 11, 2015 at 10:36 AM, Kostya Serebryany <kcc at google.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Sep 11, 2015 at 10:14 AM, Yonghong Song <yhs at plumgrid.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Currently, it starts with an empty corpus. One option may be
>>>>>>>> converting the existing test_verifier test cases into corpus so that fuzzer
>>>>>>>> can start with better initial coverage.
>>>>>>>>
>>>>>>>
>>>>>>> Yes, I would certainly start from that.
>>>>>>> Please let me know how it goes.
>>>>>>>
>>>>>>>
>>>>>>>> I have not experimented this yet. Any suggestions?
>>>>>>>>
>>>>>>>> On Fri, Sep 11, 2015 at 10:09 AM, Kostya Serebryany <kcc at google.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Sep 11, 2015 at 10:07 AM, Alexei Starovoitov <
>>>>>>>>> alexei.starovoitov at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> On Fri, Sep 11, 2015 at 9:34 AM, Kostya Serebryany via
>>>>>>>>>> llvm-commits
>>>>>>>>>> <llvm-commits at lists.llvm.org> wrote:
>>>>>>>>>> > +
>>>>>>>>>> > +* Linux Kernel's BPF verifier:
>>>>>>>>>> https://github.com/iovisor/bpf-fuzzer
>>>>>>>>>>
>>>>>>>>>> yep :)
>>>>>>>>>> It found one bug so far, but looks like we need custom
>>>>>>>>>> instruction generation. Pure random fuzzing cannot generate
>>>>>>>>>> long enough instruction sequences to stress all pieces of
>>>>>>>>>> verifier.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Did you fuzz starting from an empty corpus, or you gave it
>>>>>>>>> something to start with?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Yonghong
>>>>>>>> http://www.plumgrid.com/plumgrid-ignition/
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Yonghong
>>>>>> http://www.plumgrid.com/plumgrid-ignition/
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20150918/fd9ce463/attachment.html>