[zorg] r248504 - [libFuzzer] take the fuzzing corpora from GCS instead of git

[Kostya Serebryany via llvm-commits]

On Thu, Sep 24, 2015 at 8:52 PM, Sean Silva <chisophugis at gmail.com> wrote:

>
>
> On Thu, Sep 24, 2015 at 7:57 PM, Kostya Serebryany <kcc at google.com> wrote:
>
>> Ah, sorry, this is google cloud storage.
>> Scales much better than git for this particular task.
>>
>
> Makes sense. Out of curiosity how large is this bot's corpus?
>
>
In bytes:

207M clang
84M clang-format
812M llvm-as

in # of inputs:

clang 52015
clang-format 21113
llvm-as 201256


For clang and clang-format the corpora are reasonable,but not for llvm-as.
I've disabled llvm-as-fuzzer for now due to excessive parasitic coverage...
https://llvm.org/bugs/show_bug.cgi?id=24639#c10



> -- Sean Silva
>
>
>>
>> --kcc
>>
>> On Thu, Sep 24, 2015 at 7:02 PM, Sean Silva <chisophugis at gmail.com>
>> wrote:
>>
>>>
>>>
>>> On Thu, Sep 24, 2015 at 8:15 AM, Kostya Serebryany via llvm-commits <
>>> llvm-commits at lists.llvm.org> wrote:
>>>
>>>> Author: kcc
>>>> Date: Thu Sep 24 10:15:15 2015
>>>> New Revision: 248504
>>>>
>>>> URL: http://llvm.org/viewvc/llvm-project?rev=248504&view=rev
>>>> Log:
>>>> [libFuzzer] take the fuzzing corpora from GCS instead of git
>>>>
>>>
>>> What is GCS?
>>>
>>> -- Sean Silva
>>>
>>>
>>>>
>>>> Modified:
>>>>     zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fuzzer.sh
>>>>
>>>> Modified:
>>>> zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fuzzer.sh
>>>> URL:
>>>> http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fuzzer.sh?rev=248504&r1=248503&r2=248504&view=diff
>>>>
>>>> ==============================================================================
>>>> --- zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fuzzer.sh
>>>> (original)
>>>> +++ zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fuzzer.sh Thu
>>>> Sep 24 10:15:15 2015
>>>> @@ -19,11 +19,22 @@ LLVM=$ROOT/llvm
>>>>  # No assertions. Need to clean up the existing assertion failures
>>>> first.
>>>>  # Also, the Fuzzer does not provide reproducers on assertion failures
>>>> yet.
>>>>  CMAKE_COMMON_OPTIONS="-GNinja -DCMAKE_BUILD_TYPE=Release
>>>> -DLLVM_ENABLE_ASSERTIONS=OFF -DLLVM_PARALLEL_LINK_JOBS=3"
>>>> -CORPUS_ROOT=$ROOT/fuzzing-with-sanitizers/llvm
>>>> +CORPUS_ROOT=$ROOT/CORPORA/llvm
>>>>  CLANG_FORMAT_CORPUS=$CORPUS_ROOT/clang-format/C1
>>>>  CLANG_CORPUS=$CORPUS_ROOT/clang/C1
>>>>  LLVM_AS_CORPUS=$CORPUS_ROOT/llvm-as/C1
>>>>
>>>> +GS_ROOT=gs://fuzzing-with-sanitizers
>>>> +
>>>> +syncFromGs() {
>>>> +  mkdir -p $CORPUS_ROOT/$1
>>>> +  gsutil -m rsync $GS_ROOT/$1 $CORPUS_ROOT/$1
>>>> +}
>>>> +
>>>> +syncToGs() {
>>>> +  gsutil -m rsync $CORPUS_ROOT/$1 $GS_ROOT/$1
>>>> +}
>>>> +
>>>>  if [ "$BUILDBOT_CLOBBER" != "" ]; then
>>>>    echo @@@BUILD_STEP clobber@@@
>>>>    rm -rf llvm
>>>> @@ -36,12 +47,6 @@ fi
>>>>  rm -rf ${STAGE2_ASAN_DIR}
>>>>  rm -rf ${STAGE2_ASAN_ASSERTIONS_DIR}
>>>>
>>>> -# Create an empty directory for the corpus if it doesn't exist yet.
>>>> -# It will get populated with examples.
>>>> -# FIXME: synchronize this directory with some external persistent
>>>> storage.
>>>> -mkdir -p $CLANG_FORMAT_CORPUS
>>>> -mkdir -p $CLANG_CORPUS
>>>> -
>>>>  # Make sure asan intercepts SIGABRT so that the fuzzer can print the
>>>> test cases
>>>>  # for assertion failures.
>>>>  export ASAN_OPTIONS=handle_abort=1
>>>> @@ -56,7 +61,9 @@ echo @@@BUILD_STEP build clang@@@
>>>>  build_stage1_clang
>>>>
>>>>  echo @@@BUILD_STEP pull test corpuses @@@
>>>> -(cd $CORPUS_ROOT && git pull --no-edit)
>>>> +syncFromGs clang/C1
>>>> +syncFromGs clang-format/C1
>>>> +syncFromGs llvm-as/C1
>>>>
>>>>  # Stage 2 / AddressSanitizer
>>>>
>>>> @@ -126,5 +133,7 @@ echo @@@BUILD_STEP stage2/asan+assertion
>>>>    echo @@@STEP_WARNINGS@@@
>>>>
>>>>  echo @@@BUILD_STEP push corpus updates@@@
>>>> -$LLVM/lib/Fuzzer/pull_and_push_fuzz_corpus.sh $CORPUS_ROOT
>>>> +syncToGs clang/C1
>>>> +syncToGs clang-format/C1
>>>> +syncToGs llvm-as/C1
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> llvm-commits mailing list
>>>> llvm-commits at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20150925/ca7e88bb/attachment.html>