[zorg] r248504 - [libFuzzer] take the fuzzing corpora from GCS instead of git

[Sean Silva via llvm-commits]

On Thu, Sep 24, 2015 at 7:57 PM, Kostya Serebryany <kcc at google.com> wrote:

> Ah, sorry, this is google cloud storage.
> Scales much better than git for this particular task.
>

Makes sense. Out of curiosity how large is this bot's corpus?

-- Sean Silva


>
> --kcc
>
> On Thu, Sep 24, 2015 at 7:02 PM, Sean Silva <chisophugis at gmail.com> wrote:
>
>>
>>
>> On Thu, Sep 24, 2015 at 8:15 AM, Kostya Serebryany via llvm-commits <
>> llvm-commits at lists.llvm.org> wrote:
>>
>>> Author: kcc
>>> Date: Thu Sep 24 10:15:15 2015
>>> New Revision: 248504
>>>
>>> URL: http://llvm.org/viewvc/llvm-project?rev=248504&view=rev
>>> Log:
>>> [libFuzzer] take the fuzzing corpora from GCS instead of git
>>>
>>
>> What is GCS?
>>
>> -- Sean Silva
>>
>>
>>>
>>> Modified:
>>>     zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fuzzer.sh
>>>
>>> Modified: zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fuzzer.sh
>>> URL:
>>> http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fuzzer.sh?rev=248504&r1=248503&r2=248504&view=diff
>>>
>>> ==============================================================================
>>> --- zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fuzzer.sh
>>> (original)
>>> +++ zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fuzzer.sh Thu
>>> Sep 24 10:15:15 2015
>>> @@ -19,11 +19,22 @@ LLVM=$ROOT/llvm
>>>  # No assertions. Need to clean up the existing assertion failures first.
>>>  # Also, the Fuzzer does not provide reproducers on assertion failures
>>> yet.
>>>  CMAKE_COMMON_OPTIONS="-GNinja -DCMAKE_BUILD_TYPE=Release
>>> -DLLVM_ENABLE_ASSERTIONS=OFF -DLLVM_PARALLEL_LINK_JOBS=3"
>>> -CORPUS_ROOT=$ROOT/fuzzing-with-sanitizers/llvm
>>> +CORPUS_ROOT=$ROOT/CORPORA/llvm
>>>  CLANG_FORMAT_CORPUS=$CORPUS_ROOT/clang-format/C1
>>>  CLANG_CORPUS=$CORPUS_ROOT/clang/C1
>>>  LLVM_AS_CORPUS=$CORPUS_ROOT/llvm-as/C1
>>>
>>> +GS_ROOT=gs://fuzzing-with-sanitizers
>>> +
>>> +syncFromGs() {
>>> +  mkdir -p $CORPUS_ROOT/$1
>>> +  gsutil -m rsync $GS_ROOT/$1 $CORPUS_ROOT/$1
>>> +}
>>> +
>>> +syncToGs() {
>>> +  gsutil -m rsync $CORPUS_ROOT/$1 $GS_ROOT/$1
>>> +}
>>> +
>>>  if [ "$BUILDBOT_CLOBBER" != "" ]; then
>>>    echo @@@BUILD_STEP clobber@@@
>>>    rm -rf llvm
>>> @@ -36,12 +47,6 @@ fi
>>>  rm -rf ${STAGE2_ASAN_DIR}
>>>  rm -rf ${STAGE2_ASAN_ASSERTIONS_DIR}
>>>
>>> -# Create an empty directory for the corpus if it doesn't exist yet.
>>> -# It will get populated with examples.
>>> -# FIXME: synchronize this directory with some external persistent
>>> storage.
>>> -mkdir -p $CLANG_FORMAT_CORPUS
>>> -mkdir -p $CLANG_CORPUS
>>> -
>>>  # Make sure asan intercepts SIGABRT so that the fuzzer can print the
>>> test cases
>>>  # for assertion failures.
>>>  export ASAN_OPTIONS=handle_abort=1
>>> @@ -56,7 +61,9 @@ echo @@@BUILD_STEP build clang@@@
>>>  build_stage1_clang
>>>
>>>  echo @@@BUILD_STEP pull test corpuses @@@
>>> -(cd $CORPUS_ROOT && git pull --no-edit)
>>> +syncFromGs clang/C1
>>> +syncFromGs clang-format/C1
>>> +syncFromGs llvm-as/C1
>>>
>>>  # Stage 2 / AddressSanitizer
>>>
>>> @@ -126,5 +133,7 @@ echo @@@BUILD_STEP stage2/asan+assertion
>>>    echo @@@STEP_WARNINGS@@@
>>>
>>>  echo @@@BUILD_STEP push corpus updates@@@
>>> -$LLVM/lib/Fuzzer/pull_and_push_fuzz_corpus.sh $CORPUS_ROOT
>>> +syncToGs clang/C1
>>> +syncToGs clang-format/C1
>>> +syncToGs llvm-as/C1
>>>
>>>
>>>
>>> _______________________________________________
>>> llvm-commits mailing list
>>> llvm-commits at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20150924/2c3fad32/attachment.html>