[zorg] r224445 - Add sanitizer-x86_64-linux-fast bot.

Mon Dec 22 06:18:32 PST 2014

Average time is ~20 min. W.r.t. # of CLs, we can't collect any meaningful
statistic over the weekend, or the upcoming holidays.

If 20 minutes is not satisfactory, we have the option of upgrading this bot
to 16 cores or splitting it into ASan/MSan parts. The former is easier
(just a couple clicks in the GCE console vs a master restart).

On Fri, Dec 19, 2014 at 10:30 PM, Kostya Serebryany <kcc at google.com> wrote:
>
> Excellent, thanks!
> Do you already have some stats?  (average time, average # of CLs covered,
> etc).
> I wonder if we can make it work for every CL?
>
>
>
> On Wed, Dec 17, 2014 at 10:22 AM, Sergey Matveev <earthdok at google.com>
> wrote:
>>
>> Author: smatveev
>> Date: Wed Dec 17 12:22:17 2014
>> New Revision: 224445
>>
>> URL: http://llvm.org/viewvc/llvm-project?rev=224445&view=rev
>> Log:
>> Add sanitizer-x86_64-linux-fast bot.
>>
>> Add a new sanitizer bot. The host compiler is clang fixed at a specific
>> revision; it will not be rebuilt unless the revision number is changed in
>> the
>> bot config. The host compiler does incremental builds of
>> (A,M)San-instrumented
>> LLVM/clang, which are then tested with check-llvm, check-clang.
>>
>> Added:
>>     zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fast.sh   (with
>> props)
>> Modified:
>>     zorg/trunk/buildbot/osuosl/master/config/builders.py
>>     zorg/trunk/buildbot/osuosl/master/config/slaves.py
>>     zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_selector.py
>>
>> Modified: zorg/trunk/buildbot/osuosl/master/config/builders.py
>> URL:
>> http://llvm.org/viewvc/llvm-project/zorg/trunk/buildbot/osuosl/master/config/builders.py?rev=224445&r1=224444&r2=224445&view=diff
>>
>> ==============================================================================
>> --- zorg/trunk/buildbot/osuosl/master/config/builders.py (original)
>> +++ zorg/trunk/buildbot/osuosl/master/config/builders.py Wed Dec 17
>> 12:22:17 2014
>> @@ -661,6 +661,11 @@ def _get_sanitizer_builders():
>>             'builddir': "sanitizer-x86_64-linux-bootstrap",
>>             'factory': SanitizerBuilder.getSanitizerBuildFactory()},
>>
>> +          {'name': "sanitizer-x86_64-linux-fast",
>> +           'slavenames' :["sanitizer-buildbot3"],
>> +           'builddir': "sanitizer-x86_64-linux-fast",
>> +           'factory': SanitizerBuilder.getSanitizerBuildFactory()},
>> +
>>            #{'name': "llvm-clang-lld-x86_64-ubuntu-sanitize-address",
>>            # 'slavenames':["hexagon-build-03"],
>>            # 'builddir':"llvm-clang-lld-x86_64-ubuntu-sanitize-address",
>>
>> Modified: zorg/trunk/buildbot/osuosl/master/config/slaves.py
>> URL:
>> http://llvm.org/viewvc/llvm-project/zorg/trunk/buildbot/osuosl/master/config/slaves.py?rev=224445&r1=224444&r2=224445&view=diff
>>
>> ==============================================================================
>> --- zorg/trunk/buildbot/osuosl/master/config/slaves.py (original)
>> +++ zorg/trunk/buildbot/osuosl/master/config/slaves.py Wed Dec 17
>> 12:22:17 2014
>> @@ -153,10 +153,12 @@ def get_build_slaves():
>>          # AArch64 Clang, Juno ARM Development Platform
>>          create_slave("juno-aarch64-01", properties={'jobs': 4},
>> max_builds=1),
>>
>> -        # Ubuntu x86-64, 12x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
>> +        # Debian 7.7 x86_64 16-core GCE instance
>>          create_slave("sanitizer-buildbot1", properties={'jobs': 8},
>> max_builds=1),
>> -        # Ubuntu x86-64, 24x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
>> +        # Debian 7.7 x86_64 16-core GCE instance
>>          create_slave("sanitizer-buildbot2", properties={'jobs': 8},
>> max_builds=1),
>> +        # Debian 7.7 x86_64 8-core GCE instance
>> +        create_slave("sanitizer-buildbot3", properties={'jobs': 8},
>> max_builds=1),
>>
>>          # zEnterprise 196 (s390x), SLES 11 SP2
>>          create_slave("systemz-1", properties={'jobs': 4}, max_builds=1),
>>
>> Added: zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fast.sh
>> URL:
>> http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fast.sh?rev=224445&view=auto
>>
>> ==============================================================================
>> --- zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fast.sh (added)
>> +++ zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fast.sh Wed Dec
>> 17 12:22:17 2014
>> @@ -0,0 +1,156 @@
>> +#!/usr/bin/env bash
>> +
>> +set -x
>> +set -e
>> +set -u
>> +
>> +HERE="$(cd $(dirname $0) && pwd)"
>> +. ${HERE}/buildbot_functions.sh
>> +
>> +ROOT=`pwd`
>> +PLATFORM=`uname`
>> +export PATH="/usr/local/bin:$PATH"
>> +
>> +if [ "$BUILDBOT_CLOBBER" != "" ]; then
>> +  echo @@@BUILD_STEP clobber@@@
>> +  rm -rf llvm
>> +  rm -rf llvm_build0
>> +fi
>> +
>> +HOST_CLANG_REVISION=223108
>> +
>> +MEMORY_SANITIZER_KIND="Memory"
>> +BUILDBOT_MSAN_ORIGINS=${BUILDBOT_MSAN_ORIGINS:-}
>> +if [ "$BUILDBOT_MSAN_ORIGINS" != "" ]; then
>> +    MEMORY_SANITIZER_KIND="MemoryWithOrigins"
>> +fi
>> +
>> +MAKE_JOBS=${MAX_MAKE_JOBS:-16}
>> +LLVM=$ROOT/llvm
>> +
>> +type -a gcc
>> +type -a g++
>> +CMAKE_COMMON_OPTIONS="-GNinja -DCMAKE_BUILD_TYPE=Release
>> -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_PARALLEL_LINK_JOBS=3"
>> +CMAKE_STAGE1_OPTIONS="${CMAKE_COMMON_OPTIONS}"
>> +
>> +# Stage 1
>> +
>> +if  [ -r host_clang_revision ] && \
>> +    [ "$(cat host_clang_revision)" == $HOST_CLANG_REVISION ]
>> +then
>> +  # Do nothing.
>> +  echo @@@BUILD_STEP using pre-built stage1 clang at
>> r$HOST_CLANG_REVISION@@@
>> +else
>> +  echo @@@BUILD_STEP sync to r$HOST_CLANG_REVISION@@@
>> +  REAL_BUILDBOT_REVISION=$BUILDBOT_REVISION
>> +  BUILDBOT_REVISION=$HOST_CLANG_REVISION
>> +  buildbot_update
>> +
>> +  echo @@@BUILD_STEP build stage1 clang at r$HOST_CLANG_REVISION@@@
>> +
>> +  rm -rf host_clang_revision
>> +
>> +  # CMake does not notice that the compiler itself has changed. Anyway,
>> +  # incremental builds of stage2 don't make sense if stage1 compiler has
>> +  # changed. Clobber the build trees.
>> +  rm -rf libcxx_build_msan
>> +  rm -rf llvm_build_msan
>> +  rm -rf llvm_build_asan
>> +
>> +  if [ ! -d llvm_build0 ]; then
>> +    mkdir llvm_build0
>> +  fi
>> +  # Build cxx/cxxabi to fool the compiler check in MSan stage 2.
>> +  (cd llvm_build0 && cmake ${CMAKE_STAGE1_OPTIONS} $LLVM && \
>> +    ninja clang compiler-rt llvm-symbolizer)
>> +  echo $HOST_CLANG_REVISION > host_clang_revision
>> +
>> +  BUILDBOT_REVISION=$REAL_BUILDBOT_REVISION
>> +fi
>> +
>> +echo @@@BUILD_STEP update@@@
>> +buildbot_update
>> +
>> +CLANG_PATH=$ROOT/llvm_build0/bin
>> +CMAKE_STAGE2_COMMON_OPTIONS="\
>> +  ${CMAKE_COMMON_OPTIONS} \
>> +  -DCMAKE_C_COMPILER=${CLANG_PATH}/clang \
>> +  -DCMAKE_CXX_COMPILER=${CLANG_PATH}/clang++ \
>> +  "
>> +LLVM_SYMBOLIZER_PATH=${CLANG_PATH}/llvm-symbolizer
>> +export ASAN_SYMBOLIZER_PATH=${LLVM_SYMBOLIZER_PATH}
>> +export MSAN_SYMBOLIZER_PATH=${LLVM_SYMBOLIZER_PATH}
>> +
>> +
>> +# Stage 2 / Memory Sanitizer
>> +
>> +echo @@@BUILD_STEP build libcxx/msan@@@
>> +if [ ! -d libcxx_build_msan ]; then
>> +  mkdir libcxx_build_msan
>> +fi
>> +
>> +(cd libcxx_build_msan && \
>> +  cmake \
>> +    ${CMAKE_STAGE2_COMMON_OPTIONS} \
>> +    -DLLVM_USE_SANITIZER=${MEMORY_SANITIZER_KIND} \
>> +    $LLVM && \
>> +  ninja cxx cxxabi) || echo @@@STEP_FAILURE@@@
>> +
>> +
>> +echo @@@BUILD_STEP build clang/msan@@@
>> +if [ ! -d llvm_build_msan ]; then
>> +  mkdir llvm_build_msan
>> +fi
>> +
>> +MSAN_LDFLAGS="-lc++abi -Wl,--rpath=${ROOT}/libcxx_build_msan/lib
>> -L${ROOT}/libcxx_build_msan/lib"
>> +# See http://llvm.org/bugs/show_bug.cgi?id=19071,
>> http://www.cmake.org/Bug/view.php?id=15264
>> +CMAKE_BUG_WORKAROUND_CFLAGS="$MSAN_LDFLAGS -fsanitize=memory -w"
>> +MSAN_CFLAGS="-I${ROOT}/libcxx_build_msan/include
>> -I${ROOT}/libcxx_build_msan/include/c++/v1 $CMAKE_BUG_WORKAROUND_CFLAGS"
>> +
>> +(cd llvm_build_msan && \
>> + cmake ${CMAKE_STAGE2_COMMON_OPTIONS} \
>> +   -DLLVM_USE_SANITIZER=${MEMORY_SANITIZER_KIND} \
>> +   -DLLVM_ENABLE_LIBCXX=ON \
>> +   -DCMAKE_C_FLAGS="${MSAN_CFLAGS}" \
>> +   -DCMAKE_CXX_FLAGS="${MSAN_CFLAGS}" \
>> +   -DCMAKE_EXE_LINKER_FLAGS="${MSAN_LDFLAGS}" \
>> +   $LLVM && \
>> + ninja clang) || echo @@@STEP_FAILURE@@@
>> +
>> +echo @@@BUILD_STEP check-llvm msan@@@
>> +
>> +(cd llvm_build_msan && ninja check-llvm) || echo @@@STEP_WARNINGS@@@
>> +
>> +echo @@@BUILD_STEP check-clang msan@@@
>> +
>> +(cd llvm_build_msan && ninja check-clang) || echo @@@STEP_FAILURE@@@
>> +
>> +
>> +# Stage 2 / AddressSanitizer
>> +
>> +echo @@@BUILD_STEP build clang/asan@@@
>> +
>> +# Turn on init-order checker as ASan runtime option.
>> +export
>> ASAN_OPTIONS="check_initialization_order=true:detect_stack_use_after_return=1:detect_leaks=1"
>> +CMAKE_ASAN_OPTIONS=" \
>> +  ${CMAKE_STAGE2_COMMON_OPTIONS} \
>> +  -DLLVM_USE_SANITIZER=Address \
>> +  "
>> +
>> +if [ ! -d llvm_build_asan ]; then
>> +  mkdir llvm_build_asan
>> +fi
>> +
>> +(cd llvm_build_asan && \
>> + cmake ${CMAKE_ASAN_OPTIONS} $LLVM && \
>> + ninja clang) || echo @@@STEP_FAILURE@@@
>> +
>> +
>> +echo @@@BUILD_STEP check-llvm asan@@@
>> +
>> +(cd llvm_build_asan && ninja check-llvm) || echo @@@STEP_FAILURE@@@
>> +
>> +
>> +echo @@@BUILD_STEP check-clang asan@@@
>> +
>> +(cd llvm_build_asan && ninja check-clang) || echo @@@STEP_FAILURE@@@
>>
>> Propchange: zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fast.sh
>>
>> ------------------------------------------------------------------------------
>>     svn:executable = *
>>
>> Modified:
>> zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_selector.py
>> URL:
>> http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_selector.py?rev=224445&r1=224444&r2=224445&view=diff
>>
>> ==============================================================================
>> --- zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_selector.py
>> (original)
>> +++ zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_selector.py Wed
>> Dec 17 12:22:17 2014
>> @@ -25,6 +25,7 @@ BOT_ASSIGNMENT = {
>>      'sanitizer-ppc64-linux1': bash('buildbot_cmake.sh'),
>>      'sanitizer-x86_64-linux': bash('buildbot_cmake.sh'),
>>      'sanitizer-x86_64-linux-bootstrap': bash('buildbot_bootstrap.sh'),
>> +    'sanitizer-x86_64-linux-fast': bash('buildbot_fast.sh'),
>>  }
>>
>>  BOT_ADDITIONAL_ENV = {
>> @@ -39,6 +40,7 @@ BOT_ADDITIONAL_ENV = {
>>      'sanitizer-ppc64-linux1': { 'HAVE_NINJA': '0' },
>>      'sanitizer-x86_64-linux': { 'BUILD_ANDROID' : '1', 'RUN_ANDROID' :
>> '1', 'HAVE_NINJA' : '1' },
>>      'sanitizer-x86_64-linux-bootstrap': {},
>> +    'sanitizer-x86_64-linux-fast': {},
>>  }
>>
>>  def Main():
>>
>>
>> _______________________________________________
>> llvm-commits mailing list
>> llvm-commits at cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20141222/6bc3f840/attachment.html>