[zorg] r224445 - Add sanitizer-x86_64-linux-fast bot.

Kostya Serebryany kcc at google.com
Fri Dec 19 11:30:41 PST 2014


Excellent, thanks!
Do you already have some stats?  (average time, average # of CLs covered,
etc).
I wonder if we can make it work for every CL?



On Wed, Dec 17, 2014 at 10:22 AM, Sergey Matveev <earthdok at google.com>
wrote:
>
> Author: smatveev
> Date: Wed Dec 17 12:22:17 2014
> New Revision: 224445
>
> URL: http://llvm.org/viewvc/llvm-project?rev=224445&view=rev
> Log:
> Add sanitizer-x86_64-linux-fast bot.
>
> Add a new sanitizer bot. The host compiler is clang fixed at a specific
> revision; it will not be rebuilt unless the revision number is changed in
> the
> bot config. The host compiler does incremental builds of
> (A,M)San-instrumented
> LLVM/clang, which are then tested with check-llvm, check-clang.
>
> Added:
>     zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fast.sh   (with
> props)
> Modified:
>     zorg/trunk/buildbot/osuosl/master/config/builders.py
>     zorg/trunk/buildbot/osuosl/master/config/slaves.py
>     zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_selector.py
>
> Modified: zorg/trunk/buildbot/osuosl/master/config/builders.py
> URL:
> http://llvm.org/viewvc/llvm-project/zorg/trunk/buildbot/osuosl/master/config/builders.py?rev=224445&r1=224444&r2=224445&view=diff
>
> ==============================================================================
> --- zorg/trunk/buildbot/osuosl/master/config/builders.py (original)
> +++ zorg/trunk/buildbot/osuosl/master/config/builders.py Wed Dec 17
> 12:22:17 2014
> @@ -661,6 +661,11 @@ def _get_sanitizer_builders():
>             'builddir': "sanitizer-x86_64-linux-bootstrap",
>             'factory': SanitizerBuilder.getSanitizerBuildFactory()},
>
> +          {'name': "sanitizer-x86_64-linux-fast",
> +           'slavenames' :["sanitizer-buildbot3"],
> +           'builddir': "sanitizer-x86_64-linux-fast",
> +           'factory': SanitizerBuilder.getSanitizerBuildFactory()},
> +
>            #{'name': "llvm-clang-lld-x86_64-ubuntu-sanitize-address",
>            # 'slavenames':["hexagon-build-03"],
>            # 'builddir':"llvm-clang-lld-x86_64-ubuntu-sanitize-address",
>
> Modified: zorg/trunk/buildbot/osuosl/master/config/slaves.py
> URL:
> http://llvm.org/viewvc/llvm-project/zorg/trunk/buildbot/osuosl/master/config/slaves.py?rev=224445&r1=224444&r2=224445&view=diff
>
> ==============================================================================
> --- zorg/trunk/buildbot/osuosl/master/config/slaves.py (original)
> +++ zorg/trunk/buildbot/osuosl/master/config/slaves.py Wed Dec 17 12:22:17
> 2014
> @@ -153,10 +153,12 @@ def get_build_slaves():
>          # AArch64 Clang, Juno ARM Development Platform
>          create_slave("juno-aarch64-01", properties={'jobs': 4},
> max_builds=1),
>
> -        # Ubuntu x86-64, 12x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
> +        # Debian 7.7 x86_64 16-core GCE instance
>          create_slave("sanitizer-buildbot1", properties={'jobs': 8},
> max_builds=1),
> -        # Ubuntu x86-64, 24x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
> +        # Debian 7.7 x86_64 16-core GCE instance
>          create_slave("sanitizer-buildbot2", properties={'jobs': 8},
> max_builds=1),
> +        # Debian 7.7 x86_64 8-core GCE instance
> +        create_slave("sanitizer-buildbot3", properties={'jobs': 8},
> max_builds=1),
>
>          # zEnterprise 196 (s390x), SLES 11 SP2
>          create_slave("systemz-1", properties={'jobs': 4}, max_builds=1),
>
> Added: zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fast.sh
> URL:
> http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fast.sh?rev=224445&view=auto
>
> ==============================================================================
> --- zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fast.sh (added)
> +++ zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fast.sh Wed Dec
> 17 12:22:17 2014
> @@ -0,0 +1,156 @@
> +#!/usr/bin/env bash
> +
> +set -x
> +set -e
> +set -u
> +
> +HERE="$(cd $(dirname $0) && pwd)"
> +. ${HERE}/buildbot_functions.sh
> +
> +ROOT=`pwd`
> +PLATFORM=`uname`
> +export PATH="/usr/local/bin:$PATH"
> +
> +if [ "$BUILDBOT_CLOBBER" != "" ]; then
> +  echo @@@BUILD_STEP clobber@@@
> +  rm -rf llvm
> +  rm -rf llvm_build0
> +fi
> +
> +HOST_CLANG_REVISION=223108
> +
> +MEMORY_SANITIZER_KIND="Memory"
> +BUILDBOT_MSAN_ORIGINS=${BUILDBOT_MSAN_ORIGINS:-}
> +if [ "$BUILDBOT_MSAN_ORIGINS" != "" ]; then
> +    MEMORY_SANITIZER_KIND="MemoryWithOrigins"
> +fi
> +
> +MAKE_JOBS=${MAX_MAKE_JOBS:-16}
> +LLVM=$ROOT/llvm
> +
> +type -a gcc
> +type -a g++
> +CMAKE_COMMON_OPTIONS="-GNinja -DCMAKE_BUILD_TYPE=Release
> -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_PARALLEL_LINK_JOBS=3"
> +CMAKE_STAGE1_OPTIONS="${CMAKE_COMMON_OPTIONS}"
> +
> +# Stage 1
> +
> +if  [ -r host_clang_revision ] && \
> +    [ "$(cat host_clang_revision)" == $HOST_CLANG_REVISION ]
> +then
> +  # Do nothing.
> +  echo @@@BUILD_STEP using pre-built stage1 clang at
> r$HOST_CLANG_REVISION@@@
> +else
> +  echo @@@BUILD_STEP sync to r$HOST_CLANG_REVISION@@@
> +  REAL_BUILDBOT_REVISION=$BUILDBOT_REVISION
> +  BUILDBOT_REVISION=$HOST_CLANG_REVISION
> +  buildbot_update
> +
> +  echo @@@BUILD_STEP build stage1 clang at r$HOST_CLANG_REVISION@@@
> +
> +  rm -rf host_clang_revision
> +
> +  # CMake does not notice that the compiler itself has changed. Anyway,
> +  # incremental builds of stage2 don't make sense if stage1 compiler has
> +  # changed. Clobber the build trees.
> +  rm -rf libcxx_build_msan
> +  rm -rf llvm_build_msan
> +  rm -rf llvm_build_asan
> +
> +  if [ ! -d llvm_build0 ]; then
> +    mkdir llvm_build0
> +  fi
> +  # Build cxx/cxxabi to fool the compiler check in MSan stage 2.
> +  (cd llvm_build0 && cmake ${CMAKE_STAGE1_OPTIONS} $LLVM && \
> +    ninja clang compiler-rt llvm-symbolizer)
> +  echo $HOST_CLANG_REVISION > host_clang_revision
> +
> +  BUILDBOT_REVISION=$REAL_BUILDBOT_REVISION
> +fi
> +
> +echo @@@BUILD_STEP update@@@
> +buildbot_update
> +
> +CLANG_PATH=$ROOT/llvm_build0/bin
> +CMAKE_STAGE2_COMMON_OPTIONS="\
> +  ${CMAKE_COMMON_OPTIONS} \
> +  -DCMAKE_C_COMPILER=${CLANG_PATH}/clang \
> +  -DCMAKE_CXX_COMPILER=${CLANG_PATH}/clang++ \
> +  "
> +LLVM_SYMBOLIZER_PATH=${CLANG_PATH}/llvm-symbolizer
> +export ASAN_SYMBOLIZER_PATH=${LLVM_SYMBOLIZER_PATH}
> +export MSAN_SYMBOLIZER_PATH=${LLVM_SYMBOLIZER_PATH}
> +
> +
> +# Stage 2 / Memory Sanitizer
> +
> +echo @@@BUILD_STEP build libcxx/msan@@@
> +if [ ! -d libcxx_build_msan ]; then
> +  mkdir libcxx_build_msan
> +fi
> +
> +(cd libcxx_build_msan && \
> +  cmake \
> +    ${CMAKE_STAGE2_COMMON_OPTIONS} \
> +    -DLLVM_USE_SANITIZER=${MEMORY_SANITIZER_KIND} \
> +    $LLVM && \
> +  ninja cxx cxxabi) || echo @@@STEP_FAILURE@@@
> +
> +
> +echo @@@BUILD_STEP build clang/msan@@@
> +if [ ! -d llvm_build_msan ]; then
> +  mkdir llvm_build_msan
> +fi
> +
> +MSAN_LDFLAGS="-lc++abi -Wl,--rpath=${ROOT}/libcxx_build_msan/lib
> -L${ROOT}/libcxx_build_msan/lib"
> +# See http://llvm.org/bugs/show_bug.cgi?id=19071,
> http://www.cmake.org/Bug/view.php?id=15264
> +CMAKE_BUG_WORKAROUND_CFLAGS="$MSAN_LDFLAGS -fsanitize=memory -w"
> +MSAN_CFLAGS="-I${ROOT}/libcxx_build_msan/include
> -I${ROOT}/libcxx_build_msan/include/c++/v1 $CMAKE_BUG_WORKAROUND_CFLAGS"
> +
> +(cd llvm_build_msan && \
> + cmake ${CMAKE_STAGE2_COMMON_OPTIONS} \
> +   -DLLVM_USE_SANITIZER=${MEMORY_SANITIZER_KIND} \
> +   -DLLVM_ENABLE_LIBCXX=ON \
> +   -DCMAKE_C_FLAGS="${MSAN_CFLAGS}" \
> +   -DCMAKE_CXX_FLAGS="${MSAN_CFLAGS}" \
> +   -DCMAKE_EXE_LINKER_FLAGS="${MSAN_LDFLAGS}" \
> +   $LLVM && \
> + ninja clang) || echo @@@STEP_FAILURE@@@
> +
> +echo @@@BUILD_STEP check-llvm msan@@@
> +
> +(cd llvm_build_msan && ninja check-llvm) || echo @@@STEP_WARNINGS@@@
> +
> +echo @@@BUILD_STEP check-clang msan@@@
> +
> +(cd llvm_build_msan && ninja check-clang) || echo @@@STEP_FAILURE@@@
> +
> +
> +# Stage 2 / AddressSanitizer
> +
> +echo @@@BUILD_STEP build clang/asan@@@
> +
> +# Turn on init-order checker as ASan runtime option.
> +export
> ASAN_OPTIONS="check_initialization_order=true:detect_stack_use_after_return=1:detect_leaks=1"
> +CMAKE_ASAN_OPTIONS=" \
> +  ${CMAKE_STAGE2_COMMON_OPTIONS} \
> +  -DLLVM_USE_SANITIZER=Address \
> +  "
> +
> +if [ ! -d llvm_build_asan ]; then
> +  mkdir llvm_build_asan
> +fi
> +
> +(cd llvm_build_asan && \
> + cmake ${CMAKE_ASAN_OPTIONS} $LLVM && \
> + ninja clang) || echo @@@STEP_FAILURE@@@
> +
> +
> +echo @@@BUILD_STEP check-llvm asan@@@
> +
> +(cd llvm_build_asan && ninja check-llvm) || echo @@@STEP_FAILURE@@@
> +
> +
> +echo @@@BUILD_STEP check-clang asan@@@
> +
> +(cd llvm_build_asan && ninja check-clang) || echo @@@STEP_FAILURE@@@
>
> Propchange: zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fast.sh
>
> ------------------------------------------------------------------------------
>     svn:executable = *
>
> Modified: zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_selector.py
> URL:
> http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_selector.py?rev=224445&r1=224444&r2=224445&view=diff
>
> ==============================================================================
> --- zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_selector.py
> (original)
> +++ zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_selector.py Wed
> Dec 17 12:22:17 2014
> @@ -25,6 +25,7 @@ BOT_ASSIGNMENT = {
>      'sanitizer-ppc64-linux1': bash('buildbot_cmake.sh'),
>      'sanitizer-x86_64-linux': bash('buildbot_cmake.sh'),
>      'sanitizer-x86_64-linux-bootstrap': bash('buildbot_bootstrap.sh'),
> +    'sanitizer-x86_64-linux-fast': bash('buildbot_fast.sh'),
>  }
>
>  BOT_ADDITIONAL_ENV = {
> @@ -39,6 +40,7 @@ BOT_ADDITIONAL_ENV = {
>      'sanitizer-ppc64-linux1': { 'HAVE_NINJA': '0' },
>      'sanitizer-x86_64-linux': { 'BUILD_ANDROID' : '1', 'RUN_ANDROID' :
> '1', 'HAVE_NINJA' : '1' },
>      'sanitizer-x86_64-linux-bootstrap': {},
> +    'sanitizer-x86_64-linux-fast': {},
>  }
>
>  def Main():
>
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20141219/4acb6421/attachment.html>


More information about the llvm-commits mailing list