[zorg] r224445 - Add sanitizer-x86_64-linux-fast bot.

Fri Dec 19 11:32:34 PST 2014

Chandler suggests that adding check-lld would be cool
(you need to make sure that ldd is checked out and that it's asan/msan
clean)
Can you do this?

On Fri, Dec 19, 2014 at 11:30 AM, Kostya Serebryany <kcc at google.com> wrote:
>
> Excellent, thanks!
> Do you already have some stats?  (average time, average # of CLs covered,
> etc).
> I wonder if we can make it work for every CL?
>
>
>
> On Wed, Dec 17, 2014 at 10:22 AM, Sergey Matveev <earthdok at google.com>
> wrote:
>>
>> Author: smatveev
>> Date: Wed Dec 17 12:22:17 2014
>> New Revision: 224445
>>
>> URL: http://llvm.org/viewvc/llvm-project?rev=224445&view=rev
>> Log:
>> Add sanitizer-x86_64-linux-fast bot.
>>
>> Add a new sanitizer bot. The host compiler is clang fixed at a specific
>> revision; it will not be rebuilt unless the revision number is changed in
>> the
>> bot config. The host compiler does incremental builds of
>> (A,M)San-instrumented
>> LLVM/clang, which are then tested with check-llvm, check-clang.
>>
>> Added:
>>     zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fast.sh   (with
>> props)
>> Modified:
>>     zorg/trunk/buildbot/osuosl/master/config/builders.py
>>     zorg/trunk/buildbot/osuosl/master/config/slaves.py
>>     zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_selector.py
>>
>> Modified: zorg/trunk/buildbot/osuosl/master/config/builders.py
>> URL:
>> http://llvm.org/viewvc/llvm-project/zorg/trunk/buildbot/osuosl/master/config/builders.py?rev=224445&r1=224444&r2=224445&view=diff
>>
>> ==============================================================================
>> --- zorg/trunk/buildbot/osuosl/master/config/builders.py (original)
>> +++ zorg/trunk/buildbot/osuosl/master/config/builders.py Wed Dec 17
>> 12:22:17 2014
>> @@ -661,6 +661,11 @@ def _get_sanitizer_builders():
>>             'builddir': "sanitizer-x86_64-linux-bootstrap",
>>             'factory': SanitizerBuilder.getSanitizerBuildFactory()},
>>
>> +          {'name': "sanitizer-x86_64-linux-fast",
>> +           'slavenames' :["sanitizer-buildbot3"],
>> +           'builddir': "sanitizer-x86_64-linux-fast",
>> +           'factory': SanitizerBuilder.getSanitizerBuildFactory()},
>> +
>>            #{'name': "llvm-clang-lld-x86_64-ubuntu-sanitize-address",
>>            # 'slavenames':["hexagon-build-03"],
>>            # 'builddir':"llvm-clang-lld-x86_64-ubuntu-sanitize-address",
>>
>> Modified: zorg/trunk/buildbot/osuosl/master/config/slaves.py
>> URL:
>> http://llvm.org/viewvc/llvm-project/zorg/trunk/buildbot/osuosl/master/config/slaves.py?rev=224445&r1=224444&r2=224445&view=diff
>>
>> ==============================================================================
>> --- zorg/trunk/buildbot/osuosl/master/config/slaves.py (original)
>> +++ zorg/trunk/buildbot/osuosl/master/config/slaves.py Wed Dec 17
>> 12:22:17 2014
>> @@ -153,10 +153,12 @@ def get_build_slaves():
>>          # AArch64 Clang, Juno ARM Development Platform
>>          create_slave("juno-aarch64-01", properties={'jobs': 4},
>> max_builds=1),
>>
>> -        # Ubuntu x86-64, 12x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
>> +        # Debian 7.7 x86_64 16-core GCE instance
>>          create_slave("sanitizer-buildbot1", properties={'jobs': 8},
>> max_builds=1),
>> -        # Ubuntu x86-64, 24x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
>> +        # Debian 7.7 x86_64 16-core GCE instance
>>          create_slave("sanitizer-buildbot2", properties={'jobs': 8},
>> max_builds=1),
>> +        # Debian 7.7 x86_64 8-core GCE instance
>> +        create_slave("sanitizer-buildbot3", properties={'jobs': 8},
>> max_builds=1),
>>
>>          # zEnterprise 196 (s390x), SLES 11 SP2
>>          create_slave("systemz-1", properties={'jobs': 4}, max_builds=1),
>>
>> Added: zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fast.sh
>> URL:
>> http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fast.sh?rev=224445&view=auto
>>
>> ==============================================================================
>> --- zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fast.sh (added)
>> +++ zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fast.sh Wed Dec
>> 17 12:22:17 2014
>> @@ -0,0 +1,156 @@
>> +#!/usr/bin/env bash
>> +
>> +set -x
>> +set -e
>> +set -u
>> +
>> +HERE="$(cd $(dirname $0) && pwd)"
>> +. ${HERE}/buildbot_functions.sh
>> +
>> +ROOT=`pwd`
>> +PLATFORM=`uname`
>> +export PATH="/usr/local/bin:$PATH"
>> +
>> +if [ "$BUILDBOT_CLOBBER" != "" ]; then
>> +  echo @@@BUILD_STEP clobber@@@
>> +  rm -rf llvm
>> +  rm -rf llvm_build0
>> +fi
>> +
>> +HOST_CLANG_REVISION=223108
>> +
>> +MEMORY_SANITIZER_KIND="Memory"
>> +BUILDBOT_MSAN_ORIGINS=${BUILDBOT_MSAN_ORIGINS:-}
>> +if [ "$BUILDBOT_MSAN_ORIGINS" != "" ]; then
>> +    MEMORY_SANITIZER_KIND="MemoryWithOrigins"
>> +fi
>> +
>> +MAKE_JOBS=${MAX_MAKE_JOBS:-16}
>> +LLVM=$ROOT/llvm
>> +
>> +type -a gcc
>> +type -a g++
>> +CMAKE_COMMON_OPTIONS="-GNinja -DCMAKE_BUILD_TYPE=Release
>> -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_PARALLEL_LINK_JOBS=3"
>> +CMAKE_STAGE1_OPTIONS="${CMAKE_COMMON_OPTIONS}"
>> +
>> +# Stage 1
>> +
>> +if  [ -r host_clang_revision ] && \
>> +    [ "$(cat host_clang_revision)" == $HOST_CLANG_REVISION ]
>> +then
>> +  # Do nothing.
>> +  echo @@@BUILD_STEP using pre-built stage1 clang at
>> r$HOST_CLANG_REVISION@@@
>> +else
>> +  echo @@@BUILD_STEP sync to r$HOST_CLANG_REVISION@@@
>> +  REAL_BUILDBOT_REVISION=$BUILDBOT_REVISION
>> +  BUILDBOT_REVISION=$HOST_CLANG_REVISION
>> +  buildbot_update
>> +
>> +  echo @@@BUILD_STEP build stage1 clang at r$HOST_CLANG_REVISION@@@
>> +
>> +  rm -rf host_clang_revision
>> +
>> +  # CMake does not notice that the compiler itself has changed. Anyway,
>> +  # incremental builds of stage2 don't make sense if stage1 compiler has
>> +  # changed. Clobber the build trees.
>> +  rm -rf libcxx_build_msan
>> +  rm -rf llvm_build_msan
>> +  rm -rf llvm_build_asan
>> +
>> +  if [ ! -d llvm_build0 ]; then
>> +    mkdir llvm_build0
>> +  fi
>> +  # Build cxx/cxxabi to fool the compiler check in MSan stage 2.
>> +  (cd llvm_build0 && cmake ${CMAKE_STAGE1_OPTIONS} $LLVM && \
>> +    ninja clang compiler-rt llvm-symbolizer)
>> +  echo $HOST_CLANG_REVISION > host_clang_revision
>> +
>> +  BUILDBOT_REVISION=$REAL_BUILDBOT_REVISION
>> +fi
>> +
>> +echo @@@BUILD_STEP update@@@
>> +buildbot_update
>> +
>> +CLANG_PATH=$ROOT/llvm_build0/bin
>> +CMAKE_STAGE2_COMMON_OPTIONS="\
>> +  ${CMAKE_COMMON_OPTIONS} \
>> +  -DCMAKE_C_COMPILER=${CLANG_PATH}/clang \
>> +  -DCMAKE_CXX_COMPILER=${CLANG_PATH}/clang++ \
>> +  "
>> +LLVM_SYMBOLIZER_PATH=${CLANG_PATH}/llvm-symbolizer
>> +export ASAN_SYMBOLIZER_PATH=${LLVM_SYMBOLIZER_PATH}
>> +export MSAN_SYMBOLIZER_PATH=${LLVM_SYMBOLIZER_PATH}
>> +
>> +
>> +# Stage 2 / Memory Sanitizer
>> +
>> +echo @@@BUILD_STEP build libcxx/msan@@@
>> +if [ ! -d libcxx_build_msan ]; then
>> +  mkdir libcxx_build_msan
>> +fi
>> +
>> +(cd libcxx_build_msan && \
>> +  cmake \
>> +    ${CMAKE_STAGE2_COMMON_OPTIONS} \
>> +    -DLLVM_USE_SANITIZER=${MEMORY_SANITIZER_KIND} \
>> +    $LLVM && \
>> +  ninja cxx cxxabi) || echo @@@STEP_FAILURE@@@
>> +
>> +
>> +echo @@@BUILD_STEP build clang/msan@@@
>> +if [ ! -d llvm_build_msan ]; then
>> +  mkdir llvm_build_msan
>> +fi
>> +
>> +MSAN_LDFLAGS="-lc++abi -Wl,--rpath=${ROOT}/libcxx_build_msan/lib
>> -L${ROOT}/libcxx_build_msan/lib"
>> +# See http://llvm.org/bugs/show_bug.cgi?id=19071,
>> http://www.cmake.org/Bug/view.php?id=15264
>> +CMAKE_BUG_WORKAROUND_CFLAGS="$MSAN_LDFLAGS -fsanitize=memory -w"
>> +MSAN_CFLAGS="-I${ROOT}/libcxx_build_msan/include
>> -I${ROOT}/libcxx_build_msan/include/c++/v1 $CMAKE_BUG_WORKAROUND_CFLAGS"
>> +
>> +(cd llvm_build_msan && \
>> + cmake ${CMAKE_STAGE2_COMMON_OPTIONS} \
>> +   -DLLVM_USE_SANITIZER=${MEMORY_SANITIZER_KIND} \
>> +   -DLLVM_ENABLE_LIBCXX=ON \
>> +   -DCMAKE_C_FLAGS="${MSAN_CFLAGS}" \
>> +   -DCMAKE_CXX_FLAGS="${MSAN_CFLAGS}" \
>> +   -DCMAKE_EXE_LINKER_FLAGS="${MSAN_LDFLAGS}" \
>> +   $LLVM && \
>> + ninja clang) || echo @@@STEP_FAILURE@@@
>> +
>> +echo @@@BUILD_STEP check-llvm msan@@@
>> +
>> +(cd llvm_build_msan && ninja check-llvm) || echo @@@STEP_WARNINGS@@@
>> +
>> +echo @@@BUILD_STEP check-clang msan@@@
>> +
>> +(cd llvm_build_msan && ninja check-clang) || echo @@@STEP_FAILURE@@@
>> +
>> +
>> +# Stage 2 / AddressSanitizer
>> +
>> +echo @@@BUILD_STEP build clang/asan@@@
>> +
>> +# Turn on init-order checker as ASan runtime option.
>> +export
>> ASAN_OPTIONS="check_initialization_order=true:detect_stack_use_after_return=1:detect_leaks=1"
>> +CMAKE_ASAN_OPTIONS=" \
>> +  ${CMAKE_STAGE2_COMMON_OPTIONS} \
>> +  -DLLVM_USE_SANITIZER=Address \
>> +  "
>> +
>> +if [ ! -d llvm_build_asan ]; then
>> +  mkdir llvm_build_asan
>> +fi
>> +
>> +(cd llvm_build_asan && \
>> + cmake ${CMAKE_ASAN_OPTIONS} $LLVM && \
>> + ninja clang) || echo @@@STEP_FAILURE@@@
>> +
>> +
>> +echo @@@BUILD_STEP check-llvm asan@@@
>> +
>> +(cd llvm_build_asan && ninja check-llvm) || echo @@@STEP_FAILURE@@@
>> +
>> +
>> +echo @@@BUILD_STEP check-clang asan@@@
>> +
>> +(cd llvm_build_asan && ninja check-clang) || echo @@@STEP_FAILURE@@@
>>
>> Propchange: zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fast.sh
>>
>> ------------------------------------------------------------------------------
>>     svn:executable = *
>>
>> Modified:
>> zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_selector.py
>> URL:
>> http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_selector.py?rev=224445&r1=224444&r2=224445&view=diff
>>
>> ==============================================================================
>> --- zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_selector.py
>> (original)
>> +++ zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_selector.py Wed
>> Dec 17 12:22:17 2014
>> @@ -25,6 +25,7 @@ BOT_ASSIGNMENT = {
>>      'sanitizer-ppc64-linux1': bash('buildbot_cmake.sh'),
>>      'sanitizer-x86_64-linux': bash('buildbot_cmake.sh'),
>>      'sanitizer-x86_64-linux-bootstrap': bash('buildbot_bootstrap.sh'),
>> +    'sanitizer-x86_64-linux-fast': bash('buildbot_fast.sh'),
>>  }
>>
>>  BOT_ADDITIONAL_ENV = {
>> @@ -39,6 +40,7 @@ BOT_ADDITIONAL_ENV = {
>>      'sanitizer-ppc64-linux1': { 'HAVE_NINJA': '0' },
>>      'sanitizer-x86_64-linux': { 'BUILD_ANDROID' : '1', 'RUN_ANDROID' :
>> '1', 'HAVE_NINJA' : '1' },
>>      'sanitizer-x86_64-linux-bootstrap': {},
>> +    'sanitizer-x86_64-linux-fast': {},
>>  }
>>
>>  def Main():
>>
>>
>> _______________________________________________
>> llvm-commits mailing list
>> llvm-commits at cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20141219/b9646d5a/attachment.html>