[zorg] r219136 - Move sanitizer bots's configs from an external repo to zorg
Kostya Serebryany
kcc at google.com
Mon Oct 6 11:41:50 PDT 2014
Author: kcc
Date: Mon Oct 6 13:41:49 2014
New Revision: 219136
URL: http://llvm.org/viewvc/llvm-project?rev=219136&view=rev
Log:
Move sanitizer bots's configs from an external repo to zorg
Summary:
Currently, we store the configs for sanitizer bots in a separate repository:
https://code.google.com/p/address-sanitizer/source/browse/#svn%2Ftrunk%2Fbuild%2Fscripts%2Fslave
We are planing to add more hardware to the bots, and before that we would like to
perform some cleanup.
First (this change) we want to move the config files from the external repo to zorg.
Next, we'd like to change zorg/buildbot/builders/SanitizerBuilder.py to use the configs
from zorg instead of the extarnal repo.
Then we'll add some new bots.
Test Plan: n/a
Reviewers: eugenis, samsonov, gkistanova
Subscribers: rnk, rengolin, jfb, llvm-commits
Differential Revision: http://reviews.llvm.org/D5531
Added:
zorg/trunk/zorg/buildbot/builders/sanitizers/
zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_bootstrap.sh (with props)
zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_cmake.sh
zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_functions.sh (with props)
zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_perf_asan.sh
zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_selector.py (with props)
zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_standard.bat
zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_standard.sh
zorg/trunk/zorg/buildbot/builders/sanitizers/test_tsan.sh (with props)
Added: zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_bootstrap.sh
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_bootstrap.sh?rev=219136&view=auto
==============================================================================
--- zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_bootstrap.sh (added)
+++ zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_bootstrap.sh Mon Oct 6 13:41:49 2014
@@ -0,0 +1,198 @@
+#!/usr/bin/env bash
+
+set -x
+set -e
+set -u
+
+HERE="$(cd $(dirname $0) && pwd)"
+. ${HERE}/buildbot_functions.sh
+
+ROOT=`pwd`
+PLATFORM=`uname`
+export PATH="/usr/local/bin:$PATH"
+
+if [ "$BUILDBOT_CLOBBER" != "" ]; then
+ echo @@@BUILD_STEP clobber@@@
+ rm -rf llvm
+ rm -rf llvm_build0
+fi
+
+# CMake does not notice that the compiler itself has changed.
+# Anyway, incremental builds of stage2 and stage3 compilers don't make sense.
+# Clobber the build trees.
+rm -rf libcxx_build_msan
+rm -rf llvm_build_msan
+rm -rf llvm_build2_msan
+rm -rf llvm_build_asan
+rm -rf llvm_build2_asan
+rm -rf llvm_build_ubsan
+
+MAKE_JOBS=${MAX_MAKE_JOBS:-16}
+LLVM=$ROOT/llvm
+LIBCXX=$LLVM/projects/libcxx
+
+type -a gcc
+type -a g++
+CMAKE_COMMON_OPTIONS="-GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON"
+CMAKE_STAGE1_OPTIONS="${CMAKE_COMMON_OPTIONS}"
+
+echo @@@BUILD_STEP update@@@
+buildbot_update
+
+# Stage 1
+
+echo @@@BUILD_STEP build stage1 clang@@@
+if [ ! -d llvm_build0 ]; then
+ mkdir llvm_build0
+fi
+(cd llvm_build0 && cmake ${CMAKE_STAGE1_OPTIONS} $LLVM && \
+ ninja clang && ninja compiler-rt && ninja llvm-symbolizer) || \
+ echo @@@STEP_FAILURE@@@
+
+CLANG_PATH=$ROOT/llvm_build0/bin
+CMAKE_STAGE2_COMMON_OPTIONS="\
+ ${CMAKE_COMMON_OPTIONS} \
+ -DCMAKE_C_COMPILER=${CLANG_PATH}/clang \
+ -DCMAKE_CXX_COMPILER=${CLANG_PATH}/clang++ \
+ "
+LLVM_SYMBOLIZER_PATH=${CLANG_PATH}/llvm-symbolizer
+export ASAN_SYMBOLIZER_PATH=${LLVM_SYMBOLIZER_PATH}
+export MSAN_SYMBOLIZER_PATH=${LLVM_SYMBOLIZER_PATH}
+
+# Stage 2 / Memory Sanitizer
+
+echo @@@BUILD_STEP build libcxx/msan@@@
+if [ ! -d libcxx_build_msan ]; then
+ mkdir libcxx_build_msan
+fi
+
+LIBCXX_INST=${LIBCXX}/inst
+(cd libcxx_build_msan && \
+ cmake ${CMAKE_STAGE2_COMMON_OPTIONS} \
+ -DLLVM_USE_SANITIZER=Memory \
+ -DLIBCXX_CXX_ABI=libstdc++ \
+ -DLIBCXX_LIBSUPCXX_INCLUDE_PATHS="/usr/local/include/c++/4.9.1;/usr/local/include/c++/4.9.1/x86_64-unknown-linux-gnu" \
+ -DCMAKE_INSTALL_PREFIX=${LIBCXX_INST} \
+ ${LIBCXX} && \
+ ninja install) || echo @@@STEP_FAILURE@@@
+
+echo @@@BUILD_STEP build clang/msan@@@
+if [ ! -d llvm_build_msan ]; then
+ mkdir llvm_build_msan
+fi
+
+MSAN_INCLUDE_FLAGS="-I${LIBCXX_INST}/include/c++/v1"
+MSAN_LINK_FLAGS="-lc++ -Wl,--rpath=${LIBCXX_INST}/lib -L${LIBCXX_INST}/lib"
+
+(cd llvm_build_msan && \
+ cmake ${CMAKE_STAGE2_COMMON_OPTIONS} \
+ -DLLVM_USE_SANITIZER=Memory \
+ -DCMAKE_C_FLAGS="${MSAN_INCLUDE_FLAGS}" \
+ -DCMAKE_CXX_FLAGS="${MSAN_INCLUDE_FLAGS}" \
+ -DCMAKE_EXE_LINKER_FLAGS="${MSAN_LINK_FLAGS}" \
+ $LLVM && \
+ ninja clang) || echo @@@STEP_FAILURE@@@
+
+echo @@@BUILD_STEP check-llvm msan@@@
+
+(cd llvm_build_msan && ninja check-llvm) || echo @@@STEP_FAILURE@@@
+
+echo @@@BUILD_STEP check-clang msan@@@
+
+(cd llvm_build_msan && ninja check-clang) || echo @@@STEP_FAILURE@@@
+
+
+# Stage 3 / MemorySanitizer
+
+echo @@@BUILD_STEP build stage3/msan clang@@@
+
+if [ ! -d llvm_build2_msan ]; then
+ mkdir llvm_build2_msan
+fi
+
+CLANG_MSAN_PATH=$ROOT/llvm_build_msan/bin
+CMAKE_STAGE3_COMMON_OPTIONS="${CMAKE_COMMON_OPTIONS}"
+CMAKE_STAGE3_MSAN_OPTIONS="${CMAKE_STAGE3_COMMON_OPTIONS} -DCMAKE_C_COMPILER=${CLANG_MSAN_PATH}/clang -DCMAKE_CXX_COMPILER=${CLANG_MSAN_PATH}/clang++"
+
+(cd llvm_build2_msan && cmake ${CMAKE_STAGE3_MSAN_OPTIONS} $LLVM && ninja) || \
+ echo @@@STEP_FAILURE@@@
+
+
+echo @@@BUILD_STEP check-all stage3/msan@@@
+
+(cd llvm_build2_msan && ninja check-all) || echo @@@STEP_FAILURE@@@
+
+
+# Stage 2 / AddressSanitizer
+
+echo @@@BUILD_STEP build clang/asan@@@
+
+# Turn on init-order checker as ASan runtime option.
+export ASAN_OPTIONS="check_initialization_order=true:detect_stack_use_after_return=1:detect_leaks=1"
+CMAKE_ASAN_OPTIONS=" \
+ ${CMAKE_STAGE2_COMMON_OPTIONS} \
+ -DLLVM_USE_SANITIZER=Address \
+ "
+
+if [ ! -d llvm_build_asan ]; then
+ mkdir llvm_build_asan
+fi
+
+(cd llvm_build_asan && \
+ cmake ${CMAKE_ASAN_OPTIONS} $LLVM && \
+ ninja clang) || echo @@@STEP_FAILURE@@@
+
+
+echo @@@BUILD_STEP check-llvm asan@@@
+
+(cd llvm_build_asan && ninja check-llvm) || echo @@@STEP_FAILURE@@@
+
+
+echo @@@BUILD_STEP check-clang asan@@@
+
+(cd llvm_build_asan && ninja check-clang) || echo @@@STEP_FAILURE@@@
+
+
+# Stage 3 / AddressSanitizer
+
+echo @@@BUILD_STEP build stage3/asan clang@@@
+
+if [ ! -d llvm_build2_asan ]; then
+ mkdir llvm_build2_asan
+fi
+
+CLANG_ASAN_PATH=$ROOT/llvm_build_asan/bin
+CMAKE_STAGE3_COMMON_OPTIONS="${CMAKE_STAGE2_COMMON_OPTIONS}"
+CMAKE_STAGE3_ASAN_OPTIONS="${CMAKE_STAGE3_COMMON_OPTIONS} -DCMAKE_C_COMPILER=${CLANG_ASAN_PATH}/clang -DCMAKE_CXX_COMPILER=${CLANG_ASAN_PATH}/clang++"
+
+(cd llvm_build2_asan && cmake ${CMAKE_STAGE3_ASAN_OPTIONS} $LLVM && ninja) || \
+ echo @@@STEP_FAILURE@@@
+
+
+echo @@@BUILD_STEP check-all stage3/asan@@@
+
+(cd llvm_build2_asan && ninja check-all) || echo @@@STEP_FAILURE@@@
+
+# Stage 2 / UndefinedBehaviorSanitizer
+echo @@@BUILD_STEP build clang/ubsan@@@
+
+export UBSAN_OPTIONS="external_symbolizer_path=${LLVM_SYMBOLIZER_PATH}:print_stacktrace=1"
+CMAKE_UBSAN_OPTIONS=" \
+ ${CMAKE_STAGE2_COMMON_OPTIONS} \
+ -DCMAKE_BUILD_TYPE=Debug \
+ -DLLVM_USE_SANITIZER=Undefined \
+ "
+
+if [ ! -d llvm_build_ubsan ]; then
+ mkdir llvm_build_ubsan
+fi
+
+(cd llvm_build_ubsan &&
+ cmake ${CMAKE_UBSAN_OPTIONS} $LLVM && \
+ ninja clang) || echo @@@STEP_FAILURE@@@
+
+echo @@@BUILD_STEP check-llvm ubsan@@@
+(cd llvm_build_ubsan && ninja check-llvm) || echo @@@STEP_FAILURE@@@
+
+echo @@@BUILD_STEP check-clang ubsan@@@
+(cd llvm_build_ubsan && ninja check-clang) || echo @@@STEP_FAILURE@@@
Propchange: zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_bootstrap.sh
------------------------------------------------------------------------------
svn:executable = *
Added: zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_cmake.sh
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_cmake.sh?rev=219136&view=auto
==============================================================================
--- zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_cmake.sh (added)
+++ zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_cmake.sh Mon Oct 6 13:41:49 2014
@@ -0,0 +1,181 @@
+#!/usr/bin/env bash
+
+set -x
+set -e
+set -u
+
+# dump buildbot env
+env
+
+HERE="$(dirname $0)"
+. ${HERE}/buildbot_functions.sh
+
+ROOT=`pwd`
+PLATFORM=`uname`
+ARCH=`uname -m`
+export PATH="/usr/local/bin:$PATH"
+export ANDROID_SDK_HOME=$ROOT/../../..
+
+if [ "$BUILDBOT_CLOBBER" != "" ]; then
+ echo @@@BUILD_STEP clobber@@@
+ rm -rf llvm
+ rm -rf clang_build
+fi
+
+# Always clobber bootstrap build trees.
+rm -rf compiler_rt_build
+rm -rf llvm_build64
+rm -rf llvm_build_ninja
+
+SUPPORTS_32_BITS=${SUPPORTS_32_BITS:-1}
+MAKE_JOBS=${MAX_MAKE_JOBS:-16}
+LLVM_CHECKOUT=$ROOT/llvm
+COMPILER_RT_CHECKOUT=$LLVM_CHECKOUT/projects/compiler-rt
+CMAKE_COMMON_OPTIONS="-DLLVM_ENABLE_ASSERTIONS=ON"
+ENABLE_LIBCXX_FLAG=
+if [ "$PLATFORM" == "Darwin" ]; then
+ CMAKE_COMMON_OPTIONS="${CMAKE_COMMON_OPTIONS} -DPYTHON_EXECUTABLE=/usr/bin/python"
+ ENABLE_LIBCXX_FLAG="-DLLVM_ENABLE_LIBCXX=ON"
+fi
+
+BUILD_ANDROID=${BUILD_ANDROID:-0}
+RUN_ANDROID=${RUN_ANDROID:-0}
+if [ $BUILD_ANDROID == 1 -o $RUN_ANDROID == 1 ] ; then
+ . ${HERE}/buildbot_android_functions.sh
+fi
+
+
+echo @@@BUILD_STEP update@@@
+buildbot_update
+
+
+echo @@@BUILD_STEP lint@@@
+CHECK_LINT=${COMPILER_RT_CHECKOUT}/lib/sanitizer_common/scripts/check_lint.sh
+(LLVM_CHECKOUT=${LLVM_CHECKOUT} ${CHECK_LINT}) || echo @@@STEP_WARNINGS@@@
+
+# Use both gcc and just-built Clang as a host compiler for sanitizer tests.
+# Assume that self-hosted build tree should compile with -Werror.
+echo @@@BUILD_STEP build fresh clang@@@
+if [ ! -d clang_build ]; then
+ mkdir clang_build
+fi
+(cd clang_build && cmake -DCMAKE_BUILD_TYPE=Release \
+ ${CMAKE_COMMON_OPTIONS} $LLVM_CHECKOUT)
+(cd clang_build && make clang -j$MAKE_JOBS) || echo @@@STEP_FAILURE@@@
+
+# If we're building with libcxx, install the headers to clang_build/include.
+if [ ! -z ${ENABLE_LIBCXX_FLAG} ]; then
+(cd clang_build && make -C ${LLVM_CHECKOUT}/projects/libcxx installheaders \
+ HEADER_DIR=${PWD}/include) || echo @@@STEP_FAILURE@@@
+fi
+
+# Do a sanity check on Linux: build and test sanitizers using gcc as a host
+# compiler.
+if [ "$PLATFORM" == "Linux" ]; then
+ echo @@@BUILD_STEP run sanitizer tests in gcc build@@@
+ (cd clang_build && make -j$MAKE_JOBS check-sanitizer) || echo @@@STEP_FAILURE@@@
+ (cd clang_build && make -j$MAKE_JOBS check-asan) || echo @@@STEP_FAILURE@@@
+ (cd clang_build && make -j$MAKE_JOBS check-lsan) || echo @@@STEP_FAILURE@@@
+ (cd clang_build && make -j$MAKE_JOBS check-msan) || echo @@@STEP_FAILURE@@@
+ (cd clang_build && make -j$MAKE_JOBS check-tsan) || echo @@@STEP_FAILURE@@@
+ (cd clang_build && make -j$MAKE_JOBS check-ubsan) || echo @@@STEP_WARNINGS@@@
+ (cd clang_build && make -j$MAKE_JOBS check-dfsan) || echo @@@STEP_WARNINGS@@@
+fi
+
+### From now on we use just-built Clang as a host compiler ###
+CLANG_PATH=${ROOT}/clang_build/bin
+# Build self-hosted tree with fresh Clang and -Werror.
+CMAKE_CLANG_OPTIONS="${CMAKE_COMMON_OPTIONS} -DLLVM_ENABLE_WERROR=ON -DCMAKE_C_COMPILER=${CLANG_PATH}/clang -DCMAKE_CXX_COMPILER=${CLANG_PATH}/clang++"
+BUILD_TYPE=Release
+
+echo @@@BUILD_STEP bootstrap clang@@@
+if [ ! -d llvm_build64 ]; then
+ mkdir llvm_build64
+fi
+(cd llvm_build64 && cmake -DCMAKE_BUILD_TYPE=$BUILD_TYPE \
+ ${CMAKE_CLANG_OPTIONS} -DLLVM_BUILD_EXTERNAL_COMPILER_RT=ON \
+ ${ENABLE_LIBCXX_FLAG} $LLVM_CHECKOUT)
+# First, build only Clang.
+(cd llvm_build64 && make -j$MAKE_JOBS clang) || echo @@@STEP_FAILURE@@@
+
+# If needed, install the headers to clang_build/include.
+if [ ! -z ${ENABLE_LIBCXX_FLAG} ]; then
+(cd llvm_build64 && make -C ${LLVM_CHECKOUT}/projects/libcxx installheaders \
+ HEADER_DIR=${PWD}/include) || echo @@@STEP_FAILURE@@@
+fi
+
+# Now build everything else.
+(cd llvm_build64 && make -j$MAKE_JOBS) || echo @@@STEP_FAILURE@@@
+FRESH_CLANG_PATH=${ROOT}/llvm_build64/bin
+COMPILER_RT_BUILD_PATH=projects/compiler-rt/src/compiler-rt-build
+
+echo @@@BUILD_STEP run asan tests@@@
+(cd llvm_build64 && make -j$MAKE_JOBS check-asan) || echo @@@STEP_FAILURE@@@
+
+if [ "$PLATFORM" == "Linux" -a "$ARCH" == "x86_64" ]; then
+ echo @@@BUILD_STEP run msan unit tests@@@
+ (cd llvm_build64 && make -j$MAKE_JOBS check-msan) || echo @@@STEP_FAILURE@@@
+fi
+
+if [ "$PLATFORM" == "Linux" -a "$ARCH" == "x86_64" ]; then
+ echo @@@BUILD_STEP run 64-bit tsan unit tests@@@
+ (cd llvm_build64 && make -j$MAKE_JOBS check-tsan) || echo @@@STEP_FAILURE@@@
+fi
+
+if [ "$PLATFORM" == "Linux" -a "$ARCH" == "x86_64" ]; then
+ echo @@@BUILD_STEP run 64-bit lsan unit tests@@@
+ (cd llvm_build64 && make -j$MAKE_JOBS check-lsan) || echo @@@STEP_FAILURE@@@
+fi
+
+echo @@@BUILD_STEP run sanitizer_common tests@@@
+(cd llvm_build64 && make -j$MAKE_JOBS check-sanitizer) || echo @@@STEP_FAILURE@@@
+
+echo @@@BUILD_STEP build standalone compiler-rt@@@
+if [ ! -d compiler_rt_build ]; then
+ mkdir compiler_rt_build
+fi
+(cd compiler_rt_build && cmake -DCMAKE_BUILD_TYPE=$BUILD_TYPE \
+ -DCMAKE_C_COMPILER=${FRESH_CLANG_PATH}/clang \
+ -DCMAKE_CXX_COMPILER=${FRESH_CLANG_PATH}/clang++ \
+ -DCOMPILER_RT_INCLUDE_TESTS=ON \
+ -DCOMPILER_RT_ENABLE_WERROR=ON \
+ -DLLVM_CONFIG_PATH=${FRESH_CLANG_PATH}/llvm-config \
+ $COMPILER_RT_CHECKOUT)
+(cd compiler_rt_build && make -j$MAKE_JOBS) || echo @@@STEP_FAILURE@@@
+
+echo @@@BUILD_STEP test standalone compiler-rt@@@
+(cd compiler_rt_build && make -j$MAKE_JOBS check-all) || echo @@@STEP_FAILURE@@@
+
+HAVE_NINJA=${HAVE_NINJA:-1}
+if [ "$PLATFORM" == "Linux" -a $HAVE_NINJA == 1 ]; then
+ echo @@@BUILD_STEP run tests in ninja build tree@@@
+ if [ ! -d llvm_build_ninja ]; then
+ mkdir llvm_build_ninja
+ fi
+ CMAKE_NINJA_OPTIONS="${CMAKE_CLANG_OPTIONS} -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -G Ninja"
+ (cd llvm_build_ninja && cmake -DCMAKE_BUILD_TYPE=$BUILD_TYPE \
+ ${CMAKE_NINJA_OPTIONS} $LLVM_CHECKOUT)
+ ln -sf llvm_build_ninja/compile_commands.json $LLVM_CHECKOUT
+ (cd llvm_build_ninja && ninja check-asan) || echo @@@STEP_FAILURE@@@
+ (cd llvm_build_ninja && ninja check-sanitizer) || echo @@@STEP_FAILURE@@@
+ (cd llvm_build_ninja && ninja check-tsan) || echo @@@STEP_FAILURE@@@
+ (cd llvm_build_ninja && ninja check-msan) || echo @@@STEP_FAILURE@@@
+ (cd llvm_build_ninja && ninja check-lsan) || echo @@@STEP_FAILURE@@@
+ (cd llvm_build_ninja && ninja check-ubsan) || echo @@@STEP_WARNINGS@@@
+ (cd llvm_build_ninja && ninja check-dfsan) || echo @@@STEP_WARNINGS@@@
+fi
+
+if [ $BUILD_ANDROID == 1 ] ; then
+ echo @@@BUILD_STEP build Android runtime and tests@@@
+
+ build_compiler_rt arm arm-linux-androideabi
+ build_llvm_symbolizer arm arm-linux-androideabi
+
+ build_compiler_rt x86 i686-linux-android
+ build_llvm_symbolizer x86 i686-linux-android
+fi
+
+if [ $RUN_ANDROID == 1 ] ; then
+ trap "android_emulator_cleanup" EXIT
+ test_android arm arm-K
+fi
Added: zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_functions.sh
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_functions.sh?rev=219136&view=auto
==============================================================================
--- zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_functions.sh (added)
+++ zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_functions.sh Mon Oct 6 13:41:49 2014
@@ -0,0 +1,95 @@
+#!/usr/bin/env bash
+
+function buildbot_update {
+ REV_ARG=
+ if [ "$BUILDBOT_REVISION" != "" ]; then
+ REV_ARG="-r$BUILDBOT_REVISION"
+ fi
+ if [ -d llvm ]; then
+ svn cleanup llvm
+ fi
+ for subtree in llvm/tools/clang llvm/projects/compiler-rt llvm/projects/libcxx llvm/projects/libcxxabi
+ do
+ if [ -d ${subtree} ]; then
+ svn cleanup "${subtree}"
+ fi
+ done
+
+ if [ -d llvm -a -d llvm/projects/libcxxabi ]; then
+ svn up llvm $REV_ARG
+ if [ "$REV_ARG" == "" ]; then
+ REV_ARG="-r"$(svn info llvm | grep '^Revision:' | awk '{print $2}')
+ fi
+ for subtree in llvm/tools/clang llvm/projects/compiler-rt llvm/projects/libcxx llvm/projects/libcxxabi
+ do
+ svn up "${subtree}" $REV_ARG
+ done
+ else
+ svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm $REV_ARG
+ if [ "$REV_ARG" == "" ]; then
+ REV_ARG="-r"$(svn info llvm | grep '^Revision:' | awk '{print $2}')
+ fi
+ svn co http://llvm.org/svn/llvm-project/cfe/trunk llvm/tools/clang $REV_ARG
+ svn co http://llvm.org/svn/llvm-project/compiler-rt/trunk llvm/projects/compiler-rt $REV_ARG
+ svn co http://llvm.org/svn/llvm-project/libcxx/trunk llvm/projects/libcxx $REV_ARG
+ svn co http://llvm.org/svn/llvm-project/libcxxabi/trunk llvm/projects/libcxxabi $REV_ARG
+ fi
+}
+
+function set_chrome_suid_sandbox {
+ export CHROME_DEVEL_SANDBOX=/usr/local/sbin/chrome-devel-sandbox
+}
+
+function fetch_depot_tools {
+ ROOT=$1
+ (
+ cd $ROOT
+ if [ ! -d depot_tools ]; then
+ git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
+ fi
+ )
+ export PATH="$ROOT/depot_tools:$PATH"
+}
+
+function check_out_chromium {
+ CHROME_CHECKOUT=$1
+ (
+ if [ ! -d $CHROME_CHECKOUT ]; then
+ mkdir $CHROME_CHECKOUT
+ pushd $CHROME_CHECKOUT
+ fetch --nohooks chromium --nosvn=True
+
+ # Sync to LKGR, see http://crbug.com/109191
+ mv .gclient .gclient-tmp
+ cat .gclient-tmp | \
+ sed 's/"safesync_url": ""/"safesync_url": "https:\/\/chromium-status.appspot.com\/git-lkgr"/' > .gclient
+ rm .gclient-tmp
+ popd
+ fi
+ cd $CHROME_CHECKOUT/src
+ git checkout master
+ git pull
+ gclient sync --nohooks --jobs=16
+ )
+}
+
+function gclient_runhooks {
+ CHROME_CHECKOUT=$1
+ CLANG_BUILD=$2
+ CUSTOM_GYP_DEFINES=$3
+ (
+ cd $CHROME_CHECKOUT/src
+
+ # Clobber Chromium to catch possible LLVM regressions early.
+ rm -rf out/Release
+
+ export COMMON_GYP_DEFINES="use_allocator=none use_aura=1 clang_use_chrome_plugins=0 component=static_library"
+ export GYP_DEFINES="$CUSTOM_GYP_DEFINES $COMMON_GYP_DEFINES"
+ export GYP_GENERATORS=ninja
+ export CLANG_BIN=$CLANG_BUILD/bin
+ export CC="$CLANG_BIN/clang"
+ export CXX="$CLANG_BIN/clang++"
+
+ gclient runhooks
+ )
+}
Propchange: zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_functions.sh
------------------------------------------------------------------------------
svn:executable = *
Added: zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_perf_asan.sh
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_perf_asan.sh?rev=219136&view=auto
==============================================================================
--- zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_perf_asan.sh (added)
+++ zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_perf_asan.sh Mon Oct 6 13:41:49 2014
@@ -0,0 +1,93 @@
+#!/usr/bin/env bash
+
+set -x
+set -e
+set -u
+
+HERE="$(cd $(dirname $0) && pwd)"
+. ${HERE}/buildbot_functions.sh
+
+ROOT=`pwd`
+PLATFORM=`uname`
+export PATH="/usr/local/bin:$PATH"
+
+LLVM_CHECKOUT=$ROOT/llvm
+CLANG_BUILD=$ROOT/clang_build
+SPEC_DIRNAME=SPEC_CPU2006v1.2
+SPEC_SRC="${ROOT}/${SPEC_DIRNAME}"
+SPEC_RUNNER=./run_spec_clang_asan.sh
+#SPEC_TESTS='perlbench bzip2'
+SPEC_TESTS='perlbench bzip2 gcc mcf gobmk hmmer sjeng libquantum h264ref omnetpp astar xalancbmk'
+
+CMAKE_COMMON_OPTIONS="-GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON"
+
+echo @@@BUILD_STEP update@@@
+buildbot_update
+
+# LLVM build requires ninja.
+# TODO(glider): make a common function to fetch ninja.
+
+echo @@@BUILD_STEP fetch depot_tools@@@
+(
+ cd $ROOT
+ if [ ! -d depot_tools ]; then
+ git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
+ fi
+)
+export PATH="$ROOT/depot_tools:$PATH"
+
+echo @@@BUILD_STEP build fresh clang@@@
+(
+if [ ! -d $CLANG_BUILD ]; then
+ mkdir $CLANG_BUILD
+fi
+cd $CLANG_BUILD
+export PATH="$PATH:$ROOT/../../../ninja"
+cmake -DCMAKE_BUILD_TYPE=Release ${CMAKE_COMMON_OPTIONS} $LLVM_CHECKOUT
+ninja clang || echo @@@STEP_FAILURE@@@
+# TODO(glider): build other targets depending on the platform.
+# See https://code.google.com/p/address-sanitizer/wiki/HowToBuild.
+ninja clang_rt.asan-x86_64 clang_rt.asan-i386
+)
+
+
+echo @@@BUILD_STEP unpack, patch and install SPEC@@@
+(
+cd $ROOT
+if [ ! -d $SPEC_SRC ]; then
+ (
+ tar -jxf "$ROOT/../../../SPEC/${SPEC_DIRNAME}.tar.bz2"
+ cd $SPEC_SRC
+ wget https://address-sanitizer.googlecode.com/svn/trunk/spec/spec2006-asan.patch
+ yes | ./install.sh
+ patch -p1 -i spec2006-asan.patch
+ )
+fi
+
+if [ ! -e $SPEC_SRC/$SPEC_RUNNER ]; then
+ (
+ cd $SPEC_SRC
+ wget https://address-sanitizer.googlecode.com/svn/trunk/spec/run_spec_clang_asan.sh
+ chmod a+x $SPEC_RUNNER
+ )
+fi
+)
+
+export ASAN_BIN=$CLANG_BUILD/bin
+#export CC="$ASAN_BIN/clang"
+#export CXX="$ASAN_BIN/clang++"
+export PATH="$ASAN_BIN:$PATH"
+
+for test_name in $SPEC_TESTS
+do
+ echo @@@BUILD_STEP running $test_name@@@
+ (
+ set +x
+ cd $SPEC_SRC
+ name=asan-spec
+ size=test
+ SPEC_WRAPPER="perf record -q -o `pwd`/perf.data" $SPEC_RUNNER $name $size $test_name 2>&1 | tee asan-$test_name.log
+ grep "ERROR: AddressSanitizer" asan-$test_name.log && echo @@@STEP_FAILURE@@@
+ perf report
+ )
+done
Added: zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_selector.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_selector.py?rev=219136&view=auto
==============================================================================
--- zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_selector.py (added)
+++ zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_selector.py Mon Oct 6 13:41:49 2014
@@ -0,0 +1,67 @@
+#!/usr/bin/python
+
+import os
+import subprocess
+import sys
+
+THIS_DIR=os.path.dirname(sys.argv[0])
+
+
+def bash(path):
+ return 'bash ' + os.path.join(THIS_DIR, path)
+
+def cmd_call(path):
+ return 'call ' + os.path.join(THIS_DIR, path)
+
+BOT_ASSIGNMENT = {
+ 'win': cmd_call('buildbot_standard.bat'),
+ 'linux': bash('buildbot_standard.sh'),
+ 'linux-bootstrap': bash('buildbot_bootstrap.sh'),
+ 'linux-chrome-asan': bash('buildbot_chrome_asan.sh'),
+ 'linux-chrome-tsan': bash('buildbot_chrome_tsan.sh'),
+ 'linux-perf-asan': bash('buildbot_perf_asan.sh'),
+ 'mac10.9-cmake': bash('buildbot_cmake.sh'),
+ 'mac10.9': bash('buildbot_standard.sh'),
+ 'sanitizer-ppc64-linux1': bash('buildbot_cmake.sh'),
+ 'sanitizer-x86_64-linux': bash('buildbot_cmake.sh'),
+ 'sanitizer-x86_64-linux-bootstrap': bash('buildbot_bootstrap.sh'),
+}
+
+BOT_ADDITIONAL_ENV = {
+ 'win': {},
+ 'linux': { 'CHECK_TSAN': '1', 'BUILD_ASAN_ANDROID' : '1' },
+ 'linux-bootstrap': {},
+ 'linux-chrome-asan': {},
+ 'linux-chrome-tsan': {},
+ 'linux-perf-asan': {},
+ 'mac10.9-cmake': { 'MAX_MAKE_JOBS': '8' },
+ 'mac10.9': { 'MAX_MAKE_JOBS': '8' },
+ 'sanitizer-ppc64-linux1': { 'HAVE_NINJA': '0' },
+ 'sanitizer-x86_64-linux': { 'BUILD_ANDROID' : '1', 'RUN_ANDROID' : '0', 'HAVE_NINJA' : '1' },
+ 'sanitizer-x86_64-linux-bootstrap': {},
+}
+
+def Main():
+ builder = os.environ.get('BUILDBOT_BUILDERNAME')
+ print "builder name: %s" % (builder)
+ cmd = BOT_ASSIGNMENT.get(builder)
+ if not cmd:
+ sys.stderr.write('ERROR - unset/invalid builder name\n')
+ sys.exit(1)
+
+ print "%s runs: %s\n" % (builder, cmd)
+ sys.stdout.flush()
+
+ bot_env = os.environ
+ add_env = BOT_ADDITIONAL_ENV.get(builder)
+ for var in add_env:
+ bot_env[var] = add_env[var]
+ if 'TMPDIR' in bot_env:
+ del bot_env['TMPDIR']
+
+ retcode = subprocess.call(cmd, env=bot_env, shell=True)
+ sys.exit(retcode)
+
+
+if __name__ == '__main__':
+ Main()
Propchange: zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_selector.py
------------------------------------------------------------------------------
svn:executable = *
Added: zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_standard.bat
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_standard.bat?rev=219136&view=auto
==============================================================================
--- zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_standard.bat (added)
+++ zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_standard.bat Mon Oct 6 13:41:49 2014
@@ -0,0 +1,58 @@
+ at echo off
+
+:: TODO(timurrrr) echo @@@BUILD_STEP clobber@@@
+
+echo @@@BUILD_STEP update@@@
+:: TODO(timurrrr)
+::if [ "$BUILDBOT_CLOBBER" != "" ]; then
+:: echo @@@BUILD_STEP clobber build@@@
+:: rmdir /S /Q llvm || goto :DIE
+:: rmdir /S /Q llvm-build || goto :DIE
+:: mkdir llvm-build || goto :DIE
+::fi
+
+set REV_ARG=
+if NOT "%BUILDBOT_REVISION%" == "" set REV_ARG="-r%BUILDBOT_REVISION%"
+
+:: call -> because "svn" might be a batch script, ouch
+call svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm %REV_ARG% || goto :DIE
+call svn co http://llvm.org/svn/llvm-project/cfe/trunk llvm/tools/clang %REV_ARG% || goto :DIE
+call svn co http://llvm.org/svn/llvm-project/compiler-rt/trunk llvm/projects/compiler-rt %REV_ARG% || goto :DIE
+
+set ROOT=%cd%
+
+echo @@@BUILD_STEP cmake llvm@@@
+mkdir llvm-build
+cd llvm-build || goto :DIE
+cmake -GNinja -DLLVM_ENABLE_ASSERTIONS=ON -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD=X86 ..\llvm || goto :DIE
+
+echo @@@BUILD_STEP build compiler-rt@@@
+:: Clean compiler-rt to get all the compile-time warnings,
+:: then rebuild it separately before anything else to help us find ASan RTL
+:: compile-time bugs quicker.
+ninja -t clean compiler-rt
+ninja compiler-rt
+
+echo @@@BUILD_STEP build llvm@@@
+ninja || goto :DIE
+
+echo @@@BUILD_STEP run tests@@@
+ninja check-asan check-sanitizer || goto :DIE
+
+cd %ROOT%
+
+:: TODO(timurrrr)
+:: echo @@@BUILD_STEP build asan RTL with clang@@@
+
+echo "ALL DONE"
+goto :EOF
+
+:DIE
+:: TODO(timurrrr) : get the current process's PID?
+taskkill /F /IM cl.exe /T 2>err
+taskkill /F /IM clang.exe /T 2>err
+taskkill /F /IM clang-cl.exe /T 2>err
+taskkill /F /IM cmake.exe /T 2>err
+taskkill /F /IM MSBuild.exe /T 2>err
+taskkill /F /IM WerFault.exe /T 2>err
+exit /b 42
Added: zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_standard.sh
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_standard.sh?rev=219136&view=auto
==============================================================================
--- zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_standard.sh (added)
+++ zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_standard.sh Mon Oct 6 13:41:49 2014
@@ -0,0 +1,86 @@
+#!/usr/bin/env bash
+
+set -x
+set -e
+set -u
+
+HERE="$(dirname $0)"
+. ${HERE}/buildbot_functions.sh
+
+GCC_BUILD=/usr/local/gcc-4.8.2
+export PATH="$GCC_BUILD/bin:$PATH"
+export LD_LIBRARY_PATH=$GCC_BUILD/lib64
+
+if [ "$BUILDBOT_CLOBBER" != "" ]; then
+ echo @@@BUILD_STEP clobber@@@
+ rm -rf llvm
+ rm -rf llvm-build
+fi
+
+PLATFORM=`uname`
+MAKE_JOBS=${MAX_MAKE_JOBS:-16}
+BUILD_ASAN_ANDROID=${BUILD_ASAN_ANDROID:-0}
+CHECK_TSAN=${CHECK_TSAN:-0}
+
+echo @@@BUILD_STEP update@@@
+buildbot_update
+
+echo @@@BUILD_STEP build llvm@@@
+if [ ! -d llvm-build ]; then
+ mkdir llvm-build
+fi
+cd llvm-build
+../llvm/configure --enable-optimized
+make -j$MAKE_JOBS
+cd ..
+BUILD_ROOT=`pwd`
+CLANG_BUILD=$BUILD_ROOT/llvm-build/Release+Asserts
+
+echo @@@BUILD_STEP test llvm@@@
+cd llvm-build
+make check-all || echo @@@STEP_WARNINGS@@@
+
+echo @@@BUILD_STEP sanity check for sanitizer tools@@@
+CLANGXX_BINARY=$CLANG_BUILD/bin/clang++
+echo -e "#include <stdio.h>\nint main(){ return 0; }" > temp.cc
+for xsan in address undefined; do
+ $CLANGXX_BINARY -fsanitize=$xsan -m64 temp.cc -o a.out
+ ./a.out
+ $CLANGXX_BINARY -fsanitize=$xsan -m32 temp.cc -o a.out
+ ./a.out
+done
+if [ "$PLATFORM" == "Linux" ]; then
+ for xsan in thread memory; do
+ $CLANGXX_BINARY -fsanitize=$xsan -m64 temp.cc -o a.out
+ ./a.out
+ done
+fi
+
+if [ $BUILD_ASAN_ANDROID == 1 ] ; then
+ echo @@@BUILD_STEP build asan/android runtime@@@
+ make -j$MAKE_JOBS -C tools/clang/runtime/ \
+ LLVM_ANDROID_TOOLCHAIN_DIR=$BUILD_ROOT/../../../android-ndk/standalone
+fi
+
+if [ $CHECK_TSAN == 1 ] ; then
+ echo @@@BUILD_STEP prepare for testing tsan@@@
+
+ TSAN_PATH=$BUILD_ROOT/llvm/projects/compiler-rt/lib/tsan/
+ (cd $TSAN_PATH && make -f Makefile.old install_deps)
+
+ export PATH=$CLANG_BUILD/bin:$GCC_BUILD/bin:$PATH
+ export MAKEFLAGS=-j$MAKE_JOBS
+ gcc -v 2>tmp && grep "version" tmp
+ clang -v 2>tmp && grep "version" tmp
+
+ cd $BUILD_ROOT
+ if [ -d tsanv2 ]; then
+ (cd tsanv2 && svn up --ignore-externals)
+ else
+ svn co http://data-race-test.googlecode.com/svn/trunk/ tsanv2
+ fi
+ export RACECHECK_UNITTEST_PATH=$BUILD_ROOT/tsanv2/unittest
+
+ cp $BUILD_ROOT/../../../scripts/slave/test_tsan.sh $TSAN_PATH
+ (cd $TSAN_PATH && ./test_tsan.sh)
+fi
Added: zorg/trunk/zorg/buildbot/builders/sanitizers/test_tsan.sh
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/builders/sanitizers/test_tsan.sh?rev=219136&view=auto
==============================================================================
--- zorg/trunk/zorg/buildbot/builders/sanitizers/test_tsan.sh (added)
+++ zorg/trunk/zorg/buildbot/builders/sanitizers/test_tsan.sh Mon Oct 6 13:41:49 2014
@@ -0,0 +1,71 @@
+#!/usr/bin/env bash
+
+set -x
+set -e
+set -u
+
+echo @@@BUILD_STEP tsan build debug-clang@@@
+make -f Makefile.old clean
+make -f Makefile.old DEBUG=1 CC=clang CXX=clang++
+
+echo @@@BUILD_STEP tsan test debug-clang@@@
+./tsan_test
+
+echo @@@BUILD_STEP tsan stats/output@@@
+make -f Makefile.old clean
+make -f Makefile.old DEBUG=1 CC=clang CXX=clang++ CFLAGS="-DTSAN_COLLECT_STATS=1 -DTSAN_DEBUG_OUTPUT=2"
+
+echo @@@BUILD_STEP tsan build SHADOW_COUNT=4@@@
+make -f Makefile.old clean
+make -f Makefile.old DEBUG=1 CC=clang CXX=clang++ CFLAGS=-DTSAN_SHADOW_COUNT=4
+
+echo @@@BUILD_STEP tsan test SHADOW_COUNT=4@@@
+./tsan_test
+
+echo @@@BUILD_STEP tsan build SHADOW_COUNT=2@@@
+make -f Makefile.old clean
+make -f Makefile.old DEBUG=1 CC=clang CXX=clang++ CFLAGS=-DTSAN_SHADOW_COUNT=2
+
+echo @@@BUILD_STEP tsan test SHADOW_COUNT=2@@@
+./tsan_test
+
+echo @@@BUILD_STEP tsan build release-gcc@@@
+make -f Makefile.old clean
+make -f Makefile.old DEBUG=0 CC=gcc CXX=g++
+
+echo @@@BUILD_STEP tsan test release-gcc@@@
+./tsan_test
+
+echo @@@BUILD_STEP tsan output_tests@@@
+(cd ../../test/tsan && ./test_output.sh)
+
+echo @@@BUILD_STEP tsan analyze@@@
+./check_analyze.sh
+
+echo @@@BUILD_STEP tsan Go runtime@@@
+(cd go && ./buildgo.sh)
+
+echo @@@BUILD_STEP tsan racecheck_unittest@@@
+TSAN_PATH=`pwd`
+LIBTSAN_A=$TSAN_PATH/rtl/libtsan.a
+SUPPRESS_WARNINGS="-Wno-format-security -Wno-null-dereference -Wno-unused-private-field"
+EXTRA_COMPILER_FLAGS="-fsanitize=thread -DTHREAD_SANITIZER -fPIC -g -O2 $SUPPRESS_WARNINGS"
+(cd $RACECHECK_UNITTEST_PATH && \
+make clean && \
+OMIT_DYNAMIC_ANNOTATIONS_IMPL=1 make l64 -j16 CC=clang CXX=clang++ LDOPT="-pie -Wl,--whole-archive $LIBTSAN_A -Wl,--no-whole-archive -ldl" OMIT_CPP0X=1 EXTRA_CFLAGS="$EXTRA_COMPILER_FLAGS" EXTRA_CXXFLAGS="$EXTRA_COMPILER_FLAGS" && \
+bin/racecheck_unittest-linux-amd64-O0 --gtest_filter=-*Ignore*:*Suppress*:*EnableRaceDetectionTest*:*Rep*Test*:*NotPhb*:*Barrier*:*Death*:*PositiveTests_RaceInSignal*:StressTests.FlushStateTest:*Mmap84GTest:*.LibcStringFunctions:LockTests.UnlockingALockHeldByAnotherThread:LockTests.UnlockTwice:PrintfTests.RaceOnPutsArgument)
+
+#Ignore: ignores do not work yet
+#Suppress: suppressions do not work yet
+#EnableRaceDetectionTest: the annotation is not supported
+#Rep*Test: uses inline assembly
+#NotPhb: not-phb is not supported
+#Barrier: pthread_barrier_t is not fully supported yet
+#Death: there is some flakyness
+#PositiveTests_RaceInSignal: signal() is not intercepted yet
+#StressTests.FlushStateTest: uses suppressions
+#Mmap84GTest: too slow, causes paging
+#LockTests.UnlockingALockHeldByAnotherThread: causes tsan report and non-zero exit code
+#LockTests.UnlockTwice: causes tsan report and non-zero exit code
+#PrintfTests.RaceOnPutsArgument: seems to be an issue with tsan shadow eviction, lit tests contain a similar test and it passes
+
Propchange: zorg/trunk/zorg/buildbot/builders/sanitizers/test_tsan.sh
------------------------------------------------------------------------------
svn:executable = *
More information about the llvm-commits
mailing list