<div dir="ltr">Excellent, thanks! <div>Do you already have some stats? (average time, average # of CLs covered, etc). </div><div>I wonder if we can make it work for every CL? </div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Dec 17, 2014 at 10:22 AM, Sergey Matveev <span dir="ltr"><<a href="mailto:earthdok@google.com" target="_blank">earthdok@google.com</a>></span> wrote:<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Author: smatveev<br>
Date: Wed Dec 17 12:22:17 2014<br>
New Revision: 224445<br>
<br>
URL: <a href="http://llvm.org/viewvc/llvm-project?rev=224445&view=rev" target="_blank">http://llvm.org/viewvc/llvm-project?rev=224445&view=rev</a><br>
Log:<br>
Add sanitizer-x86_64-linux-fast bot.<br>
<br>
Add a new sanitizer bot. The host compiler is clang fixed at a specific<br>
revision; it will not be rebuilt unless the revision number is changed in the<br>
bot config. The host compiler does incremental builds of (A,M)San-instrumented<br>
LLVM/clang, which are then tested with check-llvm, check-clang.<br>
<br>
Added:<br>
zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fast.sh (with props)<br>
Modified:<br>
zorg/trunk/buildbot/osuosl/master/config/builders.py<br>
zorg/trunk/buildbot/osuosl/master/config/slaves.py<br>
zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_selector.py<br>
<br>
Modified: zorg/trunk/buildbot/osuosl/master/config/builders.py<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/zorg/trunk/buildbot/osuosl/master/config/builders.py?rev=224445&r1=224444&r2=224445&view=diff" target="_blank">http://llvm.org/viewvc/llvm-project/zorg/trunk/buildbot/osuosl/master/config/builders.py?rev=224445&r1=224444&r2=224445&view=diff</a><br>
==============================================================================<br>
--- zorg/trunk/buildbot/osuosl/master/config/builders.py (original)<br>
+++ zorg/trunk/buildbot/osuosl/master/config/builders.py Wed Dec 17 12:22:17 2014<br>
@@ -661,6 +661,11 @@ def _get_sanitizer_builders():<br>
'builddir': "sanitizer-x86_64-linux-bootstrap",<br>
'factory': SanitizerBuilder.getSanitizerBuildFactory()},<br>
<br>
+ {'name': "sanitizer-x86_64-linux-fast",<br>
+ 'slavenames' :["sanitizer-buildbot3"],<br>
+ 'builddir': "sanitizer-x86_64-linux-fast",<br>
+ 'factory': SanitizerBuilder.getSanitizerBuildFactory()},<br>
+<br>
#{'name': "llvm-clang-lld-x86_64-ubuntu-sanitize-address",<br>
# 'slavenames':["hexagon-build-03"],<br>
# 'builddir':"llvm-clang-lld-x86_64-ubuntu-sanitize-address",<br>
<br>
Modified: zorg/trunk/buildbot/osuosl/master/config/slaves.py<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/zorg/trunk/buildbot/osuosl/master/config/slaves.py?rev=224445&r1=224444&r2=224445&view=diff" target="_blank">http://llvm.org/viewvc/llvm-project/zorg/trunk/buildbot/osuosl/master/config/slaves.py?rev=224445&r1=224444&r2=224445&view=diff</a><br>
==============================================================================<br>
--- zorg/trunk/buildbot/osuosl/master/config/slaves.py (original)<br>
+++ zorg/trunk/buildbot/osuosl/master/config/slaves.py Wed Dec 17 12:22:17 2014<br>
@@ -153,10 +153,12 @@ def get_build_slaves():<br>
# AArch64 Clang, Juno ARM Development Platform<br>
create_slave("juno-aarch64-01", properties={'jobs': 4}, max_builds=1),<br>
<br>
- # Ubuntu x86-64, 12x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz<br>
+ # Debian 7.7 x86_64 16-core GCE instance<br>
create_slave("sanitizer-buildbot1", properties={'jobs': 8}, max_builds=1),<br>
- # Ubuntu x86-64, 24x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz<br>
+ # Debian 7.7 x86_64 16-core GCE instance<br>
create_slave("sanitizer-buildbot2", properties={'jobs': 8}, max_builds=1),<br>
+ # Debian 7.7 x86_64 8-core GCE instance<br>
+ create_slave("sanitizer-buildbot3", properties={'jobs': 8}, max_builds=1),<br>
<br>
# zEnterprise 196 (s390x), SLES 11 SP2<br>
create_slave("systemz-1", properties={'jobs': 4}, max_builds=1),<br>
<br>
Added: zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fast.sh<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fast.sh?rev=224445&view=auto" target="_blank">http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fast.sh?rev=224445&view=auto</a><br>
==============================================================================<br>
--- zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fast.sh (added)<br>
+++ zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fast.sh Wed Dec 17 12:22:17 2014<br>
@@ -0,0 +1,156 @@<br>
+#!/usr/bin/env bash<br>
+<br>
+set -x<br>
+set -e<br>
+set -u<br>
+<br>
+HERE="$(cd $(dirname $0) && pwd)"<br>
+. ${HERE}/buildbot_functions.sh<br>
+<br>
+ROOT=`pwd`<br>
+PLATFORM=`uname`<br>
+export PATH="/usr/local/bin:$PATH"<br>
+<br>
+if [ "$BUILDBOT_CLOBBER" != "" ]; then<br>
+ echo @@@BUILD_STEP clobber@@@<br>
+ rm -rf llvm<br>
+ rm -rf llvm_build0<br>
+fi<br>
+<br>
+HOST_CLANG_REVISION=223108<br>
+<br>
+MEMORY_SANITIZER_KIND="Memory"<br>
+BUILDBOT_MSAN_ORIGINS=${BUILDBOT_MSAN_ORIGINS:-}<br>
+if [ "$BUILDBOT_MSAN_ORIGINS" != "" ]; then<br>
+ MEMORY_SANITIZER_KIND="MemoryWithOrigins"<br>
+fi<br>
+<br>
+MAKE_JOBS=${MAX_MAKE_JOBS:-16}<br>
+LLVM=$ROOT/llvm<br>
+<br>
+type -a gcc<br>
+type -a g++<br>
+CMAKE_COMMON_OPTIONS="-GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_PARALLEL_LINK_JOBS=3"<br>
+CMAKE_STAGE1_OPTIONS="${CMAKE_COMMON_OPTIONS}"<br>
+<br>
+# Stage 1<br>
+<br>
+if [ -r host_clang_revision ] && \<br>
+ [ "$(cat host_clang_revision)" == $HOST_CLANG_REVISION ]<br>
+then<br>
+ # Do nothing.<br>
+ echo @@@BUILD_STEP using pre-built stage1 clang at r$HOST_CLANG_REVISION@@@<br>
+else<br>
+ echo @@@BUILD_STEP sync to r$HOST_CLANG_REVISION@@@<br>
+ REAL_BUILDBOT_REVISION=$BUILDBOT_REVISION<br>
+ BUILDBOT_REVISION=$HOST_CLANG_REVISION<br>
+ buildbot_update<br>
+<br>
+ echo @@@BUILD_STEP build stage1 clang at r$HOST_CLANG_REVISION@@@<br>
+<br>
+ rm -rf host_clang_revision<br>
+<br>
+ # CMake does not notice that the compiler itself has changed. Anyway,<br>
+ # incremental builds of stage2 don't make sense if stage1 compiler has<br>
+ # changed. Clobber the build trees.<br>
+ rm -rf libcxx_build_msan<br>
+ rm -rf llvm_build_msan<br>
+ rm -rf llvm_build_asan<br>
+<br>
+ if [ ! -d llvm_build0 ]; then<br>
+ mkdir llvm_build0<br>
+ fi<br>
+ # Build cxx/cxxabi to fool the compiler check in MSan stage 2.<br>
+ (cd llvm_build0 && cmake ${CMAKE_STAGE1_OPTIONS} $LLVM && \<br>
+ ninja clang compiler-rt llvm-symbolizer)<br>
+ echo $HOST_CLANG_REVISION > host_clang_revision<br>
+<br>
+ BUILDBOT_REVISION=$REAL_BUILDBOT_REVISION<br>
+fi<br>
+<br>
+echo @@@BUILD_STEP update@@@<br>
+buildbot_update<br>
+<br>
+CLANG_PATH=$ROOT/llvm_build0/bin<br>
+CMAKE_STAGE2_COMMON_OPTIONS="\<br>
+ ${CMAKE_COMMON_OPTIONS} \<br>
+ -DCMAKE_C_COMPILER=${CLANG_PATH}/clang \<br>
+ -DCMAKE_CXX_COMPILER=${CLANG_PATH}/clang++ \<br>
+ "<br>
+LLVM_SYMBOLIZER_PATH=${CLANG_PATH}/llvm-symbolizer<br>
+export ASAN_SYMBOLIZER_PATH=${LLVM_SYMBOLIZER_PATH}<br>
+export MSAN_SYMBOLIZER_PATH=${LLVM_SYMBOLIZER_PATH}<br>
+<br>
+<br>
+# Stage 2 / Memory Sanitizer<br>
+<br>
+echo @@@BUILD_STEP build libcxx/msan@@@<br>
+if [ ! -d libcxx_build_msan ]; then<br>
+ mkdir libcxx_build_msan<br>
+fi<br>
+<br>
+(cd libcxx_build_msan && \<br>
+ cmake \<br>
+ ${CMAKE_STAGE2_COMMON_OPTIONS} \<br>
+ -DLLVM_USE_SANITIZER=${MEMORY_SANITIZER_KIND} \<br>
+ $LLVM && \<br>
+ ninja cxx cxxabi) || echo @@@STEP_FAILURE@@@<br>
+<br>
+<br>
+echo @@@BUILD_STEP build clang/msan@@@<br>
+if [ ! -d llvm_build_msan ]; then<br>
+ mkdir llvm_build_msan<br>
+fi<br>
+<br>
+MSAN_LDFLAGS="-lc++abi -Wl,--rpath=${ROOT}/libcxx_build_msan/lib -L${ROOT}/libcxx_build_msan/lib"<br>
+# See <a href="http://llvm.org/bugs/show_bug.cgi?id=19071" target="_blank">http://llvm.org/bugs/show_bug.cgi?id=19071</a>, <a href="http://www.cmake.org/Bug/view.php?id=15264" target="_blank">http://www.cmake.org/Bug/view.php?id=15264</a><br>
+CMAKE_BUG_WORKAROUND_CFLAGS="$MSAN_LDFLAGS -fsanitize=memory -w"<br>
+MSAN_CFLAGS="-I${ROOT}/libcxx_build_msan/include -I${ROOT}/libcxx_build_msan/include/c++/v1 $CMAKE_BUG_WORKAROUND_CFLAGS"<br>
+<br>
+(cd llvm_build_msan && \<br>
+ cmake ${CMAKE_STAGE2_COMMON_OPTIONS} \<br>
+ -DLLVM_USE_SANITIZER=${MEMORY_SANITIZER_KIND} \<br>
+ -DLLVM_ENABLE_LIBCXX=ON \<br>
+ -DCMAKE_C_FLAGS="${MSAN_CFLAGS}" \<br>
+ -DCMAKE_CXX_FLAGS="${MSAN_CFLAGS}" \<br>
+ -DCMAKE_EXE_LINKER_FLAGS="${MSAN_LDFLAGS}" \<br>
+ $LLVM && \<br>
+ ninja clang) || echo @@@STEP_FAILURE@@@<br>
+<br>
+echo @@@BUILD_STEP check-llvm msan@@@<br>
+<br>
+(cd llvm_build_msan && ninja check-llvm) || echo @@@STEP_WARNINGS@@@<br>
+<br>
+echo @@@BUILD_STEP check-clang msan@@@<br>
+<br>
+(cd llvm_build_msan && ninja check-clang) || echo @@@STEP_FAILURE@@@<br>
+<br>
+<br>
+# Stage 2 / AddressSanitizer<br>
+<br>
+echo @@@BUILD_STEP build clang/asan@@@<br>
+<br>
+# Turn on init-order checker as ASan runtime option.<br>
+export ASAN_OPTIONS="check_initialization_order=true:detect_stack_use_after_return=1:detect_leaks=1"<br>
+CMAKE_ASAN_OPTIONS=" \<br>
+ ${CMAKE_STAGE2_COMMON_OPTIONS} \<br>
+ -DLLVM_USE_SANITIZER=Address \<br>
+ "<br>
+<br>
+if [ ! -d llvm_build_asan ]; then<br>
+ mkdir llvm_build_asan<br>
+fi<br>
+<br>
+(cd llvm_build_asan && \<br>
+ cmake ${CMAKE_ASAN_OPTIONS} $LLVM && \<br>
+ ninja clang) || echo @@@STEP_FAILURE@@@<br>
+<br>
+<br>
+echo @@@BUILD_STEP check-llvm asan@@@<br>
+<br>
+(cd llvm_build_asan && ninja check-llvm) || echo @@@STEP_FAILURE@@@<br>
+<br>
+<br>
+echo @@@BUILD_STEP check-clang asan@@@<br>
+<br>
+(cd llvm_build_asan && ninja check-clang) || echo @@@STEP_FAILURE@@@<br>
<br>
Propchange: zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_fast.sh<br>
------------------------------------------------------------------------------<br>
svn:executable = *<br>
<br>
Modified: zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_selector.py<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_selector.py?rev=224445&r1=224444&r2=224445&view=diff" target="_blank">http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_selector.py?rev=224445&r1=224444&r2=224445&view=diff</a><br>
==============================================================================<br>
--- zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_selector.py (original)<br>
+++ zorg/trunk/zorg/buildbot/builders/sanitizers/buildbot_selector.py Wed Dec 17 12:22:17 2014<br>
@@ -25,6 +25,7 @@ BOT_ASSIGNMENT = {<br>
'sanitizer-ppc64-linux1': bash('buildbot_cmake.sh'),<br>
'sanitizer-x86_64-linux': bash('buildbot_cmake.sh'),<br>
'sanitizer-x86_64-linux-bootstrap': bash('buildbot_bootstrap.sh'),<br>
+ 'sanitizer-x86_64-linux-fast': bash('buildbot_fast.sh'),<br>
}<br>
<br>
BOT_ADDITIONAL_ENV = {<br>
@@ -39,6 +40,7 @@ BOT_ADDITIONAL_ENV = {<br>
'sanitizer-ppc64-linux1': { 'HAVE_NINJA': '0' },<br>
'sanitizer-x86_64-linux': { 'BUILD_ANDROID' : '1', 'RUN_ANDROID' : '1', 'HAVE_NINJA' : '1' },<br>
'sanitizer-x86_64-linux-bootstrap': {},<br>
+ 'sanitizer-x86_64-linux-fast': {},<br>
}<br>
<br>
def Main():<br>
<br>
<br>
_______________________________________________<br>
llvm-commits mailing list<br>
<a href="mailto:llvm-commits@cs.uiuc.edu">llvm-commits@cs.uiuc.edu</a><br>
<a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a><br>
</blockquote></div></div>