<div dir="ltr">I cannot reproduce this. I managed to build and run the "standalone compiler-rt" test environment locally (the docs are missing several required pieces...) but everything passes. Does someone have ssh access to this bot to investigate on the bot itself?<br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 25, 2016 at 3:43 PM, Evgenii Stepanov <span dir="ltr"><<a href="mailto:eugeni.stepanov@gmail.com" target="_blank">eugeni.stepanov@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
this test is failing on the sanitizer bot:<br>
<a href="http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/24103/steps/test%20standalone%20compiler-rt/logs/stdio" rel="noreferrer" target="_blank">http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/24103/steps/test%20standalone%20compiler-rt/logs/stdio</a><br>
<br>
You probably did not get the mail from the bot because there were<br>
other build errors.<br>
<br>
<br>
On Tue, May 24, 2016 at 7:04 PM, Derek Bruening via llvm-commits<br>
<<a href="mailto:llvm-commits@lists.llvm.org">llvm-commits@lists.llvm.org</a>> wrote:<br>
> Author: bruening<br>
> Date: Tue May 24 21:04:04 2016<br>
> New Revision: 270650<br>
><br>
> URL: <a href="http://llvm.org/viewvc/llvm-project?rev=270650&view=rev" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project?rev=270650&view=rev</a><br>
> Log:<br>
> Add working set base runtime library<br>
><br>
> Summary:<br>
> Adds the base runtime library for the working set tool.<br>
> Adds slowpath code for updating the shadow memory.<br>
><br>
> To be added in the future:<br>
> + Scan memory and report the total size.<br>
> + Take samples for intermediate values.<br>
><br>
> Reviewers: aizatsky<br>
><br>
> Subscribers: kubabrecka, vitalybuka, zhaoqin, kcc, eugenis, llvm-commits<br>
><br>
> Differential Revision: <a href="http://reviews.llvm.org/D20485" rel="noreferrer" target="_blank">http://reviews.llvm.org/D20485</a><br>
><br>
> Added:<br>
> compiler-rt/trunk/lib/esan/working_set.cpp (with props)<br>
> compiler-rt/trunk/lib/esan/working_set.h (with props)<br>
> compiler-rt/trunk/test/esan/TestCases/workingset-memset.cpp (with props)<br>
> Modified:<br>
> compiler-rt/trunk/lib/esan/CMakeLists.txt<br>
> compiler-rt/trunk/lib/esan/esan.cpp<br>
> compiler-rt/trunk/lib/esan/esan_interface_internal.h<br>
> compiler-rt/trunk/test/esan/lit.cfg<br>
><br>
> Modified: compiler-rt/trunk/lib/esan/CMakeLists.txt<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/esan/CMakeLists.txt?rev=270650&r1=270649&r2=270650&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/esan/CMakeLists.txt?rev=270650&r1=270649&r2=270650&view=diff</a><br>
> ==============================================================================<br>
> --- compiler-rt/trunk/lib/esan/CMakeLists.txt (original)<br>
> +++ compiler-rt/trunk/lib/esan/CMakeLists.txt Tue May 24 21:04:04 2016<br>
> @@ -11,7 +11,8 @@ set(ESAN_SOURCES<br>
> esan.cpp<br>
> esan_flags.cpp<br>
> esan_interface.cpp<br>
> - esan_interceptors.cpp)<br>
> + esan_interceptors.cpp<br>
> + working_set.cpp)<br>
><br>
> foreach (arch ${ESAN_SUPPORTED_ARCH})<br>
> add_compiler_rt_runtime(clang_rt.esan<br>
><br>
> Modified: compiler-rt/trunk/lib/esan/esan.cpp<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/esan/esan.cpp?rev=270650&r1=270649&r2=270650&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/esan/esan.cpp?rev=270650&r1=270649&r2=270650&view=diff</a><br>
> ==============================================================================<br>
> --- compiler-rt/trunk/lib/esan/esan.cpp (original)<br>
> +++ compiler-rt/trunk/lib/esan/esan.cpp Tue May 24 21:04:04 2016<br>
> @@ -19,6 +19,7 @@<br>
> #include "sanitizer_common/sanitizer_common.h"<br>
> #include "sanitizer_common/sanitizer_flag_parser.h"<br>
> #include "sanitizer_common/sanitizer_flags.h"<br>
> +#include "working_set.h"<br>
><br>
> // See comment below.<br>
> extern "C" {<br>
> @@ -31,6 +32,15 @@ bool EsanIsInitialized;<br>
> ToolType WhichTool;<br>
> ShadowMapping Mapping;<br>
><br>
> +// Different tools use different scales within the same shadow mapping scheme.<br>
> +// The scale used here must match that used by the compiler instrumentation.<br>
> +// This array is indexed by the ToolType enum.<br>
> +static const uptr ShadowScale[] = {<br>
> + 0, // ESAN_None.<br>
> + 2, // ESAN_CacheFrag: 4B:1B, so 4 to 1 == >>2.<br>
> + 6, // ESAN_WorkingSet: 64B:1B, so 64 to 1 == >>6.<br>
> +};<br>
> +<br>
> // We are combining multiple performance tuning tools under the umbrella of<br>
> // one EfficiencySanitizer super-tool. Most of our tools have very similar<br>
> // memory access instrumentation, shadow memory mapping, libc interception,<br>
> @@ -57,6 +67,8 @@ void processRangeAccess(uptr PC, uptr Ad<br>
> if (WhichTool == ESAN_CacheFrag) {<br>
> // TODO(bruening): add shadow mapping and update shadow bits here.<br>
> // We'll move this to cache_frag.cpp once we have something.<br>
> + } else if (WhichTool == ESAN_WorkingSet) {<br>
> + processRangeAccessWorkingSet(PC, Addr, Size, IsWrite);<br>
> }<br>
> }<br>
><br>
> @@ -113,10 +125,7 @@ static bool verifyShadowScheme() {<br>
> static void initializeShadow() {<br>
> DCHECK(verifyShadowScheme());<br>
><br>
> - if (WhichTool == ESAN_CacheFrag)<br>
> - Mapping.initialize(2); // 4B:1B, so 4 to 1 == >>2.<br>
> - else<br>
> - UNREACHABLE("unknown tool shadow mapping");<br>
> + Mapping.initialize(ShadowScale[WhichTool]);<br>
><br>
> VPrintf(1, "Shadow scale=%d offset=%p\n", Mapping.Scale, Mapping.Offset);<br>
><br>
> @@ -157,7 +166,7 @@ void initializeLibrary(ToolType Tool) {<br>
> ::__cxa_atexit((void (*)())finalizeLibrary);<br>
><br>
> VPrintf(1, "in esan::%s\n", __FUNCTION__);<br>
> - if (WhichTool != ESAN_CacheFrag) {<br>
> + if (WhichTool <= ESAN_None || WhichTool >= ESAN_Max) {<br>
> Printf("ERROR: unknown tool %d requested\n", WhichTool);<br>
> Die();<br>
> }<br>
> @@ -165,6 +174,12 @@ void initializeLibrary(ToolType Tool) {<br>
> initializeShadow();<br>
> initializeInterceptors();<br>
><br>
> + if (WhichTool == ESAN_CacheFrag) {<br>
> + // FIXME: add runtime code for this tool<br>
> + } else if (WhichTool == ESAN_WorkingSet) {<br>
> + initializeWorkingSet();<br>
> + }<br>
> +<br>
> EsanIsInitialized = true;<br>
> }<br>
><br>
> @@ -175,6 +190,8 @@ int finalizeLibrary() {<br>
> // strategy for how to generate a final report.<br>
> // We'll move this to cache_frag.cpp once we have something.<br>
> Report("%s is not finished: nothing yet to report\n", SanitizerToolName);<br>
> + } else if (WhichTool == ESAN_WorkingSet) {<br>
> + return finalizeWorkingSet();<br>
> }<br>
> return 0;<br>
> }<br>
><br>
> Modified: compiler-rt/trunk/lib/esan/esan_interface_internal.h<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/esan/esan_interface_internal.h?rev=270650&r1=270649&r2=270650&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/esan/esan_interface_internal.h?rev=270650&r1=270649&r2=270650&view=diff</a><br>
> ==============================================================================<br>
> --- compiler-rt/trunk/lib/esan/esan_interface_internal.h (original)<br>
> +++ compiler-rt/trunk/lib/esan/esan_interface_internal.h Tue May 24 21:04:04 2016<br>
> @@ -28,6 +28,8 @@ extern "C" {<br>
> typedef enum Type : u32 {<br>
> ESAN_None = 0,<br>
> ESAN_CacheFrag,<br>
> + ESAN_WorkingSet,<br>
> + ESAN_Max,<br>
> } ToolType;<br>
><br>
> // This function should be called at the very beginning of the process,<br>
><br>
> Added: compiler-rt/trunk/lib/esan/working_set.cpp<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/esan/working_set.cpp?rev=270650&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/esan/working_set.cpp?rev=270650&view=auto</a><br>
> ==============================================================================<br>
> --- compiler-rt/trunk/lib/esan/working_set.cpp (added)<br>
> +++ compiler-rt/trunk/lib/esan/working_set.cpp Tue May 24 21:04:04 2016<br>
> @@ -0,0 +1,90 @@<br>
> +//===-- working_set.cpp ---------------------------------------------------===//<br>
> +//<br>
> +// The LLVM Compiler Infrastructure<br>
> +//<br>
> +// This file is distributed under the University of Illinois Open Source<br>
> +// License. See LICENSE.TXT for details.<br>
> +//<br>
> +//===----------------------------------------------------------------------===//<br>
> +//<br>
> +// This file is a part of EfficiencySanitizer, a family of performance tuners.<br>
> +//<br>
> +// This file contains working-set-specific code.<br>
> +//===----------------------------------------------------------------------===//<br>
> +<br>
> +#include "working_set.h"<br>
> +#include "esan.h"<br>
> +#include "esan_flags.h"<br>
> +#include "esan_shadow.h"<br>
> +<br>
> +// We shadow every cache line of app memory with one shadow byte.<br>
> +// - The highest bit of each shadow byte indicates whether the corresponding<br>
> +// cache line has ever been accessed.<br>
> +// - The lowest bit of each shadow byte indicates whether the corresponding<br>
> +// cache line was accessed since the last sample.<br>
> +// - The other bits can be used either for a single working set snapshot<br>
> +// between two consecutive samples, or an aggregate working set snapshot<br>
> +// over multiple sample periods (future work).<br>
> +// We live with races in accessing each shadow byte.<br>
> +typedef unsigned char byte;<br>
> +<br>
> +namespace __esan {<br>
> +<br>
> +// See the shadow byte layout description above.<br>
> +static const u32 TotalWorkingSetBitIdx = 7;<br>
> +static const u32 CurWorkingSetBitIdx = 0;<br>
> +static const byte ShadowAccessedVal =<br>
> + (1 << TotalWorkingSetBitIdx) | (1 << CurWorkingSetBitIdx);<br>
> +<br>
> +void processRangeAccessWorkingSet(uptr PC, uptr Addr, SIZE_T Size,<br>
> + bool IsWrite) {<br>
> + if (Size == 0)<br>
> + return;<br>
> + SIZE_T I = 0;<br>
> + uptr LineSize = getFlags()->cache_line_size;<br>
> + // As Addr+Size could overflow at the top of a 32-bit address space,<br>
> + // we avoid the simpler formula that rounds the start and end.<br>
> + SIZE_T NumLines = Size / LineSize +<br>
> + // Add any extra at the start or end adding on an extra line:<br>
> + (LineSize - 1 + Addr % LineSize + Size % LineSize) / LineSize;<br>
> + byte *Shadow = (byte *)appToShadow(Addr);<br>
> + // Write shadow bytes until we're word-aligned.<br>
> + while (I < NumLines && (uptr)Shadow % 4 != 0) {<br>
> + if ((*Shadow & ShadowAccessedVal) != ShadowAccessedVal)<br>
> + *Shadow |= ShadowAccessedVal;<br>
> + ++Shadow;<br>
> + ++I;<br>
> + }<br>
> + // Write whole shadow words at a time.<br>
> + // Using a word-stride loop improves the runtime of a microbenchmark of<br>
> + // memset calls by 10%.<br>
> + u32 WordValue = ShadowAccessedVal | ShadowAccessedVal << 8 |<br>
> + ShadowAccessedVal << 16 | ShadowAccessedVal << 24;<br>
> + while (I + 4 <= NumLines) {<br>
> + if ((*(u32*)Shadow & WordValue) != WordValue)<br>
> + *(u32*)Shadow |= WordValue;<br>
> + Shadow += 4;<br>
> + I += 4;<br>
> + }<br>
> + // Write any trailing shadow bytes.<br>
> + while (I < NumLines) {<br>
> + if ((*Shadow & ShadowAccessedVal) != ShadowAccessedVal)<br>
> + *Shadow |= ShadowAccessedVal;<br>
> + ++Shadow;<br>
> + ++I;<br>
> + }<br>
> +}<br>
> +<br>
> +void initializeWorkingSet() {<br>
> + // The shadow mapping assumes 64 so this cannot be changed.<br>
> + CHECK(getFlags()->cache_line_size == 64);<br>
> +}<br>
> +<br>
> +int finalizeWorkingSet() {<br>
> + // FIXME NYI: we need to add memory scanning to report the total lines<br>
> + // touched, and later add sampling to get intermediate values.<br>
> + Report("%s is not finished: nothing yet to report\n", SanitizerToolName);<br>
> + return 0;<br>
> +}<br>
> +<br>
> +} // namespace __esan<br>
><br>
> Propchange: compiler-rt/trunk/lib/esan/working_set.cpp<br>
> ------------------------------------------------------------------------------<br>
> svn:eol-style = LF<br>
><br>
> Added: compiler-rt/trunk/lib/esan/working_set.h<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/esan/working_set.h?rev=270650&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/esan/working_set.h?rev=270650&view=auto</a><br>
> ==============================================================================<br>
> --- compiler-rt/trunk/lib/esan/working_set.h (added)<br>
> +++ compiler-rt/trunk/lib/esan/working_set.h Tue May 24 21:04:04 2016<br>
> @@ -0,0 +1,30 @@<br>
> +//===-- working_set.h -------------------------------------------*- C++ -*-===//<br>
> +//<br>
> +// The LLVM Compiler Infrastructure<br>
> +//<br>
> +// This file is distributed under the University of Illinois Open Source<br>
> +// License. See LICENSE.TXT for details.<br>
> +//<br>
> +//===----------------------------------------------------------------------===//<br>
> +//<br>
> +// This file is a part of EfficiencySanitizer, a family of performance tuners.<br>
> +//<br>
> +// Header for working-set-specific code.<br>
> +//===----------------------------------------------------------------------===//<br>
> +<br>
> +#ifndef WORKING_SET_H<br>
> +#define WORKING_SET_H<br>
> +<br>
> +#include "interception/interception.h"<br>
> +#include "sanitizer_common/sanitizer_internal_defs.h"<br>
> +<br>
> +namespace __esan {<br>
> +<br>
> +void initializeWorkingSet();<br>
> +int finalizeWorkingSet();<br>
> +void processRangeAccessWorkingSet(uptr PC, uptr Addr, SIZE_T Size,<br>
> + bool IsWrite);<br>
> +<br>
> +} // namespace __esan<br>
> +<br>
> +#endif // WORKING_SET_H<br>
><br>
> Propchange: compiler-rt/trunk/lib/esan/working_set.h<br>
> ------------------------------------------------------------------------------<br>
> svn:eol-style = LF<br>
><br>
> Added: compiler-rt/trunk/test/esan/TestCases/workingset-memset.cpp<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/test/esan/TestCases/workingset-memset.cpp?rev=270650&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/test/esan/TestCases/workingset-memset.cpp?rev=270650&view=auto</a><br>
> ==============================================================================<br>
> --- compiler-rt/trunk/test/esan/TestCases/workingset-memset.cpp (added)<br>
> +++ compiler-rt/trunk/test/esan/TestCases/workingset-memset.cpp Tue May 24 21:04:04 2016<br>
> @@ -0,0 +1,21 @@<br>
> +// RUN: %clang_esan_wset -O0 %s -o %t 2>&1<br>
> +// RUN: %run %t 2>&1 | FileCheck %s<br>
> +<br>
> +#include <stdlib.h><br>
> +#include <string.h><br>
> +#include <sys/mman.h><br>
> +#include <assert.h><br>
> +#include <string.h><br>
> +<br>
> +int main(int argc, char **argv) {<br>
> + const int size = 128*1024*1024;<br>
> + char *p = (char *)mmap(0, size, PROT_READ | PROT_WRITE,<br>
> + MAP_ANON | MAP_PRIVATE, -1, 0);<br>
> + // Test the slowpath at different cache line boundaries.<br>
> + for (int i = 0; i < 630; i++)<br>
> + memset((char *)p + 63*i, i, 63*i);<br>
> + munmap(p, size);<br>
> + return 0;<br>
> + // FIXME: once the memory scan and size report is in place add it here.<br>
> + // CHECK: {{.*}}EfficiencySanitizer is not finished: nothing yet to report<br>
> +}<br>
><br>
> Propchange: compiler-rt/trunk/test/esan/TestCases/workingset-memset.cpp<br>
> ------------------------------------------------------------------------------<br>
> svn:eol-style = LF<br>
><br>
> Modified: compiler-rt/trunk/test/esan/lit.cfg<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/test/esan/lit.cfg?rev=270650&r1=270649&r2=270650&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/test/esan/lit.cfg?rev=270650&r1=270649&r2=270650&view=diff</a><br>
> ==============================================================================<br>
> --- compiler-rt/trunk/test/esan/lit.cfg (original)<br>
> +++ compiler-rt/trunk/test/esan/lit.cfg Tue May 24 21:04:04 2016<br>
> @@ -13,12 +13,15 @@ base_cflags = ([config.target_cflags] +<br>
> base_cxxflags = config.cxx_mode_flags + base_cflags<br>
><br>
> frag_cflags = (["-fsanitize=efficiency-cache-frag"] + base_cflags)<br>
> +wset_cflags = (["-fsanitize=efficiency-working-set"] + base_cflags)<br>
><br>
> def build_invocation(compile_flags):<br>
> return " " + " ".join([config.clang] + compile_flags) + " "<br>
><br>
> config.substitutions.append( ("%clang_esan_frag ",<br>
> build_invocation(frag_cflags)) )<br>
> +config.substitutions.append( ("%clang_esan_wset ",<br>
> + build_invocation(wset_cflags)) )<br>
><br>
> default_esan_opts = ''<br>
> config.substitutions.append(('%env_esan_opts=',<br>
><br>
><br>
> _______________________________________________<br>
> llvm-commits mailing list<br>
> <a href="mailto:llvm-commits@lists.llvm.org">llvm-commits@lists.llvm.org</a><br>
> <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits</a><br>
</blockquote></div><br></div></div>