[compiler-rt] r272076 - [esan] Intercept calloc to avoid deadlocks with tcmalloc

[Derek Bruening via llvm-commits]

Author: bruening
Date: Tue Jun  7 19:00:27 2016
New Revision: 272076

URL: http://llvm.org/viewvc/llvm-project?rev=272076&view=rev
Log:
[esan] Intercept calloc to avoid deadlocks with tcmalloc

Summary:
When tcmalloc initializes before esan, esan's initialization ends up
calling back into tcmalloc due to the calloc done by dlsym.  This results
in a deadlock.  We avoid this by special-casing this single allocation.

Intercepting calloc also gives us the opportunity to act on its zeroing as
stores by the application.

Reviewers: aizatsky

Subscribers: vitalybuka, zhaoqin, kcc, eugenis, llvm-commits, kubabrecka

Differential Revision: http://reviews.llvm.org/D21086

Modified:
    compiler-rt/trunk/lib/esan/esan_interceptors.cpp

Modified: compiler-rt/trunk/lib/esan/esan_interceptors.cpp
URL: http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/esan/esan_interceptors.cpp?rev=272076&r1=272075&r2=272076&view=diff
==============================================================================
--- compiler-rt/trunk/lib/esan/esan_interceptors.cpp (original)
+++ compiler-rt/trunk/lib/esan/esan_interceptors.cpp Tue Jun  7 19:00:27 2016
@@ -408,6 +408,56 @@ int real_sigaction(int signum, const voi
 #define ESAN_MAYBE_INTERCEPT_SIGACTION
 #endif
 
+//===----------------------------------------------------------------------===//
+// Malloc interceptors
+//===----------------------------------------------------------------------===//
+
+static char early_alloc_buf[128];
+static bool used_early_alloc_buf;
+
+static void *handleEarlyAlloc(uptr size) {
+  // If esan is initialized during an interceptor (which happens with some
+  // tcmalloc implementations that call pthread_mutex_lock), the call from
+  // dlsym to calloc will deadlock.  There is only one such calloc (dlsym
+  // allocates a single pthread key), so we work around it by using a
+  // static buffer for the calloc request.  The loader currently needs
+  // 32 bytes but we size at 128 to allow for future changes.
+  // This solution will also allow us to deliberately intercept malloc & family
+  // in the future (to perform tool actions on each allocation, without
+  // replacing the allocator), as it also solves the problem of intercepting
+  // calloc when it will itself be called before its REAL pointer is
+  // initialized.
+  CHECK(!used_early_alloc_buf && size < sizeof(early_alloc_buf));
+  // We do not handle multiple threads here.  This only happens at process init
+  // time, and while it's possible for a shared library to create early threads
+  // that race here, we consider that to be a corner case extreme enough that
+  // it's not worth the effort to handle.
+  used_early_alloc_buf = true;
+  return (void *)early_alloc_buf;
+}
+
+INTERCEPTOR(void*, calloc, uptr size, uptr n) {
+  if (EsanDuringInit && REAL(calloc) == nullptr)
+    return handleEarlyAlloc(size * n);
+  void *ctx;
+  COMMON_INTERCEPTOR_ENTER(ctx, calloc, size, n);
+  void *res = REAL(calloc)(size, n);
+  // The memory is zeroed and thus is all written.
+  COMMON_INTERCEPTOR_WRITE_RANGE(nullptr, (uptr)res, size * n);
+  return res;
+}
+
+INTERCEPTOR(void, free, void *p) {
+  void *ctx;
+  COMMON_INTERCEPTOR_ENTER(ctx, free, p);
+  if (p == (void *)early_alloc_buf) {
+    // We expect just a singleton use but we clear this for cleanliness.
+    used_early_alloc_buf = false;
+    return;
+  }
+  REAL(free)(p);
+}
+
 namespace __esan {
 
 void initializeInterceptors() {
@@ -432,8 +482,8 @@ void initializeInterceptors() {
   ESAN_MAYBE_INTERCEPT_SIGNAL;
   ESAN_MAYBE_INTERCEPT_SIGACTION;
 
-  // TODO(bruening): we should intercept calloc() and other memory allocation
-  // routines that zero memory and update our shadow memory appropriately.
+  INTERCEPT_FUNCTION(calloc);
+  INTERCEPT_FUNCTION(free);
 
   // TODO(bruening): intercept routines that other sanitizers intercept that
   // are not in the common pool or here yet, ideally by adding to the common