[compiler-rt] [tsan] Fix dead lock when starting StackDepot thread (PR #96456)

[Vitaly Buka via llvm-commits]

https://github.com/vitalybuka created https://github.com/llvm/llvm-project/pull/96456

Sometime tsan runtimes calls, like
`__tsan_mutex_create ()`, need to store a stack
in the StackDepot, and the Depot may need to start
and maintenance thread.

Example:
```
__sanitizer::FutexWait ()
__sanitizer::Semaphore::Wait ()
__sanitizer::Mutex::Lock ()
__tsan::SlotLock ()
__tsan::SlotLocker::SlotLocker ()
__tsan::Acquire ()
__tsan::CallUserSignalHandler ()
__tsan::ProcessPendingSignalsImpl ()
__tsan::ProcessPendingSignals ()
__tsan::ScopedInterceptor::~ScopedInterceptor ()
___interceptor_mmap ()
pthread_create ()
__sanitizer::internal_start_thread ()
__sanitizer::(anonymous namespace)::CompressThread::NewWorkNotify ()
__sanitizer::StackDepotNode::store ()
__sanitizer::StackDepotBase<__sanitizer::StackDepotNode, 1, 20>::Put ()
__tsan::CurrentStackId ()
__tsan::MutexCreate ()
__tsan_mutex_create ()
```

pthread_create() implementation may hit other
interceptors recursively, which may invoke
ProcessPendingSignals, which deadlocks.

Alternative solution could be block interceptors
closer to TSAN runtime API function, like
`__tsan_mutex_create`, or just before
`StackDepotPut``, but it's not needed for most
calls, only when new thread is created using
`real_pthread_create`.

I don't see a reasonable way to create a
regression test.


>From 403f5166260b9334c1d13ae1795932fa6b08e404 Mon Sep 17 00:00:00 2001
From: Vitaly Buka <vitalybuka at google.com>
Date: Sun, 23 Jun 2024 23:59:47 -0700
Subject: [PATCH] =?UTF-8?q?[=F0=9D=98=80=F0=9D=97=BD=F0=9D=97=BF]=20initia?=
 =?UTF-8?q?l=20version?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Created using spr 1.3.4
---
 .../lib/tsan/rtl/tsan_interceptors_posix.cpp        | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp b/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp
index 94adea777cafd..151693112158b 100644
--- a/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp
+++ b/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp
@@ -1088,7 +1088,18 @@ TSAN_INTERCEPTOR(int, pthread_join, void *th, void **ret) {
   return res;
 }
 
-DEFINE_REAL_PTHREAD_FUNCTIONS
+// DEFINE_REAL_PTHREAD_FUNCTIONS
+namespace __sanitizer {
+int real_pthread_create(void *th, void *attr, void *(*callback)(void *),
+                        void *param) {
+  ScopedIgnoreInterceptors ignore;
+  return REAL(pthread_create)(th, attr, callback, param);
+}
+int real_pthread_join(void *th, void **ret) {
+  ScopedIgnoreInterceptors ignore;
+  return REAL(pthread_join(th, ret));
+}
+}  // namespace __sanitizer
 
 TSAN_INTERCEPTOR(int, pthread_detach, void *th) {
   SCOPED_INTERCEPTOR_RAW(pthread_detach, th);