[compiler-rt] [ASan] Disable InstallAtForkHandler on Linux/sparc64 (PR #108542)

[Rainer Orth via llvm-commits]

https://github.com/rorth created https://github.com/llvm/llvm-project/pull/108542

When SPARC Asan testing is enabled by PR #107405, many Linux/sparc64 tests just hang like
```
#0  0xf7ae8e90 in syscall () from /usr/lib32/libc.so.6
#1  0x701065e8 in __sanitizer::FutexWait(__sanitizer::atomic_uint32_t*, unsigned int) ()
    at compiler-rt/lib/sanitizer_common/sanitizer_linux.cpp:766
#2  0x70107c90 in Wait ()
    at compiler-rt/lib/sanitizer_common/sanitizer_mutex.cpp:35
#3  0x700f7cac in Lock ()
    at compiler-rt/lib/asan/../sanitizer_common/sanitizer_mutex.h:196
#4  Lock ()
    at compiler-rt/lib/asan/../sanitizer_common/sanitizer_thread_registry.h:98
#5  LockThreads ()
    at compiler-rt/lib/asan/asan_thread.cpp:489
#6  0x700e9c8c in __asan::BeforeFork() ()
    at compiler-rt/lib/asan/asan_posix.cpp:157
#7  0xf7ac83f4 in ?? () from /usr/lib32/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
```
It turns out that this happens in tests using `internal_fork` (e.g. invoking `llvm-symbolizer`): unlike most other Linux targets, which use `clone`, Linux/sparc64 has to use `__fork` instead.  While `clone` doesn't trigger `pthread_atfork` handlers, `__fork` obviously does, causing the hang.

To avoid this, this patch disables `InstallAtForkHandler` and lets the ASan tests run to completion.

Tested on `sparc64-unknown-linux-gnu`.

>From 164a81de0d5e4c48ad85398d0c8908f882559f74 Mon Sep 17 00:00:00 2001
From: Rainer Orth <ro at gcc.gnu.org>
Date: Fri, 13 Sep 2024 13:49:37 +0200
Subject: [PATCH] [ASan] Disable InstallAtForkHandler on Linux/sparc64

When SPARC Asan testing is enabled by PR #107405, many Linux/sparc64 tests
just hang like
```
#0  0xf7ae8e90 in syscall () from /usr/lib32/libc.so.6
#1  0x701065e8 in __sanitizer::FutexWait(__sanitizer::atomic_uint32_t*, unsigned int) ()
    at compiler-rt/lib/sanitizer_common/sanitizer_linux.cpp:766
#2  0x70107c90 in Wait ()
    at compiler-rt/lib/sanitizer_common/sanitizer_mutex.cpp:35
#3  0x700f7cac in Lock ()
    at compiler-rt/lib/asan/../sanitizer_common/sanitizer_mutex.h:196
#4  Lock ()
    at compiler-rt/lib/asan/../sanitizer_common/sanitizer_thread_registry.h:98
#5  LockThreads ()
    at compiler-rt/lib/asan/asan_thread.cpp:489
#6  0x700e9c8c in __asan::BeforeFork() ()
    at compiler-rt/lib/asan/asan_posix.cpp:157
#7  0xf7ac83f4 in ?? () from /usr/lib32/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
```
It turns out that this happens in tests using `internal_fork`
(e.g. invoking `llvm-symbolizer`): unlike most other Linux targets, which
use `clone`, Linux/sparc64 has to use `__fork` instead.  While `clone`
doesn't trigger `pthread_atfork` handlers, `__fork` obviously does, causing
the hang.

To avoid this, this patch disables `InstallAtForkHandler` and lets the ASan
tests run to completion.

Tested on `sparc64-unknown-linux-gnu`.
---
 compiler-rt/lib/asan/asan_posix.cpp | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/compiler-rt/lib/asan/asan_posix.cpp b/compiler-rt/lib/asan/asan_posix.cpp
index 76564538bd5d77..c42c0472592b78 100644
--- a/compiler-rt/lib/asan/asan_posix.cpp
+++ b/compiler-rt/lib/asan/asan_posix.cpp
@@ -171,7 +171,11 @@ static void AfterFork(bool fork_child) {
 }
 
 void InstallAtForkHandler() {
-#  if SANITIZER_SOLARIS || SANITIZER_NETBSD || SANITIZER_APPLE
+#  if SANITIZER_SOLARIS || SANITIZER_NETBSD || SANITIZER_APPLE || \
+      (SANITIZER_LINUX && SANITIZER_SPARC)
+  // While other Linux targets use clone in internal_fork which doesn't
+  // trigger pthread_atfork handlers, Linux/sparc64 uses __fork, causing a
+  // hang.
   return;  // FIXME: Implement FutexWait.
 #  endif
   pthread_atfork(