[compiler-rt] [ASan] Disable InstallAtForkHandler on Linux/sparc64 (PR #108542)

[via llvm-commits]

llvmbot wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-compiler-rt-sanitizer

Author: Rainer Orth (rorth)

<details>
<summary>Changes</summary>

When SPARC Asan testing is enabled by PR #<!-- -->107405, many Linux/sparc64 tests just hang like
```
#<!-- -->0  0xf7ae8e90 in syscall () from /usr/lib32/libc.so.6
#<!-- -->1  0x701065e8 in __sanitizer::FutexWait(__sanitizer::atomic_uint32_t*, unsigned int) ()
    at compiler-rt/lib/sanitizer_common/sanitizer_linux.cpp:766
#<!-- -->2  0x70107c90 in Wait ()
    at compiler-rt/lib/sanitizer_common/sanitizer_mutex.cpp:35
#<!-- -->3  0x700f7cac in Lock ()
    at compiler-rt/lib/asan/../sanitizer_common/sanitizer_mutex.h:196
#<!-- -->4  Lock ()
    at compiler-rt/lib/asan/../sanitizer_common/sanitizer_thread_registry.h:98
#<!-- -->5  LockThreads ()
    at compiler-rt/lib/asan/asan_thread.cpp:489
#<!-- -->6  0x700e9c8c in __asan::BeforeFork() ()
    at compiler-rt/lib/asan/asan_posix.cpp:157
#<!-- -->7  0xf7ac83f4 in ?? () from /usr/lib32/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
```
It turns out that this happens in tests using `internal_fork` (e.g. invoking `llvm-symbolizer`): unlike most other Linux targets, which use `clone`, Linux/sparc64 has to use `__fork` instead.  While `clone` doesn't trigger `pthread_atfork` handlers, `__fork` obviously does, causing the hang.

To avoid this, this patch disables `InstallAtForkHandler` and lets the ASan tests run to completion.

Tested on `sparc64-unknown-linux-gnu`.

---
Full diff: https://github.com/llvm/llvm-project/pull/108542.diff


1 Files Affected:

- (modified) compiler-rt/lib/asan/asan_posix.cpp (+5-1) 


``````````diff
diff --git a/compiler-rt/lib/asan/asan_posix.cpp b/compiler-rt/lib/asan/asan_posix.cpp
index 76564538bd5d77..c42c0472592b78 100644
--- a/compiler-rt/lib/asan/asan_posix.cpp
+++ b/compiler-rt/lib/asan/asan_posix.cpp
@@ -171,7 +171,11 @@ static void AfterFork(bool fork_child) {
 }
 
 void InstallAtForkHandler() {
-#  if SANITIZER_SOLARIS || SANITIZER_NETBSD || SANITIZER_APPLE
+#  if SANITIZER_SOLARIS || SANITIZER_NETBSD || SANITIZER_APPLE || \
+      (SANITIZER_LINUX && SANITIZER_SPARC)
+  // While other Linux targets use clone in internal_fork which doesn't
+  // trigger pthread_atfork handlers, Linux/sparc64 uses __fork, causing a
+  // hang.
   return;  // FIXME: Implement FutexWait.
 #  endif
   pthread_atfork(

``````````

</details>


https://github.com/llvm/llvm-project/pull/108542