[compiler-rt] [sanitizer] Print diagnostic if ptrace syscall fails (PR #151406)

[Thurston Dang via llvm-commits]

https://github.com/thurstond updated https://github.com/llvm/llvm-project/pull/151406

>From 924c8ae63b95781cf436b540a682782c9c7de11f Mon Sep 17 00:00:00 2001
From: Thurston Dang <thurston at google.com>
Date: Wed, 30 Jul 2025 21:55:40 +0000
Subject: [PATCH 1/3] [sanitizer] Print diagnostic if ptrace syscall fails

StopTheWorld() spins up a thread that calls ptrace before releasing a
mutex; the other thread yields until the mutex is unlocked. If seccomp
kills the thread that calls ptrace, the other thread will silently hang.
The latter thread cannot use waitpid to detect that the other thread has
been killed, because the threads share errno.

This patch forks the process to test whether ptrace is allowed. If it
fails, it prints an informational message (though it does not abort the
sanitizer).

Fixes https://github.com/llvm/llvm-project/issues/150380 and https://github.com/google/sanitizers/issues/777
---
 .../sanitizer_stoptheworld_linux_libcdep.cpp  | 41 +++++++++++++++++++
 1 file changed, 41 insertions(+)

diff --git a/compiler-rt/lib/sanitizer_common/sanitizer_stoptheworld_linux_libcdep.cpp b/compiler-rt/lib/sanitizer_common/sanitizer_stoptheworld_linux_libcdep.cpp
index 24929b8c4bd7c..ce21a156d28b6 100644
--- a/compiler-rt/lib/sanitizer_common/sanitizer_stoptheworld_linux_libcdep.cpp
+++ b/compiler-rt/lib/sanitizer_common/sanitizer_stoptheworld_linux_libcdep.cpp
@@ -403,7 +403,48 @@ struct ScopedSetTracerPID {
   }
 };
 
+// This detects whether ptrace is blocked (e.g., by seccomp), by forking and
+// then attempting ptrace.
+// This separate check is necessary because StopTheWorld() creates a *thread*,
+// and therefore cannot use waitpid() due to the shared errno.
+static void TestPTrace() {
+  // Only check the first time this is called.
+  static bool checked = false;
+  if (checked)
+    return;
+  checked = true;
+
+  // fork() should be cheap because of copy-on-write. Besides, this is only
+  // called the first time.
+  int pid = internal_fork();
+
+  if (pid < 0) {
+    int rverrno;
+    if (internal_iserror(pid, &rverrno)) {
+      Report("WARNING: TestPTrace() failed to fork (errno %d)\n", rverrno);
+    }
+    _exit(-1);
+  }
+
+  if (pid == 0) {
+    // Child subprocess
+    internal_ptrace(PTRACE_ATTACH, 0, nullptr, nullptr);
+    _exit (0);
+  } else {
+    int wstatus;
+    internal_waitpid(pid, &wstatus, 0);
+
+    if (WIFSIGNALED(wstatus)) {
+      VReport(0, "Warning: ptrace appears to be blocked (is seccomp enabled?). LeakSanitizer may hang.\n");
+      VReport(0, "Child exited with signal %d.\n", WTERMSIG(wstatus));
+      // We don't abort the sanitizer - it's still worth letting the sanitizer try.
+    }
+  }
+}
+
 void StopTheWorld(StopTheWorldCallback callback, void *argument) {
+  TestPTrace();
+
   StopTheWorldScope in_stoptheworld;
   // Prepare the arguments for TracerThread.
   struct TracerThreadArgument tracer_thread_argument;

>From bb743a5ebe5d896d6ac2da580f65ae54be2db74b Mon Sep 17 00:00:00 2001
From: Thurston Dang <thurston at google.com>
Date: Wed, 30 Jul 2025 22:25:44 +0000
Subject: [PATCH 2/3] clang-format

---
 .../sanitizer_stoptheworld_linux_libcdep.cpp             | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/compiler-rt/lib/sanitizer_common/sanitizer_stoptheworld_linux_libcdep.cpp b/compiler-rt/lib/sanitizer_common/sanitizer_stoptheworld_linux_libcdep.cpp
index ce21a156d28b6..59fcd1a037655 100644
--- a/compiler-rt/lib/sanitizer_common/sanitizer_stoptheworld_linux_libcdep.cpp
+++ b/compiler-rt/lib/sanitizer_common/sanitizer_stoptheworld_linux_libcdep.cpp
@@ -429,15 +429,18 @@ static void TestPTrace() {
   if (pid == 0) {
     // Child subprocess
     internal_ptrace(PTRACE_ATTACH, 0, nullptr, nullptr);
-    _exit (0);
+    _exit(0);
   } else {
     int wstatus;
     internal_waitpid(pid, &wstatus, 0);
 
     if (WIFSIGNALED(wstatus)) {
-      VReport(0, "Warning: ptrace appears to be blocked (is seccomp enabled?). LeakSanitizer may hang.\n");
+      VReport(0,
+              "Warning: ptrace appears to be blocked (is seccomp enabled?). "
+              "LeakSanitizer may hang.\n");
       VReport(0, "Child exited with signal %d.\n", WTERMSIG(wstatus));
-      // We don't abort the sanitizer - it's still worth letting the sanitizer try.
+      // We don't abort the sanitizer - it's still worth letting the sanitizer
+      // try.
     }
   }
 }

>From 8a77e9c8617abc97a2f6861e2142a2b01fdbdc2a Mon Sep 17 00:00:00 2001
From: Thurston Dang <thurston at google.com>
Date: Mon, 4 Aug 2025 19:58:32 +0000
Subject: [PATCH 3/3] Begun, the clone wars have

---
 .../sanitizer_stoptheworld_linux_libcdep.cpp        | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/compiler-rt/lib/sanitizer_common/sanitizer_stoptheworld_linux_libcdep.cpp b/compiler-rt/lib/sanitizer_common/sanitizer_stoptheworld_linux_libcdep.cpp
index 59fcd1a037655..91ea1b1212044 100644
--- a/compiler-rt/lib/sanitizer_common/sanitizer_stoptheworld_linux_libcdep.cpp
+++ b/compiler-rt/lib/sanitizer_common/sanitizer_stoptheworld_linux_libcdep.cpp
@@ -405,17 +405,20 @@ struct ScopedSetTracerPID {
 
 // This detects whether ptrace is blocked (e.g., by seccomp), by forking and
 // then attempting ptrace.
-// This separate check is necessary because StopTheWorld() creates a *thread*,
-// and therefore cannot use waitpid() due to the shared errno.
+// This separate check is necessary because StopTheWorld() creates a child
+// process with a shared virtual address space, and therefore cannot use
+// waitpid() due to the shared errno.
 static void TestPTrace() {
-  // Only check the first time this is called.
+  // Heuristic: only check the first time this is called. This is not always
+  // correct (e.g., user manually triggers leak detection, then updates
+  // seccomp, then leak detection is triggered again).
   static bool checked = false;
   if (checked)
     return;
   checked = true;
 
-  // fork() should be cheap because of copy-on-write. Besides, this is only
-  // called the first time.
+  // We hope that fork() is not too expensive, because of copy-on-write.
+  // Besides, this is only called the first time.
   int pid = internal_fork();
 
   if (pid < 0) {