[compiler-rt] [ASan] Prevent ASan/LSan deadlock by preloading modules before error reporting (PR #131756)
via llvm-commits
llvm-commits at lists.llvm.org
Tue Mar 18 01:26:05 PDT 2025
https://github.com/Camsyn created https://github.com/llvm/llvm-project/pull/131756
### Description
This PR resolves a deadlock between AddressSanitizer (ASan) and LeakSanitizer (LSan)
that occurs when both sanitizers attempt to acquire locks in conflicting orders across
threads. The fix ensures safe lock acquisition ordering by preloading module information
before error reporting.
---
### Issue Details
**Reproducer**
```cpp
// Thread 1: ASan error path
int arr[1] = {0};
std::thread t([&]() {
arr[1] = 1; // Triggers ASan OOB error
});
// Thread 2: LSan check path
__lsan_do_leak_check();
```
**Lock Order Conflict**:
- Thread 1 (ASan error reporting):
1. Acquires ASan thread registry lock (B)
1. Attempts to acquire libdl lock (A) via `dl_iterate_phdr`
- Thread 2 (LSan leak check):
1. Acquires libdl lock (A) via `dl_iterate_phdr`
1. Attempts to acquire ASan thread registry lock (B)
This creates a circular wait condition (A -> B -> A) meeting all four Coffman deadlock criteria.
---
### Fix Strategy
The root cause lies in ASan's error reporting path needing `dl_iterate_phdr` (requiring lock A)
while already holding its thread registry lock (B). The solution:
1. **Preload Modules Early**: Force module list initialization _before_ acquiring ASan's thread lock
2. **Avoid Nested Locking**: Ensure symbolization (via dl_iterate_phdr) completes before error reporting locks
Key code change:
```cpp
// Before acquiring ASan's thread registry lock:
Symbolizer::GetOrInit()->GetRefreshedListOfModules();
```
This guarantees module information is cached before lock acquisition, eliminating
the need for `dl_iterate_phdr` calls during error reporting.
---
### Testing
Added **asan_lsan_deadlock.cpp** test case:
- Reproduces deadlock reliably without fix **under idle system conditions**
- Uses watchdog thread to detect hangs
- Verifies ASan error reports correctly without deadlock
**Note**: Due to the inherent non-determinism of thread scheduling and lock acquisition timing,
this test may not reliably reproduce the deadlock on busy systems (e.g., during parallel
`ninja check-asan` runs).
---
### Impact
- Fixes rare but severe deadlocks in mixed ASan+LSan environments
- Maintains thread safety guarantees for both sanitizers
- No user-visible behavior changes except deadlock elimination
---
### Relevant Buggy Code
- Code in ASan's asan_report.cpp
```cpp
explicit ScopedInErrorReport(bool fatal = false)
: halt_on_error_(fatal || flags()->halt_on_error) {
// Acquire lock B
asanThreadRegistry().Lock();
}
~ScopedInErrorReport() {
...
// Try to acquire lock A under holding lock B via the following path
// #4 0x000071a353d83e93 in __GI___dl_iterate_phdr (
// callback=0x5d1a07a39580 <__sanitizer::dl_iterate_phdr_cb(dl_phdr_info*, unsigned long, void*)>,
// data=0x6da3510fd3f0) at ./elf/dl-iteratephdr.c:39
// #5 0x00005d1a07a39574 in __sanitizer::ListOfModules::init (this=0x71a353ebc080)
// at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cpp:784
// #6 0x00005d1a07a429e3 in __sanitizer::Symbolizer::RefreshModules (this=0x71a353ebc058)
// at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_libcdep.cpp:188
// #7 __sanitizer::Symbolizer::FindModuleForAddress (this=this at entry=0x71a353ebc058,
// address=address at entry=102366378805727)
// at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_libcdep.cpp:214
// #8 0x00005d1a07a4291b in __sanitizer::Symbolizer::SymbolizePC (this=0x71a353ebc058, addr=102366378805727)
// at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_libcdep.cpp:88
// #9 0x00005d1a07a40df7 in __sanitizer::(anonymous namespace)::StackTraceTextPrinter::ProcessAddressFrames (
// this=this at entry=0x6da3510fd520, pc=102366378805727)
// at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_stacktrace_libcdep.cpp:37
// #10 0x00005d1a07a40d27 in __sanitizer::StackTrace::PrintTo (this=this at entry=0x6da3510fd5e8,
// output=output at entry=0x6da3510fd588)
// at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_stacktrace_libcdep.cpp:110
// #11 0x00005d1a07a410a1 in __sanitizer::StackTrace::Print (this=0x6da3510fd5e8)
// at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_stacktrace_libcdep.cpp:133
// #12 0x00005d1a0798758d in __asan::ErrorGeneric::Print (
// this=0x5d1a07aa4e08 <__asan::ScopedInErrorReport::current_error_+8>)
// at llvm-project/compiler-rt/lib/asan/asan_errors.cpp:617
current_error_.Print();
...
}
```
- Code in LSan's lsan_common_linux.cpp
```cpp
void LockStuffAndStopTheWorld(StopTheWorldCallback callback,
CheckForLeaksParam *argument) {
// Acquire lock A
dl_iterate_phdr(LockStuffAndStopTheWorldCallback, ¶m);
}
static int LockStuffAndStopTheWorldCallback(struct dl_phdr_info *info,
size_t size, void *data) {
// Try to acquire lock B under holding lock A via the following path
// #3 0x000055555562b34a in __sanitizer::ThreadRegistry::Lock (this=<optimized out>)
// at llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_thread_registry.h:99
// #4 __lsan::LockThreads () at llvm-project/compiler-rt/lib/asan/asan_thread.cpp:484
// #5 0x0000555555652629 in __lsan::ScopedStopTheWorldLock::ScopedStopTheWorldLock (this=<optimized out>)
// at llvm-project/compiler-rt/lib/lsan/lsan_common.h:164
// #6 __lsan::LockStuffAndStopTheWorldCallback (info=<optimized out>, size=<optimized out>, data=0x0,
// data at entry=0x7fffffffd158) at llvm-project/compiler-rt/lib/lsan/lsan_common_linux.cpp:120
ScopedStopTheWorldLock lock;
DoStopTheWorldParam *param = reinterpret_cast<DoStopTheWorldParam *>(data);
StopTheWorld(param->callback, param->argument);
return 1;
}
```
>From 943948f05369c333066b7f04356df82d516dc1ea Mon Sep 17 00:00:00 2001
From: Camsyn <65994555+Camsyn at users.noreply.github.com>
Date: Tue, 18 Mar 2025 16:02:24 +0800
Subject: [PATCH] [ASan] Prevent ASan/LSan deadlock by preloading modules
before error reporting
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
This fixes a circular lock dependency between ASan's thread registry
lock (B) and libdl's dl_iterate_phdr lock (A) that occurs when:
1. ASan error reporting (holding B) calls dl_iterate_phdr (needs A)
2. LSan leak check (holding A) acquires ASan thread registry lock (B)
The fix proactively initializes module information before acquiring
ASan's thread registry lock during error reporting. This eliminates
the need to call `dl_iterate_phdr` while already holding lock B,
breaking the dangerous A→B/B→A cross-thread locking pattern.
Add test case (compiler-rt/test/asan/TestCases/asan_lsan_deadlock.cpp)
that reproduces the deadlock without this patch.
---
compiler-rt/lib/asan/asan_report.cpp | 21 ++++++
.../asan/TestCases/asan_lsan_deadlock.cpp | 73 +++++++++++++++++++
2 files changed, 94 insertions(+)
create mode 100644 compiler-rt/test/asan/TestCases/asan_lsan_deadlock.cpp
diff --git a/compiler-rt/lib/asan/asan_report.cpp b/compiler-rt/lib/asan/asan_report.cpp
index 45aa607dcda07..8aeb938b2ee3d 100644
--- a/compiler-rt/lib/asan/asan_report.cpp
+++ b/compiler-rt/lib/asan/asan_report.cpp
@@ -126,6 +126,27 @@ class ScopedInErrorReport {
public:
explicit ScopedInErrorReport(bool fatal = false)
: halt_on_error_(fatal || flags()->halt_on_error) {
+ /*
+ * Deadlock Prevention Between ASan and LSan
+ *
+ * Background:
+ * - The `dl_iterate_phdr` function requires holding libdl's internal lock (Lock A).
+ * - LSan acquires the ASan thread registry lock (Lock B) *after* calling `dl_iterate_phdr`.
+ *
+ * Problem Scenario:
+ * When ASan attempts to call `dl_iterate_phdr` while holding Lock B (e.g., during
+ * error reporting via `ErrorDescription::Print`), a circular lock dependency may occur:
+ * 1. Thread 1: Holds Lock B → Requests Lock A (via dl_iterate_phdr)
+ * 2. Thread 2: Holds Lock A → Requests Lock B (via LSan operations)
+ *
+ * Solution:
+ * Proactively load all required modules before acquiring Lock B. This ensures:
+ * 1. Any `dl_iterate_phdr` calls during module loading complete before locking
+ * 2. Subsequent error reporting avoids nested lock acquisition patterns
+ * 3. Eliminates the lock order inversion risk between libdl and ASan's thread registry
+ */
+ Symbolizer::GetOrInit()->GetRefreshedListOfModules();
+
// Make sure the registry and sanitizer report mutexes are locked while
// we're printing an error report.
// We can lock them only here to avoid self-deadlock in case of
diff --git a/compiler-rt/test/asan/TestCases/asan_lsan_deadlock.cpp b/compiler-rt/test/asan/TestCases/asan_lsan_deadlock.cpp
new file mode 100644
index 0000000000000..6d4c91eb49241
--- /dev/null
+++ b/compiler-rt/test/asan/TestCases/asan_lsan_deadlock.cpp
@@ -0,0 +1,73 @@
+// Test for potential deadlock in LeakSanitizer+AddressSanitizer.
+// REQUIRES: leak-detection
+//
+// RUN: %clangxx_asan -O0 %s -o %t
+// RUN: %env_asan_opts=detect_leaks=1 not %run %t 2>&1 | FileCheck %s
+
+/*
+ * Purpose: Verify deadlock prevention between ASan error reporting and LSan leak checking.
+ *
+ * Test Design:
+ * 1. Creates contention scenario between:
+ * - ASan's error reporting (requires lock B -> lock A ordering)
+ * - LSan's leak check (requires lock A -> lock B ordering)
+ * 2. Thread timing:
+ * - Main thread: Holds 'in' mutex -> Triggers LSan check (lock A then B)
+ * - Worker thread: Triggers ASan OOB error (lock B then A via symbolization)
+ *
+ * Deadlock Condition (if unfixed):
+ * Circular lock dependency forms when:
+ * [Main Thread] LSan: lock A -> requests lock B
+ * [Worker Thread] ASan: lock B -> requests lock A
+ *
+ * Success Criteria:
+ * With proper lock ordering enforcement, watchdog should NOT trigger - test exits normally.
+ * If deadlock occurs, watchdog terminates via _exit(1) after 10s timeout.
+ */
+
+#include <mutex>
+#include <sanitizer/lsan_interface.h>
+#include <stdio.h>
+#include <thread>
+#include <unistd.h>
+
+std::mutex in;
+
+void Watchdog() {
+ // Safety mechanism: Turn infinite deadlock into finite test failure
+ usleep(10000000);
+ // CHECK-NOT: Timeout! Deadlock detected.
+ puts("Timeout! Deadlock detected.");
+ fflush(stdout);
+ _exit(1);
+}
+
+int main(int argc, char **argv) {
+ int arr[1] = {0};
+ in.lock();
+
+ std::thread w(Watchdog);
+ w.detach();
+
+ std::thread t([&]() {
+ in.unlock();
+ /*
+ * Provoke ASan error: ASan's error reporting acquires:
+ * 1. ASan's thread registry lock (B) during the reporting
+ * 2. dl_iterate_phdr lock (A) during symbolization
+ */
+ // CHECK: SUMMARY: AddressSanitizer: stack-buffer-overflow
+ arr[argc] = 1; // Deliberate OOB access
+ });
+
+ in.lock();
+ /*
+ * Critical section: LSan's check acquires:
+ * 1. dl_iterate_phdr lock (A)
+ * 2. ASan's thread registry lock (B)
+ * before Stop The World.
+ */
+ __lsan_do_leak_check();
+ t.join();
+ return 0;
+}
\ No newline at end of file
More information about the llvm-commits
mailing list