[compiler-rt] [asan] Re-exec without ASLR if needed on 32-bit Linux (PR #132682)

[Thurston Dang via llvm-commits]

https://github.com/thurstond created https://github.com/llvm/llvm-project/pull/132682

This generalizes https://github.com/llvm/llvm-project/pull/131975 to non-32-bit Linux (i.e., 64-bit Linux).

This works around an edge case in 64-bit Linux, where the memory layout is incompatible if the stack size is unlimited AND ASLR entropy is 32 bits (see
https://github.com/google/sanitizers/issues/856#issuecomment-2747076811).

More generally, this "re-exec if layout is incompatible" is a hammer that can work around most shadow mapping issues, without the overhead of using a dynamic shadow.

>From c83975ba265f320a92dec5dd846441df86d0cd7d Mon Sep 17 00:00:00 2001
From: Thurston Dang <thurston at google.com>
Date: Mon, 24 Mar 2025 07:04:35 +0000
Subject: [PATCH] [asan] Re-exec without ASLR if needed on 32-bit Linux

This generalizes https://github.com/llvm/llvm-project/pull/131975 to
non-32-bit Linux (i.e., 64-bit Linux).

This works around an edge case in 64-bit Linux, where the memory layout
is incompatible if the stack size is unlimited AND ASLR entropy is 32
bits (see
https://github.com/google/sanitizers/issues/856#issuecomment-2747076811).

More generally, this "re-exec if layout is incompatible" is a hammer
that can work around most shadow mapping issues, without the overhead of
using a dynamic shadow.
---
 compiler-rt/lib/asan/asan_shadow_setup.cpp | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

diff --git a/compiler-rt/lib/asan/asan_shadow_setup.cpp b/compiler-rt/lib/asan/asan_shadow_setup.cpp
index e66b8af1d2c30..ba61dc2c7fa6e 100644
--- a/compiler-rt/lib/asan/asan_shadow_setup.cpp
+++ b/compiler-rt/lib/asan/asan_shadow_setup.cpp
@@ -109,12 +109,14 @@ void InitializeShadowMemory() {
     ProtectGap(kShadowGap2Beg, kShadowGap2End - kShadowGap2Beg + 1);
     ProtectGap(kShadowGap3Beg, kShadowGap3End - kShadowGap3Beg + 1);
   } else {
-    // The shadow mappings can shadow the entire user address space. However,
-    // on 32-bit systems, the maximum ASLR entropy (currently up to 16-bits
-    // == 256MB) is a significant chunk of the address space; reclaiming it by
-    // disabling ASLR might allow chonky binaries to run.
-    if (sizeof(uptr) == 32)
-      TryReExecWithoutASLR();
+    // ASan's mappings can usually shadow the entire address space, even with
+    // maximum ASLR entropy. However:
+    // - On 32-bit systems, the maximum ASLR entropy (currently up to 16-bits
+    //   == 256MB) is a significant chunk of the address space; reclaiming it
+    //   by disabling ASLR might allow chonky binaries to run.
+    // - On 64-bit systems, some settings (e.g., for Linux, unlimited stack
+    //   size plus maximum ASLR entropy) can lead to an incompatible layout.
+    TryReExecWithoutASLR();
 
     Report(
         "Shadow memory range interleaves with an existing memory mapping. "