[compiler-rt] [asan] Re-exec without ASLR if needed on 32-bit Linux (PR #132682)

[via llvm-commits]

llvmbot wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-compiler-rt-sanitizer

Author: Thurston Dang (thurstond)

<details>
<summary>Changes</summary>

This generalizes https://github.com/llvm/llvm-project/pull/131975 to non-32-bit Linux (i.e., 64-bit Linux).

This works around an edge case in 64-bit Linux, where the memory layout is incompatible if the stack size is unlimited AND ASLR entropy is 32 bits (see
https://github.com/google/sanitizers/issues/856#issuecomment-2747076811).

More generally, this "re-exec if layout is incompatible" is a hammer that can work around most shadow mapping issues, without the overhead of using a dynamic shadow.

---
Full diff: https://github.com/llvm/llvm-project/pull/132682.diff


1 Files Affected:

- (modified) compiler-rt/lib/asan/asan_shadow_setup.cpp (+8-6) 


``````````diff
diff --git a/compiler-rt/lib/asan/asan_shadow_setup.cpp b/compiler-rt/lib/asan/asan_shadow_setup.cpp
index e66b8af1d2c30..ba61dc2c7fa6e 100644
--- a/compiler-rt/lib/asan/asan_shadow_setup.cpp
+++ b/compiler-rt/lib/asan/asan_shadow_setup.cpp
@@ -109,12 +109,14 @@ void InitializeShadowMemory() {
     ProtectGap(kShadowGap2Beg, kShadowGap2End - kShadowGap2Beg + 1);
     ProtectGap(kShadowGap3Beg, kShadowGap3End - kShadowGap3Beg + 1);
   } else {
-    // The shadow mappings can shadow the entire user address space. However,
-    // on 32-bit systems, the maximum ASLR entropy (currently up to 16-bits
-    // == 256MB) is a significant chunk of the address space; reclaiming it by
-    // disabling ASLR might allow chonky binaries to run.
-    if (sizeof(uptr) == 32)
-      TryReExecWithoutASLR();
+    // ASan's mappings can usually shadow the entire address space, even with
+    // maximum ASLR entropy. However:
+    // - On 32-bit systems, the maximum ASLR entropy (currently up to 16-bits
+    //   == 256MB) is a significant chunk of the address space; reclaiming it
+    //   by disabling ASLR might allow chonky binaries to run.
+    // - On 64-bit systems, some settings (e.g., for Linux, unlimited stack
+    //   size plus maximum ASLR entropy) can lead to an incompatible layout.
+    TryReExecWithoutASLR();
 
     Report(
         "Shadow memory range interleaves with an existing memory mapping. "

``````````

</details>


https://github.com/llvm/llvm-project/pull/132682