[compiler-rt] 8e1afda - [ASan] Skip high-shadow and gap setup when HighMem region is empty (#202037)

[via llvm-commits]

Author: Matt Turner
Date: 2026-06-06T11:34:31-07:00
New Revision: 8e1afdada5d09cccb18b3ff17dac04e30645b93e

URL: https://github.com/llvm/llvm-project/commit/8e1afdada5d09cccb18b3ff17dac04e30645b93e
DIFF: https://github.com/llvm/llvm-project/commit/8e1afdada5d09cccb18b3ff17dac04e30645b93e.diff

LOG: [ASan] Skip high-shadow and gap setup when HighMem region is empty (#202037)

On targets where the shadow offset sits above all addressable user
memory (e.g. Alpha with ASAN_SHADOW_OFFSET=0x70000000000 and a 42-bit
user VAS), kHighMemBeg is set above kHighMemEnd so the HighMem region is
empty. Since MEM_TO_SHADOW is monotonically increasing, kHighMemBeg >
kHighMemEnd implies kHighShadowBeg > kHighShadowEnd. Calling
ReserveShadowMemoryRange(kHighShadowBeg, kHighShadowEnd) passes size =
kHighShadowEnd - kHighShadowBeg + 1, which underflows to a large
negative value, and mmap() fails with ENOMEM.

ProtectGap is also skipped: there is no meaningful shadow gap between
LowShadow and an empty HighShadow.

Guard both operations on kHighMemBeg <= kHighMemEnd.

Added: 
    

Modified: 
    compiler-rt/lib/asan/asan_shadow_setup.cpp

Removed: 
    


################################################################################
diff  --git a/compiler-rt/lib/asan/asan_shadow_setup.cpp b/compiler-rt/lib/asan/asan_shadow_setup.cpp
index 5b3591da067bd..de3fb3910c56e 100644
--- a/compiler-rt/lib/asan/asan_shadow_setup.cpp
+++ b/compiler-rt/lib/asan/asan_shadow_setup.cpp
@@ -89,11 +89,19 @@ void InitializeShadowMemory() {
     // mmap the low shadow plus at least one page at the left.
     if (kLowShadowBeg)
       ReserveShadowMemoryRange(shadow_start, kLowShadowEnd, "low shadow");
-    // mmap the high shadow.
-    ReserveShadowMemoryRange(kHighShadowBeg, kHighShadowEnd, "high shadow");
-    // protect the gap.
-    ProtectGap(kShadowGapBeg, kShadowGapEnd - kShadowGapBeg + 1);
-    CHECK_EQ(kShadowGapEnd, kHighShadowBeg - 1);
+    // mmap the high shadow and protect the gap.
+    // On targets where the shadow offset sits above all addressable memory
+    // (e.g. Alpha's 42-bit user VAS with offset 0x70000000000), the shadow of
+    // the highest address exceeds the highest address itself, so there is no
+    // high memory region.  Skip both the high-shadow reservation and the gap
+    // protect.
+    if (MEM_TO_SHADOW(GetMaxUserVirtualAddress()) <
+        GetMaxUserVirtualAddress()) {
+      DCHECK_LE(kHighMemBeg, kHighMemEnd);
+      ReserveShadowMemoryRange(kHighShadowBeg, kHighShadowEnd, "high shadow");
+      ProtectGap(kShadowGapBeg, kShadowGapEnd - kShadowGapBeg + 1);
+      CHECK_EQ(kShadowGapEnd, kHighShadowBeg - 1);
+    }
   } else if (kMidMemBeg &&
              MemoryRangeIsAvailable(shadow_start, kMidMemBeg - 1) &&
              MemoryRangeIsAvailable(kMidMemEnd + 1, kHighShadowEnd)) {