[compiler-rt] [ASan] Skip high-shadow and gap setup when HighMem region is empty (PR #202037)

[Matt Turner via llvm-commits]

https://github.com/mattst88 updated https://github.com/llvm/llvm-project/pull/202037

>From d801796ba1166dcdfc859d23415bdd92e565d5d9 Mon Sep 17 00:00:00 2001
From: Matt Turner <mattst88 at gmail.com>
Date: Thu, 4 Jun 2026 23:34:38 -0400
Subject: [PATCH] [ASan] Skip high-shadow and gap setup when HighMem region is
 empty

On targets where the shadow offset sits above all addressable user memory
(e.g. Alpha with ASAN_SHADOW_OFFSET=0x70000000000 and a 42-bit user VAS),
kHighMemBeg is set above kHighMemEnd so the HighMem region is empty.
Since MEM_TO_SHADOW is monotonically increasing, kHighMemBeg > kHighMemEnd
implies kHighShadowBeg > kHighShadowEnd.  Calling
ReserveShadowMemoryRange(kHighShadowBeg, kHighShadowEnd) passes
size = kHighShadowEnd - kHighShadowBeg + 1, which underflows to a large
negative value, and mmap() fails with ENOMEM.

ProtectGap is also skipped: there is no meaningful shadow gap between
LowShadow and an empty HighShadow.

Guard both operations on kHighMemBeg <= kHighMemEnd.
---
 compiler-rt/lib/asan/asan_shadow_setup.cpp | 18 +++++++++++++-----
 1 file changed, 13 insertions(+), 5 deletions(-)

diff --git a/compiler-rt/lib/asan/asan_shadow_setup.cpp b/compiler-rt/lib/asan/asan_shadow_setup.cpp
index 5b3591da067bd..de3fb3910c56e 100644
--- a/compiler-rt/lib/asan/asan_shadow_setup.cpp
+++ b/compiler-rt/lib/asan/asan_shadow_setup.cpp
@@ -89,11 +89,19 @@ void InitializeShadowMemory() {
     // mmap the low shadow plus at least one page at the left.
     if (kLowShadowBeg)
       ReserveShadowMemoryRange(shadow_start, kLowShadowEnd, "low shadow");
-    // mmap the high shadow.
-    ReserveShadowMemoryRange(kHighShadowBeg, kHighShadowEnd, "high shadow");
-    // protect the gap.
-    ProtectGap(kShadowGapBeg, kShadowGapEnd - kShadowGapBeg + 1);
-    CHECK_EQ(kShadowGapEnd, kHighShadowBeg - 1);
+    // mmap the high shadow and protect the gap.
+    // On targets where the shadow offset sits above all addressable memory
+    // (e.g. Alpha's 42-bit user VAS with offset 0x70000000000), the shadow of
+    // the highest address exceeds the highest address itself, so there is no
+    // high memory region.  Skip both the high-shadow reservation and the gap
+    // protect.
+    if (MEM_TO_SHADOW(GetMaxUserVirtualAddress()) <
+        GetMaxUserVirtualAddress()) {
+      DCHECK_LE(kHighMemBeg, kHighMemEnd);
+      ReserveShadowMemoryRange(kHighShadowBeg, kHighShadowEnd, "high shadow");
+      ProtectGap(kShadowGapBeg, kShadowGapEnd - kShadowGapBeg + 1);
+      CHECK_EQ(kShadowGapEnd, kHighShadowBeg - 1);
+    }
   } else if (kMidMemBeg &&
              MemoryRangeIsAvailable(shadow_start, kMidMemBeg - 1) &&
              MemoryRangeIsAvailable(kMidMemEnd + 1, kHighShadowEnd)) {