[compiler-rt] [asan] Plumb may_return_null through asan_memalign (PR #196413)

[Justin T. Gibbs via llvm-commits]

https://github.com/scsiguy updated https://github.com/llvm/llvm-project/pull/196413

>From aae3f662641fb817123a2ab843ab7d1690230e63 Mon Sep 17 00:00:00 2001
From: "Justin T. Gibbs" <gibbs at scsiguy.com>
Date: Thu, 28 May 2026 14:55:04 -0700
Subject: [PATCH] [asan] Allow OOM Policy Injection in Allocator::Allocate API

Allocator::Allocate() previously unconditionally applied the global
OOM policy as dictated by the AllocatorMayReturnNull() flag: return
nellptr or abort via Report*+Die(). Introduce Allocator::AllocateImpl()
which delegates this decision to the caller via its 'may_return_null'
boolean parameter, allowing future changes to the operator new
implementation to fully control this behavior.

Implementation:

  * Rename Allocator::Allocate(...) to Allocator::AllocateImpl(...) and
    add a may_return_null parameter that replaces the three internal
    AllocatorMayReturnNull() calls (RSS limit, oversize, OOM).

  * Reintroduce Allocator::Allocate(...) as a thin wrapper that calls
    AllocateImpl(..., AllocatorMayReturnNull()), preserving the
    flag-deferring semantics for every existing caller.

NFC.
---
 compiler-rt/lib/asan/asan_allocator.cpp | 20 +++++++++++++++-----
 1 file changed, 15 insertions(+), 5 deletions(-)

diff --git a/compiler-rt/lib/asan/asan_allocator.cpp b/compiler-rt/lib/asan/asan_allocator.cpp
index ce8311c05c5da..6af197ab4cb1c 100644
--- a/compiler-rt/lib/asan/asan_allocator.cpp
+++ b/compiler-rt/lib/asan/asan_allocator.cpp
@@ -536,12 +536,15 @@ struct Allocator {
   }
 
   // -------------------- Allocation/Deallocation routines ---------------
-  void* Allocate(uptr size, uptr alignment, BufferedStackTrace* stack,
-                 AllocType alloc_type, bool can_fill) {
+  // may_return_null tells AllocateImpl() whether OOM should produce a nullptr
+  // (true) or a fatal Report*+Die() (false).
+  void* AllocateImpl(uptr size, uptr alignment, BufferedStackTrace* stack,
+                     AllocType alloc_type, bool can_fill,
+                     bool may_return_null) {
     if (UNLIKELY(!AsanInited()))
       AsanInitFromRtl();
     if (UNLIKELY(IsRssLimitExceeded())) {
-      if (AllocatorMayReturnNull())
+      if (may_return_null)
         return nullptr;
       ReportRssLimitExceeded(stack);
     }
@@ -578,7 +581,7 @@ struct Allocator {
     CHECK(IsAligned(needed_size, min_alignment));
     if (size > kMaxAllowedMallocSize || needed_size > kMaxAllowedMallocSize ||
         size > max_user_defined_malloc_size) {
-      if (AllocatorMayReturnNull()) {
+      if (may_return_null) {
         Report("WARNING: AddressSanitizer failed to allocate 0x%zx bytes\n",
                size);
         return nullptr;
@@ -600,7 +603,7 @@ struct Allocator {
     }
     if (UNLIKELY(!allocated)) {
       SetAllocatorOutOfMemory();
-      if (AllocatorMayReturnNull())
+      if (may_return_null)
         return nullptr;
       ReportOutOfMemory(size, stack);
     }
@@ -679,6 +682,13 @@ struct Allocator {
     return res;
   }
 
+  // Defer to the global, flag controlled, OOM policy.
+  void* Allocate(uptr size, uptr alignment, BufferedStackTrace* stack,
+                 AllocType alloc_type, bool can_fill) {
+    return AllocateImpl(size, alignment, stack, alloc_type, can_fill,
+                        AllocatorMayReturnNull());
+  }
+
   // Set quarantine flag if chunk is allocated, issue ASan error report on
   // available and quarantined chunks. Return true on success, false otherwise.
   bool AtomicallySetQuarantineFlagIfAllocated(AsanChunk* m, void* ptr,