[compiler-rt] [sanitizer_common] Add unit tests for operator new failure handlings (PR #226832)
Justin T. Gibbs via llvm-commits
llvm-commits at lists.llvm.org
Sun Sep 27 13:16:52 PDT 2026
https://github.com/scsiguy created https://github.com/llvm/llvm-project/pull/226832
Depends on #225847.
Test `RunNewHandlerChain`, `NewImplThrowing`, `NewImplNothrow` and
`InvokeOnExhausted` with fake allocation and exhaustion callbacks. The
tests cover handler retries, chain exhaustion, exceptions thrown by a
`std::new_handler`, and the `allocator_may_return_null` exhaustion
policy for throwing and nothrow `operator new`. Sanitizers that adopt
the framework then only need to test their own wiring.
The tests build only where the header includes `<new>` and supports
exceptions, which excludes Windows.
Assisted-by: Claude Opus 5.5
>From 24ddd5aa5630df892cebe03034956d0efe2583cd Mon Sep 17 00:00:00 2001
From: "Justin T. Gibbs" <gibbs at scsiguy.com>
Date: Tue, 22 Sep 2026 09:44:19 -0700
Subject: [PATCH 1/2] [sanitizer] Always report invalid aligned operator new
requests
Aligned `operator new` in every sanitizer now reports an invalid
`std::align_val_t` (not a positive power of two) with
`ReportInvalidAllocationAlignment()`, regardless of
`allocator_may_return_null`. This ensures the caller's UB API usage is
clearly reported instead of being treated as an out-of-storage failure.
Previously, all sanitizers but ASan also silently accepted an alignment
of zero.
The operator new framework comments now state the resulting contract:
an allocation callback returns nullptr only when storage cannot be
obtained.
Test Plan:
- New sanitizer_common test covers all four aligned overloads.
- NSan, MemProf and DFSan (unsupported by sanitizer_common tests) have
their own tests of the single-object overloads, which share one
implementation with the array overloads.
History:
Every sanitizer implemented aligned `operator new` on top of its C
`memalign`, inheriting that API's flag-gated return of
`nullptr+errno=EINVAL` for invalid alignment. ASan later moved aligned
new to a dedicated helper that copied the policy (see table); the other
runtimes still call `memalign` directly. C++ does not require a program
to continue after this UB. Removing the flag-controlled path makes a
nullptr return *always* mean an out-of-storage failure, one the
`std::get_new_handler()` loop can retry.
ASan's history, as an example:
| Commit | Year | Change |
| -------------- | ---- | -------------------------------------------- |
| `c7cc93ad0723` | 2016 | Aligned `operator new` added as a call to |
| | | `asan_memalign()` (D24771). |
| `31e8173c9425` | 2017 | `asan_memalign()` gains the flag-gated |
| | | `EINVAL` path for C `memalign` (D35440). |
| `10f50a44c1fa` | 2018 | That path becomes an explicit |
| | | `AllocatorMayReturnNull()` check plus |
| | | `ReportInvalidAllocationAlignment()` |
| | | (D44404). |
| `681c2ee4dfbf` | 2025 | The dedicated `asan_new_aligned()` helper |
| | | copies the branch (#145087). |
Assisted-by: Claude Opus 5.5
---
compiler-rt/lib/asan/asan_allocator.cpp | 6 +-
compiler-rt/lib/dfsan/dfsan_new_delete.cpp | 17 ++++--
compiler-rt/lib/hwasan/hwasan_new_delete.cpp | 20 +++++--
compiler-rt/lib/lsan/lsan_interceptors.cpp | 14 +++--
.../lib/memprof/memprof_new_delete.cpp | 3 +
compiler-rt/lib/msan/msan_new_delete.cpp | 21 ++++---
compiler-rt/lib/nsan/nsan_new_delete.cpp | 6 ++
.../sanitizer_allocator_checks.h | 6 ++
.../sanitizer_common/sanitizer_new_handler.h | 14 +++--
.../sanitizer_new_operators.inc | 13 +++--
compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp | 31 ++++++-----
.../test/dfsan/invalid_aligned_new.cpp | 43 +++++++++++++++
.../memprof/TestCases/invalid_aligned_new.cpp | 34 ++++++++++++
compiler-rt/test/nsan/invalid_aligned_new.cpp | 34 ++++++++++++
.../TestCases/invalid_aligned_new.cpp | 55 +++++++++++++++++++
15 files changed, 263 insertions(+), 54 deletions(-)
create mode 100644 compiler-rt/test/dfsan/invalid_aligned_new.cpp
create mode 100644 compiler-rt/test/memprof/TestCases/invalid_aligned_new.cpp
create mode 100644 compiler-rt/test/nsan/invalid_aligned_new.cpp
create mode 100644 compiler-rt/test/sanitizer_common/TestCases/invalid_aligned_new.cpp
diff --git a/compiler-rt/lib/asan/asan_allocator.cpp b/compiler-rt/lib/asan/asan_allocator.cpp
index 6af197ab4cb1c..b75a8ea09e34a 100644
--- a/compiler-rt/lib/asan/asan_allocator.cpp
+++ b/compiler-rt/lib/asan/asan_allocator.cpp
@@ -1214,12 +1214,8 @@ void* asan_new(uptr size, BufferedStackTrace* stack, bool array) {
void* asan_new_aligned(uptr size, uptr alignment, BufferedStackTrace* stack,
bool array) {
- if (UNLIKELY(alignment == 0 || !IsPowerOfTwo(alignment))) {
- errno = errno_EINVAL;
- if (AllocatorMayReturnNull())
- return nullptr;
+ if (UNLIKELY(!CheckAlignedNewAlignment(alignment)))
ReportInvalidAllocationAlignment(alignment, stack);
- }
return SetErrnoOnNull(instance.Allocate(size, alignment, stack,
array ? FROM_NEW_BR : FROM_NEW,
/*can_fill=*/true));
diff --git a/compiler-rt/lib/dfsan/dfsan_new_delete.cpp b/compiler-rt/lib/dfsan/dfsan_new_delete.cpp
index 4482e22951040..0031ae9737b0f 100644
--- a/compiler-rt/lib/dfsan/dfsan_new_delete.cpp
+++ b/compiler-rt/lib/dfsan/dfsan_new_delete.cpp
@@ -16,6 +16,7 @@
#include "dfsan.h"
#include "interception/interception.h"
#include "sanitizer_common/sanitizer_allocator.h"
+#include "sanitizer_common/sanitizer_allocator_checks.h"
#include "sanitizer_common/sanitizer_allocator_report.h"
using namespace __dfsan;
@@ -34,12 +35,16 @@ enum class align_val_t : size_t {};
ReportOutOfMemory(size, &stack); \
} \
return res
-#define OPERATOR_NEW_BODY_ALIGN(nothrow) \
- void *res = dfsan_memalign((uptr)align, size); \
- if (!nothrow && UNLIKELY(!res)) { \
- UNINITIALIZED BufferedStackTrace stack; \
- ReportOutOfMemory(size, &stack); \
- } \
+#define OPERATOR_NEW_BODY_ALIGN(nothrow) \
+ if (UNLIKELY(!CheckAlignedNewAlignment((uptr)align))) { \
+ UNINITIALIZED BufferedStackTrace stack; \
+ ReportInvalidAllocationAlignment((uptr)align, &stack); \
+ } \
+ void* res = dfsan_memalign((uptr)align, size); \
+ if (!nothrow && UNLIKELY(!res)) { \
+ UNINITIALIZED BufferedStackTrace stack; \
+ ReportOutOfMemory(size, &stack); \
+ } \
return res;
INTERCEPTOR_ATTRIBUTE
diff --git a/compiler-rt/lib/hwasan/hwasan_new_delete.cpp b/compiler-rt/lib/hwasan/hwasan_new_delete.cpp
index 232eb0eb6da67..f4a32476e3775 100644
--- a/compiler-rt/lib/hwasan/hwasan_new_delete.cpp
+++ b/compiler-rt/lib/hwasan/hwasan_new_delete.cpp
@@ -11,14 +11,15 @@
// Interceptors for operators new and delete.
//===----------------------------------------------------------------------===//
+#include <stddef.h>
+#include <stdlib.h>
+
#include "hwasan.h"
#include "interception/interception.h"
#include "sanitizer_common/sanitizer_allocator.h"
+#include "sanitizer_common/sanitizer_allocator_checks.h"
#include "sanitizer_common/sanitizer_allocator_report.h"
-#include <stddef.h>
-#include <stdlib.h>
-
#if HWASAN_REPLACE_OPERATORS_NEW_AND_DELETE
// TODO(alekseys): throw std::bad_alloc instead of dying on OOM.
@@ -40,23 +41,32 @@
# define OPERATOR_NEW_BODY_ARRAY_NOTHROW \
GET_MALLOC_STACK_TRACE; \
return hwasan_malloc(size, &stack)
+# define CHECK_ALIGNED_NEW_ALIGNMENT \
+ do { \
+ if (UNLIKELY(!CheckAlignedNewAlignment(static_cast<uptr>(align)))) \
+ ReportInvalidAllocationAlignment(static_cast<uptr>(align), &stack); \
+ } while (0)
# define OPERATOR_NEW_BODY_ALIGN \
GET_MALLOC_STACK_TRACE; \
- void *res = hwasan_memalign(static_cast<uptr>(align), size, &stack); \
+ CHECK_ALIGNED_NEW_ALIGNMENT; \
+ void* res = hwasan_memalign(static_cast<uptr>(align), size, &stack); \
if (UNLIKELY(!res)) \
ReportOutOfMemory(size, &stack); \
return res
# define OPERATOR_NEW_BODY_ALIGN_NOTHROW \
GET_MALLOC_STACK_TRACE; \
+ CHECK_ALIGNED_NEW_ALIGNMENT; \
return hwasan_memalign(static_cast<uptr>(align), size, &stack)
# define OPERATOR_NEW_BODY_ALIGN_ARRAY \
GET_MALLOC_STACK_TRACE; \
- void *res = hwasan_memalign(static_cast<uptr>(align), size, &stack); \
+ CHECK_ALIGNED_NEW_ALIGNMENT; \
+ void* res = hwasan_memalign(static_cast<uptr>(align), size, &stack); \
if (UNLIKELY(!res)) \
ReportOutOfMemory(size, &stack); \
return res
# define OPERATOR_NEW_BODY_ALIGN_ARRAY_NOTHROW \
GET_MALLOC_STACK_TRACE; \
+ CHECK_ALIGNED_NEW_ALIGNMENT; \
return hwasan_memalign(static_cast<uptr>(align), size, &stack)
# define OPERATOR_DELETE_BODY \
diff --git a/compiler-rt/lib/lsan/lsan_interceptors.cpp b/compiler-rt/lib/lsan/lsan_interceptors.cpp
index 5340c6ffba607..13bc6f22b2060 100644
--- a/compiler-rt/lib/lsan/lsan_interceptors.cpp
+++ b/compiler-rt/lib/lsan/lsan_interceptors.cpp
@@ -13,6 +13,7 @@
#include "interception/interception.h"
#include "sanitizer_common/sanitizer_allocator.h"
+#include "sanitizer_common/sanitizer_allocator_checks.h"
#include "sanitizer_common/sanitizer_allocator_dlsym.h"
#include "sanitizer_common/sanitizer_allocator_report.h"
#include "sanitizer_common/sanitizer_atomic.h"
@@ -256,11 +257,14 @@ INTERCEPTOR(int, mprobe, void *ptr) {
void *res = lsan_malloc(size, stack);\
if (!nothrow && UNLIKELY(!res)) ReportOutOfMemory(size, &stack);\
return res;
-#define OPERATOR_NEW_BODY_ALIGN(nothrow)\
- ENSURE_LSAN_INITED;\
- GET_STACK_TRACE_MALLOC;\
- void *res = lsan_memalign((uptr)align, size, stack);\
- if (!nothrow && UNLIKELY(!res)) ReportOutOfMemory(size, &stack);\
+#define OPERATOR_NEW_BODY_ALIGN(nothrow) \
+ ENSURE_LSAN_INITED; \
+ GET_STACK_TRACE_MALLOC; \
+ if (UNLIKELY(!CheckAlignedNewAlignment((uptr)align))) \
+ ReportInvalidAllocationAlignment((uptr)align, &stack); \
+ void* res = lsan_memalign((uptr)align, size, stack); \
+ if (!nothrow && UNLIKELY(!res)) \
+ ReportOutOfMemory(size, &stack); \
return res;
#define OPERATOR_DELETE_BODY\
diff --git a/compiler-rt/lib/memprof/memprof_new_delete.cpp b/compiler-rt/lib/memprof/memprof_new_delete.cpp
index cae5de301367a..6def879c959c1 100644
--- a/compiler-rt/lib/memprof/memprof_new_delete.cpp
+++ b/compiler-rt/lib/memprof/memprof_new_delete.cpp
@@ -14,6 +14,7 @@
#include "memprof_allocator.h"
#include "memprof_internal.h"
#include "memprof_stack.h"
+#include "sanitizer_common/sanitizer_allocator_checks.h"
#include "sanitizer_common/sanitizer_allocator_report.h"
#include "interception/interception.h"
@@ -38,6 +39,8 @@ enum class align_val_t : size_t {};
return res;
#define OPERATOR_NEW_BODY_ALIGN(type, nothrow) \
GET_STACK_TRACE_MALLOC; \
+ if (UNLIKELY(!CheckAlignedNewAlignment((uptr)align))) \
+ ReportInvalidAllocationAlignment((uptr)align, &stack); \
void *res = memprof_memalign((uptr)align, size, &stack, type); \
if (!nothrow && UNLIKELY(!res)) \
ReportOutOfMemory(size, &stack); \
diff --git a/compiler-rt/lib/msan/msan_new_delete.cpp b/compiler-rt/lib/msan/msan_new_delete.cpp
index 7daa55474b7da..177e02e8044cc 100644
--- a/compiler-rt/lib/msan/msan_new_delete.cpp
+++ b/compiler-rt/lib/msan/msan_new_delete.cpp
@@ -11,9 +11,10 @@
// Interceptors for operators new and delete.
//===----------------------------------------------------------------------===//
-#include "msan.h"
#include "interception/interception.h"
+#include "msan.h"
#include "sanitizer_common/sanitizer_allocator.h"
+#include "sanitizer_common/sanitizer_allocator_checks.h"
#include "sanitizer_common/sanitizer_allocator_report.h"
#if MSAN_REPLACE_OPERATORS_NEW_AND_DELETE
@@ -38,13 +39,17 @@ namespace std {
ReportOutOfMemory(size, &stack); \
} \
return res
-# define OPERATOR_NEW_BODY_ALIGN(nothrow) \
- GET_MALLOC_STACK_TRACE; \
- void *res = msan_memalign((uptr)align, size, &stack); \
- if (!nothrow && UNLIKELY(!res)) { \
- GET_FATAL_STACK_TRACE_IF_EMPTY(&stack); \
- ReportOutOfMemory(size, &stack); \
- } \
+# define OPERATOR_NEW_BODY_ALIGN(nothrow) \
+ GET_MALLOC_STACK_TRACE; \
+ if (UNLIKELY(!CheckAlignedNewAlignment((uptr)align))) { \
+ GET_FATAL_STACK_TRACE_IF_EMPTY(&stack); \
+ ReportInvalidAllocationAlignment((uptr)align, &stack); \
+ } \
+ void* res = msan_memalign((uptr)align, size, &stack); \
+ if (!nothrow && UNLIKELY(!res)) { \
+ GET_FATAL_STACK_TRACE_IF_EMPTY(&stack); \
+ ReportOutOfMemory(size, &stack); \
+ } \
return res;
INTERCEPTOR_ATTRIBUTE
diff --git a/compiler-rt/lib/nsan/nsan_new_delete.cpp b/compiler-rt/lib/nsan/nsan_new_delete.cpp
index f203a583f2c44..0101e19d202c0 100644
--- a/compiler-rt/lib/nsan/nsan_new_delete.cpp
+++ b/compiler-rt/lib/nsan/nsan_new_delete.cpp
@@ -15,6 +15,7 @@
#include "nsan.h"
#include "nsan_allocator.h"
#include "sanitizer_common/sanitizer_allocator.h"
+#include "sanitizer_common/sanitizer_allocator_checks.h"
#include "sanitizer_common/sanitizer_allocator_report.h"
#include <stddef.h>
@@ -36,6 +37,11 @@ enum class align_val_t : size_t {};
} \
return res
#define OPERATOR_NEW_BODY_ALIGN(nothrow) \
+ if (UNLIKELY(!CheckAlignedNewAlignment((uptr)align))) { \
+ BufferedStackTrace stack; \
+ GET_FATAL_STACK_TRACE_IF_EMPTY(&stack); \
+ ReportInvalidAllocationAlignment((uptr)align, &stack); \
+ } \
void *res = nsan_memalign((uptr)align, size); \
if (!nothrow && UNLIKELY(!res)) { \
BufferedStackTrace stack; \
diff --git a/compiler-rt/lib/sanitizer_common/sanitizer_allocator_checks.h b/compiler-rt/lib/sanitizer_common/sanitizer_allocator_checks.h
index 1cc3992c4c9fa..af3f3a278c2a5 100644
--- a/compiler-rt/lib/sanitizer_common/sanitizer_allocator_checks.h
+++ b/compiler-rt/lib/sanitizer_common/sanitizer_allocator_checks.h
@@ -57,6 +57,12 @@ inline bool CheckPosixMemalignAlignment(uptr alignment) {
(alignment % sizeof(void *)) == 0;
}
+// Checks the std::align_val_t argument of aligned operator new, verifies that
+// the alignment is a power of two.
+inline bool CheckAlignedNewAlignment(uptr alignment) {
+ return alignment != 0 && IsPowerOfTwo(alignment);
+}
+
// Returns true if calloc(size, n) call overflows on size*n calculation.
inline bool CheckForCallocOverflow(uptr size, uptr n) {
if (!size)
diff --git a/compiler-rt/lib/sanitizer_common/sanitizer_new_handler.h b/compiler-rt/lib/sanitizer_common/sanitizer_new_handler.h
index 0b52796836c96..706b38fc7cdd4 100644
--- a/compiler-rt/lib/sanitizer_common/sanitizer_new_handler.h
+++ b/compiler-rt/lib/sanitizer_common/sanitizer_new_handler.h
@@ -13,10 +13,12 @@
// throwing / nothrow exhaustion policies that compose with it. Each
// sanitizer's operator new wrapper supplies two small lambdas:
//
-// * Alloc — invokes the sanitizer's internal allocator, returning
-// nullptr on OOM (never aborting on OOM). Other detected
-// failure modes (e.g. invalid alignment) should abort with
-// a diagnostic.
+// * Alloc — invokes the sanitizer's internal allocator. Returns
+// nullptr, rather than aborting, only when storage cannot
+// be obtained: every nullptr return is treated as a
+// storage failure by the std::get_new_handler() loop.
+// Other detected failures (e.g. invalid alignment) must
+// abort with a diagnostic.
//
// * OnExhausted — invokes the sanitizer's "abort with diagnostic"
// handler (e.g. asan's ReportOutOfMemory + Die()).
@@ -51,8 +53,8 @@ new_handler get_new_handler() noexcept;
namespace __sanitizer {
// Runs std::get_new_handler() per [new.delete.single]/3+/4 until the
-// allocation succeeds or the chain is exhausted. Returns the allocated
-// pointer on success, nullptr if the handler chain is exhausted.
+// allocation succeeds (returns the allocated pointer) or the chain is
+// exhausted (returns nullptr).
//
// NOTE: Exceptions thrown by Alloc or std::new_handler callbacks escape this
// function. Callers that need to convert exceptions to a nullptr return
diff --git a/compiler-rt/lib/sanitizer_common/sanitizer_new_operators.inc b/compiler-rt/lib/sanitizer_common/sanitizer_new_operators.inc
index 66dd83f2fddc5..a81426bc17c74 100644
--- a/compiler-rt/lib/sanitizer_common/sanitizer_new_operators.inc
+++ b/compiler-rt/lib/sanitizer_common/sanitizer_new_operators.inc
@@ -11,9 +11,9 @@
// All eight variants compose the same per-call setup (a sanitizer-specific
// stack-trace acquisition) with one of the two chain-handling templates from
// sanitizer_new_handler.h and two sanitizer-supplied lambdas: an Alloc that
-// invokes the sanitizer's internal allocator (returning nullptr on failure)
-// and an OnExhausted that calls the sanitizer's "abort with diagnostic"
-// handler.
+// invokes the sanitizer's internal allocator (returning nullptr only when
+// storage cannot be obtained) and an OnExhausted that calls the sanitizer's
+// "abort with diagnostic" handler.
//
// This file is included after the consuming TU has defined the prerequisite
// macros listed below. Names reference symbols (e.g. `stack`, `size`) that
@@ -40,9 +40,10 @@
// SANITIZER_NEW_ALIGNED(size, align)
// SANITIZER_NEW_ARRAY_ALIGNED(size, align)
// The sanitizer's four internal alloc helpers. Each must return nullptr
-// on OOM (so the chain-handling templates can drive the
-// std::get_new_handler() loop). Other failure modes (e.g. invalid
-// alignment) should cause the helper to abort with a diagnostic.
+// only when storage cannot be obtained: every nullptr return is treated
+// as a storage failure by the std::get_new_handler() loop. Other failure
+// modes (e.g. invalid alignment) must cause the helper to abort with a
+// diagnostic.
//
//===----------------------------------------------------------------------===//
diff --git a/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp b/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp
index fc44a5221b5b0..6314348e5d4c5 100644
--- a/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp
+++ b/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp
@@ -12,6 +12,7 @@
//===----------------------------------------------------------------------===//
#include "interception/interception.h"
#include "sanitizer_common/sanitizer_allocator.h"
+#include "sanitizer_common/sanitizer_allocator_checks.h"
#include "sanitizer_common/sanitizer_allocator_report.h"
#include "sanitizer_common/sanitizer_internal_defs.h"
#include "tsan_interceptors.h"
@@ -43,19 +44,23 @@ DECLARE_REAL(void, free, void *ptr)
invoke_malloc_hook(p, size); \
return p;
-#define OPERATOR_NEW_BODY_ALIGN(mangled_name, nothrow) \
- if (in_symbolizer()) \
- return InternalAlloc(size, nullptr, (uptr)align); \
- void *p = 0; \
- { \
- SCOPED_INTERCEPTOR_RAW(mangled_name, size); \
- p = user_memalign(thr, pc, (uptr)align, size); \
- if (!nothrow && UNLIKELY(!p)) { \
- GET_STACK_TRACE_FATAL(thr, pc); \
- ReportOutOfMemory(size, &stack); \
- } \
- } \
- invoke_malloc_hook(p, size); \
+#define OPERATOR_NEW_BODY_ALIGN(mangled_name, nothrow) \
+ if (in_symbolizer()) \
+ return InternalAlloc(size, nullptr, (uptr)align); \
+ void* p = 0; \
+ { \
+ SCOPED_INTERCEPTOR_RAW(mangled_name, size); \
+ if (UNLIKELY(!CheckAlignedNewAlignment((uptr)align))) { \
+ GET_STACK_TRACE_FATAL(thr, pc); \
+ ReportInvalidAllocationAlignment((uptr)align, &stack); \
+ } \
+ p = user_memalign(thr, pc, (uptr)align, size); \
+ if (!nothrow && UNLIKELY(!p)) { \
+ GET_STACK_TRACE_FATAL(thr, pc); \
+ ReportOutOfMemory(size, &stack); \
+ } \
+ } \
+ invoke_malloc_hook(p, size); \
return p;
SANITIZER_INTERFACE_ATTRIBUTE
diff --git a/compiler-rt/test/dfsan/invalid_aligned_new.cpp b/compiler-rt/test/dfsan/invalid_aligned_new.cpp
new file mode 100644
index 0000000000000..1018615224d53
--- /dev/null
+++ b/compiler-rt/test/dfsan/invalid_aligned_new.cpp
@@ -0,0 +1,43 @@
+// Invalid alignment is a caller error, not a failure to obtain storage. DFSan
+// must report it, even with allocator_may_return_null=1, rather than return
+// nullptr. DFSan's operator new overrides serve uninstrumented code, so the
+// allocation happens in an uninstrumented object linked into the program.
+
+// RUN: %clangxx_dfsan -fno-sanitize=dataflow -std=c++17 -DLIB -c %s -o %t-lib.o
+// RUN: echo 'fun:AllocAligned=uninstrumented' > %t.abilist
+// RUN: echo 'fun:AllocAligned=discard' >> %t.abilist
+// RUN: %clangxx_dfsan -std=c++17 -fsanitize-ignorelist=%t.abilist %s %t-lib.o -o %t
+// RUN: env DFSAN_OPTIONS=allocator_may_return_null=1 not %run %t new 0 2>&1 | FileCheck %s -DALIGN=0
+// RUN: env DFSAN_OPTIONS=allocator_may_return_null=1 not %run %t new-nothrow 0 2>&1 | FileCheck %s -DALIGN=0
+// RUN: env DFSAN_OPTIONS=allocator_may_return_null=1 not %run %t new 3 2>&1 | FileCheck %s -DALIGN=3
+// RUN: env DFSAN_OPTIONS=allocator_may_return_null=1 not %run %t new-nothrow 3 2>&1 | FileCheck %s -DALIGN=3
+
+#include <cstddef>
+
+extern "C" void *AllocAligned(bool nothrow, std::size_t alignment);
+
+#ifdef LIB
+# include <new>
+
+extern "C" void *AllocAligned(bool nothrow, std::size_t alignment) {
+ const auto align = static_cast<std::align_val_t>(alignment);
+ return nothrow ? ::operator new(16, align, std::nothrow)
+ : ::operator new(16, align);
+}
+#else
+# include <cassert>
+# include <cstdio>
+# include <cstdlib>
+# include <cstring>
+
+int main(int argc, char **argv) {
+ assert(argc == 3);
+ const bool nothrow = std::strcmp(argv[1], "new-nothrow") == 0;
+ assert(nothrow || std::strcmp(argv[1], "new") == 0);
+ void *p = AllocAligned(nothrow, std::strtoull(argv[2], nullptr, 0));
+ std::fprintf(stderr, "allocation unexpectedly returned %p\n", p);
+ return 1;
+}
+#endif
+
+// CHECK: ERROR: DataflowSanitizer: invalid allocation alignment: [[ALIGN]],
diff --git a/compiler-rt/test/memprof/TestCases/invalid_aligned_new.cpp b/compiler-rt/test/memprof/TestCases/invalid_aligned_new.cpp
new file mode 100644
index 0000000000000..5bc237c693c09
--- /dev/null
+++ b/compiler-rt/test/memprof/TestCases/invalid_aligned_new.cpp
@@ -0,0 +1,34 @@
+// Invalid alignment is a caller error, not a failure to obtain storage.
+// MemProf must report it, even with allocator_may_return_null=1, rather than
+// return nullptr.
+
+// RUN: %clangxx_memprof -O0 -std=c++17 %s -o %t
+// RUN: %env_memprof_opts=log_path=stderr:allocator_may_return_null=1 not %run %t new 0 2>&1 | FileCheck %s -DALIGN=0
+// RUN: %env_memprof_opts=log_path=stderr:allocator_may_return_null=1 not %run %t new-nothrow 0 2>&1 | FileCheck %s -DALIGN=0
+// RUN: %env_memprof_opts=log_path=stderr:allocator_may_return_null=1 not %run %t new 3 2>&1 | FileCheck %s -DALIGN=3
+// RUN: %env_memprof_opts=log_path=stderr:allocator_may_return_null=1 not %run %t new-nothrow 3 2>&1 | FileCheck %s -DALIGN=3
+
+#include <cassert>
+#include <cstdio>
+#include <cstdlib>
+#include <cstring>
+#include <new>
+
+int main(int argc, char **argv) {
+ assert(argc == 3);
+ const auto alignment =
+ static_cast<std::align_val_t>(std::strtoull(argv[2], nullptr, 0));
+
+ void *p;
+ if (std::strcmp(argv[1], "new") == 0)
+ p = ::operator new(16, alignment);
+ else if (std::strcmp(argv[1], "new-nothrow") == 0)
+ p = ::operator new(16, alignment, std::nothrow);
+ else
+ assert(false);
+
+ std::fprintf(stderr, "allocation unexpectedly returned %p\n", p);
+ return 1;
+}
+
+// CHECK: ERROR: MemProfiler: invalid allocation alignment: [[ALIGN]],
diff --git a/compiler-rt/test/nsan/invalid_aligned_new.cpp b/compiler-rt/test/nsan/invalid_aligned_new.cpp
new file mode 100644
index 0000000000000..e431024b56e8d
--- /dev/null
+++ b/compiler-rt/test/nsan/invalid_aligned_new.cpp
@@ -0,0 +1,34 @@
+// Invalid alignment is a caller error, not a failure to obtain storage. NSan
+// must report it, even with allocator_may_return_null=1, rather than return
+// nullptr.
+
+// RUN: %clangxx_nsan -O0 %s -o %t
+// RUN: env NSAN_OPTIONS=allocator_may_return_null=1 not %run %t new 0 2>&1 | FileCheck %s -DALIGN=0
+// RUN: env NSAN_OPTIONS=allocator_may_return_null=1 not %run %t new-nothrow 0 2>&1 | FileCheck %s -DALIGN=0
+// RUN: env NSAN_OPTIONS=allocator_may_return_null=1 not %run %t new 3 2>&1 | FileCheck %s -DALIGN=3
+// RUN: env NSAN_OPTIONS=allocator_may_return_null=1 not %run %t new-nothrow 3 2>&1 | FileCheck %s -DALIGN=3
+
+#include <cassert>
+#include <cstdio>
+#include <cstdlib>
+#include <cstring>
+#include <new>
+
+int main(int argc, char **argv) {
+ assert(argc == 3);
+ const auto alignment =
+ static_cast<std::align_val_t>(std::strtoull(argv[2], nullptr, 0));
+
+ void *p;
+ if (std::strcmp(argv[1], "new") == 0)
+ p = ::operator new(16, alignment);
+ else if (std::strcmp(argv[1], "new-nothrow") == 0)
+ p = ::operator new(16, alignment, std::nothrow);
+ else
+ assert(false);
+
+ std::fprintf(stderr, "allocation unexpectedly returned %p\n", p);
+ return 1;
+}
+
+// CHECK: ERROR: NumericalStabilitySanitizer: invalid allocation alignment: [[ALIGN]],
diff --git a/compiler-rt/test/sanitizer_common/TestCases/invalid_aligned_new.cpp b/compiler-rt/test/sanitizer_common/TestCases/invalid_aligned_new.cpp
new file mode 100644
index 0000000000000..b47d0ca0c2c6d
--- /dev/null
+++ b/compiler-rt/test/sanitizer_common/TestCases/invalid_aligned_new.cpp
@@ -0,0 +1,55 @@
+// Invalid alignment is a caller error, not a failure to obtain storage. The
+// sanitizer must report it, even with allocator_may_return_null=1, rather than
+// return nullptr or call std::new_handler.
+
+// RUN: %clangxx -O0 -std=c++17 %s -o %t
+// RUN: %env_tool_opts=allocator_may_return_null=1 not %run %t new 0 2>&1 | FileCheck %s -DALIGN=0
+// RUN: %env_tool_opts=allocator_may_return_null=1 not %run %t new-array 0 2>&1 | FileCheck %s -DALIGN=0
+// RUN: %env_tool_opts=allocator_may_return_null=1 not %run %t new-nothrow 0 2>&1 | FileCheck %s -DALIGN=0
+// RUN: %env_tool_opts=allocator_may_return_null=1 not %run %t new-array-nothrow 0 2>&1 | FileCheck %s -DALIGN=0
+// RUN: %env_tool_opts=allocator_may_return_null=1 not %run %t new 3 2>&1 | FileCheck %s -DALIGN=3
+// RUN: %env_tool_opts=allocator_may_return_null=1 not %run %t new-array 3 2>&1 | FileCheck %s -DALIGN=3
+// RUN: %env_tool_opts=allocator_may_return_null=1 not %run %t new-nothrow 3 2>&1 | FileCheck %s -DALIGN=3
+// RUN: %env_tool_opts=allocator_may_return_null=1 not %run %t new-array-nothrow 3 2>&1 | FileCheck %s -DALIGN=3
+
+// ubsan does not replace operator new. ASan and LSan do not override aligned
+// operator new on Darwin, nor ASan on Windows (MSVC).
+// UNSUPPORTED: ubsan, (asan || lsan) && darwin, target={{.*windows-msvc.*}}
+// REQUIRES: stable-runtime
+
+#include <cassert>
+#include <cstdio>
+#include <cstdlib>
+#include <cstring>
+#include <new>
+
+static void NewHandler() {
+ std::fputs("new_handler called\n", stderr);
+ std::abort();
+}
+
+int main(int argc, char **argv) {
+ assert(argc == 3);
+ std::set_new_handler(NewHandler);
+ const auto alignment =
+ static_cast<std::align_val_t>(std::strtoull(argv[2], nullptr, 0));
+
+ void *p;
+ if (std::strcmp(argv[1], "new") == 0)
+ p = ::operator new(16, alignment);
+ else if (std::strcmp(argv[1], "new-array") == 0)
+ p = ::operator new[](16, alignment);
+ else if (std::strcmp(argv[1], "new-nothrow") == 0)
+ p = ::operator new(16, alignment, std::nothrow);
+ else if (std::strcmp(argv[1], "new-array-nothrow") == 0)
+ p = ::operator new[](16, alignment, std::nothrow);
+ else
+ assert(false);
+
+ std::fprintf(stderr, "allocation unexpectedly returned %p\n", p);
+ return 1;
+}
+
+// CHECK-NOT: new_handler called
+// CHECK: ERROR: {{.*}}Sanitizer: invalid allocation alignment: [[ALIGN]],
+// CHECK: SUMMARY: {{.*}}Sanitizer: invalid-allocation-alignment
>From 5860ba43f5962906f7640dd72530435dfd5b496b Mon Sep 17 00:00:00 2001
From: "Justin T. Gibbs" <gibbs at scsiguy.com>
Date: Fri, 25 Sep 2026 10:20:17 -0700
Subject: [PATCH 2/2] [sanitizer_common] Add unit tests for operator new
failure handling
Test `RunNewHandlerChain`, `NewImplThrowing`, `NewImplNothrow` and
`InvokeOnExhausted` with fake allocation and exhaustion callbacks. The
tests cover handler retries, chain exhaustion, exceptions thrown by a
`std::new_handler`, and the `allocator_may_return_null` exhaustion
policy for throwing and nothrow `operator new`. Sanitizers that adopt
the framework then only need to test their own wiring.
The tests build only where the header includes `<new>` and supports
exceptions, which excludes Windows.
Assisted-by: Claude Opus 5.5
---
.../lib/sanitizer_common/tests/CMakeLists.txt | 1 +
.../tests/sanitizer_new_handler_test.cpp | 202 ++++++++++++++++++
2 files changed, 203 insertions(+)
create mode 100644 compiler-rt/lib/sanitizer_common/tests/sanitizer_new_handler_test.cpp
diff --git a/compiler-rt/lib/sanitizer_common/tests/CMakeLists.txt b/compiler-rt/lib/sanitizer_common/tests/CMakeLists.txt
index bf3e32c3f7786..ce5a1019144aa 100644
--- a/compiler-rt/lib/sanitizer_common/tests/CMakeLists.txt
+++ b/compiler-rt/lib/sanitizer_common/tests/CMakeLists.txt
@@ -34,6 +34,7 @@ set(SANITIZER_UNITTESTS
sanitizer_mac_test.cpp
sanitizer_module_uuid_size.cpp
sanitizer_mutex_test.cpp
+ sanitizer_new_handler_test.cpp
sanitizer_nolibc_test.cpp
sanitizer_posix_test.cpp
sanitizer_printf_test.cpp
diff --git a/compiler-rt/lib/sanitizer_common/tests/sanitizer_new_handler_test.cpp b/compiler-rt/lib/sanitizer_common/tests/sanitizer_new_handler_test.cpp
new file mode 100644
index 0000000000000..39a4f557856a8
--- /dev/null
+++ b/compiler-rt/lib/sanitizer_common/tests/sanitizer_new_handler_test.cpp
@@ -0,0 +1,202 @@
+//===-- sanitizer_new_handler_test.cpp ------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Tests for the operator new failure handling in sanitizer_new_handler.h,
+// driven by fake allocation and exhaustion callbacks.
+//
+//===----------------------------------------------------------------------===//
+#include "sanitizer_common/sanitizer_platform.h"
+
+#if !SANITIZER_WINDOWS && defined(__cpp_exceptions)
+
+# include <stdio.h>
+# include <stdlib.h>
+
+# include <new>
+# include <stdexcept>
+
+# include "gtest/gtest.h"
+# include "sanitizer_common/sanitizer_new_handler.h"
+
+namespace __sanitizer {
+namespace {
+
+int handler_calls;
+// Number of calls after which CountingHandler uninstalls itself.
+int handler_budget;
+
+void CountingHandler() {
+ if (++handler_calls >= handler_budget)
+ std::set_new_handler(nullptr);
+}
+
+void ThrowingHandler() { throw std::runtime_error("handler"); }
+
+// Ends the retry loop by throwing std::bad_alloc while staying installed.
+void BadAllocHandler() {
+ ++handler_calls;
+ throw std::bad_alloc();
+}
+
+void ReportExhausted() {
+ fprintf(stderr, "exhausted after %d handler call(s)\n", handler_calls);
+ abort();
+}
+
+// Exhaustion callbacks must not return; throwing records the failure without
+// killing the test binary.
+struct UnexpectedExhaustion {};
+
+void UnexpectedExhausted() {
+ ADD_FAILURE() << "unexpected exhaustion";
+ throw UnexpectedExhaustion();
+}
+
+class NewHandlerTest : public ::testing::Test {
+ protected:
+ // The new_handler and allocator_may_return_null are process-wide; restore
+ // them so other tests in this binary are unaffected.
+ void SetUp() override {
+ saved_handler_ = std::get_new_handler();
+ saved_may_return_null_ = AllocatorMayReturnNull();
+ handler_calls = 0;
+ handler_budget = 1;
+ attempts_ = 0;
+ }
+
+ void TearDown() override {
+ std::set_new_handler(saved_handler_);
+ SetAllocatorMayReturnNull(saved_may_return_null_);
+ }
+
+ // Returns an allocation callback that fails `failures` times, then succeeds.
+ auto Alloc(int failures) {
+ return [this, failures]() -> void* {
+ return ++attempts_ > failures ? &storage_ : nullptr;
+ };
+ }
+
+ // "Always" means the first ~million attempts. Eventually succeeding makes an
+ // infinite retry loop fail with a clear diagnostic instead of timing out.
+ static constexpr int kAlwaysFail = 1 << 20;
+ int attempts_;
+ char storage_;
+
+ private:
+ std::new_handler saved_handler_;
+ bool saved_may_return_null_;
+};
+
+TEST_F(NewHandlerTest, SuccessSkipsHandler) {
+ std::set_new_handler(CountingHandler);
+ EXPECT_EQ(&storage_, RunNewHandlerChain(Alloc(0)));
+ EXPECT_EQ(1, attempts_);
+ EXPECT_EQ(0, handler_calls);
+}
+
+TEST_F(NewHandlerTest, RetriesAfterEachHandlerCall) {
+ handler_budget = 10;
+ std::set_new_handler(CountingHandler);
+ EXPECT_EQ(&storage_, RunNewHandlerChain(Alloc(3)));
+ EXPECT_EQ(4, attempts_);
+ EXPECT_EQ(3, handler_calls);
+}
+
+TEST_F(NewHandlerTest, NoHandlerReturnsNull) {
+ std::set_new_handler(nullptr);
+ EXPECT_EQ(nullptr, RunNewHandlerChain(Alloc(kAlwaysFail)));
+ EXPECT_EQ(1, attempts_);
+}
+
+TEST_F(NewHandlerTest, ExhaustedChainReturnsNull) {
+ handler_budget = 2;
+ std::set_new_handler(CountingHandler);
+ EXPECT_EQ(nullptr, RunNewHandlerChain(Alloc(kAlwaysFail)));
+ EXPECT_EQ(3, attempts_);
+ EXPECT_EQ(2, handler_calls);
+}
+
+TEST_F(NewHandlerTest, ThrowingThrowsBadAllocOnExhaustionWhenMayReturnNull) {
+ SetAllocatorMayReturnNull(true);
+ std::set_new_handler(CountingHandler);
+ EXPECT_THROW((void)NewImplThrowing(Alloc(kAlwaysFail), UnexpectedExhausted),
+ std::bad_alloc);
+ EXPECT_EQ(1, handler_calls);
+}
+
+TEST_F(NewHandlerTest, ThrowingReportsExhaustion) {
+ SetAllocatorMayReturnNull(false);
+ std::set_new_handler(CountingHandler);
+ EXPECT_DEATH((void)NewImplThrowing(Alloc(kAlwaysFail), ReportExhausted),
+ "exhausted after 1 handler call");
+}
+
+TEST_F(NewHandlerTest, ThrowingPropagatesHandlerException) {
+ std::set_new_handler(ThrowingHandler);
+ for (bool may_return_null : {false, true}) {
+ SetAllocatorMayReturnNull(may_return_null);
+ EXPECT_THROW((void)NewImplThrowing(Alloc(kAlwaysFail), UnexpectedExhausted),
+ std::runtime_error);
+ }
+}
+
+TEST_F(NewHandlerTest, ThrowingPropagatesHandlerBadAlloc) {
+ std::set_new_handler(BadAllocHandler);
+ for (bool may_return_null : {false, true}) {
+ SetAllocatorMayReturnNull(may_return_null);
+ handler_calls = attempts_ = 0;
+ EXPECT_THROW((void)NewImplThrowing(Alloc(kAlwaysFail), UnexpectedExhausted),
+ std::bad_alloc);
+ EXPECT_EQ(1, attempts_);
+ EXPECT_EQ(1, handler_calls);
+ EXPECT_EQ(&BadAllocHandler, std::get_new_handler());
+ }
+}
+
+TEST_F(NewHandlerTest, NothrowReturnsNullOnExhaustionWhenMayReturnNull) {
+ SetAllocatorMayReturnNull(true);
+ std::set_new_handler(CountingHandler);
+ EXPECT_EQ(nullptr, NewImplNothrow(Alloc(kAlwaysFail), UnexpectedExhausted));
+ EXPECT_EQ(1, handler_calls);
+}
+
+TEST_F(NewHandlerTest, NothrowReportsExhaustion) {
+ SetAllocatorMayReturnNull(false);
+ std::set_new_handler(CountingHandler);
+ EXPECT_DEATH((void)NewImplNothrow(Alloc(kAlwaysFail), ReportExhausted),
+ "exhausted after 1 handler call");
+}
+
+TEST_F(NewHandlerTest, NothrowSwallowsHandlerException) {
+ std::set_new_handler(ThrowingHandler);
+ for (bool may_return_null : {false, true}) {
+ SetAllocatorMayReturnNull(may_return_null);
+ EXPECT_EQ(nullptr, NewImplNothrow(Alloc(kAlwaysFail), UnexpectedExhausted));
+ }
+}
+
+TEST_F(NewHandlerTest, NothrowReturnsNullOnHandlerBadAlloc) {
+ std::set_new_handler(BadAllocHandler);
+ for (bool may_return_null : {false, true}) {
+ SetAllocatorMayReturnNull(may_return_null);
+ handler_calls = attempts_ = 0;
+ EXPECT_EQ(nullptr, NewImplNothrow(Alloc(kAlwaysFail), UnexpectedExhausted));
+ EXPECT_EQ(1, attempts_);
+ EXPECT_EQ(1, handler_calls);
+ EXPECT_EQ(&BadAllocHandler, std::get_new_handler());
+ }
+}
+
+TEST_F(NewHandlerTest, ReturningExhaustionCallbackDies) {
+ EXPECT_DEATH(InvokeOnExhausted([] {}), "OnExhausted callable returned");
+}
+
+} // namespace
+} // namespace __sanitizer
+
+#endif // !SANITIZER_WINDOWS && defined(__cpp_exceptions)
More information about the llvm-commits
mailing list