[compiler-rt] [asan] Throw std::bad_alloc from operator new on allocation failure (PR #196388)

[Justin T. Gibbs via llvm-commits]

================
@@ -83,8 +83,17 @@ COMMON_FLAG(
     "detect_leaks=false, or if __lsan_do_leak_check() is called before the "
     "handler has a chance to run.")
 COMMON_FLAG(bool, allocator_may_return_null, false,
-            "If false, the allocator will crash instead of returning 0 on "
-            "out-of-memory.")
+            "Controls allocator behavior on out-of-memory. C-allocator entry "
+            "points (malloc / calloc / realloc / aligned_alloc / "
+            "posix_memalign): default=false reports and aborts, true returns "
+            "nullptr. AddressSanitizer's operator new always runs the "
+            "std::get_new_handler() chain first per [new.delete.single]/3+/4 "
+            "regardless of this flag; on chain exhaustion default=false "
+            "reports and aborts for both forms, true throws std::bad_alloc "
+            "(throwing form) or returns nullptr (nothrow form). On Windows "
+            "the asan runtime is built without exceptions, so the throwing "
+            "form aborts even with true; the nothrow form still returns "
+            "nullptr.")
----------------
scsiguy wrote:

Hopefully we can strike a good balance between brevity and accurately documenting the flag behavior.

The doc string for the existing implementation says "If false, the allocator will crash instead of returning 0 on out-of-memory", but that isn't accurate. You can set the flag to true and still have asan crash.  For folks hoping to be ignorant consumers of ASAN, that can be really frustrating . A comprehensive doc string makes the behavior discoverable and explicit without the end user having to go dig into the source code.

Saying that the allocator "may return 'null'" implies that there are times when it won't. What should a user then do if they find the flag doesn't work as documented for their program — e.g. for the throwing versions of new, they either get a std::bad_alloc exception or get a crash on Windows. It seems prudent to document that somewhere that is easy to find.

Something like this I think would be ok:

"""
Controls allocator behavior on out-of-memory. If 'true', allocators conform to the C/C++ standard and either return 'null' or throw the C++ std::bad_alloc exception as expected. If 'false', out of memory exceptions force a crash. NOTE: On Windows the asan runtime is built without exceptions so throwing allocator forms always abort, regardless of the value of this flag.
"""

I originally added the extra detail in the doc string about how asan now handles standards compliance because it is a change in behavior and I wanted that change to be easy to find. But maybe there is a better place for this?

https://github.com/llvm/llvm-project/pull/196388