[llvm] [Support] Use malloc instead of non-throwing new (PR #92157)

[Aaron Ballman via llvm-commits]

https://github.com/AaronBallman updated https://github.com/llvm/llvm-project/pull/92157

>From 165a2f5ff93c411893725ec733f96f981e7e477c Mon Sep 17 00:00:00 2001
From: Aaron Ballman <aaron at aaronballman.com>
Date: Tue, 14 May 2024 14:05:17 -0400
Subject: [PATCH 1/2] Use malloc instead of non-throwing new

When allocating a memory buffer, we use a non-throwing new so that we
can explicitly handle memory buffers that are too large to fit into
memory. However, when exceptions are disabled, LLVM installs a custom
new handler
(https://github.com/llvm/llvm-project/blob/90109d444839683b09f0aafdc50b749cb4b3203b/llvm/lib/Support/InitLLVM.cpp#L61)
that explicitly crashes when we run out of memory
(https://github.com/llvm/llvm-project/blob/de14b749fee41d4ded711e771e43043ae3100cb3/llvm/lib/Support/ErrorHandling.cpp#L188)
and that means this particular out-of-memory situation cannot be
gracefully handled.

This was discovered while working on #embed
(https://github.com/llvm/llvm-project/pull/68620) on Windows and
resulted in a crash rather than the preprocessor issuing a diagnostic
as expected.

This patch switches away from the non-throwing new to a call to malloc
(and free), which will return a null pointer without calling a custom
new handler. It is the only instance in Clang or LLVM that I could find
which used a non-throwing new, so I did not think we would need
anything more involved than this change.

Testing this would be highly platform dependent and so it does not come
with test coverage. And because it doesn't change behavior that users
are likely to be able to observe, it does not come with a release note.
---
 llvm/lib/Support/MemoryBuffer.cpp | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/llvm/lib/Support/MemoryBuffer.cpp b/llvm/lib/Support/MemoryBuffer.cpp
index 4cc4fe019b75b..e30c3acdb8814 100644
--- a/llvm/lib/Support/MemoryBuffer.cpp
+++ b/llvm/lib/Support/MemoryBuffer.cpp
@@ -98,7 +98,7 @@ class MemoryBufferMem : public MB {
 
   /// Disable sized deallocation for MemoryBufferMem, because it has
   /// tail-allocated data.
-  void operator delete(void *p) { ::operator delete(p); }
+  void operator delete(void *p) { std::free(p); }
 
   StringRef getBufferIdentifier() const override {
     // The name is stored after the class itself.
@@ -315,7 +315,7 @@ WritableMemoryBuffer::getNewUninitMemBuffer(size_t Size,
   size_t RealLen = StringLen + Size + 1 + BufAlign.value();
   if (RealLen <= Size) // Check for rollover.
     return nullptr;
-  char *Mem = static_cast<char*>(operator new(RealLen, std::nothrow));
+  char *Mem = static_cast<char*>(std::malloc(RealLen));
   if (!Mem)
     return nullptr;
 

>From 1961c82cbf0633813a2c2089285b27e4d689846c Mon Sep 17 00:00:00 2001
From: Aaron Ballman <aaron at aaronballman.com>
Date: Wed, 15 May 2024 08:03:31 -0400
Subject: [PATCH 2/2] Add comment, reformat code; NFC

---
 llvm/lib/Support/MemoryBuffer.cpp | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/llvm/lib/Support/MemoryBuffer.cpp b/llvm/lib/Support/MemoryBuffer.cpp
index e30c3acdb8814..50308bd2bf4a3 100644
--- a/llvm/lib/Support/MemoryBuffer.cpp
+++ b/llvm/lib/Support/MemoryBuffer.cpp
@@ -315,7 +315,14 @@ WritableMemoryBuffer::getNewUninitMemBuffer(size_t Size,
   size_t RealLen = StringLen + Size + 1 + BufAlign.value();
   if (RealLen <= Size) // Check for rollover.
     return nullptr;
-  char *Mem = static_cast<char*>(std::malloc(RealLen));
+  // We use a call to malloc() rather than a call to a non-throwing operator
+  // new() because LLVM unconditionally installs an out of memory new handler
+  // when exceptions are disabled. This new handler intentionally crashes to
+  // aid with debugging, but that makes non-throwing new calls unhelpful.
+  // See MemoryBufferMem::operator delete() for the paired call to free(), and
+  // llvm::install_out_of_memory_new_handler() for the installation of the
+  // custom new handler.
+  char *Mem = static_cast<char *>(std::malloc(RealLen));
   if (!Mem)
     return nullptr;