[PATCH] D155598: [libc++abi] Use std::abort() instead of std::terminate() on failure to allocate

[Louis Dionne via Phabricator via llvm-commits]

ldionne created this revision.
ldionne added reviewers: nickdesaulniers, MaskRay.
Herald added a project: All.
ldionne requested review of this revision.
Herald added subscribers: llvm-commits, libcxx-commits, wangpc.
Herald added projects: libc++abi, LLVM.
Herald added a reviewer: libc++abi.

Inside the Itanium demangler, we would previously call std::terminate()
after failing to (re)allocate. However, programs are free to install a
custom terminate_handler which does non-trivial things. In fact, by
default libc++abi itself installs a demangling_terminate_handler() which
tries to demangle the currently active exception before aborting the
program.

In case of an out-of-memory exception caused by the system truly having
no more memory (as opposed to attempting to allocate INT_MAX just once),
we will end up trying to demangle the exception, failing to do so because
we can't grow the OutputBuffer inside ItaniumDemangle.h, and then calling
std::terminate() there. That will call the demangling_terminate_handler(),
which will then start this loop again. We eventually end up crashing due
to a stack overflow.

To fix this problem, this patch calls std::abort() directly from the
demangler instead of going through std::terminate(). After all, calling
std::abort() is the default behavior for std::terminate() according
to the Standard, so this behavior is definitely valid. The downside of
this approach is that in case of a "true" out-of-memory condition:

1. the program will throw an exception
2. std::terminate() will be called if uncaught
3. demangling_terminate_handler() will be called and will fail
4. abort() will be called

We'll end up aborting the program without mentioning the cause, which
normally looks like:

  terminating due to uncaught exception of type <TYPE>: <what()-MESSAGE>

Another option would be to properly handle failure-to-allocate inside
ItaniumDemangle.h and to propagate something like an error code or a
std::expected to the caller of all functions in the demangler that
can allocate. Then, we could make sure that __cxa_demangle returns
nullptr when it fails to demangle the input due to any error, as it is
supposed to (but today "true" out-of-memory conditions are not handled
properly). The demangling_terminate_handler() would then see that
__cxa_demangle failed to do its job and would still print the
appropriate message, simply using the non-demangled exception type.
However, this is akin to a partial rewrite of the demangler code since
a large number of functions would now have to return a std::expected
to account for out-of-memory conditions.

Using exceptions would be a lot simpler in terms of code changes and
would achieve the same result, however the demangler can't use exceptions
because it is used inside LLVM and libc++abi implements the exception
runtime anyway (so while it might be possible to use them in that
context, I'd argue we'd only be playing with fire).

rdar://110767664


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D155598

Files:
  libcxxabi/src/demangle/ItaniumDemangle.h
  libcxxabi/src/demangle/README.txt
  libcxxabi/src/demangle/Utility.h
  llvm/include/llvm/Demangle/ItaniumDemangle.h
  llvm/include/llvm/Demangle/README.txt
  llvm/include/llvm/Demangle/Utility.h

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D155598.541512.patch
Type: text/x-patch
Size: 3840 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20230718/72d7bbb5/attachment.bin>