[libcxx-commits] [libcxxabi] [libcxxabi] Make fallback malloc heap size configurable via CMake (PR #181251)

[Louis Dionne via libcxx-commits]

ldionne wrote:

That's very interesting, thanks for the context! It's great to hear that libc++ will be welcoming a new major user. I would like to extend an invitation to meet so we can chat about your overall migration project if you're interested. I wasn't the one who made Apple's transition from libstdc++ to libc++ back in the days and the constraints were also different (ABI wise), but I suspect it might be useful to sync up on the general task ahead (both for you and for libc++, in case there are many such contributions planned). Obviously, my interest is in ensuring that the process goes as smoothly as possible for the libc++ contributions. If you're interested, you can reach out to me via email or Discord.

> Given the above context, what do you think we should do?

I agree this is not an easy question. On the one hand I'd be tempted to mirror the libstdc++ behavior, which seems more useful in the general case. On the other hand, I also don't know whether some memory-conscious programs might be relying on the current implementation. I kind of doubt that is the case since I've never heard of anything like that since I've been around the project *and* I wouldn't be surprised if there were other issues that made OOM recovery not work in a very memory-constrained case. Note that I'm just saying that because I've never witnessed a case where we consciously considered that case and designed to make it work, which makes me think this is probably a blind spot -- but I might very well be wrong.

Anyway, my initial reaction to this would be to introduce a way to support both behaviors that is more flexible than a simple CMake define. Instead of increasing the complexity of the library by adding another configuration option, create a new extension point that can be used to "externally" extend/change the behavior of the library. We use a similar approach for the low-level system APIs like localization and threading, for which we have https://github.com/llvm/llvm-project/blob/main/libcxx/include/__thread/support.h and https://github.com/llvm/llvm-project/blob/main/libcxx/include/__locale_dir/locale_base_api.h. We also do something similar for the [assertion handler](https://github.com/llvm/llvm-project/blob/main/libcxx/vendor/llvm/default_assertion_handler.in).

In this case, I think the "extension point" would be basically this API https://github.com/llvm/llvm-project/blob/main/libcxxabi/src/fallback_malloc.h:

```c++
namespace __cxxabiv1 {

// Allocate some memory from _somewhere_
_LIBCXXABI_HIDDEN void * __aligned_malloc_with_fallback(size_t size);

// Allocate and zero-initialize memory from _somewhere_
_LIBCXXABI_HIDDEN void * __calloc_with_fallback(size_t count, size_t size);

_LIBCXXABI_HIDDEN void __aligned_free_with_fallback(void *ptr);
_LIBCXXABI_HIDDEN void __free_with_fallback(void *ptr);

} // namespace __cxxabiv1
```

Any vendor could define a custom implementation for this API without impacting upstream libc++. This is flexible and it reduces the likelihood of downstream conflicts when we make upstream changes since the extension point is clearly defined. Libc++ could provide its current implementation (and potentially a new one that is better suited for general purpose applications, employing a strategy similar to libstdc++), but this mechanism would make it easy to accommodate new behaviors in the future without increasing the complexity of the library.

WDYT?

https://github.com/llvm/llvm-project/pull/181251