[PATCH] D71786: RFC: [Support] On Windows, add optional support for {rpmalloc|snmalloc|mimalloc}

[Alexandre Ganea via Phabricator via llvm-commits]

aganea updated this revision to Diff 274989.
aganea retitled this revision from "RFC: [Support] On Windows, add optional support for rpmalloc" to "RFC: [Support] On Windows, add optional support for {rpmalloc|snmalloc|mimalloc}".
aganea edited the summary of this revision.
aganea added a comment.

Obviously this needs a bit more polishing, but just quick WIP before going further into the discussions.

- Removed the rpmalloc source code.
- Added support for mimalloc and snmalloc.
- The path to the replacement allocator can be set with `-DLLVM_INTEGRATED_CRT_ALLOC`.

This patch should work on Linux as well, probably with minor tweaks.

- If using rpmalloc, after git checkout set both `ENABLE_OVERRIDE` and `ENABLE_PRELOAD` to 1 in https://github.com/mjansson/rpmalloc/blob/develop/rpmalloc/rpmalloc.c
- snmalloc works out of the box after git checkout.
- If using mimalloc, after git checkout, this diff needs to be applied: https://github.com/microsoft/mimalloc/issues/268 - and the .lib be compiled through: https://github.com/microsoft/mimalloc/blob/master/ide/vs2019/mimalloc.sln

After previous steps, compile LLVM such as `cmake ... -DLLVM_USE_CRT_RELEASE=MT -DLLVM_INTEGRATED_CRT_ALLOC=D:/git/mimalloc`

Latest timings - LLD linking `clang.exe` with ThinLTO on 36-core CPU:

|                                      | Wall clock       | Page ranges commited/decommited | Total touched pages | Peak Mem    |
| Windows 10 version 2004 default heap | 38 min 47 sec    |                                 |                     | 14.9 GB     |
| mimalloc                             | 2 min 22 sec     | 1,449,501                       | 174,3 GB            | **19,8 GB** |
| rpmalloc                             | **2 min 15 sec** | 270,796                         | 45,9 GB             | 31,9 GB     |
| snmalloc                             | 2 min 19 sec     | 102,839                         | 47,0 GB             | 42,0 GB     |
|



> ! In D71786#2125487 <https://reviews.llvm.org/D71786#2125487>, @Meinersbur wrote:
>  That diverges from what we have done in other cases (ConvertUTF, gtest, gmock, google benchmark, isl, ...), adds to the effort to build LLVM and adds a dimension to the cmake configuration space.

I would imagine a downstream fork of LLVM could simply embed the allocator's checkout into the LLVM tree. Not ideal, but that leaves latitude to choose the allocator and/or one with a suitable licence.

> ! In D71786#2124330 <https://reviews.llvm.org/D71786#2124330>, @MaskRay wrote:
>  Finally, let me end with a hopefully constructive argument:) If the malloc interface is well defined on Windows, can -fintegrated-crt-alloc be generalized to take a value - path to the precompiled malloc implementation, so that we don't need to check in a specific implementation & users can easily use whatever implementations they want?

Replacing the allocator dynamically at startup could be possible, but that's a lot of trouble. If that's we want, I could check if we can load a DLL from a TLS callback <https://stackoverflow.com/a/36891752>, because we need the allocator very early, before the CRT entry point is called.

But since the allocator is more a LLVM-developer-facing feature, and not really a Clang/LLD-user feature, I would tend to prefer `cmake -DLLVM_INTEGRATED_CRT_ALLOC=D:/git/rpmalloc` and perhaps `-fno-integrated-crt-alloc` at runtime to disable. But if you disagree and/or have arguments for a dynamic loader, let me know.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D71786/new/

https://reviews.llvm.org/D71786

Files:
  llvm/CMakeLists.txt
  llvm/lib/Support/CMakeLists.txt
  llvm/tools/llvm-shlib/CMakeLists.txt
  llvm/tools/remarks-shlib/CMakeLists.txt
  llvm/unittests/Support/DynamicLibrary/CMakeLists.txt

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D71786.274989.patch
Type: text/x-patch
Size: 4338 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20200702/92f2f1bd/attachment.bin>