[PATCH] D71786: RFC: [Support] On Windows, add optional support for rpmalloc

[Alexandre Ganea via Phabricator via llvm-commits]

aganea updated this revision to Diff 236639.
aganea marked 5 inline comments as done.
aganea added a comment.

- Changes as suggested by @rnk
- Fixed linking `remarks-shlib` as reported by @russell.gallop
- Reverted back to default rpmalloc settings (no more unlimited mode), ie. `ENABLE_UNLIMITED_CACHE` is 0 now.
- Support rpmalloc's `ENABLE_STATISTICS` - when enabled, allocation statistics will be printed on process exit.

---

In D71786#1807883 <https://reviews.llvm.org/D71786#1807883>, @maniccoder wrote:

> It does seem that @aganea has tried it on more service facing hardware, perhaps he can give additional input.


All I can say is that we're using it in production for the past year (on editor/production builds) and we're thrilled, the gains are absolutely massive. All multi-threaded workloads are largely improved. Some figures from early last year:
F11182646: ac_rpmalloc.png <https://reviews.llvm.org/F11182646>

But there's an extra memory cost. When building AC Odyssey's Greece world data, it requires 220 GB of RAM instead of 180 with dlmalloc.

In D71786#1802970 <https://reviews.llvm.org/D71786#1802970>, @russell.gallop wrote:

> 2). This can significantly increase memory usage. Part of this can be attributed to the increased parallelism but I don't think that entirely explains it. I think that rpmalloc is using more memory.


At first, I configured rpmalloc in the "unlimited" mode because I wanted maximum performance. I now reverted to default rpmalloc flags, and the memory usage has decreased (albeit still higher than the Windows Heap), with no impact on link time (meaning that timings are still the same as in the Summary). Below are "Max" values when linking `clang.exe` with ThinLTO:

F11182735: rpmalloc_vs_crt_mem.png <https://reviews.llvm.org/F11182735>


CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D71786/new/

https://reviews.llvm.org/D71786

Files:
  llvm/CMakeLists.txt
  llvm/include/llvm/Config/config.h.cmake
  llvm/lib/Support/CMakeLists.txt
  llvm/lib/Support/Windows/Memory.inc
  llvm/lib/Support/rpmalloc/LICENSE
  llvm/lib/Support/rpmalloc/malloc.c
  llvm/lib/Support/rpmalloc/rpmalloc.c
  llvm/lib/Support/rpmalloc/rpmalloc.h
  llvm/tools/remarks-shlib/CMakeLists.txt
  llvm/unittests/Support/DynamicLibrary/CMakeLists.txt

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D71786.236639.patch
Type: text/x-patch
Size: 126577 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20200107/d53fefa1/attachment.bin>