[clang-tools-extra] [llvm] [clangd][Support] Release rpmalloc caches when the background index goes idle (PR #242843)

Олександр Федоров via cfe-commits cfe-commits at lists.llvm.org
Sun Nov 15 07:13:13 PST 2026


https://github.com/namefreezers created https://github.com/llvm/llvm-project/pull/242843

With LLVM_ENABLE_RPMALLOC, which the Windows x64 release uses, freed memory
stays in per-thread and global span caches, and a block freed by a thread
other than the one that allocated it is left for the allocating thread to
reclaim. clangd's background-index workers allocate and free a lot while
indexing and then block until new work arrives, so most of that memory stays
committed. After indexing LLVM itself (about 7,200 TUs, 12 workers) clangd
sits at 14.8 GiB of commit, of which its $/memoryUsage accounts for 3.0 GiB.
With this change it settles at 5.2 GiB.

Support: implement rpmalloc_thread_collect(), which the vendored rpmalloc
1.4.5 declares but leaves empty. It adopts the calling heap's deferred frees
and unmaps the free spans that heap caches. Add rpmalloc_global_collect(),
which takes the free spans out of the global cache under its existing lock
and unmaps them after releasing it. Both keep master spans that still have
mapped subspans, because unmapping those releases nothing yet.
sys::Process::releaseHeapCaches() calls the two and does nothing unless LLVM
is built with LLVM_ENABLE_RPMALLOC. rpmalloc_global_collect() exists only in
LLVM's copy of rpmalloc, so an update of the vendored library has to port or
replace it.

clangd: BackgroundQueue::work() takes an OnWorkerIdle callback. After the
queue went idle and OnIdle returned, each worker that still finds the queue
empty runs it once. BackgroundIndex uses it to call releaseHeapCaches().

Pre-merge CI doesn't build with rpmalloc, so it runs only
ProcessTest.ReleaseHeapCaches and BackgroundQueueTest.WorkerIdle. The other
three tests (two measure the commit charge after a release, one checks block
contents while threads release concurrently) are compiled only on Windows with
LLVM_ENABLE_RPMALLOC, which the Windows release build sets before it runs
check-llvm. I ran them locally, also with LLVM and rpmalloc assertions,
together with all of ClangdTests, and checked that a build with an external
rpmalloc (-DLLVM_INTEGRATED_CRT_ALLOC=<checkout>) still links.

This covers the background-index workers only. The per-file AST and preamble
threads keep their caches while a file is open.

Fixes #242835

Assisted-by: Claude Fable 5.2

>From a028552f9f97b86d4ef6bc81a6d1608171f108c2 Mon Sep 17 00:00:00 2001
From: Oleksandr Fedorov <freezers at ukr.net>
Date: Thu, 12 Nov 2026 21:37:47 +0100
Subject: [PATCH] [clangd][Support] Release rpmalloc caches when the background
 index goes idle

With LLVM_ENABLE_RPMALLOC, which the Windows x64 release uses, freed memory
stays in per-thread and global span caches, and a block freed by a thread
other than the one that allocated it is left for the allocating thread to
reclaim. clangd's background-index workers allocate and free a lot while
indexing and then block until new work arrives, so most of that memory stays
committed. After indexing LLVM itself (about 7,200 TUs, 12 workers) clangd
sits at 14.8 GiB of commit, of which its $/memoryUsage accounts for 3.0 GiB.
With this change it settles at 5.2 GiB.

Support: implement rpmalloc_thread_collect(), which the vendored rpmalloc
1.4.5 declares but leaves empty. It adopts the calling heap's deferred frees
and unmaps the free spans that heap caches. Add rpmalloc_global_collect(),
which takes the free spans out of the global cache under its existing lock
and unmaps them after releasing it. Both keep master spans that still have
mapped subspans, because unmapping those releases nothing yet.
sys::Process::releaseHeapCaches() calls the two and does nothing unless LLVM
is built with LLVM_ENABLE_RPMALLOC. rpmalloc_global_collect() exists only in
LLVM's copy of rpmalloc, so an update of the vendored library has to port or
replace it.

clangd: BackgroundQueue::work() takes an OnWorkerIdle callback. After the
queue went idle and OnIdle returned, each worker that still finds the queue
empty runs it once. BackgroundIndex uses it to call releaseHeapCaches().

Assisted-by: Claude Fable 5.2
---
 clang-tools-extra/clangd/index/Background.cpp |  12 +-
 clang-tools-extra/clangd/index/Background.h   |   6 +-
 .../clangd/index/BackgroundQueue.cpp          |  22 ++-
 .../clangd/unittests/BackgroundIndexTests.cpp |  64 +++++++++
 clang-tools-extra/docs/ReleaseNotes.md        |   6 +
 llvm/include/llvm/Support/Process.h           |  16 +++
 llvm/lib/Support/CMakeLists.txt               |   5 +
 llvm/lib/Support/Unix/Process.inc             |   2 +
 llvm/lib/Support/Windows/Process.inc          |  12 ++
 llvm/lib/Support/rpmalloc/rpmalloc.c          | 107 +++++++++++++-
 llvm/lib/Support/rpmalloc/rpmalloc.h          |   6 +-
 llvm/unittests/Support/CMakeLists.txt         |   6 +
 llvm/unittests/Support/ProcessTest.cpp        | 131 ++++++++++++++++++
 13 files changed, 389 insertions(+), 6 deletions(-)

diff --git a/clang-tools-extra/clangd/index/Background.cpp b/clang-tools-extra/clangd/index/Background.cpp
index 17a8097394492e..1ca896e3a1bb34 100644
--- a/clang-tools-extra/clangd/index/Background.cpp
+++ b/clang-tools-extra/clangd/index/Background.cpp
@@ -39,6 +39,7 @@
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/Path.h"
+#include "llvm/Support/Process.h"
 #include "llvm/Support/Threading.h"
 #include "llvm/Support/xxhash.h"
 
@@ -88,6 +89,14 @@ bool shardIsStale(const LoadedShard &LS, llvm::vfs::FileSystem *FS) {
   return digest(Buf->get()->getBuffer()) != LS.Digest;
 }
 
+// Indexing allocates and frees a lot of memory. The heap allocator may retain
+// freed memory in per-thread caches (see
+// llvm::sys::Process::releaseHeapCaches), so each worker gives it back when
+// the queue goes idle.
+void releaseWorkerHeapCaches() {
+  llvm::sys::Process::releaseHeapCaches(/*IncludeShared=*/true);
+}
+
 } // namespace
 
 BackgroundIndex::BackgroundIndex(
@@ -111,7 +120,8 @@ BackgroundIndex::BackgroundIndex(
                         [this, Ctx(Context::current().clone())]() mutable {
                           clang::noteBottomOfStack();
                           WithContext BGContext(std::move(Ctx));
-                          Queue.work([&] { Rebuilder.idle(); });
+                          Queue.work(/*OnIdle=*/[&] { Rebuilder.idle(); },
+                                     /*OnWorkerIdle=*/releaseWorkerHeapCaches);
                         });
   }
 }
diff --git a/clang-tools-extra/clangd/index/Background.h b/clang-tools-extra/clangd/index/Background.h
index 448e911201575f..43ffba81ec68b6 100644
--- a/clang-tools-extra/clangd/index/Background.h
+++ b/clang-tools-extra/clangd/index/Background.h
@@ -103,7 +103,11 @@ class BackgroundQueue {
 
   // Process items on the queue until the queue is stopped.
   // If the queue becomes empty, OnIdle will be called (on one worker).
-  void work(std::function<void()> OnIdle = nullptr);
+  // After OnIdle returns, each worker that still finds the queue empty calls
+  // OnWorkerIdle once, e.g. to release per-thread resources. Other workers
+  // may already run new tasks at that time.
+  void work(std::function<void()> OnIdle = nullptr,
+            std::function<void()> OnWorkerIdle = nullptr);
 
   // Stop processing new tasks, allowing all work() calls to return soon.
   void stop();
diff --git a/clang-tools-extra/clangd/index/BackgroundQueue.cpp b/clang-tools-extra/clangd/index/BackgroundQueue.cpp
index 6023f48ccd3b3d..132fa036adbed1 100644
--- a/clang-tools-extra/clangd/index/BackgroundQueue.cpp
+++ b/clang-tools-extra/clangd/index/BackgroundQueue.cpp
@@ -19,17 +19,35 @@ void BackgroundQueue::preventThreadStarvationInTests() {
   PreventStarvation.store(true);
 }
 
-void BackgroundQueue::work(std::function<void()> OnIdle) {
+void BackgroundQueue::work(std::function<void()> OnIdle,
+                           std::function<void()> OnWorkerIdle) {
+  // Stat.LastIdle when this worker last ran OnWorkerIdle. Stat.LastIdle is set
+  // to Stat.Completed whenever the queue goes idle, so it is 0 until the first
+  // idle period and different for each one after that.
+  unsigned WorkerIdleAt = 0;
   while (true) {
     std::optional<Task> Task;
     {
       std::unique_lock<std::mutex> Lock(Mu);
-      CV.wait(Lock, [&] { return ShouldStop || !Queue.empty(); });
+      CV.wait(Lock, [&] {
+        // Stat.Active is still 1 while OnIdle runs.
+        return ShouldStop || !Queue.empty() ||
+               (OnWorkerIdle && Stat.Active == 0 &&
+                WorkerIdleAt != Stat.LastIdle);
+      });
       if (ShouldStop) {
         Queue.clear();
         CV.notify_all();
         return;
       }
+      if (Queue.empty()) {
+        // The queue went idle, and OnIdle finished, since this worker last ran
+        // OnWorkerIdle.
+        WorkerIdleAt = Stat.LastIdle;
+        Lock.unlock();
+        OnWorkerIdle();
+        continue;
+      }
       ++Stat.Active;
       std::pop_heap(Queue.begin(), Queue.end());
       Task = std::move(Queue.back());
diff --git a/clang-tools-extra/clangd/unittests/BackgroundIndexTests.cpp b/clang-tools-extra/clangd/unittests/BackgroundIndexTests.cpp
index 0eb4acf0469b79..f9a0f83105e324 100644
--- a/clang-tools-extra/clangd/unittests/BackgroundIndexTests.cpp
+++ b/clang-tools-extra/clangd/unittests/BackgroundIndexTests.cpp
@@ -14,7 +14,13 @@
 #include "llvm/Support/ScopedPrinter.h"
 #include "gmock/gmock.h"
 #include "gtest/gtest.h"
+#include <atomic>
+#include <chrono>
+#include <condition_variable>
 #include <deque>
+#include <map>
+#include <mutex>
+#include <thread>
 
 using ::testing::_;
 using ::testing::AllOf;
@@ -1018,6 +1024,64 @@ TEST(BackgroundQueueTest, Progress) {
   EXPECT_EQ(S.LastIdle, 2000u);
 }
 
+TEST(BackgroundQueueTest, WorkerIdle) {
+  // Each time the queue goes idle, OnIdle runs on one worker, and after it
+  // returns, OnWorkerIdle runs once on every worker.
+  constexpr unsigned Workers = 4, Rounds = 3;
+  BackgroundQueue Q;
+  std::mutex Mu;
+  std::condition_variable CV;
+  bool InOnIdle = false;
+  unsigned IdleCalls = 0, WorkerIdleCalls = 0;
+  std::map<std::thread::id, unsigned> WorkerIdleCallsByThread;
+  std::atomic<unsigned> Ran(0);
+  auto OnIdle = [&] {
+    std::unique_lock<std::mutex> Lock(Mu);
+    InOnIdle = true;
+    Lock.unlock();
+    // Wake the waiting workers, as a spurious wakeup would, and give them
+    // time to run OnWorkerIdle too early.
+    Q.append({});
+    Lock.lock();
+    unsigned Calls = WorkerIdleCalls;
+    CV.wait_for(Lock, std::chrono::milliseconds(100),
+                [&] { return WorkerIdleCalls != Calls; });
+    InOnIdle = false;
+    ++IdleCalls;
+  };
+  auto OnWorkerIdle = [&] {
+    std::lock_guard<std::mutex> Lock(Mu);
+    // OnIdle returned for this idle period, and the next round hasn't
+    // started.
+    EXPECT_FALSE(InOnIdle);
+    EXPECT_EQ(IdleCalls, WorkerIdleCalls / Workers + 1);
+    ++WorkerIdleCalls;
+    ++WorkerIdleCallsByThread[std::this_thread::get_id()];
+    CV.notify_all();
+  };
+  AsyncTaskRunner ThreadPool;
+  for (unsigned I = 0; I < Workers; ++I)
+    ThreadPool.runAsync("worker", [&] { Q.work(OnIdle, OnWorkerIdle); });
+
+  for (unsigned Round = 1; Round <= Rounds; ++Round) {
+    Q.append(std::vector<BackgroundQueue::Task>(
+        10, BackgroundQueue::Task([&] { ++Ran; })));
+    EXPECT_TRUE(Q.blockUntilIdleForTest(/*TimeoutSeconds=*/60));
+    std::unique_lock<std::mutex> Lock(Mu);
+    EXPECT_TRUE(CV.wait_for(Lock, std::chrono::seconds(60), [&] {
+      return WorkerIdleCalls >= Round * Workers;
+    }));
+  }
+  Q.stop();
+  ThreadPool.wait();
+  EXPECT_EQ(Ran.load(), 10 * Rounds);
+  EXPECT_EQ(IdleCalls, Rounds);
+  EXPECT_EQ(WorkerIdleCalls, Rounds * Workers);
+  EXPECT_EQ(WorkerIdleCallsByThread.size(), Workers);
+  for (const auto &Entry : WorkerIdleCallsByThread)
+    EXPECT_EQ(Entry.second, Rounds);
+}
+
 TEST(BackgroundIndex, Profile) {
   MockFS FS;
   MockCompilationDatabase CDB;
diff --git a/clang-tools-extra/docs/ReleaseNotes.md b/clang-tools-extra/docs/ReleaseNotes.md
index b6774370511feb..301c66a86daf0f 100644
--- a/clang-tools-extra/docs/ReleaseNotes.md
+++ b/clang-tools-extra/docs/ReleaseNotes.md
@@ -124,6 +124,12 @@ infrastructure are described first, followed by tool-specific sections.
 - Added support for loading dynamic plugins via the `-load` command-line
   option.
 
+- On Windows, clangd built with rpmalloc (`LLVM_ENABLE_RPMALLOC`, as the
+  official release is) now returns memory to the operating system when
+  background indexing finishes. Previously most of the memory used for
+  indexing stayed committed afterwards.
+  ([#242835](https://github.com/llvm/llvm-project/issues/242835))
+
 ### Improvements to clang-doc
 
 ### Improvements to clang-query
diff --git a/llvm/include/llvm/Support/Process.h b/llvm/include/llvm/Support/Process.h
index f644fe7336f74c..a37500a0ce00a0 100644
--- a/llvm/include/llvm/Support/Process.h
+++ b/llvm/include/llvm/Support/Process.h
@@ -75,6 +75,22 @@ class Process {
   /// allocated space.
   LLVM_ABI static size_t GetMallocUsage();
 
+  /// Release free memory that the heap allocator keeps cached, giving it back
+  /// to the operating system. This is best effort: the allocator may still
+  /// hold some free memory afterwards.
+  ///
+  /// This only has an effect when LLVM is built with LLVM_ENABLE_RPMALLOC.
+  /// rpmalloc caches freed memory per thread, and leaves a block freed by a
+  /// thread other than the one that allocated it for the allocating thread to
+  /// reclaim, e.g. when it allocates again. Only the calling thread's cache is
+  /// released, so a long-running program should call this on every thread
+  /// that goes idle after allocating a lot.
+  ///
+  /// \param IncludeShared Also release the cache shared by all threads.
+  /// Threads that are allocating at the same time may then have to get memory
+  /// from the operating system again.
+  LLVM_ABI static void releaseHeapCaches(bool IncludeShared);
+
   /// This static function will set \p user_time to the amount of CPU time
   /// spent in user (non-kernel) mode and \p sys_time to the amount of CPU
   /// time spent in system (kernel) mode.  If the operating system does not
diff --git a/llvm/lib/Support/CMakeLists.txt b/llvm/lib/Support/CMakeLists.txt
index 27853d858d9d15..3a06dd548b5ae9 100644
--- a/llvm/lib/Support/CMakeLists.txt
+++ b/llvm/lib/Support/CMakeLists.txt
@@ -134,6 +134,11 @@ if(LLVM_INTEGRATED_CRT_ALLOC)
   if((LLVM_INTEGRATED_CRT_ALLOC MATCHES "rpmalloc$") OR LLVM_ENABLE_RPMALLOC)
     add_compile_definitions(ENABLE_OVERRIDE ENABLE_PRELOAD)
     add_compile_definitions(LLVM_INTEGRATED_CRT_ALLOC="rpmalloc")
+    if(LLVM_ENABLE_RPMALLOC)
+      # sys::Process::releaseHeapCaches() needs functions that an external
+      # rpmalloc may not have.
+      add_compile_definitions(LLVM_ENABLE_RPMALLOC)
+    endif()
     set(ALLOCATOR_FILES "${LLVM_INTEGRATED_CRT_ALLOC}/rpmalloc/rpmalloc.c")
     list(APPEND delayload_flags "${WL}-INCLUDE:malloc")
   elseif(LLVM_INTEGRATED_CRT_ALLOC MATCHES "snmalloc$")
diff --git a/llvm/lib/Support/Unix/Process.inc b/llvm/lib/Support/Unix/Process.inc
index 1c0a29787b9f9b..25d9f3b1678e04 100644
--- a/llvm/lib/Support/Unix/Process.inc
+++ b/llvm/lib/Support/Unix/Process.inc
@@ -121,6 +121,8 @@ size_t Process::GetMallocUsage() {
 #endif
 }
 
+void Process::releaseHeapCaches(bool IncludeShared) {}
+
 void Process::GetTimeUsage(TimePoint<> &elapsed,
                            std::chrono::nanoseconds &user_time,
                            std::chrono::nanoseconds &sys_time) {
diff --git a/llvm/lib/Support/Windows/Process.inc b/llvm/lib/Support/Windows/Process.inc
index cab08326273047..60675abadb4127 100644
--- a/llvm/lib/Support/Windows/Process.inc
+++ b/llvm/lib/Support/Windows/Process.inc
@@ -27,6 +27,10 @@
 #include <psapi.h>
 #include <shellapi.h>
 
+#ifdef LLVM_ENABLE_RPMALLOC
+#include "rpmalloc/rpmalloc.h"
+#endif
+
 #if !defined(__MINGW32__)
 #pragma comment(lib, "psapi.lib")
 #if !LLVM_WINDOWS_RESTRICTED_APIS
@@ -77,6 +81,14 @@ size_t Process::GetMallocUsage() {
   return size;
 }
 
+void Process::releaseHeapCaches(bool IncludeShared) {
+#ifdef LLVM_ENABLE_RPMALLOC
+  rpmalloc_thread_collect();
+  if (IncludeShared)
+    rpmalloc_global_collect();
+#endif
+}
+
 void Process::GetTimeUsage(TimePoint<> &elapsed,
                            std::chrono::nanoseconds &user_time,
                            std::chrono::nanoseconds &sys_time) {
diff --git a/llvm/lib/Support/rpmalloc/rpmalloc.c b/llvm/lib/Support/rpmalloc/rpmalloc.c
index a0f99f3e0184ec..60e8a4c4b3d506 100644
--- a/llvm/lib/Support/rpmalloc/rpmalloc.c
+++ b/llvm/lib/Support/rpmalloc/rpmalloc.c
@@ -3532,7 +3532,112 @@ extern inline size_t rpmalloc_usable_size(void *ptr) {
   return (ptr ? _rpmalloc_usable_size(ptr) : 0);
 }
 
-extern inline void rpmalloc_thread_collect(void) {}
+//! Check if the span is a master span that still has mapped subspans.
+//  Unmapping such a span releases nothing, its pages stay committed until
+//  the last subspan is unmapped
+static int _rpmalloc_span_is_master_with_subspans(span_t *span) {
+  return (span->flags & SPAN_FLAG_MASTER) &&
+         (atomic_load32(&span->remaining_spans) > (int32_t)span->span_count);
+}
+
+void rpmalloc_thread_collect(void) {
+  heap_t *heap = get_thread_heap_raw();
+  if (!heap || heap->finalize)
+    return;
+  // Complete the frees other threads deferred to this heap. Without this,
+  // spans (including huge blocks) freed by other threads stay mapped until
+  // this thread allocates again
+  _rpmalloc_heap_cache_adopt_deferred(heap, 0);
+  // Spans kept for reuse by a size class go to the span cache below
+  for (size_t iclass = 0; iclass < SIZE_CLASS_COUNT; ++iclass) {
+    span_t *span = heap->size_class[iclass].cache;
+    if (!span)
+      continue;
+    heap->size_class[iclass].cache = 0;
+    _rpmalloc_heap_cache_insert(heap, span);
+  }
+  // Spans reserved for this heap go to the span cache too, as when a new
+  // mapping replaces them. The reserve can be a freed master span that still
+  // has mapped subspans, which the loops below keep. A reserve larger than the
+  // largest span cache class (possible with a span map count above 64) stays
+  // reserved
+  if (heap->spans_reserved && heap->spans_reserved <= LARGE_CLASS_COUNT)
+    _rpmalloc_heap_cache_insert(heap,
+                                _rpmalloc_span_map(heap, heap->spans_reserved));
+#if ENABLE_THREAD_CACHE
+  // Two passes, because a master span is kept while it has mapped subspans,
+  // and they can come after it in the caches
+  for (int pass = 0; pass < 2; ++pass) {
+    for (size_t iclass = 0; iclass < LARGE_CLASS_COUNT; ++iclass) {
+      span_cache_t *span_cache;
+      if (!iclass)
+        span_cache = &heap->span_cache;
+      else
+        span_cache = (span_cache_t *)(heap->span_large_cache + (iclass - 1));
+      size_t keep_count = 0;
+      for (size_t ispan = 0; ispan < span_cache->count; ++ispan) {
+        span_t *span = span_cache->span[ispan];
+        if (_rpmalloc_span_is_master_with_subspans(span))
+          span_cache->span[keep_count++] = span;
+        else
+          _rpmalloc_span_unmap(span);
+      }
+      span_cache->count = keep_count;
+    }
+  }
+#endif
+}
+
+#if ENABLE_GLOBAL_CACHE
+//! Unmap the free spans of a global cache, except masters with mapped subspans
+static void _rpmalloc_global_cache_collect(global_cache_t *cache) {
+  // The spans to unmap are taken out of the cache under its lock and unmapped
+  // after the lock is released
+  span_t *unmap = 0;
+  while (!atomic_cas32_acquire(&cache->lock, 1, 0))
+    _rpmalloc_spin();
+  size_t keep_count = 0;
+  for (size_t ispan = 0; ispan < cache->count; ++ispan) {
+    span_t *span = cache->span[ispan];
+    if (_rpmalloc_span_is_master_with_subspans(span)) {
+      cache->span[keep_count++] = span;
+    } else {
+      span->next = unmap;
+      unmap = span;
+    }
+  }
+  cache->count = (uint32_t)keep_count;
+  span_t **link = &cache->overflow;
+  while (*link) {
+    span_t *span = *link;
+    if (_rpmalloc_span_is_master_with_subspans(span)) {
+      link = &span->next;
+    } else {
+      *link = span->next;
+      span->next = unmap;
+      unmap = span;
+    }
+  }
+  atomic_store32_release(&cache->lock, 0);
+  while (unmap) {
+    // Read the link before unmapping, the span header may be decommitted
+    span_t *span = unmap;
+    unmap = span->next;
+    _rpmalloc_span_unmap(span);
+  }
+}
+#endif
+
+void rpmalloc_global_collect(void) {
+#if ENABLE_GLOBAL_CACHE
+  // Two passes, because a master span is kept while it has mapped subspans,
+  // and they can come after it in the caches
+  for (int pass = 0; pass < 2; ++pass) {
+    for (size_t iclass = 0; iclass < LARGE_CLASS_COUNT; ++iclass)
+      _rpmalloc_global_cache_collect(&_memory_span_cache[iclass]);
+  }
+#endif
+}
 
 void rpmalloc_thread_statistics(rpmalloc_thread_statistics_t *stats) {
   memset(stats, 0, sizeof(rpmalloc_thread_statistics_t));
diff --git a/llvm/lib/Support/rpmalloc/rpmalloc.h b/llvm/lib/Support/rpmalloc/rpmalloc.h
index 5b7fe1ff4286ba..3ace20975e8914 100644
--- a/llvm/lib/Support/rpmalloc/rpmalloc.h
+++ b/llvm/lib/Support/rpmalloc/rpmalloc.h
@@ -243,9 +243,13 @@ RPMALLOC_EXPORT void rpmalloc_thread_initialize(void);
 //! Finalize allocator for calling thread
 RPMALLOC_EXPORT void rpmalloc_thread_finalize(int release_caches);
 
-//! Perform deferred deallocations pending for the calling thread heap
+//! Perform deferred deallocations pending for the calling thread heap, and
+//  unmap the free spans cached by the calling thread heap
 RPMALLOC_EXPORT void rpmalloc_thread_collect(void);
 
+//! Unmap the free spans held in the global cache
+RPMALLOC_EXPORT void rpmalloc_global_collect(void);
+
 //! Query if allocator is initialized for calling thread
 RPMALLOC_EXPORT int rpmalloc_is_thread_initialized(void);
 
diff --git a/llvm/unittests/Support/CMakeLists.txt b/llvm/unittests/Support/CMakeLists.txt
index f9732ff4e0e4bb..591826f97eb170 100644
--- a/llvm/unittests/Support/CMakeLists.txt
+++ b/llvm/unittests/Support/CMakeLists.txt
@@ -156,6 +156,12 @@ endif()
 # ManagedStatic.cpp uses <pthread>.
 target_link_libraries(SupportTests PRIVATE LLVMTestingSupport ${LLVM_PTHREAD_LIB})
 
+# ProcessTest.cpp checks that Process::releaseHeapCaches() returns memory to the
+# OS, which it only does with the in-tree rpmalloc.
+if(LLVM_ENABLE_RPMALLOC)
+  target_compile_definitions(SupportTests PRIVATE LLVM_ENABLE_RPMALLOC)
+endif()
+
 if(NOT LLVM_INTEGRATED_CRT_ALLOC)
   # The test doesn't pass when using a custom allocator, PR47881.
   add_subdirectory(DynamicLibrary)
diff --git a/llvm/unittests/Support/ProcessTest.cpp b/llvm/unittests/Support/ProcessTest.cpp
index d508a67c57cddc..1759517fe5ff6c 100644
--- a/llvm/unittests/Support/ProcessTest.cpp
+++ b/llvm/unittests/Support/ProcessTest.cpp
@@ -7,14 +7,24 @@
 //===----------------------------------------------------------------------===//
 
 #include "llvm/Support/Process.h"
+#include "llvm/Config/llvm-config.h" // for LLVM_ENABLE_THREADS
 #include "llvm/Support/Error.h"
 #include "llvm/TargetParser/Host.h"
 #include "llvm/TargetParser/Triple.h"
 #include "gtest/gtest.h"
+#include <atomic>
+#include <cstdlib>
+#include <cstring>
+#include <mutex>
 #include <optional>
+#include <random>
+#include <thread>
+#include <vector>
 
 #ifdef _WIN32
 #include <windows.h>
+// psapi.h needs windows.h.
+#include <psapi.h>
 #endif
 
 namespace {
@@ -110,4 +120,125 @@ TEST_F(PageSizeTest, PageSize) {
   ASSERT_TRUE(pageSizeAsExpected(*Result));
 }
 
+#if LLVM_ENABLE_THREADS
+TEST(ProcessTest, ReleaseHeapCaches) {
+  // Callable on any thread, whether or not the allocator caches anything.
+  Process::releaseHeapCaches(/*IncludeShared=*/false);
+  Process::releaseHeapCaches(/*IncludeShared=*/true);
+  std::thread([] {
+    Process::releaseHeapCaches(/*IncludeShared=*/true);
+  }).join();
+  void *P = std::malloc(100);
+  EXPECT_NE(P, nullptr);
+  std::free(P);
+}
+
+#if defined(_WIN32) && defined(LLVM_ENABLE_RPMALLOC)
+// Threads allocate blocks of all size classes, free each other's blocks and
+// release heap caches concurrently. Checks that no block is corrupted.
+TEST(ProcessTest, ReleaseHeapCachesConcurrent) {
+  constexpr unsigned Threads = 8, Iterations = 2000;
+  struct Block {
+    unsigned char *Data;
+    size_t Size;
+    unsigned char Fill;
+  };
+  std::mutex Mu;
+  std::vector<Block> Shared;
+  std::atomic<unsigned> Corrupted(0);
+  auto Check = [&](const Block &B) {
+    if (B.Data[0] != B.Fill || B.Data[B.Size / 2] != B.Fill ||
+        B.Data[B.Size - 1] != B.Fill)
+      ++Corrupted;
+    std::free(B.Data);
+  };
+  std::vector<std::thread> Workers;
+  for (unsigned T = 0; T < Threads; ++T)
+    Workers.emplace_back([&, T] {
+      std::mt19937 Rng(T);
+      for (unsigned I = 0; I < Iterations; ++I) {
+        // Mostly small blocks, some medium, large and huge ones.
+        size_t MaxSize = 256;
+        if (I % 50 == 0)
+          MaxSize = 8 << 20;
+        else if (I % 10 == 0)
+          MaxSize = 2 << 20;
+        else if (I % 4 == 0)
+          MaxSize = 32 << 10;
+        size_t Size = 1 + Rng() % MaxSize;
+        Block B{static_cast<unsigned char *>(std::malloc(Size)), Size,
+                static_cast<unsigned char>(Rng())};
+        ASSERT_NE(B.Data, nullptr);
+        std::memset(B.Data, B.Fill, Size);
+        std::optional<Block> Other;
+        {
+          std::lock_guard<std::mutex> Lock(Mu);
+          Shared.push_back(B);
+          if (Shared.size() > 64) {
+            size_t Pick = Rng() % Shared.size();
+            Other = Shared[Pick];
+            Shared[Pick] = Shared.back();
+            Shared.pop_back();
+          }
+        }
+        if (Other)
+          Check(*Other);
+        if (I % 256 == 0)
+          Process::releaseHeapCaches(/*IncludeShared=*/(I / 256) % 2 == 1);
+      }
+      Process::releaseHeapCaches(/*IncludeShared=*/true);
+    });
+  for (std::thread &W : Workers)
+    W.join();
+  for (const Block &B : Shared)
+    Check(B);
+  EXPECT_EQ(Corrupted.load(), 0u);
+}
+
+// The commit charge of the process. It changes when memory is committed or
+// released, not when pages are touched.
+int64_t committedBytes() {
+  PROCESS_MEMORY_COUNTERS_EX C = {};
+  EXPECT_TRUE(::GetProcessMemoryInfo(
+      ::GetCurrentProcess(), reinterpret_cast<PROCESS_MEMORY_COUNTERS *>(&C),
+      sizeof(C)));
+  return static_cast<int64_t>(C.PrivateUsage);
+}
+
+// Blocks freed by the thread that allocated them stay in rpmalloc's span
+// caches. One call has to return practically all of them.
+TEST(ProcessTest, ReleaseHeapCachesFreedBlocks) {
+  constexpr size_t Size = 1 << 20, Count = 256;
+  // Start with empty caches, so that the blocks need new memory.
+  Process::releaseHeapCaches(/*IncludeShared=*/true);
+  int64_t Before = committedBytes();
+  std::vector<void *> Blocks;
+  for (size_t I = 0; I < Count; ++I)
+    Blocks.push_back(std::malloc(Size));
+  // rpmalloc commits the memory it maps for the blocks.
+  EXPECT_GE(committedBytes() - Before, static_cast<int64_t>(Size * Count / 2));
+  for (void *B : Blocks)
+    std::free(B);
+  Process::releaseHeapCaches(/*IncludeShared=*/true);
+  EXPECT_LT(committedBytes() - Before, static_cast<int64_t>(Size * Count / 16));
+}
+
+// rpmalloc only completes a free of a block that another thread allocated on
+// the allocating thread, so the block stays committed until that thread
+// allocates again or calls releaseHeapCaches.
+TEST(ProcessTest, ReleaseHeapCachesCrossThreadFree) {
+  constexpr size_t Size = 256 << 20;
+  int64_t AfterFree = 0, AfterRelease = 0;
+  std::thread([&] {
+    void *Block = std::malloc(Size);
+    std::thread([Block] { std::free(Block); }).join();
+    AfterFree = committedBytes();
+    Process::releaseHeapCaches(/*IncludeShared=*/false);
+    AfterRelease = committedBytes();
+  }).join();
+  EXPECT_GE(AfterFree - AfterRelease, static_cast<int64_t>(Size / 2));
+}
+#endif // _WIN32 && LLVM_ENABLE_RPMALLOC
+#endif // LLVM_ENABLE_THREADS
+
 } // end anonymous namespace



More information about the cfe-commits mailing list