[PATCH] D93452: [clangd] Trim memory after buildINdex

[Quentin Chateau via Phabricator via cfe-commits]

qchateau created this revision.
qchateau added a reviewer: sammccall.
Herald added subscribers: usaxena95, kadircet, arphaman.
qchateau requested review of this revision.
Herald added subscribers: cfe-commits, MaskRay, ilya-biryukov.
Herald added a project: clang.

This diff addresses the issue of the ever increasing memory usage of clangd. The key to understand what happens is to use `malloc_stats()`: malloc arenas keep getting bigger, although the actual memory used does not. It seems some operations while bulding the indices (both dynamic and background) create this problem. Specifically, 'FileSymbols::update' and 'FileSymbols::buildIndex' seem especially affected.

This diff adds a call to `malloc_trim(0)` at the end of `FileSymbols::buildIndex`.

For reference:
https://github.com/clangd/clangd/issues/251
https://github.com/clangd/clangd/issues/115

----------------------------

I'm not sure how much I'm allowed to use GNU extensions but this diff has been a game changer for me and my 8GB RAM laptop.

In any case, I think I've properly diagnosed to issue. Once trimmed, the actual memory usage of the process is approximately what clangd reports with the "memoryUsage" feature.

The solution is either what I suggest here (use `malloc_trim`) or rework a ton of code, change containers and allocators and hope the problem won't happen again in the future as we add features.


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D93452

Files:
  clang-tools-extra/clangd/index/FileIndex.cpp


Index: clang-tools-extra/clangd/index/FileIndex.cpp
===================================================================
--- clang-tools-extra/clangd/index/FileIndex.cpp
+++ clang-tools-extra/clangd/index/FileIndex.cpp
@@ -32,6 +32,7 @@
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/ScopeExit.h"
 #include "llvm/ADT/StringMap.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/Error.h"
@@ -40,11 +41,17 @@
 #include <tuple>
 #include <utility>
 #include <vector>
+#include <malloc.h>
 
 namespace clang {
 namespace clangd {
 namespace {
 
+void trimMemory() {
+  if (malloc_trim(0))
+    vlog("Trimmed memory");
+}
+
 SlabTuple indexSymbols(ASTContext &AST, std::shared_ptr<Preprocessor> PP,
                        llvm::ArrayRef<Decl *> DeclsToIndex,
                        const MainFileMacros *MacroRefsToIndex,
@@ -263,6 +270,10 @@
 std::unique_ptr<SymbolIndex>
 FileSymbols::buildIndex(IndexType Type, DuplicateHandling DuplicateHandle,
                         size_t *Version) {
+  // Building the index often leaves a lof of freed but unreleased memory
+  // (at the OS level): manually trim the memory after the index is built
+  auto _ = llvm::make_scope_exit(trimMemory);
+
   std::vector<std::shared_ptr<SymbolSlab>> SymbolSlabs;
   std::vector<std::shared_ptr<RefSlab>> RefSlabs;
   std::vector<std::shared_ptr<RelationSlab>> RelationSlabs;


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D93452.312449.patch
Type: text/x-patch
Size: 1436 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/cfe-commits/attachments/20201217/e2d6220a/attachment.bin>