[clang-tools-extra] dad804a - [clangd] Improve clangd-indexer performance

[Aleksandr Platonov via cfe-commits]

Author: Aleksandr Platonov
Date: 2020-11-11T14:38:06+03:00
New Revision: dad804a193edf092322d80bb404fabb2f6f2c888

URL: https://github.com/llvm/llvm-project/commit/dad804a193edf092322d80bb404fabb2f6f2c888
DIFF: https://github.com/llvm/llvm-project/commit/dad804a193edf092322d80bb404fabb2f6f2c888.diff

LOG: [clangd] Improve clangd-indexer performance

This is a try to improve clangd-indexer tool performance:
- avoid processing already processed files.
- use different mutexes for different entities (e.g. do not block insertion of references while symbols are inserted)

Results for LLVM project indexing:
- before: ~30 minutes
- after: ~10 minutes

Reviewed By: kadircet

Differential Revision: https://reviews.llvm.org/D91051

Added: 
    

Modified: 
    clang-tools-extra/clangd/indexer/IndexerMain.cpp

Removed: 
    


################################################################################
diff  --git a/clang-tools-extra/clangd/indexer/IndexerMain.cpp b/clang-tools-extra/clangd/indexer/IndexerMain.cpp
index 46224238c3fc..fd8404be677a 100644
--- a/clang-tools-extra/clangd/indexer/IndexerMain.cpp
+++ b/clang-tools-extra/clangd/indexer/IndexerMain.cpp
@@ -43,6 +43,16 @@ class IndexActionFactory : public tooling::FrontendActionFactory {
   std::unique_ptr<FrontendAction> create() override {
     SymbolCollector::Options Opts;
     Opts.CountReferences = true;
+    Opts.FileFilter = [&](const SourceManager &SM, FileID FID) {
+      const auto *F = SM.getFileEntryForID(FID);
+      if (!F)
+        return false; // Skip invalid files.
+      auto AbsPath = getCanonicalPath(F, SM);
+      if (!AbsPath)
+        return false; // Skip files without absolute path.
+      std::lock_guard<std::mutex> Lock(FilesMu);
+      return Files.insert(*AbsPath).second; // Skip already processed files.
+    };
     return createStaticIndexingAction(
         Opts,
         [&](SymbolSlab S) {
@@ -56,7 +66,7 @@ class IndexActionFactory : public tooling::FrontendActionFactory {
           }
         },
         [&](RefSlab S) {
-          std::lock_guard<std::mutex> Lock(SymbolsMu);
+          std::lock_guard<std::mutex> Lock(RefsMu);
           for (const auto &Sym : S) {
             // Deduplication happens during insertion.
             for (const auto &Ref : Sym.second)
@@ -64,7 +74,7 @@ class IndexActionFactory : public tooling::FrontendActionFactory {
           }
         },
         [&](RelationSlab S) {
-          std::lock_guard<std::mutex> Lock(SymbolsMu);
+          std::lock_guard<std::mutex> Lock(RelsMu);
           for (const auto &R : S) {
             Relations.insert(R);
           }
@@ -82,9 +92,13 @@ class IndexActionFactory : public tooling::FrontendActionFactory {
 
 private:
   IndexFileIn &Result;
+  std::mutex FilesMu;
+  llvm::StringSet<> Files;
   std::mutex SymbolsMu;
   SymbolSlab::Builder Symbols;
+  std::mutex RefsMu;
   RefSlab::Builder Refs;
+  std::mutex RelsMu;
   RelationSlab::Builder Relations;
 };