[clang-tools-extra] [clangd] [Modules] Support Reusable Modules Builder (PR #106683)

[kadir çetinkaya via cfe-commits]

================
@@ -316,36 +287,169 @@ llvm::Error buildModuleFile(llvm::StringRef ModuleName,
   if (Clang->getDiagnostics().hasErrorOccurred())
     return llvm::createStringError("Compilation failed");
 
-  BuiltModuleFiles.addModuleFile(ModuleName, Inputs.CompileCommand.Output);
-  return llvm::Error::success();
+  return ModuleFile{ModuleName, Inputs.CompileCommand.Output};
+}
+
+bool ReusablePrerequisiteModules::canReuse(
+    const CompilerInvocation &CI,
+    llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem> VFS) const {
+  if (RequiredModules.empty())
+    return true;
+
+  SmallVector<StringRef> BMIPaths;
+  for (auto &MF : RequiredModules)
+    BMIPaths.push_back(MF->ModuleFilePath);
+  return IsModuleFilesUpToDate(BMIPaths, *this, VFS);
 }
 } // namespace
 
+class ModulesBuilder::ModuleFileCache {
+public:
+  ModuleFileCache(const GlobalCompilationDatabase &CDB) : CDB(CDB) {}
+
+  llvm::Error
+  getOrBuildModuleFile(StringRef ModuleName, const ThreadsafeFS &TFS,
+                       ProjectModules &MDB,
+                       ReusablePrerequisiteModules &RequiredModules);
+  const GlobalCompilationDatabase &getCDB() const { return CDB; }
+
+private:
+  std::shared_ptr<ModuleFile>
+  getValidModuleFile(StringRef ModuleName, ProjectModules &MDB,
+                     const ThreadsafeFS &TFS,
+                     PrerequisiteModules &BuiltModuleFiles);
+
+  /// This should only be called by getValidModuleFile. This is unlocked version
+  /// of getValidModuleFile. The function is extracted to avoid dead locks when
+  /// recursing.
+  std::shared_ptr<ModuleFile>
+  isValidModuleFileUnlocked(StringRef ModuleName, ProjectModules &MDB,
+                            const ThreadsafeFS &TFS,
+                            PrerequisiteModules &BuiltModuleFiles);
+
+  const GlobalCompilationDatabase &CDB;
+
+  llvm::StringMap<std::shared_ptr<ModuleFile>> ModuleFiles;
----------------
kadircet wrote:

> The cache should be an owner. Otherwise, if we close all tabs and we open a tab again, we may have to build the same module file that could be in the cache.

Yes, and I think that's WAI. That's what clangd has today with preambles, if you close a file and re-open it, you'll pay for the warm up cost.

This ensures we can keep a long editing session going on. We ensure we're clearing up the resources as people close their files. Doing it differently here will imply clangd can no longer be used in workflows that open tons of files while navigating the codebase. As nothing will clear up the dependent modules built in the meanwhile.

We're still going to provide better performance here, compared to preambles. As we'll re-use any "core" modules that we keep in the cache.

https://github.com/llvm/llvm-project/pull/106683