<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Looks good now. Thanks Kadir!<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Jul 19, 2019, at 3:18 AM, Kadir Çetinkaya <<a href="mailto:kadircet@google.com" class="">kadircet@google.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Hi Azhar, D64980 should fix the problem. I am reverting your revert while adding the fix in r366559.</div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jul 19, 2019 at 11:29 AM Azhar Mohammed <<a href="mailto:azhar@apple.com" class="">azhar@apple.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class="">Reverted in r366551. <span class=""><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div class=""> Revert r366458, r366467 and r366468</div><div class=""><br class=""></div><div class=""> r366458 is causing test failures. r366467 and r366468 had to be reverted as</div><div class=""> they were casuing conflict while reverting r366458.</div><div class=""><br class=""></div><div class=""> r366468 [clangd] Remove dead code from BackgroundIndex</div><div class=""> r366467 [clangd] BackgroundIndex stores shards to the closest project</div><div class=""> r366458 [clangd] Refactor background-index shard loading</div></div><div class=""><br class=""></div></span><span class=""><div class=""><div class=""><div class=""><blockquote type="cite" class=""><div class="">On Jul 18, 2019, at 6:21 PM, Azhar Mohammed <<a href="mailto:azhar@apple.com" target="_blank" class="">azhar@apple.com</a>> wrote:</div><br class="gmail-m_1817645778349330322Apple-interchange-newline"><div class=""><div style="overflow-wrap: break-word;" class="">Hi Kadir<div class=""><br class=""></div><div class="">This change is causing test failures, can you please look into it. Refer to <a href="http://green.lab.llvm.org/green/job/clang-stage1-configure-RA/58104/testReport/" target="_blank" class="">http://green.lab.llvm.org/green/job/clang-stage1-configure-RA/58104/testReport/</a>. </div><div class=""><br class=""></div><div class=""><pre style="box-sizing:border-box;white-space:pre-wrap;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:15px" class="">Assertion failed: (TUsIt != FileToTU.end() && "No TU registered for the shard"), function takeResult, file /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/llvm/tools/clang/tools/extra/clangd/index/BackgroundIndexLoader.cpp, line 131.</pre><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><pre style="box-sizing:border-box;white-space:pre-wrap;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:15px" class="gmail-m_1817645778349330322console-output">Failing Tests (10):
Clangd :: did-change-configuration-params.test
Clangd Unit Tests :: ./ClangdTests/BackgroundIndexTest.CmdLineHash
Clangd Unit Tests :: ./ClangdTests/BackgroundIndexTest.DirectIncludesTest
Clangd Unit Tests :: ./ClangdTests/BackgroundIndexTest.IndexTwoFiles
Clangd Unit Tests :: ./ClangdTests/BackgroundIndexTest.NoCrashOnErrorFile
Clangd Unit Tests :: ./ClangdTests/BackgroundIndexTest.NoDotsInAbsPath
Clangd Unit Tests :: ./ClangdTests/BackgroundIndexTest.ShardStorageEmptyFile
Clangd Unit Tests :: ./ClangdTests/BackgroundIndexTest.ShardStorageLoad
Clangd Unit Tests :: ./ClangdTests/BackgroundIndexTest.ShardStorageTest
Clangd Unit Tests :: ./ClangdTests/BackgroundIndexTest.UncompilableFiles</pre><div class=""><br class=""></div></div><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Jul 18, 2019, at 9:25 AM, Kadir Cetinkaya via cfe-commits <<a href="mailto:cfe-commits@lists.llvm.org" target="_blank" class="">cfe-commits@lists.llvm.org</a>> wrote:</div><br class="gmail-m_1817645778349330322Apple-interchange-newline"><div class=""><div class="">Author: kadircet<br class="">Date: Thu Jul 18 09:25:36 2019<br class="">New Revision: 366458<br class=""><br class="">URL: <a href="http://llvm.org/viewvc/llvm-project?rev=366458&view=rev" target="_blank" class="">http://llvm.org/viewvc/llvm-project?rev=366458&view=rev</a><br class="">Log:<br class="">[clangd] Refactor background-index shard loading<br class=""><br class="">Reviewers: sammccall<br class=""><br class="">Subscribers: mgorny, ilya-biryukov, MaskRay, jkorous, arphaman, cfe-commits<br class=""><br class="">Tags: #clang<br class=""><br class="">Differential Revision: <a href="https://reviews.llvm.org/D64712" target="_blank" class="">https://reviews.llvm.org/D64712</a><br class=""><br class="">Added:<br class=""> clang-tools-extra/trunk/clangd/index/BackgroundIndexLoader.cpp<br class=""> clang-tools-extra/trunk/clangd/index/BackgroundIndexLoader.h<br class="">Modified:<br class=""> clang-tools-extra/trunk/clangd/CMakeLists.txt<br class=""> clang-tools-extra/trunk/clangd/index/Background.cpp<br class=""> clang-tools-extra/trunk/clangd/index/Background.h<br class=""> clang-tools-extra/trunk/clangd/index/BackgroundRebuild.cpp<br class=""> clang-tools-extra/trunk/clangd/index/BackgroundRebuild.h<br class=""> clang-tools-extra/trunk/clangd/unittests/BackgroundIndexTests.cpp<br class=""><br class="">Modified: clang-tools-extra/trunk/clangd/CMakeLists.txt<br class="">URL: <a href="http://llvm.org/viewvc/llvm-project/clang-tools-extra/trunk/clangd/CMakeLists.txt?rev=366458&r1=366457&r2=366458&view=diff" target="_blank" class="">http://llvm.org/viewvc/llvm-project/clang-tools-extra/trunk/clangd/CMakeLists.txt?rev=366458&r1=366457&r2=366458&view=diff</a><br class="">==============================================================================<br class="">--- clang-tools-extra/trunk/clangd/CMakeLists.txt (original)<br class="">+++ clang-tools-extra/trunk/clangd/CMakeLists.txt Thu Jul 18 09:25:36 2019<br class="">@@ -73,6 +73,7 @@ add_clang_library(clangDaemon<br class=""> XRefs.cpp<br class=""><br class=""> index/Background.cpp<br class="">+ index/BackgroundIndexLoader.cpp<br class=""> index/BackgroundIndexStorage.cpp<br class=""> index/BackgroundQueue.cpp<br class=""> index/BackgroundRebuild.cpp<br class=""><br class="">Modified: clang-tools-extra/trunk/clangd/index/Background.cpp<br class="">URL: <a href="http://llvm.org/viewvc/llvm-project/clang-tools-extra/trunk/clangd/index/Background.cpp?rev=366458&r1=366457&r2=366458&view=diff" target="_blank" class="">http://llvm.org/viewvc/llvm-project/clang-tools-extra/trunk/clangd/index/Background.cpp?rev=366458&r1=366457&r2=366458&view=diff</a><br class="">==============================================================================<br class="">--- clang-tools-extra/trunk/clangd/index/Background.cpp (original)<br class="">+++ clang-tools-extra/trunk/clangd/index/Background.cpp Thu Jul 18 09:25:36 2019<br class="">@@ -10,6 +10,7 @@<br class=""> #include "ClangdUnit.h"<br class=""> #include "Compiler.h"<br class=""> #include "Context.h"<br class="">+#include "FSProvider.h"<br class=""> #include "Headers.h"<br class=""> #include "Logger.h"<br class=""> #include "Path.h"<br class="">@@ -18,6 +19,7 @@<br class=""> #include "Threading.h"<br class=""> #include "Trace.h"<br class=""> #include "URI.h"<br class="">+#include "index/BackgroundIndexLoader.h"<br class=""> #include "index/FileIndex.h"<br class=""> #include "index/IndexAction.h"<br class=""> #include "index/MemIndex.h"<br class="">@@ -28,6 +30,8 @@<br class=""> #include "clang/Basic/SourceLocation.h"<br class=""> #include "clang/Basic/SourceManager.h"<br class=""> #include "clang/Driver/Types.h"<br class="">+#include "llvm/ADT/ArrayRef.h"<br class="">+#include "llvm/ADT/DenseSet.h"<br class=""> #include "llvm/ADT/Hashing.h"<br class=""> #include "llvm/ADT/STLExtras.h"<br class=""> #include "llvm/ADT/ScopeExit.h"<br class="">@@ -42,6 +46,7 @@<br class=""> #include <atomic><br class=""> #include <chrono><br class=""> #include <condition_variable><br class="">+#include <cstddef><br class=""> #include <memory><br class=""> #include <mutex><br class=""> #include <numeric><br class="">@@ -49,6 +54,8 @@<br class=""> #include <random><br class=""> #include <string><br class=""> #include <thread><br class="">+#include <utility><br class="">+#include <vector><br class=""><br class=""> namespace clang {<br class=""> namespace clangd {<br class="">@@ -119,6 +126,18 @@ llvm::SmallString<128> getAbsolutePath(c<br class=""> }<br class=""> return AbsolutePath;<br class=""> }<br class="">+<br class="">+bool shardIsStale(const LoadedShard &LS, llvm::vfs::FileSystem *FS) {<br class="">+ auto Buf = FS->getBufferForFile(LS.AbsolutePath);<br class="">+ if (!Buf) {<br class="">+ elog("Background-index: Couldn't read {0} to validate stored index: {1}",<br class="">+ LS.AbsolutePath, Buf.getError().message());<br class="">+ // There is no point in indexing an unreadable file.<br class="">+ return false;<br class="">+ }<br class="">+ return digest(Buf->get()->getBuffer()) != LS.Digest;<br class="">+}<br class="">+<br class=""> } // namespace<br class=""><br class=""> BackgroundIndex::BackgroundIndex(<br class="">@@ -156,7 +175,7 @@ BackgroundQueue::Task BackgroundIndex::c<br class=""> log("Enqueueing {0} commands for indexing", ChangedFiles.size());<br class=""> SPAN_ATTACH(Tracer, "files", int64_t(ChangedFiles.size()));<br class=""><br class="">- auto NeedsReIndexing = loadShards(std::move(ChangedFiles));<br class="">+ auto NeedsReIndexing = loadProject(std::move(ChangedFiles));<br class=""> // Run indexing for files that need to be updated.<br class=""> std::shuffle(NeedsReIndexing.begin(), NeedsReIndexing.end(),<br class=""> std::mt19937(std::random_device{}()));<br class="">@@ -431,169 +450,77 @@ llvm::Error BackgroundIndex::index(tooli<br class=""> return llvm::Error::success();<br class=""> }<br class=""><br class="">-std::vector<BackgroundIndex::Source><br class="">-BackgroundIndex::loadShard(const tooling::CompileCommand &Cmd,<br class="">- BackgroundIndexStorage *IndexStorage,<br class="">- llvm::StringSet<> &LoadedShards) {<br class="">- struct ShardInfo {<br class="">- std::string AbsolutePath;<br class="">- std::unique_ptr<IndexFileIn> Shard;<br class="">- FileDigest Digest = {};<br class="">- bool CountReferences = false;<br class="">- bool HadErrors = false;<br class="">- };<br class="">- std::vector<ShardInfo> IntermediateSymbols;<br class="">- // Make sure we don't have duplicate elements in the queue. Keys are absolute<br class="">- // paths.<br class="">- llvm::StringSet<> InQueue;<br class="">- auto FS = FSProvider.getFileSystem();<br class="">- // Dependencies of this TU, paired with the information about whether they<br class="">- // need to be re-indexed or not.<br class="">- std::vector<Source> Dependencies;<br class="">- std::queue<Source> ToVisit;<br class="">- std::string AbsolutePath = getAbsolutePath(Cmd).str();<br class="">- // Up until we load the shard related to a dependency it needs to be<br class="">- // re-indexed.<br class="">- ToVisit.emplace(AbsolutePath, true);<br class="">- InQueue.insert(AbsolutePath);<br class="">- // Goes over each dependency.<br class="">- while (!ToVisit.empty()) {<br class="">- Dependencies.push_back(std::move(ToVisit.front()));<br class="">- // Dependencies is not modified during the rest of the loop, so it is safe<br class="">- // to keep the reference.<br class="">- auto &CurDependency = Dependencies.back();<br class="">- ToVisit.pop();<br class="">- // If we have already seen this shard before(either loaded or failed) don't<br class="">- // re-try again. Since the information in the shard won't change from one TU<br class="">- // to another.<br class="">- if (!LoadedShards.try_emplace(CurDependency.Path).second) {<br class="">- // If the dependency needs to be re-indexed, first occurence would already<br class="">- // have detected that, so we don't need to issue it again.<br class="">- CurDependency.NeedsReIndexing = false;<br class="">- continue;<br class="">- }<br class="">-<br class="">- auto Shard = IndexStorage->loadShard(CurDependency.Path);<br class="">- if (!Shard || !Shard->Sources) {<br class="">- // File will be returned as requiring re-indexing to caller.<br class="">- vlog("Failed to load shard: {0}", CurDependency.Path);<br class="">- continue;<br class="">- }<br class="">- // These are the edges in the include graph for current dependency.<br class="">- for (const auto &I : *Shard->Sources) {<br class="">- auto U = URI::parse(I.getKey());<br class="">- if (!U)<br class="">- continue;<br class="">- auto AbsolutePath = URI::resolve(*U, CurDependency.Path);<br class="">- if (!AbsolutePath)<br class="">- continue;<br class="">- // Add file as dependency if haven't seen before.<br class="">- if (InQueue.try_emplace(*AbsolutePath).second)<br class="">- ToVisit.emplace(*AbsolutePath, true);<br class="">- // The node contains symbol information only for current file, the rest is<br class="">- // just edges.<br class="">- if (*AbsolutePath != CurDependency.Path)<br class="">- continue;<br class="">+// Restores shards for \p MainFiles from index storage. Then checks staleness of<br class="">+// those shards and returns a list of TUs that needs to be indexed to update<br class="">+// staleness.<br class="">+std::vector<std::pair<tooling::CompileCommand, BackgroundIndexStorage *>><br class="">+BackgroundIndex::loadProject(std::vector<std::string> MainFiles) {<br class="">+ std::vector<std::pair<tooling::CompileCommand, BackgroundIndexStorage *>><br class="">+ NeedsReIndexing;<br class=""><br class="">- // We found source file info for current dependency.<br class="">- assert(I.getValue().Digest != FileDigest{{0}} && "Digest is empty?");<br class="">- ShardInfo SI;<br class="">- SI.AbsolutePath = CurDependency.Path;<br class="">- SI.Shard = std::move(Shard);<br class="">- SI.Digest = I.getValue().Digest;<br class="">- SI.CountReferences =<br class="">- I.getValue().Flags & IncludeGraphNode::SourceFlag::IsTU;<br class="">- SI.HadErrors =<br class="">- I.getValue().Flags & IncludeGraphNode::SourceFlag::HadErrors;<br class="">- IntermediateSymbols.push_back(std::move(SI));<br class="">- // Check if the source needs re-indexing.<br class="">- // Get the digest, skip it if file doesn't exist.<br class="">- auto Buf = FS->getBufferForFile(CurDependency.Path);<br class="">- if (!Buf) {<br class="">- elog("Couldn't get buffer for file: {0}: {1}", CurDependency.Path,<br class="">- Buf.getError().message());<br class="">- continue;<br class="">- }<br class="">- // If digests match then dependency doesn't need re-indexing.<br class="">- // FIXME: Also check for dependencies(sources) of this shard and compile<br class="">- // commands for cache invalidation.<br class="">- CurDependency.NeedsReIndexing =<br class="">- digest(Buf->get()->getBuffer()) != I.getValue().Digest;<br class="">- }<br class="">- }<br class="">- // Load shard information into background-index.<br class="">+ Rebuilder.startLoading();<br class="">+ // Load shards for all of the mainfiles.<br class="">+ const std::vector<LoadedShard> Result =<br class="">+ loadIndexShards(MainFiles, IndexStorageFactory, CDB);<br class="">+ size_t LoadedShards = 0;<br class=""> {<br class="">+ // Update in-memory state.<br class=""> std::lock_guard<std::mutex> Lock(ShardVersionsMu);<br class="">- // This can override a newer version that is added in another thread,<br class="">- // if this thread sees the older version but finishes later. This<br class="">- // should be rare in practice.<br class="">- for (const ShardInfo &SI : IntermediateSymbols) {<br class="">+ for (auto &LS : Result) {<br class="">+ if (!LS.Shard)<br class="">+ continue;<br class=""> auto SS =<br class="">- SI.Shard->Symbols<br class="">- ? llvm::make_unique<SymbolSlab>(std::move(*SI.Shard->Symbols))<br class="">+ LS.Shard->Symbols<br class="">+ ? llvm::make_unique<SymbolSlab>(std::move(*LS.Shard->Symbols))<br class=""> : nullptr;<br class="">- auto RS = SI.Shard->Refs<br class="">- ? llvm::make_unique<RefSlab>(std::move(*SI.Shard->Refs))<br class="">+ auto RS = LS.Shard->Refs<br class="">+ ? llvm::make_unique<RefSlab>(std::move(*LS.Shard->Refs))<br class=""> : nullptr;<br class=""> auto RelS =<br class="">- SI.Shard->Relations<br class="">- ? llvm::make_unique<RelationSlab>(std::move(*SI.Shard->Relations))<br class="">+ LS.Shard->Relations<br class="">+ ? llvm::make_unique<RelationSlab>(std::move(*LS.Shard->Relations))<br class=""> : nullptr;<br class="">- ShardVersion &SV = ShardVersions[SI.AbsolutePath];<br class="">- SV.Digest = SI.Digest;<br class="">- SV.HadErrors = SI.HadErrors;<br class="">+ ShardVersion &SV = ShardVersions[LS.AbsolutePath];<br class="">+ SV.Digest = LS.Digest;<br class="">+ SV.HadErrors = LS.HadErrors;<br class="">+ ++LoadedShards;<br class=""><br class="">- IndexedSymbols.update(SI.AbsolutePath, std::move(SS), std::move(RS),<br class="">- std::move(RelS), SI.CountReferences);<br class="">+ IndexedSymbols.update(LS.AbsolutePath, std::move(SS), std::move(RS),<br class="">+ std::move(RelS), LS.CountReferences);<br class=""> }<br class=""> }<br class="">- if (!IntermediateSymbols.empty())<br class="">- Rebuilder.loadedTU();<br class="">+ Rebuilder.loadedShard(LoadedShards);<br class="">+ Rebuilder.doneLoading();<br class=""><br class="">- return Dependencies;<br class="">-}<br class="">+ auto FS = FSProvider.getFileSystem();<br class="">+ llvm::DenseSet<PathRef> TUsToIndex;<br class="">+ // We'll accept data from stale shards, but ensure the files get reindexed<br class="">+ // soon.<br class="">+ for (auto &LS : Result) {<br class="">+ if (!shardIsStale(LS, FS.get()))<br class="">+ continue;<br class="">+ PathRef TUForFile = LS.DependentTU;<br class="">+ assert(!TUForFile.empty() && "File without a TU!");<br class=""><br class="">-// Goes over each changed file and loads them from index. Returns the list of<br class="">-// TUs that had out-of-date/no shards.<br class="">-std::vector<std::pair<tooling::CompileCommand, BackgroundIndexStorage *>><br class="">-BackgroundIndex::loadShards(std::vector<std::string> ChangedFiles) {<br class="">- std::vector<std::pair<tooling::CompileCommand, BackgroundIndexStorage *>><br class="">- NeedsReIndexing;<br class="">- // Keeps track of the files that will be reindexed, to make sure we won't<br class="">- // re-index same dependencies more than once. Keys are AbsolutePaths.<br class="">- llvm::StringSet<> FilesToIndex;<br class="">- // Keeps track of the loaded shards to make sure we don't perform redundant<br class="">- // disk IO. Keys are absolute paths.<br class="">- llvm::StringSet<> LoadedShards;<br class="">- Rebuilder.startLoading();<br class="">- for (const auto &File : ChangedFiles) {<br class="">- auto Cmd = CDB.getCompileCommand(File);<br class="">+ // FIXME: Currently, we simply schedule indexing on a TU whenever any of<br class="">+ // its dependencies needs re-indexing. We might do it smarter by figuring<br class="">+ // out a minimal set of TUs that will cover all the stale dependencies.<br class="">+ // FIXME: Try looking at other TUs if no compile commands are available<br class="">+ // for this TU, i.e TU was deleted after we performed indexing.<br class="">+ TUsToIndex.insert(TUForFile);<br class="">+ }<br class="">+<br class="">+ for (PathRef TU : TUsToIndex) {<br class="">+ auto Cmd = CDB.getCompileCommand(TU);<br class=""> if (!Cmd)<br class=""> continue;<br class="">-<br class=""> std::string ProjectRoot;<br class="">- if (auto PI = CDB.getProjectInfo(File))<br class="">+ if (auto PI = CDB.getProjectInfo(TU))<br class=""> ProjectRoot = std::move(PI->SourceRoot);<br class="">-<br class="">- BackgroundIndexStorage *IndexStorage = IndexStorageFactory(ProjectRoot);<br class="">- auto Dependencies = loadShard(*Cmd, IndexStorage, LoadedShards);<br class="">- for (const auto &Dependency : Dependencies) {<br class="">- if (!Dependency.NeedsReIndexing || FilesToIndex.count(Dependency.Path))<br class="">- continue;<br class="">- // FIXME: Currently, we simply schedule indexing on a TU whenever any of<br class="">- // its dependencies needs re-indexing. We might do it smarter by figuring<br class="">- // out a minimal set of TUs that will cover all the stale dependencies.<br class="">- vlog("Enqueueing TU {0} because its dependency {1} needs re-indexing.",<br class="">- Cmd->Filename, Dependency.Path);<br class="">- NeedsReIndexing.push_back({std::move(*Cmd), IndexStorage});<br class="">- // Mark all of this TU's dependencies as to-be-indexed so that we won't<br class="">- // try to re-index those.<br class="">- for (const auto &Dependency : Dependencies)<br class="">- FilesToIndex.insert(Dependency.Path);<br class="">- break;<br class="">- }<br class="">+ BackgroundIndexStorage *Storage = IndexStorageFactory(ProjectRoot);<br class="">+ NeedsReIndexing.emplace_back(std::move(*Cmd), Storage);<br class=""> }<br class="">- Rebuilder.doneLoading();<br class="">+<br class=""> return NeedsReIndexing;<br class=""> }<br class=""><br class=""><br class="">Modified: clang-tools-extra/trunk/clangd/index/Background.h<br class="">URL: <a href="http://llvm.org/viewvc/llvm-project/clang-tools-extra/trunk/clangd/index/Background.h?rev=366458&r1=366457&r2=366458&view=diff" target="_blank" class="">http://llvm.org/viewvc/llvm-project/clang-tools-extra/trunk/clangd/index/Background.h?rev=366458&r1=366457&r2=366458&view=diff</a><br class="">==============================================================================<br class="">--- clang-tools-extra/trunk/clangd/index/Background.h (original)<br class="">+++ clang-tools-extra/trunk/clangd/index/Background.h Thu Jul 18 09:25:36 2019<br class="">@@ -12,6 +12,7 @@<br class=""> #include "Context.h"<br class=""> #include "FSProvider.h"<br class=""> #include "GlobalCompilationDatabase.h"<br class="">+#include "Path.h"<br class=""> #include "SourceCode.h"<br class=""> #include "Threading.h"<br class=""> #include "index/BackgroundRebuild.h"<br class="">@@ -173,20 +174,9 @@ private:<br class=""> std::mutex ShardVersionsMu;<br class=""><br class=""> BackgroundIndexStorage::Factory IndexStorageFactory;<br class="">- struct Source {<br class="">- std::string Path;<br class="">- bool NeedsReIndexing;<br class="">- Source(llvm::StringRef Path, bool NeedsReIndexing)<br class="">- : Path(Path), NeedsReIndexing(NeedsReIndexing) {}<br class="">- };<br class="">- // Loads the shards for a single TU and all of its dependencies. Returns the<br class="">- // list of sources and whether they need to be re-indexed.<br class="">- std::vector<Source> loadShard(const tooling::CompileCommand &Cmd,<br class="">- BackgroundIndexStorage *IndexStorage,<br class="">- llvm::StringSet<> &LoadedShards);<br class="">- // Tries to load shards for the ChangedFiles.<br class="">+ // Tries to load shards for the MainFiles and their dependencies.<br class=""> std::vector<std::pair<tooling::CompileCommand, BackgroundIndexStorage *>><br class="">- loadShards(std::vector<std::string> ChangedFiles);<br class="">+ loadProject(std::vector<std::string> MainFiles);<br class=""><br class=""> BackgroundQueue::Task<br class=""> changedFilesTask(const std::vector<std::string> &ChangedFiles);<br class=""><br class="">Added: clang-tools-extra/trunk/clangd/index/BackgroundIndexLoader.cpp<br class="">URL: <a href="http://llvm.org/viewvc/llvm-project/clang-tools-extra/trunk/clangd/index/BackgroundIndexLoader.cpp?rev=366458&view=auto" target="_blank" class="">http://llvm.org/viewvc/llvm-project/clang-tools-extra/trunk/clangd/index/BackgroundIndexLoader.cpp?rev=366458&view=auto</a><br class="">==============================================================================<br class="">--- clang-tools-extra/trunk/clangd/index/BackgroundIndexLoader.cpp (added)<br class="">+++ clang-tools-extra/trunk/clangd/index/BackgroundIndexLoader.cpp Thu Jul 18 09:25:36 2019<br class="">@@ -0,0 +1,153 @@<br class="">+//===-- BackgroundIndexLoader.cpp - ---------------------------------------===//<br class="">+//<br class="">+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.<br class="">+// See <a href="https://llvm.org/LICENSE.txt" target="_blank" class="">https://llvm.org/LICENSE.txt</a> for license information.<br class="">+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception<br class="">+//<br class="">+//===----------------------------------------------------------------------===//<br class="">+<br class="">+#include "index/BackgroundIndexLoader.h"<br class="">+#include "GlobalCompilationDatabase.h"<br class="">+#include "Logger.h"<br class="">+#include "Path.h"<br class="">+#include "index/Background.h"<br class="">+#include "llvm/ADT/DenseMap.h"<br class="">+#include "llvm/ADT/DenseSet.h"<br class="">+#include "llvm/ADT/SmallString.h"<br class="">+#include "llvm/ADT/StringMap.h"<br class="">+#include "llvm/Support/Path.h"<br class="">+#include <string><br class="">+#include <utility><br class="">+#include <vector><br class="">+<br class="">+namespace clang {<br class="">+namespace clangd {<br class="">+namespace {<br class="">+<br class="">+llvm::Optional<Path> uriToAbsolutePath(llvm::StringRef URI, PathRef HintPath) {<br class="">+ auto U = URI::parse(URI);<br class="">+ if (!U)<br class="">+ return llvm::None;<br class="">+ auto AbsolutePath = URI::resolve(*U, HintPath);<br class="">+ if (!AbsolutePath)<br class="">+ return llvm::None;<br class="">+ return *AbsolutePath;<br class="">+}<br class="">+<br class="">+/// A helper class to cache BackgroundIndexStorage operations and keep the<br class="">+/// inverse dependency mapping.<br class="">+class BackgroundIndexLoader {<br class="">+public:<br class="">+ /// Load the shards for \p MainFile and all of its dependencies.<br class="">+ void load(PathRef MainFile, BackgroundIndexStorage *Storage);<br class="">+<br class="">+ /// Consumes the loader and returns all shards.<br class="">+ std::vector<LoadedShard> takeResult() &&;<br class="">+<br class="">+private:<br class="">+ /// Returns the Shard for \p StartSourceFile from cache or loads it from \p<br class="">+ /// Storage. Also returns paths for dependencies of \p StartSourceFile if it<br class="">+ /// wasn't cached yet.<br class="">+ std::pair<const LoadedShard &, std::vector<Path>><br class="">+ loadShard(PathRef StartSourceFile, BackgroundIndexStorage *Storage);<br class="">+<br class="">+ /// Cache for Storage lookups.<br class="">+ llvm::StringMap<LoadedShard> LoadedShards;<br class="">+<br class="">+ /// References are into the AbsolutePaths in LoadedShards.<br class="">+ llvm::DenseMap<PathRef, PathRef> FileToTU;<br class="">+};<br class="">+<br class="">+std::pair<const LoadedShard &, std::vector<Path>><br class="">+BackgroundIndexLoader::loadShard(PathRef StartSourceFile,<br class="">+ BackgroundIndexStorage *Storage) {<br class="">+ auto It = LoadedShards.try_emplace(StartSourceFile);<br class="">+ LoadedShard &LS = It.first->getValue();<br class="">+ std::vector<Path> Edges = {};<br class="">+ // Return the cached shard.<br class="">+ if (!It.second)<br class="">+ return {LS, Edges};<br class="">+<br class="">+ LS.AbsolutePath = StartSourceFile.str();<br class="">+ auto Shard = Storage->loadShard(StartSourceFile);<br class="">+ if (!Shard || !Shard->Sources) {<br class="">+ vlog("Failed to load shard: {0}", StartSourceFile);<br class="">+ return {LS, Edges};<br class="">+ }<br class="">+<br class="">+ LS.Shard = std::move(Shard);<br class="">+ for (const auto &It : *LS.Shard->Sources) {<br class="">+ auto AbsPath = uriToAbsolutePath(It.getKey(), StartSourceFile);<br class="">+ if (!AbsPath)<br class="">+ continue;<br class="">+ // A shard contains only edges for non main-file sources.<br class="">+ if (*AbsPath != StartSourceFile) {<br class="">+ Edges.push_back(*AbsPath);<br class="">+ continue;<br class="">+ }<br class="">+<br class="">+ // Fill in shard metadata.<br class="">+ const IncludeGraphNode &IGN = It.getValue();<br class="">+ LS.Digest = IGN.Digest;<br class="">+ LS.CountReferences = IGN.Flags & IncludeGraphNode::SourceFlag::IsTU;<br class="">+ LS.HadErrors = IGN.Flags & IncludeGraphNode::SourceFlag::HadErrors;<br class="">+ }<br class="">+ assert(LS.Digest != FileDigest{{0}} && "Digest is empty?");<br class="">+ return {LS, Edges};<br class="">+}<br class="">+<br class="">+void BackgroundIndexLoader::load(PathRef MainFile,<br class="">+ BackgroundIndexStorage *Storage) {<br class="">+ llvm::StringSet<> InQueue;<br class="">+ // Following containers points to strings inside InQueue.<br class="">+ std::queue<PathRef> ToVisit;<br class="">+ InQueue.insert(MainFile);<br class="">+ ToVisit.push(MainFile);<br class="">+<br class="">+ while (!ToVisit.empty()) {<br class="">+ PathRef SourceFile = ToVisit.front();<br class="">+ ToVisit.pop();<br class="">+<br class="">+ auto ShardAndEdges = loadShard(SourceFile, Storage);<br class="">+ FileToTU[ShardAndEdges.first.AbsolutePath] = MainFile;<br class="">+ for (PathRef Edge : ShardAndEdges.second) {<br class="">+ auto It = InQueue.insert(Edge);<br class="">+ if (It.second)<br class="">+ ToVisit.push(It.first->getKey());<br class="">+ }<br class="">+ }<br class="">+}<br class="">+<br class="">+std::vector<LoadedShard> BackgroundIndexLoader::takeResult() && {<br class="">+ std::vector<LoadedShard> Result;<br class="">+ Result.reserve(LoadedShards.size());<br class="">+ for (auto &It : LoadedShards) {<br class="">+ Result.push_back(std::move(It.getValue()));<br class="">+ LoadedShard &LS = Result.back();<br class="">+ auto TUsIt = FileToTU.find(LS.AbsolutePath);<br class="">+ assert(TUsIt != FileToTU.end() && "No TU registered for the shard");<br class="">+ Result.back().DependentTU = TUsIt->second;<br class="">+ }<br class="">+ return Result;<br class="">+}<br class="">+} // namespace<br class="">+<br class="">+std::vector<LoadedShard><br class="">+loadIndexShards(llvm::ArrayRef<Path> MainFiles,<br class="">+ BackgroundIndexStorage::Factory &IndexStorageFactory,<br class="">+ const GlobalCompilationDatabase &CDB) {<br class="">+ BackgroundIndexLoader Loader;<br class="">+ for (llvm::StringRef MainFile : MainFiles) {<br class="">+ assert(llvm::sys::path::is_absolute(MainFile));<br class="">+<br class="">+ std::string ProjectRoot;<br class="">+ if (auto PI = CDB.getProjectInfo(MainFile))<br class="">+ ProjectRoot = std::move(PI->SourceRoot);<br class="">+ BackgroundIndexStorage *Storage = IndexStorageFactory(ProjectRoot);<br class="">+ Loader.load(MainFile, Storage);<br class="">+ }<br class="">+ return std::move(Loader).takeResult();<br class="">+}<br class="">+<br class="">+} // namespace clangd<br class="">+} // namespace clang<br class=""><br class="">Added: clang-tools-extra/trunk/clangd/index/BackgroundIndexLoader.h<br class="">URL: <a href="http://llvm.org/viewvc/llvm-project/clang-tools-extra/trunk/clangd/index/BackgroundIndexLoader.h?rev=366458&view=auto" target="_blank" class="">http://llvm.org/viewvc/llvm-project/clang-tools-extra/trunk/clangd/index/BackgroundIndexLoader.h?rev=366458&view=auto</a><br class="">==============================================================================<br class="">--- clang-tools-extra/trunk/clangd/index/BackgroundIndexLoader.h (added)<br class="">+++ clang-tools-extra/trunk/clangd/index/BackgroundIndexLoader.h Thu Jul 18 09:25:36 2019<br class="">@@ -0,0 +1,54 @@<br class="">+//===--- BackgroundIndexLoader.h - Load shards from index storage-*- C++-*-===//<br class="">+//<br class="">+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.<br class="">+// See <a href="https://llvm.org/LICENSE.txt" target="_blank" class="">https://llvm.org/LICENSE.txt</a> for license information.<br class="">+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception<br class="">+//<br class="">+//===----------------------------------------------------------------------===//<br class="">+<br class="">+#ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_INDEX_BACKGROUND_INDEX_LOADER_H<br class="">+#define LLVM_CLANG_TOOLS_EXTRA_CLANGD_INDEX_BACKGROUND_INDEX_LOADER_H<br class="">+<br class="">+#include "Path.h"<br class="">+#include "index/Background.h"<br class="">+#include "llvm/ADT/ArrayRef.h"<br class="">+#include "llvm/ADT/DenseMap.h"<br class="">+#include "llvm/ADT/Optional.h"<br class="">+#include "llvm/ADT/StringMap.h"<br class="">+#include "llvm/ADT/StringRef.h"<br class="">+#include "llvm/Support/VirtualFileSystem.h"<br class="">+#include <memory><br class="">+#include <vector><br class="">+<br class="">+namespace clang {<br class="">+namespace clangd {<br class="">+<br class="">+/// Represents a shard loaded from storage, stores contents in \p Shard and<br class="">+/// metadata about the source file that generated this shard.<br class="">+struct LoadedShard {<br class="">+ /// Path of the source file that produced this shard.<br class="">+ Path AbsolutePath;<br class="">+ /// Digest of the source file contents that produced this shard.<br class="">+ FileDigest Digest = {};<br class="">+ /// Whether the RefSlab in Shard should be used for updating symbol reference<br class="">+ /// counts when building an index.<br class="">+ bool CountReferences = false;<br class="">+ /// Whether the indexing action producing that shard had errors.<br class="">+ bool HadErrors = false;<br class="">+ /// Path to a TU that is depending on this shard.<br class="">+ Path DependentTU;<br class="">+ /// Will be nullptr when index storage couldn't provide a valid shard for<br class="">+ /// AbsolutePath.<br class="">+ std::unique_ptr<IndexFileIn> Shard;<br class="">+};<br class="">+<br class="">+/// Loads all shards for the TU \p MainFile from \p Storage.<br class="">+std::vector<LoadedShard><br class="">+loadIndexShards(llvm::ArrayRef<Path> MainFiles,<br class="">+ BackgroundIndexStorage::Factory &IndexStorageFactory,<br class="">+ const GlobalCompilationDatabase &CDB);<br class="">+<br class="">+} // namespace clangd<br class="">+} // namespace clang<br class="">+<br class="">+#endif<br class=""><br class="">Modified: clang-tools-extra/trunk/clangd/index/BackgroundRebuild.cpp<br class="">URL: <a href="http://llvm.org/viewvc/llvm-project/clang-tools-extra/trunk/clangd/index/BackgroundRebuild.cpp?rev=366458&r1=366457&r2=366458&view=diff" target="_blank" class="">http://llvm.org/viewvc/llvm-project/clang-tools-extra/trunk/clangd/index/BackgroundRebuild.cpp?rev=366458&r1=366457&r2=366458&view=diff</a><br class="">==============================================================================<br class="">--- clang-tools-extra/trunk/clangd/index/BackgroundRebuild.cpp (original)<br class="">+++ clang-tools-extra/trunk/clangd/index/BackgroundRebuild.cpp Thu Jul 18 09:25:36 2019<br class="">@@ -78,13 +78,13 @@ void BackgroundIndexRebuilder::idle() {<br class=""> void BackgroundIndexRebuilder::startLoading() {<br class=""> std::lock_guard<std::mutex> Lock(Mu);<br class=""> if (!Loading)<br class="">- LoadedTUs = 0;<br class="">+ LoadedShards = 0;<br class=""> ++Loading;<br class=""> }<br class="">-void BackgroundIndexRebuilder::loadedTU() {<br class="">+void BackgroundIndexRebuilder::loadedShard(size_t ShardCount) {<br class=""> std::lock_guard<std::mutex> Lock(Mu);<br class=""> assert(Loading);<br class="">- ++LoadedTUs;<br class="">+ LoadedShards += ShardCount;<br class=""> }<br class=""> void BackgroundIndexRebuilder::doneLoading() {<br class=""> maybeRebuild("after loading index from disk", [this] {<br class="">@@ -93,7 +93,7 @@ void BackgroundIndexRebuilder::doneLoadi<br class=""> if (Loading) // was loading multiple batches concurrently<br class=""> return false; // rebuild once the last batch is done.<br class=""> // Rebuild if we loaded any shards, or if we stopped an indexedTU rebuild.<br class="">- return LoadedTUs > 0 || enoughTUsToRebuild();<br class="">+ return LoadedShards > 0 || enoughTUsToRebuild();<br class=""> });<br class=""> }<br class=""><br class=""><br class="">Modified: clang-tools-extra/trunk/clangd/index/BackgroundRebuild.h<br class="">URL: <a href="http://llvm.org/viewvc/llvm-project/clang-tools-extra/trunk/clangd/index/BackgroundRebuild.h?rev=366458&r1=366457&r2=366458&view=diff" target="_blank" class="">http://llvm.org/viewvc/llvm-project/clang-tools-extra/trunk/clangd/index/BackgroundRebuild.h?rev=366458&r1=366457&r2=366458&view=diff</a><br class="">==============================================================================<br class="">--- clang-tools-extra/trunk/clangd/index/BackgroundRebuild.h (original)<br class="">+++ clang-tools-extra/trunk/clangd/index/BackgroundRebuild.h Thu Jul 18 09:25:36 2019<br class="">@@ -17,6 +17,7 @@<br class=""> #include "index/FileIndex.h"<br class=""> #include "index/Index.h"<br class=""> #include "llvm/Support/Threading.h"<br class="">+#include <cstddef><br class=""><br class=""> namespace clang {<br class=""> namespace clangd {<br class="">@@ -61,7 +62,7 @@ public:<br class=""> // sessions may happen concurrently.<br class=""> void startLoading();<br class=""> // Called to indicate some shards were actually loaded from disk.<br class="">- void loadedTU();<br class="">+ void loadedShard(size_t ShardCount);<br class=""> // Called to indicate we're finished loading shards from disk.<br class=""> // May rebuild (if any were loaded).<br class=""> void doneLoading();<br class="">@@ -89,7 +90,7 @@ private:<br class=""> unsigned IndexedTUsAtLastRebuild = 0;<br class=""> // Are we loading shards? May be multiple concurrent sessions.<br class=""> unsigned Loading = 0;<br class="">- unsigned LoadedTUs; // In the current loading session.<br class="">+ unsigned LoadedShards; // In the current loading session.<br class=""><br class=""> SwapIndex *Target;<br class=""> FileSymbols *Source;<br class=""><br class="">Modified: clang-tools-extra/trunk/clangd/unittests/BackgroundIndexTests.cpp<br class="">URL: <a href="http://llvm.org/viewvc/llvm-project/clang-tools-extra/trunk/clangd/unittests/BackgroundIndexTests.cpp?rev=366458&r1=366457&r2=366458&view=diff" target="_blank" class="">http://llvm.org/viewvc/llvm-project/clang-tools-extra/trunk/clangd/unittests/BackgroundIndexTests.cpp?rev=366458&r1=366457&r2=366458&view=diff</a><br class="">==============================================================================<br class="">--- clang-tools-extra/trunk/clangd/unittests/BackgroundIndexTests.cpp (original)<br class="">+++ clang-tools-extra/trunk/clangd/unittests/BackgroundIndexTests.cpp Thu Jul 18 09:25:36 2019<br class="">@@ -211,7 +211,7 @@ TEST_F(BackgroundIndexTest, ShardStorage<br class=""> OverlayCDB CDB(/*Base=*/nullptr);<br class=""> BackgroundIndex Idx(Context::empty(), FS, CDB,<br class=""> [&](llvm::StringRef) { return &MSS; });<br class="">- CDB.setCompileCommand(testPath("root"), Cmd);<br class="">+ CDB.setCompileCommand(testPath("root/<a href="http://a.cc/" target="_blank" class="">A.cc</a>"), Cmd);<br class=""> ASSERT_TRUE(Idx.blockUntilIdleForTest());<br class=""> }<br class=""> EXPECT_EQ(CacheHits, 2U); // Check both <a href="http://a.cc/" target="_blank" class="">A.cc</a> and A.h loaded from cache.<br class="">@@ -335,7 +335,7 @@ TEST_F(BackgroundIndexTest, ShardStorage<br class=""> OverlayCDB CDB(/*Base=*/nullptr);<br class=""> BackgroundIndex Idx(Context::empty(), FS, CDB,<br class=""> [&](llvm::StringRef) { return &MSS; });<br class="">- CDB.setCompileCommand(testPath("root"), Cmd);<br class="">+ CDB.setCompileCommand(testPath("root/<a href="http://a.cc/" target="_blank" class="">A.cc</a>"), Cmd);<br class=""> ASSERT_TRUE(Idx.blockUntilIdleForTest());<br class=""> }<br class=""> EXPECT_EQ(CacheHits, 2U); // Check both <a href="http://a.cc/" target="_blank" class="">A.cc</a> and A.h loaded from cache.<br class="">@@ -353,7 +353,7 @@ TEST_F(BackgroundIndexTest, ShardStorage<br class=""> OverlayCDB CDB(/*Base=*/nullptr);<br class=""> BackgroundIndex Idx(Context::empty(), FS, CDB,<br class=""> [&](llvm::StringRef) { return &MSS; });<br class="">- CDB.setCompileCommand(testPath("root"), Cmd);<br class="">+ CDB.setCompileCommand(testPath("root/<a href="http://a.cc/" target="_blank" class="">A.cc</a>"), Cmd);<br class=""> ASSERT_TRUE(Idx.blockUntilIdleForTest());<br class=""> }<br class=""> EXPECT_EQ(CacheHits, 2U); // Check both <a href="http://a.cc/" target="_blank" class="">A.cc</a> and A.h loaded from cache.<br class="">@@ -621,8 +621,8 @@ TEST_F(BackgroundIndexRebuilderTest, Ind<br class=""><br class=""> TEST_F(BackgroundIndexRebuilderTest, LoadingShards) {<br class=""> Rebuilder.startLoading();<br class="">- Rebuilder.loadedTU();<br class="">- Rebuilder.loadedTU();<br class="">+ Rebuilder.loadedShard(10);<br class="">+ Rebuilder.loadedShard(20);<br class=""> EXPECT_TRUE(checkRebuild([&] { Rebuilder.doneLoading(); }));<br class=""><br class=""> // No rebuild for no shards.<br class="">@@ -631,11 +631,11 @@ TEST_F(BackgroundIndexRebuilderTest, Loa<br class=""><br class=""> // Loads can overlap.<br class=""> Rebuilder.startLoading();<br class="">- Rebuilder.loadedTU();<br class="">+ Rebuilder.loadedShard(1);<br class=""> Rebuilder.startLoading();<br class="">- Rebuilder.loadedTU();<br class="">+ Rebuilder.loadedShard(1);<br class=""> EXPECT_FALSE(checkRebuild([&] { Rebuilder.doneLoading(); }));<br class="">- Rebuilder.loadedTU();<br class="">+ Rebuilder.loadedShard(1);<br class=""> EXPECT_TRUE(checkRebuild([&] { Rebuilder.doneLoading(); }));<br class=""><br class=""> // No rebuilding for indexed files while loading.<br class=""><br class=""><br class="">_______________________________________________<br class="">cfe-commits mailing list<br class=""><a href="mailto:cfe-commits@lists.llvm.org" target="_blank" class="">cfe-commits@lists.llvm.org</a><br class=""><a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits" target="_blank" class="">https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits</a><br class=""></div></div></blockquote></div><br class=""></div></div></div></blockquote></div><br class=""></div></div></span></div></blockquote></div>
</div></blockquote></div><br class=""></body></html>