[PATCH] D50576: [clangd] Allow consumption of DocIDs without overhead

[Kirill Bobyrev via Phabricator via cfe-commits]

kbobyrev created this revision.
kbobyrev added reviewers: ioeric, ilya-biryukov.
Herald added subscribers: arphaman, jkorous, MaskRay.

This patch allows processing DocIDs from iterator using callback so that they are not stored in a vector if actual DocIDs are not needed.

Such overhead is the case for https://reviews.llvm.org/D50337 patch: `fuzzyFindLongQuery` stores `SymbolDocIDs` but they are thrown away later, because what the index really needs is `std::vector<std::pair<float, const Symbol *>> Scores;` and it can be filled on-the-fly in `matchSymbols`.


https://reviews.llvm.org/D50576

Files:
  clang-tools-extra/clangd/index/dex/Iterator.cpp
  clang-tools-extra/clangd/index/dex/Iterator.h


Index: clang-tools-extra/clangd/index/dex/Iterator.h
===================================================================
--- clang-tools-extra/clangd/index/dex/Iterator.h
+++ clang-tools-extra/clangd/index/dex/Iterator.h
@@ -101,8 +101,14 @@
   virtual llvm::raw_ostream &dump(llvm::raw_ostream &OS) const = 0;
 };
 
-/// Advances the iterator until it is either exhausted or the number of
-/// requested items is reached. The result contains sorted DocumentIDs.
+/// Advances given iterator until it is exhausted or the requested number of
+/// symbols is already seen while using callback on each processed DocID.
+void matchSymbols(Iterator &It,
+                  llvm::function_ref<void(const DocID &)> Callback,
+                  size_t Limit = std::numeric_limits<size_t>::max());
+
+/// Advances given iterator until it is either exhausted or the number of
+/// requested items is reached and returns all seen DocIDs.
 std::vector<DocID> consume(Iterator &It,
                            size_t Limit = std::numeric_limits<size_t>::max());
 
Index: clang-tools-extra/clangd/index/dex/Iterator.cpp
===================================================================
--- clang-tools-extra/clangd/index/dex/Iterator.cpp
+++ clang-tools-extra/clangd/index/dex/Iterator.cpp
@@ -220,10 +220,16 @@
 
 std::vector<DocID> consume(Iterator &It, size_t Limit) {
   std::vector<DocID> Result;
+  matchSymbols(It, [&](const DocID &ID) { Result.push_back(ID); }, Limit);
+  return Result;
+}
+
+void matchSymbols(Iterator &It,
+                  llvm::function_ref<void(const DocID &)> Callback,
+                  size_t Limit) {
   for (size_t Retrieved = 0; !It.reachedEnd() && Retrieved < Limit;
        It.advance(), ++Retrieved)
-    Result.push_back(It.peek());
-  return Result;
+    Callback(It.peek());
 }
 
 std::unique_ptr<Iterator> create(PostingListRef Documents) {


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D50576.160158.patch
Type: text/x-patch
Size: 1875 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/cfe-commits/attachments/20180810/fe113316/attachment.bin>