[PATCH] D52300: [clangd] Implement VByte PostingList compression

[Kirill Bobyrev via Phabricator via cfe-commits]

kbobyrev created this revision.
kbobyrev added reviewers: ioeric, sammccall, ilya-biryukov.
kbobyrev added a project: clang-tools-extra.
Herald added subscribers: kadircet, arphaman, jkorous, MaskRay, mgorny.
kbobyrev edited the summary of this revision.

This patch implements Variable-length Byte compression of `PostingList`s to sacrifice some performance for lower memory consumption.

`PostingList` compression and decompression was extensively tested using fuzzer for multiple hours and runnning significant number of realistic `FuzzyFindRequests`. AddressSanitizer and UndefinedBehaviorSanitizer were used to ensure the correct behaviour.

Performance evaluation was conducted with recent LLVM symbol index (292k symbols) and the collection of user-recorded queries (5540 `FuzzyFindRequest` JSON dumps):

| Metrics                             | Before | After | Change (%) |
| ----------------------------------- | ------ | ----- | ---------- |
| Memory consumption (index only), MB | 65     | 52    | -20%       |
| Time to process queries, sec        | 5.370  | 7.145 | +25%       |


https://reviews.llvm.org/D52300

Files:
  clang-tools-extra/clangd/CMakeLists.txt
  clang-tools-extra/clangd/index/dex/Dex.cpp
  clang-tools-extra/clangd/index/dex/PostingList.cpp
  clang-tools-extra/clangd/index/dex/PostingList.h
  clang-tools-extra/clangd/index/dex/fuzzer/CMakeLists.txt
  clang-tools-extra/clangd/index/dex/fuzzer/VByteFuzzer.cpp

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D52300.166275.patch
Type: text/x-patch
Size: 19106 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/cfe-commits/attachments/20180920/bd8dc596/attachment-0001.bin>