[clangd-dev] Building and sharing a clangd global index
William Wagner (BLOOMBERG/ 731 LEX) via clangd-dev
clangd-dev at lists.llvm.org
Thu Apr 4 18:36:47 PDT 2019
Ah awesome, thanks for the clarification! I think going forward I’ll prefer the background-indexer over clangd-indexer - since it sounds like it’s the future and I’d rather not use something that may soon be deprecated.
> If you always want an up-to-date index and storing the shared snapshot is
> just a performance optimization, use background-indexer.
Sharing the index was just a performance optimization, I think we may be able to avoid doing this for now. I outlined in a previous post that my team uses docker containers for development, and seeing as they are ephemeral, I didn’t want to be continually re-building the entire index across docker container start/stops. So this brings up another question: if we have a workflow where we
1)Start up a development container and clangd, building a background-index in the process.
2)Bring down said containers (but persisting the built index by e.g. a docker volume)
3)Bring up new containers with updated source code and mount the background index built by the previous docker run into the container (at the same path)
Is the background-indexer smart enough to do a rescan of the code base, and only update the files that changed? My assumption is yes, because the paths are the same and the digests(?) will be the same for the unchanged files, but confirmation here would be great.
> Dex is an efficient implementation of running search queries (e.g. it models fuzzy-matching algorithm, > etc.).
Ah okay, so if I wanted to use the background-index and dex, are the only arguments I have to pass (in clang-8) `-background-index` and `-use-dex-index`?
Also one last question (sorry for the bombardment!), will background-index functionality be invokable by a separate binary in the future, or will it always be invoked by clangd? If we do want to build and share the index produced by the background-indexer in the future, it seems like the easiest way.
Thanks,
William
From: ibiryukov at google.com At: 04/03/19 04:27:23To: William Wagner (BLOOMBERG/ 731 LEX )
Cc: clangd-dev at lists.llvm.org
Subject: Re: [clangd-dev] Building and sharing a clangd global index
Hi William,
The difference between background-indexer and clangd-indexer is the layout
of the output:
- background-indexer would put the resulting index into the folder
<project-root>/.clangd/index.
The index is split per-file, i.e. it's incremental and clangd would be
able to update the files that changed after the index was built.
You would need to run clangd with '-background-index' to load the index,
it will also automatically update the index for files that changed on load.
- clangd-indexer would produce a *merged *index, it can't be incrementally
updated and you have more control for the location of the output:
./bin/clangd-indexer -executor=all-TUs path/to/compile_commands.json >
path/to/output.riff
You would need to run clangd with '-index-file=path/to/output.riff' to
load the index.
Note that both indexes store absolute paths, so sharing the produced index
across multiple machines would only be possible if the directory structure
is kept the same.
If having the same directory structure is plausible, please try it out and
let us know if it works, we haven't tried sharing the same index across
multiple machines.
Which option to prefer? Depending on your situation, either of the two
might be better:
- If you always want an up-to-date index and storing the shared snapshot is
just a performance optimization, use background-indexer.
- If you not wasting resources to rebuild the index for changed files is
more important than the fact that some results are stale (e.g. it's too
expensive, you want to save laptop battery, etc.), clangd-indexer might be
a better choice.
Here's a short summary on what each index means:
- Static index is an index that is persisted across multiple runs of
clangd. There are two flavours of it:
1. Background index. Incremental (split per-file) index living in
'<project-root>/.clangd/index'. Built automatically by clangd when
-background-index is specified. Long-term, we want this to be enabled by
default (and possibly be the only option).
2. Old-style "merged" index produced by clangd-indexer. The results will
not get updated by clangd automatically, you can ask clangd to load it with
'-index-file=path/to/index.riff'.
- Dynamic index is an overlay for a small number of updated files
(currently the open files for which we built the AST). Kept in memory, not
persisted across multiple runs. We use to adjust for the fact that static
index might be stale. We want the correct results for the open files in all
cases.
- Dex is an efficient implementation of running search queries (e.g. it
models fuzzy-matching algorithm, etc.). It's an "index" in an information
retrieval sense, it is not actually specific to C++ or clangd.
On Mon, Apr 1, 2019 at 6:36 PM William Wagner (BLOOMBERG/ 731 LEX) via
clangd-dev <clangd-dev at lists.llvm.org> wrote:
> Hello!
>
> I work on a fairly large C++ project and wanted to figure out a way to
> regularly build (e.g. nightly via Jenkins) a global project index that can
> be shared with all the members of my team. I want to share it because it
> takes a fairly long time to build the index after starting up, and it seems
> pretty redundant to have each team member doing so, seeing as most of the
> code is not changing on a day-to-day basis. I’ve tried peeking around the
> mailing lists and commit history of clangd, but I’m not sure whether this
> is possible yet - and if it was, what flags to use, what indexer etc.
>
> I see there’s background-indexer WIP (https://reviews.llvm.org/D59605)
> and an existing clangd-indexer
>
https://github.com/llvm-mirror/clang-tools-extra/blob/master/clangd/indexer/Inde
xerMain.cpp
> What is the difference between these?
>
> Additionally, if anyone could provide some clarification on the different
> types of indexes clangd currently has (dex, background, static, etc.) that
> would be great :)
>
> Thanks!
>
> _______________________________________________
> clangd-dev mailing list
> clangd-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/clangd-dev
>
--
Regards,
Ilya Biryukov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/clangd-dev/attachments/20190405/b1e67cab/attachment-0001.html>
More information about the clangd-dev
mailing list