[clang-tools-extra] [clangd] Add background index format support to clangd-indexer (PR #175209)

[Nathan Ridge via cfe-commits]

================
@@ -48,6 +64,23 @@ static llvm::cl::list<std::string> QueryDriverGlobs{
     llvm::cl::CommaSeparated,
 };
 
+static llvm::cl::opt<std::string> ProjectRoot{
----------------
HighCommander4 wrote:

There are a couple of issues with this flag and how we're using it.

 * The semantics of writing **all** shards to `<project-root>/.cache/clangd/index` is only an approximate match for what clangd's background indexer does.
    * Clangd's background indexer will write shards for source and header files in the project's directory tree to `<project-root>/.cache/clangd/index`, but shards for a header in a dependent project with its own `compile_commands.json` go in `<dependent-project>/.cache/clangd/index`, and shards for standard library and other systems headers will go in `~/.cache/clangd/index`.
    * (This is arguably a questionable choice on clangd's part, but this PR is not the place to fix clangd's questionable choices. Note that the background index **loader** will look for files in the above locations, so writing them elsewhere will just result in clangd reindexing the files in question on startup. For this feature to work as intended, we need to match the loader's behaviour quirk-for-quirk, so to speak.)
 * The current directory is a poor default. You want the default to be the directory containing the `compile_commands.json` file, and then you almost never need to pass this flag.

Here are my suggestions for what to do instead.

First, to match clangd's shard-locating behaviour closely, pass something like this to `createDiskBasedStorageFactory`:

```
  RealThreadsafeFS TFS;
  DirectoryBasedGlobalCompilationDatabase::Options CDBOpts(TFS);
  DirectoryBasedGlobalCompilationDatabase CDB(CDBOpts);
  ...createDiskBackedStorageFactory(
       [&CDB](PathRef File) { return CDB.getProjectInfo(File); });
 ```

This is what [clangd itself](https://searchfox.org/llvm/rev/aff1ba103ab3006242c72af33420a3437f4ff1db/clang-tools-extra/clangd/ClangdServer.cpp#266-268) does.

Second, instead of having a `--project-root` flag, have a `--compile-commands-dir` flag matching [clangd's](https://searchfox.org/llvm/rev/aff1ba103ab3006242c72af33420a3437f4ff1db/clang-tools-extra/clangd/tool/ClangdMain.cpp#121-127), which is wired up to the above `DirectoryBasedGlobalCompilationDatabase` in the same way clangd does:

```
  CDBOpts.CompileCommandsDir = CompileCommandsDir;
```

And now the guidance for using the flag is:

 * In most cases, you don't need it
 * If you're running clangd with a `--compile-commands-dir` option, run `clangd-indexer` with the same option

https://github.com/llvm/llvm-project/pull/175209