[clang-tools-extra] [clangd] Add background index format support to clangd-indexer (PR #175209)

[Nathan Ridge via cfe-commits]

================
@@ -123,6 +156,107 @@ class IndexActionFactory : public tooling::FrontendActionFactory {
   RelationSlab::Builder Relations;
 };
 
+// Action factory that writes per-file shards (for sharded index format).
+// Each TU's index data is sharded independently — no merging across TUs.
+// Header shards are deduplicated: if a header's content hasn't changed since
+// the last time we wrote its shard, we skip writing it again.
+class ShardedIndexActionFactory : public tooling::FrontendActionFactory {
+public:
+  ShardedIndexActionFactory(BackgroundIndexStorage &Storage)
+      : Storage(Storage) {}
+
+  std::unique_ptr<FrontendAction> create() override {
+    // Snapshot the current shard versions so the callback can check staleness
+    // without holding the lock during indexing.
+    llvm::StringMap<ShardVersion> Snapshot;
+    {
+      std::lock_guard<std::mutex> Lock(ShardVersionsMu);
+      Snapshot = ShardVersions;
+    }
+
+    SymbolCollector::Options Opts;
+    Opts.CountReferences = true;
----------------
HighCommander4 wrote:

`CountReferences` should be false for a sharded index, because loading a sharded index goes through `FileSymbols` which [does its own](https://searchfox.org/llvm/rev/aff1ba103ab3006242c72af33420a3437f4ff1db/clang-tools-extra/clangd/index/FileIndex.cpp#320) bookkeeping of reference counts, and indeed [asserts](https://searchfox.org/llvm/rev/aff1ba103ab3006242c72af33420a3437f4ff1db/clang-tools-extra/clangd/index/FileIndex.cpp#307-308) that the incoming shards have reference counts of zero.

On the other hand, there are two options we **should** be specifying here:

  * `CollectMainFileRefs` should be set to `true`, to match the [background indexer](https://searchfox.org/llvm/rev/aff1ba103ab3006242c72af33420a3437f4ff1db/clang-tools-extra/clangd/index/Background.cpp#308). (The fact that the monolithic index doesn't use it seems like an oversight to me; feel free to add it there too.)
  * Like the background indexer, we should use a `FileFilter` which [skips files](https://searchfox.org/llvm/rev/aff1ba103ab3006242c72af33420a3437f4ff1db/clang-tools-extra/clangd/index/Background.cpp#289-307) whose contents haven't changed, using the `ShardVersions` machinery we already have.

https://github.com/llvm/llvm-project/pull/175209