[clang-tools-extra] [clangd] Drop the optimization where only shards for modified files are updated in the background index (PR #140651)

[via cfe-commits]

JVApen wrote:

> FWIW, my expectation as a user is that when a header is modified, that **does** trigger re-indexing of all source files that include the header. I don't think the performance impact of this is that catastrophic, as the number of headers in a project tends to vary inversely with how widely they're included (i.e. most headers will have a few includers, and a few will have many). In my mind, the scenario is comparable to modifying a header and then **rebuilding**: if the header is widely included, you will rebuild many TUs, and programmers expect that (and adopt practices like forward declarations, pimpl, etc. to avoid unnecessary header dependencies).

At ours, indexing the whole project is measured is hours. There are a couple of headers that are included a lot, like the header that suppresses compiler warnings. I don't think it adds value to reindex everything when such a commonly inserted header is touched.

I also don't believe that you can compare reindexing with recompilation. We do distributed compilation only on remote machines, such that your local machine is not unresponsive when you do touch that kind of headers. Assuming indexing and compilation take the same time, a 10x speedup would for a build of 10 minutes give you 1h40m of indexing. (Without any other changes)

As such, I think it would be reasonable to fallback to the current behavior if the header touches more than X files. Once X is large enough, I don't think that it adds a lot of value, as you don't want to wait for 1h for it to catch up in order to make use of it.

https://github.com/llvm/llvm-project/pull/140651