[PATCH] D39050: Add index-while-building support to Clang

[Marc-Andre Laperle via Phabricator via cfe-commits]

malaperle added a comment.

In https://reviews.llvm.org/D39050#1040501, @akyrtzi wrote:

> > That would be good. How would one go about asking Clang to generate this extra information? Would a Clang Plugin be suitable for this?
>
> That's an interesting idea that we could explore, but I don't have much experience with that mechanism to comment on.
>
> > Only the lexer was needed to get the end of the token
>
> Ok, that's interesting, not sure why Xcode is so fast to highlight, did you reuse same SourceManager/Lexer/buffers for occurrences from same file ? We'd definitely add the end-loc if we cannot come up with a mechanism to highlight fast enough without it.


I don't think Xcode is quite fast, it's about 10 times slower (although I'm not sure it really finished) than when I use my branch that has the end-loc. I would try end-locs in Xcode if I could, to compare :) So I don't really know where the bottleneck is in Xcode. Comparing oranges to oranges, it's 4 times slower without end-locs compared to with end-locs on my branch. I does use the same SourceManager for the 46K references and I verified that it uses the same buffers, etc.
I'll put the numbers here again for readability.

> For my little benchmark, I used the LLVM/Clang/Clangd code base which I queried for all references of "std" (the namespace) which is around 46K references in the index.
> 
> With end-loc in index: 3.45s on average (20 samples)
>  With end-loc computed on the fly: 11.33s on average (20 samples)
>  I also tried with Xcode but without too much success: it took about 30 secs to reach 45K results and then carried on for a long time and hung (although I didn't try to leave it for hours to see if it finished).



>> I think it's useful to highlight something even when the name is not there. For example in "MyClass o1, o2;" it feels natural that o1 and o2 would get highlighted.
> 
> To clarify, even with implicit references the start loc points to something. In this case the implicit references can have start locs for the o1 and o2 identifiers and the end result for the UI will be the same (o1 and o2 get highlighted) even without having end-locs for all references.

It's the same but slower. IMO, the trade off is not great. It's entirely subjective but I think 4-10 times slower in order to save an integer per occurrence is not worth it from my point of view.

> Even without end-loc, the start loc could point to MyStringRef and you could highlight it.

(Same here, it's doable but faster if already in the index.)

>> It does? I can only seem to do a textual search.
> 
> The example I tried is the following. If you could file a bug report for the test case that did not work as you expected it would be much appreciated!

Sure, I'll give that a try and isolate it as much as I can. BTW, does it work for you on the LLVM code base?


https://reviews.llvm.org/D39050