[clangd-dev] RFC: Semantic highlighting

[Jan Korous via clangd-dev]

> On Jan 16, 2019, at 1:06 AM, Sam McCall <sammccall at google.com> wrote:
> 
> Sounds great, Jan.
> This has been one of those things we'd hoped to get around to, but never quite made it to the top of the pile.
> The protocol looks pretty sensible to me. I'm a bit nervous that there doesn't seem to be a high-quality VSCode implementation to check our work against yet - I guess we can use Theia.

Our intended use-case is clangd talking to the recently announced SourceKit-LSP service:
https://github.com/apple/sourcekit-lsp <https://github.com/apple/sourcekit-lsp>

I haven’t really thought about details yet (including how to approach testing) but I assume we’ll have some integration tests in SourceKit-LSP. But it makes sense to test (at least manually) with some editor client too.

> Still, I think the best way to start is go ahead with a minimal implementation (without worrying about deltas etc yet) to demonstrate it'll work.

I agree.

> 
> Some issues to think about:
> 
> What's the basic implementation idea? Traverse the main-file AST after every edit, collect spans, diff them at the end? Do we do this synchronously as part of the diagnostics cycle? Latency seems likely to be ~1 second with this approach, and never under 500ms (because of the debouncing of continuous AST rebuilds).
> I don't see something fundamentally better, other than tracking dirty regions and only traversing decls that intersect them. My intuition says that wouldn't save much, as we have to build the whole AST, which should be much slower than traversing it.

Haven’t looked into this yet as I wanted to know what people think first. I’ll start by looking at how libclang does this.

> 
> What to do before first parse? (In particular when preamble is slow)
> I guess the client could fall back to its standard heuristic-parsing-based highlighting, use of TextMate scopes seems to suggest that.
> An alternative: for more uniformity/control, we could instead implement such parsing on the server side as a cheap fallback when ASTs aren't available. There may be other reasons to have such a heuristic parser. Probably doesn't make much sense to worry about initially.

I agree this is an enhancement which we can implement later.

> 
> How are re-highlights around the cursor going to feel in practice? I think there's a danger that if we highlight highlighting partially-formed expressions from a more-or-less-random point in editing, and the editor fills in the gaps, then we'll end up with basically random flashing colors on the line being edited.
> We should probably wait and see how it feels after a first implementation. If this is an issue, one idea would be to attempt some low-latency highlighting of the edited section (using heuristic parsing and/or the old AST) while we wait for the AST rebuild.

Sounds reasonable - on server side we can either annotate the hot zone quickly or not annotate it at all.

> 
> On Wed, Jan 16, 2019 at 12:08 AM Jan Korous via clangd-dev <clangd-dev at lists.llvm.org <mailto:clangd-dev at lists.llvm.org>> wrote:
> Hi,
> 
> I am starting to work on semantic highlighting in clangd.
> 
> There's currently no mention of that in LSP but it seems like this proposal is on the right path to get accepted:
> https://github.com/Microsoft/vscode-languageserver-node/pull/367 <https://github.com/Microsoft/vscode-languageserver-node/pull/367> 
> 
> TLDR: the proposal introduces new capabilities and server notification.
> - New client boolean capability flag.
> - New server capability - server shares a lookup table of semantic categories with client in response.
> - Semantic categories are defined by TextMate scopes. https://www.sublimetext.com/docs/3/scope_naming.html <https://www.sublimetext.com/docs/3/scope_naming.html>
> - Server sends notifications containing line number, column number, length and semantic category (index in the lookup table).
> - Server is responsible for sending only delta information (not full document every time).
> - Server is responsible for ignoring trivial shift edits to documents (e. g. newline at the start of the document).
> I think you mean *client* is responsible for that last one.

No, it’s really the server who’s responsible for ignoring these. I guess it makes sense since client can’t just decide to not send 'textDocument/didChange’ to server as they would get out of sync.
https://github.com/Microsoft/vscode-languageserver-node/pull/367/files#diff-0253ecfa5ea7304bd5d168e2665e2fd1R54
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/clangd-dev/attachments/20190116/cd0cebff/attachment-0001.html>