[clangd-dev] Clangd + XPC

Tue Aug 14 05:37:13 PDT 2018

> On 13 Aug 2018, at 17:10, Ilya Biryukov <ibiryukov at google.com> wrote:
> 
> Thanks for the details.
> The "json-inside-xpc" looks a bit hacky on first notice, but the adapter is actually trying to solve a different problem of dispatching requests to the correct clangd instance based on the workspace.
> In that case, it does make sense to avoid losing time converting between the protocols.

Right, the other motivation is that XPC is a prescribed form of IPC for client-service on macOS and actually provides more user-friendly API than POSIX. So we need to use it but there’s no specific requirement *how* to use it and actually this form of optimization is also used by some other services.
Anyway, we could consider this just an implementation detail and add a minimal JSON string -> xpc_dictionary conversion for LSP as a library for clangd XPC consumers to use. Then we could label the XPC representation as a stable API. Not sure if that’s necessary though. What do you think?

> 
> One thing that might turn out to be problematic with the 'clangd-per-workspace' approach is that clangd manages some resource (ASTs, worker threads) and, obviously, can only do that inside a single process, so there are possibilities of resource starvation if too many clangd instances are running at the same time and don't know about each other.

We were also discussing that and our main concern were possibly duplicate ASTs in different clangd processes’ memory. Since our supported use-case is "just a few workspaces” (think 1-3 with let’s say 5 being already an extreme case) we concluded that for the time being it’s fine. In general we’d just use some reasonable limits for resources (e. g. -j 10 or whatever value we find optimal for worker threads). For general memory consumption we’ll need to implement some strategy anyway since in our case clangd would be a long-lived process running on developer’s machine. We were thinking about keeping N least recently used ASTs in memory and dropping the rest. 

> 
> 
> On Fri, Aug 10, 2018 at 7:01 PM Jan Korous <jkorous at apple.com <mailto:jkorous at apple.com>> wrote:
>> 
>> On 9 Aug 2018, at 18:00, Ilya Biryukov <ibiryukov at google.com <mailto:ibiryukov at google.com>> wrote:
>> 
>> Hi Jan,
>> 
>> Thanks for the update! It's great that we're after a solution that works at the protocol boundary, that does sound like something that should be easier to maintain and not too complicated.
>> I have skimmed through the patch, and as far as I can tell the idea is to forward raw JSON messages back and forth via XPC without actually converting from JSON to XPC and back.
>> Am I reading it correctly?
>> 
> 
> Hi Ilya,
> 
> Yes that’s correct - we did some performance measurements and it turned out that for JSON containing a lot of sub-objects (think 40k CompletionItem[]) the cost of conversion to XPC dictionary is actually higher than what we gain at IPC. Just sending string over XPC is faster.
> Since we might possibly need to use ranking and filtering logic on client side this use-case is interesting for us.
> 
> Cheers,
> 
> Jan
> 
>> On Wed, Aug 8, 2018 at 5:38 PM Jan Korous via clangd-dev <clangd-dev at lists.llvm.org <mailto:clangd-dev at lists.llvm.org>> wrote:
>> Hi all,
>> 
>> Based on our discussion with Sam in https://reviews.llvm.org/D48559 <https://reviews.llvm.org/D48559> we talked about our requirements internally and ultimately decided to change our approach.
>> 
>> We are going to use serialized JSON LSP over XPC instead of custom transport layer and we aren’t going to indroduce XPC-specific code to clangd binary itself which has several advantages.
>> 
>> I put our current design for review to continue the discussion.
>> https://reviews.llvm.org/D50452 <https://reviews.llvm.org/D50452>
>> 
>> I would like to thank you all for the great feedback you gave us! It helped us to precisely pinpoint our requirements.
>> 
>> Thanks.
>> 
>> Jan
>> _______________________________________________
>> clangd-dev mailing list
>> clangd-dev at lists.llvm.org <mailto:clangd-dev at lists.llvm.org>
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/clangd-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/clangd-dev>
>> 
>> 
>> -- 
>> Regards,
>> Ilya Biryukov
> 
> 
> 
> -- 
> Regards,
> Ilya Biryukov

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/clangd-dev/attachments/20180814/43b097f0/attachment-0001.html>