[cfe-dev] ClangD

Thu Jan 26 06:32:54 PST 2017

On 01/25/2017 02:11 PM, Manuel Klimek via cfe-dev wrote:
> Hi fellow clang devs,
>
> we wanted to let you know that we're (finally) starting up work on
> ClangD, which you might know from email threads such as [1] (June 2012!).
>
> In the meantime, YCM had done a good enough job at packaging up a
> libclang interface to our favorite editors, and other priorities (like
> modules) have eaten up a lot of folks priority lunches.
>
> What has changed?
> 1. YCM is starting to develop more and more into a language multiplexer,
> with other languages (Go, Typescript, etc) providing their own servers
> to talk to
> 2. MS has created a language server protocol [2], which already has both
> a bunch of client and server implementations
> 3. Debugging through python into libclang crashers is a pain and eating
> our support resources
> 4. While libclang is a good abstraction if you want to have something
> run in your process with close coupling, a standard protocol like the
> language server protocol seems like a better way to allow fast
> iterations on the server implementation without the need to keep
> backward-compatibility hacks through a restrictive C API.
>
> One of the cool things about the MS language server protocol is that it
> seems to not actually do any networking, which means that we do not need
> to introduce any new dependencies into clang-tools-extra, which is where
> we want to start developing this.
>
> If you have any thoughts / concerns let me know; otherwise look forward
> to code reviews on initial ClangD dropping by :D

[Huch, my first association with "ClangD" was support for the D language]

Hi!

We from the Qt Creator IDE team are also interested in this. :)

Some notes:

1. You've indirectly referenced the initial design document at [1]. Will 
this get an update or is it already updated somewhere else? An updated 
document clarifying the scope and goals would be probably useful to all 
interested parties, also in regard to contribution and how to align own 
current development that uses libclang/libtooling/clang tools.

E.g. Qt Creator also already launches a separate process 
("clangbackend") that makes use of libclang to provide the basic 
functionality like updated diagnostics, highlighting information and 
results for completion requests.

2. How is clangd related to "clang-refactor" [2]?

If it's not related at all, what's the state of clang-refactor?

3. I take this opportunity to share our biggest problems with libclang 
from the Qt Creator perspective since clangd users will probably face 
them also (as far as I understand, clangd will be based upon libclang - 
please correct me if I'm wrong). I share this in hope that they will be 
addressed or taken into consideration during the development of clangd 
for the benefit of all interested parties now that the development of 
clangd is announced. Please keep in mind that the actual problems might 
be in other components that libclang makes use of.

The problems we are facing with libclang, in this order, are:

3.1 Windows-specific issues

The whole performance of parse/reparse in general is worser than on 
Mac/Linux (probably also because of slow IO on Windows). At the end of 
the day, this makes it less usable on Windows for real-world 
projects/files. There are also other Windows-specific issues.

My gut feeling is that there aren't that many libclang users on Windows.

@Manual: To which extend do you or your team use and develop on Windows? 
Are any Windows developers working on clangd?

Concrete issues are:

  * Preamble is also generated for the second reparse on Windows
    https://llvm.org/bugs/show_bug.cgi?id=28535

  * Problems with FileManager caches leading to leaking file descriptors
    https://reviews.llvm.org/D27810

  * Very slow reparsing for big header and even slower with
    preamble-generation enabled (might be related to the one above)
    https://llvm.org/bugs/show_bug.cgi?id=31353

  * Windows: Crash with absolute path with forward slashes
    https://llvm.org/bugs/show_bug.cgi?id=28381

3.2 Synchronous/blocking API
Low-latency completion results at all times are impossible with a single 
CXTranslationUnit that is just being parsed or reparsed. This gets worse 
the longer the parse/reparse takes (hello Windows).

One can mitigate this by triggering the parse/reparse after a 
significant timeout of user-typing-inactivity, but this leads to a 
sluggish experience regarding new diagnostics and highlighting 
informations (you don't want to wait more than 2s for new diagnostics to 
see if your code is valid now).

Another approach is to use two CXTranslationUnits per file, so at least 
there is always a non-blocking CXTranslationUnits ready for completion 
requests. Downside is doubled memory consumption and potentially 
threading issues (@Manual: do they still exist or are they already 
resolved?).

3.3 libclang requires unsaved files to exist on disk.

If the unsaved file does not exist a preamble generation will happen for 
every reparse which simply takes to long.

For the concrete use case, refer to

	https://llvm.org/bugs/show_bug.cgi?id=31112

This one might be also a problem if you do version control checkouts and 
files suddenly vanish.

Best regards,
    Nikolai

[1] https://github.com/chandlerc/llvm-designs/blob/master/ClangService.rst
[2] http://lists.llvm.org/pipermail/cfe-dev/2016-September/050641.html

>
> Cheers,
> /Manuel
>
> [1] http://lists.llvm.org/pipermail/cfe-dev/2012-June/022028.html
> [2] https://github.com/Microsoft/language-server-protocol
>
>
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev