[clangd-dev] VS Code Clangd and CMake integration

Thu Nov 29 08:36:33 PST 2018

>
> But in VSCode, the build system is just another plugin that may or may not
> be installed, so a decoupled design seems most likely to work in many
> configurations.
>
If we could find a solution, it would of course be completely optional.
Clangd would use its normal discovery mechanism of compile_commands.json,
unless another plugin tells it to use a specific file or build dir. Maybe
the build system extension could even give Clangd separate info for each
C++ file if we want to be really flexible, but again, I don't know exactly
how that info would be sent to Clangd. I agree that it's not reasonable
that this info would be shared between different projects.

Something like this would be ideal:

   1. User opens a folder
   2. CMake runs
   3. A compile_commands.json is generated
   4. CMake Tools somehow tells Clangd where to find the file (this one is
   where the problem lies)
   5. Repeat steps 2-4 whenever CMake needs to generate again

Clangd needs to be able to determine the compile command given a filename,
> at any time, within the clangd process (i.e. without exchanging LSP
> messages with the editor).
>
I'll admit that this sounds rather tricky. But the build plugin would only
need to notify Clangd when it actually generates its compile_commands.json,
after which Clangd could keep and reuse that state.

> It is in fact possible as a command flag (it's spelled
> -compile-commands-dir).
>
Would this only be passed once when the process starts and then never
touched again? If so, then yeah this obviously doesn't work.

each tool (clangd, clang-tidy etc) needs to get this configuration in some
> appropriate way, it's not shared (whereas all clang-based tools share
> discovery logic for compile_commands.json)
>
Yeah, this does sound like a scalability issue if a user wants to use
multiple Clang tools, as the build system extension would have to
communicate with every one of them.

Maybe one idea is to stick to compile_commands.json but extend the way they
> are discovered, e.g. let the Cmake Tools plugin notify Clangd that a
> certain compile_commands file is relevant and should be considered, even if
> it's not in an ancestor dir of the source file.
>
This sounds like a fairly reasonable compromise. But then again, we would
still need a way for the build system extension to communicate this to
Clangd.

Given the following circumstances:

   - There is always only one Clangd instance which is shared between
   different projects/folders
   - Command line arguments can only be passed to Clangd when launching it
   - Command line arguments passed to Clangd will affect all different
   projects/folders
   - The only ways for VSCode to communicate with Clangd after launching it
   is via LSP or files that both VSCode and Clangd already know about
   - LSP can not be used for this purpose

then this problem might not actually be possible to solve. If all of this
would mean completely redesigning the way that Clangd is invoked, then yeah
maybe generating a symlink is the only way to go about this. I'm still not
fond of polluting the source tree with something that is not super obvious
that it was generated by an IDE/editor or what generated it, but maybe this
is just an inherent downside of using LSP rather than an integrated C++
library (not trying to discredit your work, it's still a great project!)...
Then the question is how the build system extension would decide where to
put the symlink. Would it always end up in the root folder? Or something
like CMAKE_SOURCE_DIR (these two are often the same, but I guess it's a
good idea to be specific)? What did you mean by symlinking src/.build to
build/? Sorry, I never really use symlinks so I'm not very familiar with
them. It seems like CMake tool actually has a config option to place a copy
(not a symlink) of compile_commands.json in a path specified by the user in
the config. Though I think this has to be done automatically without the
user having to go into the config and set it up, maybe after clicking a
"yes" button that pops up in VSCode or something. Otherwise most users will
probably just assume that the two plugins simply don't work well together
(like I did). Could even have a notification to warn the user to include
the file in their VCS ignore file. Of course, this is in the hands of CMake
Tools and other build system extensions, just thought it was relevant to
the discussion.

It's... not trivial. (Other clang tools don't handle this as they tend to
> be batch)
> So we'll fix it, but it might take a little time :-)
>
I'm not too worried about time, just good to hear that someone is working
on it. :)

On Wed, 28 Nov 2018 at 23:59, Sam McCall <sammccall at google.com> wrote:

> On Wed, Nov 28, 2018, 23:00 Anton Södergren <anton.sodergren at gmail.com
> wrote:
>
>> I'm not familiar with CMake Tools, I guess it helps with editing
>>> CMakeLists, CMake configs, and running configure/generate?
>>
>> Except for the file editing part (that's handled by another extension),
>> you're pretty much right. It also allows for building and launching
>> debuggers. But it's the configure/generate part that's the most interesting
>> in our case.
>>
>> Clangd works like other clang-based tools. Starting at the source file in
>>> question, it walks up directories until it finds compile_commands.json
>>> <https://clang.llvm.org/docs/JSONCompilationDatabase.html>.
>>>
>> So it turned out that it indeed seemed to generate a
>> compile_commands.json file in the build folder (at least yesterday, today
>> it doesn't seem to work well at all). I tried copying that file over to my
>> project root folder and Clangd found it and started working correctly
>> indeed!
>>
>> The current best-practice is to manually create a symlink
>>> src/compile_commands.json -> build/compile_commands.json. I think
>>> symlinking src/.build to build/ is a slightly more principled approach, but
>>> something still has to create a symlink. (Maybe we could get CMake or CMake
>>> Tools to do it?)
>>>
>> While this is a possible approach, I think is sounds a little unwieldy
>> (though that might just be personal preference). When I looked around a bit
>> more, I saw that the Clang tooling docs
>> <https://clang.llvm.org/docs/HowToSetupToolingForLLVM.html> mentioned
>> that it's possible to pass the build path directly to Clang tools with a -p
>> parameter, is this possible in Clangd too?
>>
> It is in fact possible as a command flag (it's spelled
> -compile-commands-dir). It works in a pinch, but it's a subtly broken idea
> and I don't think it's the right long-term direction.
>
> 1) a clangd instance is typically 1:1 with an editor session. But an
> editor can open files from multiple projects, so what should the flag value
> be?
> 2) how does the editor know which dir to pass to clangd? If it's an editor
> preference, how does this work for users who work in different projects at
> different times?
> 3) each tool (clangd, clang-tidy etc) needs to get this configuration in
> some appropriate way, it's not shared (whereas all clang-based tools share
> discovery logic for compile_commands.json)
>
> Requiring the symlink (and adding it to gitignore etc) is a little fiddly
> and certainly unconventional, but it's the best thing we've found so far.
>
> For example, other IDE's that utilize clang-tidy (e.g. CLion) seem to do
>> so without any visible symlinks.
>>
> Yes, if you're an IDE that already knows how to build the code (has
> opinions/integrations for the build system) then directly injecting the
> configuration is feasible. (Clangd has LSP extensions for this). But in
> VSCode, the build system is just another plugin that may or may not be
> installed, so a decoupled design seems most likely to work in many
> configurations.
>
> To me, that sounds more robust and clean than having to pollute the source
>> tree with symlinks. Of course, that would require some work probably on
>> both of the plugins involved. Also, having a general and flexible approach
>> for this would be nice, as I'm also interested in getting work on a build2
>> extension going, which would require the same functionality.
>>
> Definitely open to new ideas here, but I'm not sure exactly what you're
> proposing.
>
> Clangd needs to be able to determine the compile command given a filename,
> at any time, within the clangd process (i.e. without exchanging LSP
> messages with the editor).
>
> Maybe one idea is to stick to compile_commands.json but extend the way
> they are discovered, e.g. let the Cmake Tools plugin notify Clangd that a
> certain compile_commands file is relevant and should be considered, even if
> it's not in an ancestor dir of the source file.
>
> clangd has to reload the data when appropriate. Today, it caches forever
>>> :-( Ilya is working on fixing it!
>>>
>> I did notice this problem when testing it out yesterday. But I trust that
>> it will get fixed then!
>>
> It's... not trivial. (Other clang tools don't handle this as they tend to
> be batch)
> So we'll fix it, but it might take a little time :-)
>
>
>> Cheers,
>> Anton
>>
>> On Tue, 27 Nov 2018 at 20:25, Sam McCall <sammccall at google.com> wrote:
>>
>>> On Tue, Nov 27, 2018 at 8:09 PM Anton Södergren via clangd-dev <
>>> clangd-dev at lists.llvm.org> wrote:
>>>
>>>> Hello! I'm not sure if this is the best place to ask, but it's better
>>>> to get a discussion going rather than just being confused.
>>>>
>>> Right place! Welcome :-)
>>>
>>>
>>>> I tried using Clangd in Visual Studio Code, but noticed that it doesn't
>>>> integrate well with the CMake Tools extension, which makes it hard to use
>>>> together with a CMake workflow. I would like a way for Clangd to
>>>> automatically get the correct info when CMake runs in the editor.
>>>> Specifically, I want seamless and automatic integration between these
>>>> two extensions without the user having to do any additional config:
>>>>
>>>>    - vscode-clangd (
>>>>    https://marketplace.visualstudio.com/items?itemName=llvm-vs-code-extensions.vscode-clangd
>>>>    )
>>>>    - CMake Tools (
>>>>    https://marketplace.visualstudio.com/items?itemName=vector-of-bool.cmake-tools
>>>>    )
>>>>
>>>> I'm not familiar with CMake Tools, I guess it helps with editing
>>> CMakeLists, CMake configs, and running configure/generate?
>>> (Sorry, I would dig into it, about to head out on vacation)
>>>
>>>
>>>> I see three possible approaches for this:
>>>>
>>>>    1. Work with the author of CMake Tools to make the extension
>>>>    generate the appropriate info for vscode-clangd
>>>>    2. Make vscode-clangd retrieve the info from CMake Tools somehow
>>>>    3. Create a new extension that acts as a bridge between the two
>>>>
>>>> Or alternatively, some combination of the different approaches.
>>>> I know that the CMake Tools extension integrates well with Microsoft's
>>>> C/C++ extension, so maybe it would be a good idea to do this in a similar
>>>> way that they did it. If my understanding is correct, a separate bridge
>>>> extension was previously used for this, but I think they now use a
>>>> combination of something similar to approach 1 and 2 for the extensions to
>>>> communicate with each other.
>>>> I would be interested to hear what you all think would be the best way
>>>> to go about this, and maybe I can get in touch with vector-of-bools (the
>>>> author of the CMake Tools extension) about it too.
>>>>
>>>> Another question I had regarding this: how does Clangd find the correct
>>>> include paths and other info in the first place? Does it use some kind of
>>>> compilation commands file for this? Is it possible to tell Clangd where to
>>>> look for such a file? I couldn't really find any documentation about this
>>>> when I looked around.
>>>>
>>> So if my understanding of CMake Tools is correct, these questions are
>>> very much related :-)
>>>
>>> Clangd works like other clang-based tools. Starting at the source file
>>> in question, it walks up directories until it finds
>>> compile_commands.json
>>> <https://clang.llvm.org/docs/JSONCompilationDatabase.html>. This lists
>>> the flags required to compile each file in its scope.
>>>
>>> So I think there are three relevant parts here:
>>>
>>>    - this file has to be created somehow. CMake can do this, it's
>>>    controlled by CMAKE_EXPORT_COMPILE_COMMANDS
>>>    <https://cmake.org/cmake/help/v3.5/variable/CMAKE_EXPORT_COMPILE_COMMANDS.html>,
>>>    which I believe is on by default now. It writes the files into the root of
>>>    the build tree.
>>>    - the file has to be discovered somehow (if the CMake build tree !=
>>>    the source tree). The current best-practice is to manually create a symlink
>>>    src/compile_commands.json -> build/compile_commands.json. I think
>>>    symlinking src/.build to build/ is a slightly more principled approach, but
>>>    something still has to create a symlink. (Maybe we could get CMake or CMake
>>>    Tools to do it?). Or maybe there's some other clever way to find the build
>>>    tree.
>>>    - clangd has to reload the data when appropriate. Today, it caches
>>>    forever :-( Ilya is working on fixing it!
>>>
>>> Gotta run, hope this helps!
>>> Patches and ideas welcome :-)
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/clangd-dev/attachments/20181129/f7686ea1/attachment-0001.html>