[clangd-dev] VS Code Clangd and CMake integration

Ilya Biryukov via clangd-dev clangd-dev at lists.llvm.org
Fri Dec 7 09:48:12 PST 2018


The idea of using an LSP extension to communicate compile_commands.json
produced by cmake to clangd should work, especially if we could also
include the root of the source tree, "covered" by this
compile_commands.json file (essentially the root of the project where
CMakeLists.txt lives). This should fit into the overall design pretty
nicely.

This way we would avoid polluting the source directory with symlinks and
will be able to communicate with clangd from VSCode using our conventional
LSP channel.

The downside is having yet another mechanism for configuring compilations,
but I would expect it to be unified with the current -compile-commands-dir
flag, so this should be acceptable.


Am Mi., 5. Dez. 2018, 19:40 hat Doug Schaefer via clangd-dev <
clangd-dev at lists.llvm.org> geschrieben:

> On Sat, 2018-12-01 at 18:40 +0100, Anton Södergren via clangd-dev wrote:
>
> So I would argue that the default lookup rather becomes the unusual
> case...
>
> I'm sorry, I don't quite understand what you mean by this. Could you
> clarify?
>
>
> I agree with Michael here. CMake generates the compile_commands.json file
> in the build directory which could be anywhere. I think it would make sense
> for the vscode extension to have an API to allow various build tools to
> configure clangd properly. We need to make sure this isn't CMake specific
> since all build tools in the end need to be able to generate this file.
> Mind you the compilation database is shared with other clang tools but
> ideally this all just works in an IDE setting without the user having to do
> additional setup.
>
> BTW, I'm still wondering if it makes sense for the vscode extension to be
> in the clangd repository. We clearly won't be adding all the client
> interfaces there. CDT has it's own to hook up to the Eclipse IDE. As the
> CDT project is creating a debug adapter with client interfaces for both
> Eclipse and VS Code (yes, we're doing both!), we occasionally have thoughts
> of doing VS Code client interfaces for other services. But we're still
> trying to get a feel for this new world.
>
> Doug.
>
>
> I bet it would be an easy thing to click into that interfacing for the
> clangd extension as well.
>
> I would agree that if Clangd could use the same mechanism as Microsoft's
> C/C++ extension, then it would probably be very easy to get CMake Tools to
> support it. However, Microsoft's extension clearly works very differently
> from Clangd.
> One difference is that their extension seems to handle a lot more things
> in TypeScript, while Clangd pretty much does the minimal amount in the
> vscode extension and delegates as much as possible to the Clangd program.
> Another difference is how the language server system processes are run. As
> stated before, Clangd launches only one process when it is first activated
> and keeps that process around until the application is closed.
> The Microsoft extension has multiple processes running at once. It seems
> to have one or two base processes or something and then also launches a new
> process for every code file that is opened. This may give the language
> server more freedom to isolate data, though I guess it also makes it harder
> to reuse AST's and such?
> The extension is open source, so one could of course look at the source
> code to get a better understanding of how they tackle the problem, but it's
> not a trivial codebase to dive into. I guess maybe even just asking them
> could work too... :P
>
> Cheers,
> Anton
>
> On Fri, 30 Nov 2018 at 02:58, Michael Steinberg via clangd-dev <
> clangd-dev at lists.llvm.org> wrote:
>
> Howdy,
>
> the "domain" of the two tools compiler and language server are completely
> different, where the former usually lives in the build-tree, the latter
> usually lives with the user inside the source tree... So I would argue that
> the default lookup rather becomes the unusual case...
>
> That being said, after long hard times the CMakeTools developer together
> with the Microsoft C++ extension team came up with an approach exactly like
> you depict somewhere around mid-year. But since the MS Extensions don't
> support "find references", it was still useless - which is why I'm
> currently using cquery. I bet it would be an easy thing to click into that
> interfacing for the clangd extension as well.
>
> Regards,
> Michael
> Am 29.11.2018 um 17:36 schrieb Anton Södergren via clangd-dev:
>
> But in VSCode, the build system is just another plugin that may or may not
> be installed, so a decoupled design seems most likely to work in many
> configurations.
>
> If we could find a solution, it would of course be completely optional.
> Clangd would use its normal discovery mechanism of compile_commands.json,
> unless another plugin tells it to use a specific file or build dir. Maybe
> the build system extension could even give Clangd separate info for each
> C++ file if we want to be really flexible, but again, I don't know exactly
> how that info would be sent to Clangd. I agree that it's not reasonable
> that this info would be shared between different projects.
>
> Something like this would be ideal:
>
>    1. User opens a folder
>    2. CMake runs
>    3. A compile_commands.json is generated
>    4. CMake Tools somehow tells Clangd where to find the file (this one
>    is where the problem lies)
>    5. Repeat steps 2-4 whenever CMake needs to generate again
>
>
> Clangd needs to be able to determine the compile command given a filename,
> at any time, within the clangd process (i.e. without exchanging LSP
> messages with the editor).
>
> I'll admit that this sounds rather tricky. But the build plugin would only
> need to notify Clangd when it actually generates its compile_commands.json,
> after which Clangd could keep and reuse that state.
>
> It is in fact possible as a command flag (it's spelled
> -compile-commands-dir).
>
> Would this only be passed once when the process starts and then never
> touched again? If so, then yeah this obviously doesn't work.
>
> each tool (clangd, clang-tidy etc) needs to get this configuration in some
> appropriate way, it's not shared (whereas all clang-based tools share
> discovery logic for compile_commands.json)
>
> Yeah, this does sound like a scalability issue if a user wants to use
> multiple Clang tools, as the build system extension would have to
> communicate with every one of them.
>
> Maybe one idea is to stick to compile_commands.json but extend the way
> they are discovered, e.g. let the Cmake Tools plugin notify Clangd that a
> certain compile_commands file is relevant and should be considered, even if
> it's not in an ancestor dir of the source file.
>
> This sounds like a fairly reasonable compromise. But then again, we would
> still need a way for the build system extension to communicate this to
> Clangd.
>
> Given the following circumstances:
>
>    - There is always only one Clangd instance which is shared between
>    different projects/folders
>    - Command line arguments can only be passed to Clangd when launching
>    it
>    - Command line arguments passed to Clangd will affect all different
>    projects/folders
>    - The only ways for VSCode to communicate with Clangd after launching
>    it is via LSP or files that both VSCode and Clangd already know about
>    - LSP can not be used for this purpose
>
> then this problem might not actually be possible to solve. If all of this
> would mean completely redesigning the way that Clangd is invoked, then yeah
> maybe generating a symlink is the only way to go about this. I'm still not
> fond of polluting the source tree with something that is not super obvious
> that it was generated by an IDE/editor or what generated it, but maybe this
> is just an inherent downside of using LSP rather than an integrated C++
> library (not trying to discredit your work, it's still a great project!)...
> Then the question is how the build system extension would decide where to
> put the symlink. Would it always end up in the root folder? Or something
> like CMAKE_SOURCE_DIR (these two are often the same, but I guess it's a
> good idea to be specific)? What did you mean by symlinking src/.build to
> build/? Sorry, I never really use symlinks so I'm not very familiar with
> them. It seems like CMake tool actually has a config option to place a copy
> (not a symlink) of compile_commands.json in a path specified by the user in
> the config. Though I think this has to be done automatically without the
> user having to go into the config and set it up, maybe after clicking a
> "yes" button that pops up in VSCode or something. Otherwise most users will
> probably just assume that the two plugins simply don't work well together
> (like I did). Could even have a notification to warn the user to include
> the file in their VCS ignore file. Of course, this is in the hands of CMake
> Tools and other build system extensions, just thought it was relevant to
> the discussion.
>
> It's... not trivial. (Other clang tools don't handle this as they tend to
> be batch)
> So we'll fix it, but it might take a little time :-)
>
> I'm not too worried about time, just good to hear that someone is working
> on it. :)
>
>
>
>
> On Wed, 28 Nov 2018 at 23:59, Sam McCall <sammccall at google.com> wrote:
>
> On Wed, Nov 28, 2018, 23:00 Anton Södergren <anton.sodergren at gmail.com
> wrote:
>
> I'm not familiar with CMake Tools, I guess it helps with editing
> CMakeLists, CMake configs, and running configure/generate?
>
> Except for the file editing part (that's handled by another extension),
> you're pretty much right. It also allows for building and launching
> debuggers. But it's the configure/generate part that's the most interesting
> in our case.
>
> Clangd works like other clang-based tools. Starting at the source file in
> question, it walks up directories until it finds compile_commands.json
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__clang.llvm.org_docs_JSONCompilationDatabase.html&d=DwMFaQ&c=yzoHOc_ZK-sxl-kfGNSEvlJYanssXN3q-lhj0sp26wE&r=NrrbvTHWa2Nbp_kAN0Hl1o3lM1WAwSes64uBjxjNhMc&m=UHKL5pgLOckV1e5mDyrMfr9QjCd_ur-DOyrz6IwNYFU&s=YNYy_n5ndALY2rhjGtZ0lK4MyZKUwC16JBPh1Y_5yjQ&e=>
> .
>
> So it turned out that it indeed seemed to generate a compile_commands.json
> file in the build folder (at least yesterday, today it doesn't seem to work
> well at all). I tried copying that file over to my project root folder and
> Clangd found it and started working correctly indeed!
>
> The current best-practice is to manually create a symlink
> src/compile_commands.json -> build/compile_commands.json. I think
> symlinking src/.build to build/ is a slightly more principled approach, but
> something still has to create a symlink. (Maybe we could get CMake or CMake
> Tools to do it?)
>
> While this is a possible approach, I think is sounds a little unwieldy
> (though that might just be personal preference). When I looked around a bit
> more, I saw that the Clang tooling docs
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__clang.llvm.org_docs_HowToSetupToolingForLLVM.html&d=DwMFaQ&c=yzoHOc_ZK-sxl-kfGNSEvlJYanssXN3q-lhj0sp26wE&r=NrrbvTHWa2Nbp_kAN0Hl1o3lM1WAwSes64uBjxjNhMc&m=UHKL5pgLOckV1e5mDyrMfr9QjCd_ur-DOyrz6IwNYFU&s=dxoqy4G7DvYguPEcwFXYTMo_pz8eocQ67O-w1MaudII&e=>
> mentioned that it's possible to pass the build path directly to Clang tools
> with a -p parameter, is this possible in Clangd too?
>
> It is in fact possible as a command flag (it's spelled
> -compile-commands-dir). It works in a pinch, but it's a subtly broken idea
> and I don't think it's the right long-term direction.
>
> 1) a clangd instance is typically 1:1 with an editor session. But an
> editor can open files from multiple projects, so what should the flag value
> be?
> 2) how does the editor know which dir to pass to clangd? If it's an editor
> preference, how does this work for users who work in different projects at
> different times?
> 3) each tool (clangd, clang-tidy etc) needs to get this configuration in
> some appropriate way, it's not shared (whereas all clang-based tools share
> discovery logic for compile_commands.json)
>
> Requiring the symlink (and adding it to gitignore etc) is a little fiddly
> and certainly unconventional, but it's the best thing we've found so far.
>
> For example, other IDE's that utilize clang-tidy (e.g. CLion) seem to do
> so without any visible symlinks.
>
> Yes, if you're an IDE that already knows how to build the code (has
> opinions/integrations for the build system) then directly injecting the
> configuration is feasible. (Clangd has LSP extensions for this). But in
> VSCode, the build system is just another plugin that may or may not be
> installed, so a decoupled design seems most likely to work in many
> configurations.
>
> To me, that sounds more robust and clean than having to pollute the source
> tree with symlinks. Of course, that would require some work probably on
> both of the plugins involved. Also, having a general and flexible approach
> for this would be nice, as I'm also interested in getting work on a build2
> extension going, which would require the same functionality.
>
> Definitely open to new ideas here, but I'm not sure exactly what you're
> proposing.
>
> Clangd needs to be able to determine the compile command given a filename,
> at any time, within the clangd process (i.e. without exchanging LSP
> messages with the editor).
>
> Maybe one idea is to stick to compile_commands.json but extend the way
> they are discovered, e.g. let the Cmake Tools plugin notify Clangd that a
> certain compile_commands file is relevant and should be considered, even if
> it's not in an ancestor dir of the source file.
>
> clangd has to reload the data when appropriate. Today, it caches forever
> :-( Ilya is working on fixing it!
>
> I did notice this problem when testing it out yesterday. But I trust that
> it will get fixed then!
>
> It's... not trivial. (Other clang tools don't handle this as they tend to
> be batch)
> So we'll fix it, but it might take a little time :-)
>
>
> Cheers,
> Anton
>
> On Tue, 27 Nov 2018 at 20:25, Sam McCall <sammccall at google.com> wrote:
>
> On Tue, Nov 27, 2018 at 8:09 PM Anton Södergren via clangd-dev <
> clangd-dev at lists.llvm.org> wrote:
>
> Hello! I'm not sure if this is the best place to ask, but it's better to
> get a discussion going rather than just being confused.
>
> Right place! Welcome :-)
>
>
> I tried using Clangd in Visual Studio Code, but noticed that it doesn't
> integrate well with the CMake Tools extension, which makes it hard to use
> together with a CMake workflow. I would like a way for Clangd to
> automatically get the correct info when CMake runs in the editor.
> Specifically, I want seamless and automatic integration between these two
> extensions without the user having to do any additional config:
>
>    - vscode-clangd (
>    https://marketplace.visualstudio.com/items?itemName=llvm-vs-code-extensions.vscode-clangd
>    <https://urldefense.proofpoint.com/v2/url?u=https-3A__marketplace.visualstudio.com_items-3FitemName-3Dllvm-2Dvs-2Dcode-2Dextensions.vscode-2Dclangd&d=DwMFaQ&c=yzoHOc_ZK-sxl-kfGNSEvlJYanssXN3q-lhj0sp26wE&r=NrrbvTHWa2Nbp_kAN0Hl1o3lM1WAwSes64uBjxjNhMc&m=UHKL5pgLOckV1e5mDyrMfr9QjCd_ur-DOyrz6IwNYFU&s=ZlqetZsnGkKa4Z9kLz9iBbv-BRQYL4q1bV2Pj3cjJKA&e=>
>    )
>    - CMake Tools (
>    https://marketplace.visualstudio.com/items?itemName=vector-of-bool.cmake-tools
>    <https://urldefense.proofpoint.com/v2/url?u=https-3A__marketplace.visualstudio.com_items-3FitemName-3Dvector-2Dof-2Dbool.cmake-2Dtools&d=DwMFaQ&c=yzoHOc_ZK-sxl-kfGNSEvlJYanssXN3q-lhj0sp26wE&r=NrrbvTHWa2Nbp_kAN0Hl1o3lM1WAwSes64uBjxjNhMc&m=UHKL5pgLOckV1e5mDyrMfr9QjCd_ur-DOyrz6IwNYFU&s=DZnchJoFbs6DeiH5qFEgRjgHOs4JHejJik568nXmogg&e=>)
>
>
>
> I'm not familiar with CMake Tools, I guess it helps with editing
> CMakeLists, CMake configs, and running configure/generate?
> (Sorry, I would dig into it, about to head out on vacation)
>
>
> I see three possible approaches for this:
>
>    1. Work with the author of CMake Tools to make the extension generate
>    the appropriate info for vscode-clangd
>    2. Make vscode-clangd retrieve the info from CMake Tools somehow
>    3. Create a new extension that acts as a bridge between the two
>
> Or alternatively, some combination of the different approaches.
> I know that the CMake Tools extension integrates well with Microsoft's
> C/C++ extension, so maybe it would be a good idea to do this in a similar
> way that they did it. If my understanding is correct, a separate bridge
> extension was previously used for this, but I think they now use a
> combination of something similar to approach 1 and 2 for the extensions to
> communicate with each other.
> I would be interested to hear what you all think would be the best way to
> go about this, and maybe I can get in touch with vector-of-bools (the
> author of the CMake Tools extension) about it too.
>
> Another question I had regarding this: how does Clangd find the correct
> include paths and other info in the first place? Does it use some kind of
> compilation commands file for this? Is it possible to tell Clangd where to
> look for such a file? I couldn't really find any documentation about this
> when I looked around.
>
> So if my understanding of CMake Tools is correct, these questions are very
> much related :-)
>
> Clangd works like other clang-based tools. Starting at the source file in
> question, it walks up directories until it finds compile_commands.json
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__clang.llvm.org_docs_JSONCompilationDatabase.html&d=DwMFaQ&c=yzoHOc_ZK-sxl-kfGNSEvlJYanssXN3q-lhj0sp26wE&r=NrrbvTHWa2Nbp_kAN0Hl1o3lM1WAwSes64uBjxjNhMc&m=UHKL5pgLOckV1e5mDyrMfr9QjCd_ur-DOyrz6IwNYFU&s=YNYy_n5ndALY2rhjGtZ0lK4MyZKUwC16JBPh1Y_5yjQ&e=>.
> This lists the flags required to compile each file in its scope.
>
> So I think there are three relevant parts here:
>
>    - this file has to be created somehow. CMake can do this, it's
>    controlled by CMAKE_EXPORT_COMPILE_COMMANDS
>    <https://urldefense.proofpoint.com/v2/url?u=https-3A__cmake.org_cmake_help_v3.5_variable_CMAKE-5FEXPORT-5FCOMPILE-5FCOMMANDS.html&d=DwMFaQ&c=yzoHOc_ZK-sxl-kfGNSEvlJYanssXN3q-lhj0sp26wE&r=NrrbvTHWa2Nbp_kAN0Hl1o3lM1WAwSes64uBjxjNhMc&m=UHKL5pgLOckV1e5mDyrMfr9QjCd_ur-DOyrz6IwNYFU&s=ukq8rlXh4A6sEO5H9T4qExkZ5nEYfkrg8hT5djzxXSg&e=>,
>    which I believe is on by default now. It writes the files into the root of
>    the build tree.
>    - the file has to be discovered somehow (if the CMake build tree !=
>    the source tree). The current best-practice is to manually create a symlink
>    src/compile_commands.json -> build/compile_commands.json. I think
>    symlinking src/.build to build/ is a slightly more principled approach, but
>    something still has to create a symlink. (Maybe we could get CMake or CMake
>    Tools to do it?). Or maybe there's some other clever way to find the build
>    tree.
>    - clangd has to reload the data when appropriate. Today, it caches
>    forever :-( Ilya is working on fixing it!
>
> Gotta run, hope this helps!
> Patches and ideas welcome :-)
>
>
>
>
> _______________________________________________
>
> clangd-dev mailing list
>
> clangd-dev at lists.llvm.org
>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/clangd-dev <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.llvm.org_cgi-2Dbin_mailman_listinfo_clangd-2Ddev&d=DwMFaQ&c=yzoHOc_ZK-sxl-kfGNSEvlJYanssXN3q-lhj0sp26wE&r=NrrbvTHWa2Nbp_kAN0Hl1o3lM1WAwSes64uBjxjNhMc&m=UHKL5pgLOckV1e5mDyrMfr9QjCd_ur-DOyrz6IwNYFU&s=FtK0UBeTqzNbxmTfoelYI2JuG8BntVEPnvq8nCeBtTI&e=>
>
>
> _______________________________________________
> clangd-dev mailing list
> clangd-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/clangd-dev
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.llvm.org_cgi-2Dbin_mailman_listinfo_clangd-2Ddev&d=DwMFaQ&c=yzoHOc_ZK-sxl-kfGNSEvlJYanssXN3q-lhj0sp26wE&r=NrrbvTHWa2Nbp_kAN0Hl1o3lM1WAwSes64uBjxjNhMc&m=UHKL5pgLOckV1e5mDyrMfr9QjCd_ur-DOyrz6IwNYFU&s=FtK0UBeTqzNbxmTfoelYI2JuG8BntVEPnvq8nCeBtTI&e=>
>
> _______________________________________________
>
> clangd-dev mailing list
>
> clangd-dev at lists.llvm.org
>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.llvm.org_cgi-2Dbin_mailman_listinfo_clangd-2Ddev&d=DwIGaQ&c=yzoHOc_ZK-sxl-kfGNSEvlJYanssXN3q-lhj0sp26wE&r=NrrbvTHWa2Nbp_kAN0Hl1o3lM1WAwSes64uBjxjNhMc&m=UHKL5pgLOckV1e5mDyrMfr9QjCd_ur-DOyrz6IwNYFU&s=FtK0UBeTqzNbxmTfoelYI2JuG8BntVEPnvq8nCeBtTI&e=
>
>
> _______________________________________________
> clangd-dev mailing list
> clangd-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/clangd-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/clangd-dev/attachments/20181207/462499c0/attachment-0001.html>


More information about the clangd-dev mailing list