[cfe-dev] RFC: A proposal for a Clang-based service architecture

[Sean Silva]

A couple things.

We don't want .clangrc files just being willy nilly all over the place, and
we don't want them tied at all to a particular person's machine. The
.clangrc files should be a part of the project and checked into VCS. I
think that the way .gitignore files work is a good model, e.g. how you have
a canonical one in the top-level directory of your source tree, and then
you can customize on a per-directory basis. I'm not sure what you expect to
be in these files, but the thought of each person having their .clangrc
files strewn about their computer frightens me (difficult to set up).
Perhaps .clangdconfig or something is a better name for these files, since
rc files generally are for personal (per-user) settings (we can have those
too, but for options like how to sort search results, which is a
user-preference).

A use case that I think is worth considering is the case of completing a
TableGen'd diagnostic (e.g. diag::err_missing_typ<complete>). By necessity,
this service will need to be able to find any generated files so that it
can look in them. Thus, clangd has to either
1) be aware of the build system
2) or that the build system has to generate files that appropriately inform
clangd about the setup.
I think that 1 is clearly undesirable since we don't want to code into
clangd the idiosyncrasies of every build system known to man. Therefore, 2
it is.

As for the proposal and implementation strategy, I disagree with the
implementation strategy and think that the rest of the proposal is a bit
premature (things like settling on using LLVM bitcode format for the
protocol). With the current implementation strategy, you have no way of
knowing whether you are building "the right thing" until it is too late.
For example, there is nothing in the proposal about having this be easy to
use and setup.

I think that the following should work before anything else happens (based
roughly on how I set up my dev environment):

git clone http://llvm.org/git/llvm.git
cd llvm/tools
git clone http://llvm.org/git/clang.git
cd ../../
mkdir release
cd release
cmake -G Ninja ../source /* ... other flags ... */
-DLLVM_ENABLE_CLANGD_INTEGRATION=ON
ninja clangd-init

At this point, I should be able to do something like:
clang-cli complete --file=llvm/tools/clang/lib/Sema/SemaStmt.cpp
--line=1302 --column=24
and get completions. Since clangd knows about the project, it knows exactly
what project to complete for based on just the filename.

At this point, stable binary interfaces don't have to happen yet, stable
IPC protocols don't have to happen yet, etc. Just something that works.

Now we're in business, and can make real progress:
* Too slow? --> Can benchmark and make it faster
* Wow it would be really useful if we could do X? --> Do X, and be able to
immediately test it and dogfood it.
* Oh shit, there are certain kinds of modifications of the source file that
cause extremely long delays that are completely unavoidable? -> come up
with a way to handle this, which might require rethinking the protocol or
client API.

Since this is primarily for interactive use, a clang-cli based RESTful
interface is all you need (I'd honestly say use JSON).  I don't like the
idea of having a persistent client-server connection since I don't want vim
to have a socket open constantly (emacs users probably would be comfortable
with that though ;). Also, what if I change directory while in Vim? Or have
two files open from different projects? Now vim has to handle the logic of
managing N connections and renegotiating sessions? That's just silly. If
the overhead of fork/exec'ing a new process ever becomes significant then
we will already be unnoticeably fast (try `:%!sort` on a large file, fast
no?).

I also think it is mind-numbingly stupid to add a socket I/O library into
LLVM/Clang. There is no reason why the Clang/LLVM part of clangd should
*ever* have to touch the network or have to include request-processing
logic. Leave that to node.js (or whatever). Let the people that are good
with that kind of thing mash it up and make cool things. We of course can
make our own cool things, but don't decrease the usefulness of LLVM/Clang
for this kind of thing by enshrining what will become "that shitty
web-server-written-from-scratch-by-compiler-writers-in-C++ that only talks
this weird binary format". Just because clang provides functionality useful
for GUI IDEs doesn't mean that you need to write a GUI library in-tree and
enshrine a specific IDE with Clang/LLVM. Similarly, it is braindead to
attempt to add a networking library to LLVM/Clang and enshrine a specific
server implementation in-tree. The networking code will always be total
crap, never production quality, and a huge pain in the ass for anyone that
wants to do anything nontrivial with it; for example, if users want to use
encryption over the network, are you going to add OpenSSL as a dependency
for LLVM? Or write a crypto library from scratch for LLVM? What if a
node.js user wants to use this? Are you going to make all the networking
code asynchronous?. What I'm talking about here of course doesn't preclude
stuff like fixing the fs layer, which is necessary for LLVM/Clang to be
useful in this capability (at production quality) in the first place.

To recap:
1) Get something working ASAP
2) Iterate, dogfooding it every step of the way.
3) Don't add things in-tree that are outside the domain of what LLVM/Clang
do (e.g. networking, GUIs, crypto, 3D-rendering, etc).
4) In-tree changes should focus (as always) on increasing the
utility/flexibility of the LLVM/Clang libraries to clients (e.g. can safely
be used multithreaded, etc), and should be motivated by compelling
use-cases derived from part 2)

--Sean Silva


On Tue, Jun 12, 2012 at 3:43 AM, Chandler Carruth <chandlerc at gmail.com>wrote:

> Greetings all!
>
> What follows is a fairly lengthy and detailed design document for a
> proposed persistent Clang server (or clangd in unix-terms) to serve as
> infrastructure for increasingly advanced and interactive C++ tools. It
> should generalize and build upon libclang, and will allow us to effectively
> target Vim, Emacs, and other editors. This is something we're planning to
> pursue in the near term, so I'd appreciate any and all feedback.
>
> Here is a Google Docs link you can use to view and comment on the proposal:
>
> https://docs.google.com/document/d/1kNv2jJK0I0JGnxJxU6w5lUOlrIBoecU4fi9d_o5e2-c/edit
>
> Its interim home is on github here, where you can see the history and the
> actual rest version in all its glory:
> https://github.com/chandlerc/llvm-designs/blob/master/ClangService.rst
>
> I've also attached the text for email-based comments.
>
> Thanks!
> -Chandler
>
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20120612/d996f5b5/attachment.html>