[cfe-dev] Clang-based indexer and code navigator

[David Röthlisberger]

On Sat, Mar 9, 2013 at 8:47 PM, Ryan Prichard <ryan.prichard at gmail.com> wrote:
> Hi,
> 
> I've been working on a source code navigation tool for C and C++ that uses
> Clang to index code.  I thought people on this list might be interested in
> it.

That's a great demo, a great teaser to draw people's attention -- people
want to know if it's going to be worth their time to install your tool,
and I think the demo helps a lot there. The installation & usage
instructions in the readme are great. Your "sw-btrace" tools looks
generally useful to other clang-based projects; we need to advertise
"this is how you generate a compile_commands.json without CMake".

What follows are a few random thoughts in no particular order:

Over the years I've tried a variety of code browsing tools (for C++ and
other languages too) but I never stuck with any of them for one simple
reason: They weren't integrated into my IDE. If I'm looking at a file I
don't want to switch to a different tool, find that file again, and
browse from there; then when I find the target, switch back to my IDE
and find *that* file. I use the term "IDE" loosely here; in my case it's
Emacs.

What we really need is a daemon that maintains an index database and
services requests from the IDE. A while ago Chandler Carruth posted a
design proposal for a "clang service daemon":
https://github.com/chandlerc/llvm-designs/blob/master/ClangService.rst

Anders Bakken wrote "rtags": https://github.com/Andersbakken/rtags/
which uses such an architecture, with (so far) a client for Emacs. It
doesn't use the compile_commands.json at all, but provides a wrapper
that you place on the PATH before cc, g++, etc.  Once rtags knows about
a file, it monitors the file for changes using inotify/kqueue.
Rtags will group source files into "projects" based on some heuristics.

Another option is to use clang to generate a database in a format
understood by existing tools. I'm thinking particularly of "cscope" here
(but I don't know the cscope database's capabilities in any detail).
That way you only need to write the indexer, and you get front-end
interfaces for several major editors for free.

As you know, the clang C++ API is unstable. What is your plan for
maintaining sourceweb? Have you evaluated the stable C api ("libclang")?
Is there anything missing from libclang that prevented you from using
it? If the missing areas aren't huge, perhaps effort would be better
spent improving the libclang API than keeping an out-of-tree tool in
sync with the C++ API.

If libclang is an option, then you can use its python bindings. I can't
help feeling that C++ is the wrong language for this kind of thing. 10k
lines is quite a lot of code. (But maybe I'm entirely mistaken here
about the capabilities of libclang.)

I realise this is all sounding rather negative and for that I apologise.
Sourceweb is certainly far more polished and feature-full than my own
attempt at solving C++ indexing (
https://github.com/drothlis/clang-ctags ). I'm just trying to share what
I, personally, look for in such a tool, for whatever that's worth. :-)

Dave.