[cfe-dev] Clang-based indexer and code navigator

[Ryan Prichard]

On 03/12/2013 04:00 AM, David Röthlisberger wrote:
> On Sat, Mar 9, 2013 at 8:47 PM, Ryan Prichard <ryan.prichard at gmail.com> wrote:
>> Hi,
>>
>> I've been working on a source code navigation tool for C and C++ that uses
>> Clang to index code.  I thought people on this list might be interested in
>> it.
>
> That's a great demo, a great teaser to draw people's attention -- people
> want to know if it's going to be worth their time to install your tool,
> and I think the demo helps a lot there. The installation & usage
> instructions in the readme are great. Your "sw-btrace" tools looks
> generally useful to other clang-based projects; we need to advertise
> "this is how you generate a compile_commands.json without CMake".

The demo is particularly useful because the tool doesn't have a web 
interface.

sw-btrace is similar to the bear tool, which was also announced on this 
list.

> What follows are a few random thoughts in no particular order:
>
> Over the years I've tried a variety of code browsing tools (for C++ and
> other languages too) but I never stuck with any of them for one simple
> reason: They weren't integrated into my IDE. If I'm looking at a file I
> don't want to switch to a different tool, find that file again, and
> browse from there; then when I find the target, switch back to my IDE
> and find *that* file. I use the term "IDE" loosely here; in my case it's
> Emacs.

Make sense.  I think the idea is that the index and the IDE/GUI should 
be independent of one another?

> What we really need is a daemon that maintains an index database and
> services requests from the IDE. A while ago Chandler Carruth posted a
> design proposal for a "clang service daemon":
> https://github.com/chandlerc/llvm-designs/blob/master/ClangService.rst
>
> Anders Bakken wrote "rtags": https://github.com/Andersbakken/rtags/
> which uses such an architecture, with (so far) a client for Emacs. It
> doesn't use the compile_commands.json at all, but provides a wrapper
> that you place on the PATH before cc, g++, etc.  Once rtags knows about
> a file, it monitors the file for changes using inotify/kqueue.
> Rtags will group source files into "projects" based on some heuristics.

I hadn't seen rtags before.  I might take a look at it sometime.

> Another option is to use clang to generate a database in a format
> understood by existing tools. I'm thinking particularly of "cscope" here
> (but I don't know the cscope database's capabilities in any detail).
> That way you only need to write the indexer, and you get front-end
> interfaces for several major editors for free.
>
> As you know, the clang C++ API is unstable. What is your plan for
> maintaining sourceweb? Have you evaluated the stable C api ("libclang")?
> Is there anything missing from libclang that prevented you from using
> it? If the missing areas aren't huge, perhaps effort would be better
> spent improving the libclang API than keeping an out-of-tree tool in
> sync with the C++ API.

I don't really have a plan, other than hope that the Clang API doesn't 
change too quickly.  So far, the code has gone through one API 
transition, from 3.1 to 3.2, and the changes were minimal:

https://github.com/rprichard/sourceweb/commit/4183ca758d602e775af0a679d15bc84a3f37f287

I was using libclang at one point.  IIRC, I switched away from it 
because I was having trouble getting some information I wanted out of 
it.  In particular, I wanted to classify different kinds of references, 
e.g.:
  - For a function: is this a declaration, a definition, a call, or an 
address-of operation?
  - For a variable: is this a declaration, a definition, a read, a 
write, or an address-of operation?

I'm open to switching back to libclang, though, but that would depend on 
having enough time to work on the project.

> If libclang is an option, then you can use its python bindings. I can't
> help feeling that C++ is the wrong language for this kind of thing. 10k
> lines is quite a lot of code. (But maybe I'm entirely mistaken here
> about the capabilities of libclang.)
>
> I realise this is all sounding rather negative and for that I apologise.
> Sourceweb is certainly far more polished and feature-full than my own
> attempt at solving C++ indexing (
> https://github.com/drothlis/clang-ctags ). I'm just trying to share what
> I, personally, look for in such a tool, for whatever that's worth. :-)

Thanks for your feedback.

-Ryan