[cfe-dev] Announcing Crange

[Anurag]

Announcing Crange: https://github.com/crange/crange

Summary
-------

Crange is a tool to index and cross-reference C/C++ source code. It
can be used to generate tags database that can help with:

* Identifier definitions
* Identifier declaraions
* References
* Expressions
* Operators
* Symbols
* Source range

The source metadata collected by Crange can help with building tools
to provide cross-referencing, syntax highlighting, code folding and
deep source code search.


Rationale
---------

I was looking for tools that can extract and index identifiers present
in C/C++ source code and can work with large code bases.

Considering the amount of data Clang can generate while traversing
very large C/C++ projects (like, Linux), I decided against using
ctags/etags style tags database. Crange uses SQLite based tags
database to store identifiers and metadata, and uses SQLite's bulk
insert capabilities wherever possible.

I've used python's multiprocessing library to parallelize translation
unit traversal and metadata extraction from identifiers. Its possible
to control the number of jobs using -j command line option.


Usage example
-------------

Generating tags database for Linux 3.13.5

  $ cd linux-3.13.5
  $ crtags -v -j 32 .
  Parsing fs/xfs/xfs_bmap_btree.c (count: 1)
  Indexing fs/xfs/xfs_bmap_btree.c (nodes: 379, qsize: 0)
  ...
  Parsing sound/soc/codecs/ak4641.h (count: 34348)
  Generating indexes

This would create a new file named tags.db containing all the
identified tags.

Search all declarations for identifier named device_create

  $ crange device_create

Search all references for identifier named device_create

  $ crange -r device_create

Not all command line options are available though (-b, -k etc.), as
the tool is still in development.

Performance
-----------

Running crtags on Linux kernel v3.13.5 sources (containing 45K files,
size 614MB) took a little less than 7 hours (415m10.974s) on 32 CPU
Xeon server with 16GB of memory and 32 jobs. The generated tags.db
file was 22GB in size and contained 60,461,329 unique identifiers.

Installation
------------

  $ sudo python setup.py install
or
  $ sudo pip install crange

Feedback
--------

I would highly appreciate any feedback on improving this tool and
making it more useful. Also, clang-ctags and python-clang test suite
were of great help, thank you guys!

Anurag