<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">Hi all,</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">If you are following the cfe-commits mailing list, you may have noticed that a new 'Index' library (referred to as 'clangIndex' from now on) has "landed" on the clang repository recently.</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">I'd like to make a proper introduction of this new library.</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">ClangIndex is meant to provide the basic infrastructure for cross-translation-unit analysis and is primarily focused on indexing related functionality.</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">It provides an API for clients that need to accurately map the AST nodes of the ASTContext to the locations in the source files.</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">It also allows them to analyze information across multiple translation units.</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">As a "general rule", ASTContexts are considered the primary source of information that a client wants about a translation unit.</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">As a consequence, there will be no such class as an "indexing database" that stores, for example, source locations of identifiers separately from ASTContext.</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">All the information that a client needs from a translation unit will be extracted from the ASTContext.</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">Entity:</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">--------</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">To be able to reason about semantically the same Decls that are contained in multiple ASTContexts, the 'Entity' class was introduced.</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">An Entity is an ASTContext-independent "token" that can be created from a Decl (and a typename in the future) with the purpose to "resolve"</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">it into a Decl belonging to another ASTContext. Some examples to make the concept of Entities more clear:</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">t1.c:</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">void foo(void);</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">void bar(void);</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">t2.c:</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">void foo(void) {</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">}</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">Translation unit 't1.c' contains 2 Entities 'foo' and 'bar', while 't2.c' contains 1 Entity 'foo'.</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">Entities are uniqued in such a way that the Entity* pointer for 't1.c/foo' is the same as the Entity* pointer for 't2.c/foo'.</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">An Entity doesn't convey any information about the declaration, it is more like an opaque pointer used only to get the</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">associated Decl out of an ASTContext so that the actual information for the declaration can be accessed.</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">Another important aspect of Entities is that they can only be created/associated for declarations that are visible outside the</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">translation unit. This means that for:</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">t3.c:</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">static void foo(void);</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">there can be no Entity (if you ask for the Entity* of the static function 'foo' you'll get a null pointer).</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">This is for 2 reasons:</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">1) To preserve the invariant that the same Entity* pointers refer to the same semantic Decls.</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"> In the above example t1.c/foo and t2.c/foo are the same, while t3.c/foo is different.</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">2) The purpose of Entity is to get the same semantic Decl from multiple ASTContexts. For a Decl that is not visible</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"> outside of its own translation unit, you don't need an Entity since it won't appear in another ASTContext.</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">ASTLocation</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">-----------</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">Encapsulates a "point" in the AST tree of the ASTContext.</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">It represents either a Decl*, or a Stmt* along with its immediate Decl* parent.</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">An example for its usage is that clangIndex will provide the references of 'foo' in the form of ASTLocations, "pointing" at the expressions that reference 'foo'.</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'"><br></font></div><div><font class="Apple-style-span" face="'Andale Mono'"><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; ">ResolveLocationInAST</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; ">-------------------</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; ">A function that accepts an ASTContext and a SourceLocation which it resolves into an ASTLocation.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; ">DeclReferenceMap</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; ">---------------</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; ">Accepts an ASTContext and creates a mapping from NamedDecls to the ASTLocations that reference them (in the same ASTContext).</div></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px; "><br></span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">AST files</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">---------</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">The precompiled headers implementation of clang (<a href="http://clang.llvm.org/docs/PCHInternals.html">http://clang.llvm.org/docs/PCHInternals.html</a>) is ideal for storing an ASTContext in a compact form that</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">will be loaded later for AST analysis. An "AST file" refers to a translation unit that was "compiled" into a precompiled header file. </span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">index-test</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">----------</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">A command-line tool that exercises the clangIndex API, useful for testing the clangIndex features.</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">As input it accepts multiple AST files (representing multiple translation units) and a few options:</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"> -point-at [file:line:column]</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">Resolves a [file:line:column] triplet into a ASTLocation from the first AST file. If no other option is specified, it prints the ASTLocation.</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">It also prints a declaration's associated doxygen comment, if one is available (courtesy of Doug).</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"> -print-refs</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">Prints the ASTLocations that reference the declaration that was resolved out of the [file:line:column] triplet</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"> -print-defs</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">Prints the ASTLocations that define the resolved declaration</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"> -print-decls</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">Prints the ASTLocations that declare the resolved declaration</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" face="'Andale Mono'"><br></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">Here's an example of using index-test:</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">You have 3 files,</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">foo.h:</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">---------------------</span></font></div><div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">extern int global_var;</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">void foo_func(int param1);</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">void bar_func(void);</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">-----------------------</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">t1.c:</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">--------------------</span></font></div><div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">#include "foo.h"</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">void foo_func(int param1) {</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"> int local_var = global_var;</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"> for (int for_var = 100; for_var < 500; ++for_var) {</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"> local_var = param1 + for_var;</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"> }</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"> bar_func();</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">}</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">--------------------</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">t2.c:</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">----------------------</span></font></div><div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">#include "foo.h"</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">int global_var = 10;</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">void bar_func(void) {</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"> global_var += 100;</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"> foo_func(global_var);</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">}</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">-------------------------</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">You first get AST files out of t1.c and t2.c:</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">$ clang-cc -emit-pch t1.c -o t1.ast</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">$ clang-cc -emit-pch t2.c -o t2.ast</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">Find the ASTLocation under this position of t1.c:</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">.................</span></font></div><div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;">void foo_func(int param1) {</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'" size="3"><span class="Apple-style-span" style="font-size: 12px;"> int local_var = global_var;</span></font></div><div><font class="Apple-style-span" face="'Andale Mono'"> ^</font></div><div><font class="Apple-style-span" face="'Andale Mono'">...................</font></div><div><font class="Apple-style-span" face="'Andale Mono'"><br></font></div><div><font class="Apple-style-span" face="'Andale Mono'"><div>$ index-test t1.ast -point-at t1.c:4:23</div><div>[Decl: Var local_var | Stmt: DeclRefExpr global_var] <t1.c:4:19, t1.c:4:19></div><div><br></div><div>Find the declaration:</div><div><br></div><div><div>$ index-test t1.ast -point-at t1.c:4:23 -print-decls</div><div>[Decl: Var global_var] <foo.h:1:12, foo.h:1:12></div><div><br></div><div>Find the references:</div><div><br></div><div><div>$ index-test t1.ast t2.ast -point-at t1.c:4:23 -print-refs</div><div>[Decl: Var local_var | Stmt: DeclRefExpr global_var] <t1.c:4:19, t1.c:4:19></div><div>[Decl: Function bar_func | Stmt: DeclRefExpr global_var] <t2.c:6:3, t2.c:6:3></div><div>[Decl: Function bar_func | Stmt: DeclRefExpr global_var] <t2.c:7:12, t2.c:7:12></div><div><br></div><div>Find definitions:</div><div><br></div><div><div>$ index-test t1.ast t2.ast -point-at t1.c:4:23 -print-defs</div><div>[Decl: Var global_var] <t2.c:3:5, t2.c:3:18></div><div><br></div><div><br></div><div><br></div><div>This concludes the introduction to the clangIndex library, if you have any questions or comments please let me know.</div><div><br></div><div>-Argiris</div></div></div></div></font></div></div></div></div></div></body></html>