[cfe-dev] libclang: Dissecting token lists.

[Manuel Holtgrewe]

Dear all,

I want to use libclang for writing a code convention checker. I already 
have code in place that checks variable names for nameConventions() etc. 
which was very straightforward with the libclang API.

Currently, I'm trying to write indentation checks and this turns out to 
be a bit more involved. I want to start with top-level functions. The 
C++11 standard basically says that a function declarations consists of 
(1) an attribute specifier sequence, (2) a declaration specifier 
sequence, (3) the function declarator, and (4) the function body.

It is straightforward to seperate (4) from (1-3), but the rest appears 
to be harder. Can anyone give me any pointers on how to dissect the 
token list of the following small program? The output of c-index-test of 
clang-3.0 is below.

Thanks,
Manuel

template <typename T>
struct Identity
{
     typedef T Type;
};

int bar() { return 5; }

inline Identity<int>::Type const &
foo(int y = bar())
{
     static int i = 0;
     return i;
}

$ /opt/clang-3.0/bin/c-index-test 
-test-annotate-tokens=examples/tiny.cpp:1:1:100:0 examples/tiny.cpp
Keyword: "template" [1:1 - 1:9] ClassTemplate=Identity:2:8 (Definition)
Punctuation: "<" [1:10 - 1:11] ClassTemplate=Identity:2:8 (Definition)
Keyword: "typename" [1:11 - 1:19] TemplateTypeParameter=T:1:20 (Definition)
Identifier: "T" [1:20 - 1:21] TemplateTypeParameter=T:1:20 (Definition)
Punctuation: ">" [1:21 - 1:22] ClassTemplate=Identity:2:8 (Definition)
Keyword: "struct" [2:1 - 2:7] ClassTemplate=Identity:2:8 (Definition)
Identifier: "Identity" [2:8 - 2:16] ClassTemplate=Identity:2:8 (Definition)
Punctuation: "{" [3:1 - 3:2] ClassTemplate=Identity:2:8 (Definition)
Keyword: "typedef" [4:5 - 4:12] ClassTemplate=Identity:2:8 (Definition)
Identifier: "T" [4:13 - 4:14] TypeRef=T:1:20
Identifier: "Type" [4:15 - 4:19] TypedefDecl=Type:4:15 (Definition)
Punctuation: ";" [4:19 - 4:20] ClassTemplate=Identity:2:8 (Definition)
Punctuation: "}" [5:1 - 5:2] ClassTemplate=Identity:2:8 (Definition)
Punctuation: ";" [5:2 - 5:3]
Keyword: "int" [7:1 - 7:4] FunctionDecl=bar:7:5 (Definition)
Identifier: "bar" [7:5 - 7:8] FunctionDecl=bar:7:5 (Definition)
Punctuation: "(" [7:8 - 7:9] FunctionDecl=bar:7:5 (Definition)
Punctuation: ")" [7:9 - 7:10] FunctionDecl=bar:7:5 (Definition)
Punctuation: "{" [7:11 - 7:12] CompoundStmt=
Keyword: "return" [7:13 - 7:19] ReturnStmt=
Literal: "5" [7:20 - 7:21] IntegerLiteral=
Punctuation: ";" [7:21 - 7:22] CompoundStmt=
Punctuation: "}" [7:23 - 7:24] CompoundStmt=
Keyword: "inline" [9:1 - 9:7]
Identifier: "Identity" [9:8 - 9:16] TemplateRef=Identity:2:8
Punctuation: "<" [9:16 - 9:17] FunctionDecl=foo:10:1 (Definition)
Keyword: "int" [9:17 - 9:20] FunctionDecl=foo:10:1 (Definition)
Punctuation: ">" [9:20 - 9:21] FunctionDecl=foo:10:1 (Definition)
Punctuation: "::" [9:21 - 9:23] FunctionDecl=foo:10:1 (Definition)
Identifier: "Type" [9:23 - 9:27] TypeRef=Type:4:15
Keyword: "const" [9:28 - 9:33] FunctionDecl=foo:10:1 (Definition)
Punctuation: "&" [9:34 - 9:35] FunctionDecl=foo:10:1 (Definition)
Identifier: "foo" [10:1 - 10:4] FunctionDecl=foo:10:1 (Definition)
Punctuation: "(" [10:4 - 10:5] FunctionDecl=foo:10:1 (Definition)
Keyword: "int" [10:5 - 10:8] ParmDecl=y:10:9 (Definition)
Identifier: "y" [10:9 - 10:10] ParmDecl=y:10:9 (Definition)
Punctuation: "=" [10:11 - 10:12] ParmDecl=y:10:9 (Definition)
Identifier: "bar" [10:13 - 10:16] DeclRefExpr=bar:7:5
Punctuation: "(" [10:16 - 10:17] CallExpr=bar:7:5
Punctuation: ")" [10:17 - 10:18] CallExpr=bar:7:5
Punctuation: ")" [10:18 - 10:19] FunctionDecl=foo:10:1 (Definition)
Punctuation: "{" [11:1 - 11:2] CompoundStmt=
Keyword: "static" [12:5 - 12:11] DeclStmt=
Keyword: "int" [12:12 - 12:15] VarDecl=i:12:16 (Definition)
Identifier: "i" [12:16 - 12:17] VarDecl=i:12:16 (Definition)
Punctuation: "=" [12:18 - 12:19] VarDecl=i:12:16 (Definition)
Literal: "0" [12:20 - 12:21] IntegerLiteral=
Punctuation: ";" [12:21 - 12:22] DeclStmt=
Keyword: "return" [13:5 - 13:11] ReturnStmt=
Identifier: "i" [13:12 - 13:13] DeclRefExpr=i:12:16
Punctuation: ";" [13:13 - 13:14] CompoundStmt=
Punctuation: "}" [14:1 - 14:2] CompoundStmt=