[cfe-users] Out-of-range tokens after clang_tokenize()

[Conrad Hoffmann]

Hello everybody,

I am new to the list, so please excuse if this isn't the right place to
ask and thanks for any pointer where to turn instead.
I am trying to write a C program that analyzes certain parts of a .c
file at token level, in this particular example I am interested in
function bodies. I use a visitor to find cursors of interest, get their
source code range (extent) and tokenize that range.
The docs for clang_tokenize() explicitly state: "All of the tokens
produced by tokenization will fall within this source range".

Unfortunately, in my case that is not working out. The last token is out
of the range and seems to be the beginning of the next function
definition. I am assuming this is my fault, yet I reduced the code to a
minimal version which reliably produces this bahaviour and still cannot
spot the mistake.

Can anybody tell me what the problem is?

I attached the minimal example, compile and run with e.g.
  $ gcc -o token -lclang -Wall token.c
  $ ./token

It will analyze its own source code and point out the relevant tokens.
In case this is a problem with my clang installation, for me the program
output is:

> range_values(CXSourceRange, uint32_t *, uint32_t *, uint32_t *) - 13
lines: 10-22
>  - Token 95/95 ('static') (lines 24-24) not in range!
> visitor_left_source_file(CXCursor, CXFile) - 7 lines: 24-30
>  - Token 43/43 ('static') (lines 32-32) not in range!
> visitor(CXCursor, CXCursor, CXClientData) - 43 lines: 32-74
>  - Token 260/260 ('int') (lines 76-76) not in range!
> main(int, char **) - 20 lines: 76-95

Output of 'clang -v':

> clang version 3.4.1 (tags/RELEASE_34/dot1-final)
> Target: x86_64-unknown-linux-gnu
> Thread model: posix

Any hints would be greatly appreciated!

Thanks a bunch,
Conrad
-------------- next part --------------
A non-text attachment was scrubbed...
Name: token.c
Type: text/x-csrc
Size: 2794 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/cfe-users/attachments/20140617/d381310a/attachment.c>