[cfe-dev] Out-of-range tokens after clang_tokenize()

[Conrad Hoffmann]

Hello everybody,

I am trying to write a C program that analyzes certain parts of a .c
file at token level, in this particular example I am interested in
function bodies. I use a visitor to find cursors of interest, get their
source code range (extent) and tokenize that range.
The docs for clang_tokenize() explicitly state: "All of the tokens
produced by tokenization will fall within this source range".

Unfortunately, in my case that is not quite working out. The last token
is out of the range and seems to be the beginning of the next function
definition. I am assuming this is my fault, yet I reduced the code to a
minimal version which reliably produces this bahaviour and still cannot
spot the mistake.

Can anybody tell me what the problem is?

I attached the minimal example, compile and run with e.g.
  $ gcc -o token -lclang -Wall token.c
  $ ./token

Note: the system-dependent includes are hardcoded for brevity (line 79).

It will analyze its own source code and point out the relevant tokens.
In case this is a problem with my clang installation, for me the program
output is:

> range_values(CXSourceRange, uint32_t *, uint32_t *, uint32_t *) - 13
> lines: 10-22
>  - Token 95/95 ('static') (lines 24-24) not in range!
> visitor_left_source_file(CXCursor, CXFile) - 7 lines: 24-30
>  - Token 43/43 ('static') (lines 32-32) not in range!
> visitor(CXCursor, CXCursor, CXClientData) - 43 lines: 32-74
>  - Token 260/260 ('int') (lines 76-76) not in range!
> main(int, char **) - 21 lines: 76-96


Output of 'clang -v':

> clang version 3.4.1 (tags/RELEASE_34/dot1-final)
> Target: x86_64-unknown-linux-gnu
> Thread model: posix

Any hints would be greatly appreciated!

Thanks a bunch,
Conrad

-------------- next part --------------
A non-text attachment was scrubbed...
Name: token.c
Type: text/x-csrc
Size: 2880 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20140708/2ee5d1fc/attachment.c>