[cfe-users] Out-of-range tokens after clang_tokenize()
ch at bitfehler.net
Tue Jun 17 13:57:22 PDT 2014
I am new to the list, so please excuse if this isn't the right place to
ask and thanks for any pointer where to turn instead.
I am trying to write a C program that analyzes certain parts of a .c
file at token level, in this particular example I am interested in
function bodies. I use a visitor to find cursors of interest, get their
source code range (extent) and tokenize that range.
The docs for clang_tokenize() explicitly state: "All of the tokens
produced by tokenization will fall within this source range".
Unfortunately, in my case that is not working out. The last token is out
of the range and seems to be the beginning of the next function
definition. I am assuming this is my fault, yet I reduced the code to a
minimal version which reliably produces this bahaviour and still cannot
spot the mistake.
Can anybody tell me what the problem is?
I attached the minimal example, compile and run with e.g.
$ gcc -o token -lclang -Wall token.c
It will analyze its own source code and point out the relevant tokens.
In case this is a problem with my clang installation, for me the program
> range_values(CXSourceRange, uint32_t *, uint32_t *, uint32_t *) - 13
> - Token 95/95 ('static') (lines 24-24) not in range!
> visitor_left_source_file(CXCursor, CXFile) - 7 lines: 24-30
> - Token 43/43 ('static') (lines 32-32) not in range!
> visitor(CXCursor, CXCursor, CXClientData) - 43 lines: 32-74
> - Token 260/260 ('int') (lines 76-76) not in range!
> main(int, char **) - 20 lines: 76-95
Output of 'clang -v':
> clang version 3.4.1 (tags/RELEASE_34/dot1-final)
> Target: x86_64-unknown-linux-gnu
> Thread model: posix
Any hints would be greatly appreciated!
Thanks a bunch,
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 2794 bytes
Desc: not available
More information about the cfe-users