[cfe-dev] Python bindings questions

[Gregory Szorc]

On 12/27/2011 5:03 AM, Manuel Holtgrewe wrote:
> (2) requires wrapping of the libclang tokenization API that is not 
> available in the clang SVN's python wrapper yet (I think). I have an 
> experimental version of this living at
> 
> https://github.com/holtgrewe/linty/blob/master/clang/cindex.py
> 
> The main "problem" is that tokens are not managed as single objects but 
> as arrays. Additionally, the tokens can be annotated back to their AST 
> cursors but this information lives in an additional array.
> 
> My take on a pythonic wrapper was to write a TokenCollection class and 
> some more icing. This new API part works pretty well for me.

Hmmm. I just noticed this after I wrote an implementation for the token
API myself:

https://github.com/indygreg/clang/tree/python_features/bindings/python/clang

I worked around the problem you mentioned by copying the individual
CXToken structs into separate Python objects then calling
clang_disposeTokens() on the original memory before returning the newly
obtained tokens. This relies on the current implementation of those APIs
where the allocated CXToken array is merely itself a shallow copy of
underlying data. Although it seems to violate an unstated API contract,
it does work and is much cleaner on the Python side.

Greg