[cfe-commits] [PATCH 2/2] [clang.py] Implement Token API

[Gregory Szorc]

Committing to https://llvm.org/svn/llvm-project/cfe/trunk ...
     A    bindings/python/clang/enumerations.py
     A    bindings/python/tests/cindex/test_token_kind.py
     A    bindings/python/tests/cindex/test_tokens.py
     M    bindings/python/clang/cindex.py
     M    bindings/python/tests/cindex/test_cursor.py
     M    bindings/python/tests/cindex/test_translation_unit.py
Committed r160111

On 7/11/12 11:47 PM, Manuel Klimek wrote:
> On Tue, Jul 10, 2012 at 3:27 PM, Manuel Klimek <klimek at google.com> wrote:
>> On Mon, Jul 9, 2012 at 8:13 AM, Gregory Szorc <gregory.szorc at gmail.com> wrote:
>>> On 7/1/12 11:48 PM, Manuel Klimek wrote:
>>>
>>> +        lib.clang_tokenize(tu, extent, byref(tokens_memory),
>>> +                byref(tokens_count))
>>>
>>> Strange indent. Hm, after reading further I see that that kind of indent is
>>> quite common. Is there a guideline for how to indent this?
>>>
>>> It appears my editor is indenting 8 spaces when wrapping lines. I'm pretty
>>> sure that's provided by a popular "macro" settings file to handle Python.
>>> I'll change it to 4 spaces, as that seems more conventional.
>>>
>>>
>>>
>>> --- /dev/null
>>> +++ b/bindings/python/clang/enumerations.py
>>>
>>> +TokenKinds = [
>>> +    ('PUNCTUATION', 0),
>>> +    ('KEYWORD', 1),
>>> +    ('IDENTIFIER', 2),
>>> +    ('LITERAL', 3),
>>> +    ('COMMENT', 4),
>>> +]
>>>
>>> This doesn't seem to match the comment. Those "TokenKinds" are different
>>> from what I see in clang/Basic/TokenKinds.def. Am I missing something?
>>>
>>> What doesn't match the comment?
>>>
>>> Also, these come from
>>> http://clang.llvm.org/doxygen/group__CINDEX__LEX.html#gaf63e37eee4280e2c039829af24bbc201,
>>> not the low-level tokens from TokenKinds.def.
>> Oh, wow. Can you add a comment on where these TokenKinds come from.
>> With the overloaded meaning it's really confusing.
> And apart from that lgtm.
>
>> Thanks!
>> /Manuel
>>
>>>
>>> On Sat, Jun 30, 2012 at 4:16 AM, Gregory Szorc <gregory.szorc at gmail.com>
>>> wrote:
>>>> Updated patch attached.
>>>>
>>>> * Added Cursor.get_tokens()
>>>> * Refactored TokenGroup out of TranslationUnit to support above
>>>> * Refactored with TranslationUnit.get_* API changes from last patch.
>>>>
>>>> On Fri, Jun 29, 2012 at 12:18 AM, Gregory Szorc <gregory.szorc at gmail.com>
>>>> wrote:
>>>>> This exposes all of libclang's token functions in the Python bindings.
>>>>>
>>>>> I'm putting the TokenKind enumerations in a new module,
>>>>> clang.enumerations. I plan to eventually move all existing
>>>>> enumerations there so they are all consolidated. And, maybe one day,
>>>>> we can even generate that file automatically by parsing the libclang
>>>>> header files. I know Anders has code around somewhere that does
>>>>> this...
>>>>>
>>>>> ---
>>>>>   bindings/python/clang/cindex.py                    | 177
>>>>> +++++++++++++++++++--
>>>>>   bindings/python/clang/enumerations.py              |  32 ++++
>>>>>   bindings/python/tests/cindex/test_token_kind.py    |  43 +++++
>>>>>   bindings/python/tests/cindex/test_tokens.py        |  52 ++++++
>>>>>   .../python/tests/cindex/test_translation_unit.py   |  24 ++-
>>>>>   5 files changed, 312 insertions(+), 16 deletions(-)
>>>>>   create mode 100644 bindings/python/clang/enumerations.py
>>>>>   create mode 100644 bindings/python/tests/cindex/test_token_kind.py
>>>>>   create mode 100644 bindings/python/tests/cindex/test_tokens.py
>>>> _______________________________________________
>>>> cfe-commits mailing list
>>>> cfe-commits at cs.uiuc.edu
>>>> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-commits
>>>>
>>>