[cfe-commits] [PATCH 2/2] [clang.py] Implement Token API
klimek at google.com
Wed Jul 11 23:47:24 PDT 2012
On Tue, Jul 10, 2012 at 3:27 PM, Manuel Klimek <klimek at google.com> wrote:
> On Mon, Jul 9, 2012 at 8:13 AM, Gregory Szorc <gregory.szorc at gmail.com> wrote:
>> On 7/1/12 11:48 PM, Manuel Klimek wrote:
>> + lib.clang_tokenize(tu, extent, byref(tokens_memory),
>> + byref(tokens_count))
>> Strange indent. Hm, after reading further I see that that kind of indent is
>> quite common. Is there a guideline for how to indent this?
>> It appears my editor is indenting 8 spaces when wrapping lines. I'm pretty
>> sure that's provided by a popular "macro" settings file to handle Python.
>> I'll change it to 4 spaces, as that seems more conventional.
>> --- /dev/null
>> +++ b/bindings/python/clang/enumerations.py
>> +TokenKinds = [
>> + ('PUNCTUATION', 0),
>> + ('KEYWORD', 1),
>> + ('IDENTIFIER', 2),
>> + ('LITERAL', 3),
>> + ('COMMENT', 4),
>> This doesn't seem to match the comment. Those "TokenKinds" are different
>> from what I see in clang/Basic/TokenKinds.def. Am I missing something?
>> What doesn't match the comment?
>> Also, these come from
>> not the low-level tokens from TokenKinds.def.
> Oh, wow. Can you add a comment on where these TokenKinds come from.
> With the overloaded meaning it's really confusing.
And apart from that lgtm.
>> On Sat, Jun 30, 2012 at 4:16 AM, Gregory Szorc <gregory.szorc at gmail.com>
>>> Updated patch attached.
>>> * Added Cursor.get_tokens()
>>> * Refactored TokenGroup out of TranslationUnit to support above
>>> * Refactored with TranslationUnit.get_* API changes from last patch.
>>> On Fri, Jun 29, 2012 at 12:18 AM, Gregory Szorc <gregory.szorc at gmail.com>
>>> > This exposes all of libclang's token functions in the Python bindings.
>>> > I'm putting the TokenKind enumerations in a new module,
>>> > clang.enumerations. I plan to eventually move all existing
>>> > enumerations there so they are all consolidated. And, maybe one day,
>>> > we can even generate that file automatically by parsing the libclang
>>> > header files. I know Anders has code around somewhere that does
>>> > this...
>>> > ---
>>> > bindings/python/clang/cindex.py | 177
>>> > +++++++++++++++++++--
>>> > bindings/python/clang/enumerations.py | 32 ++++
>>> > bindings/python/tests/cindex/test_token_kind.py | 43 +++++
>>> > bindings/python/tests/cindex/test_tokens.py | 52 ++++++
>>> > .../python/tests/cindex/test_translation_unit.py | 24 ++-
>>> > 5 files changed, 312 insertions(+), 16 deletions(-)
>>> > create mode 100644 bindings/python/clang/enumerations.py
>>> > create mode 100644 bindings/python/tests/cindex/test_token_kind.py
>>> > create mode 100644 bindings/python/tests/cindex/test_tokens.py
>>> cfe-commits mailing list
>>> cfe-commits at cs.uiuc.edu
More information about the cfe-commits