<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">I’m experimenting with syntax highlighting using the AST. What I want to do is end up with a stream of tokens for a file; each annotated with a scope, which represents the AST cursors it belongs to (broadest to narrowest) and finally its token kind.<div class=""><br class=""></div><div class="">The idea is then to apply scoping rules to these, similar to how TextMate’s parses does it. Thus I can colour an IDENTIFIER inside a PARM_DECL different to an IDENTIFIER on it’s own.</div><div class=""><br class=""></div><div class="">The problems I’ve got are:</div><div class=""><ol class="MailOutline"><li class="">I can’t just walk the cursor tree, as Cursors don’t cover all tokens; thus don’t include things like comments</li><li class="">I can’t just walk the token stream, as Tokens don’t include the AST context and you can only get the most detailed cursor from clang_annotateTokens</li><ol class=""><li class="">I can’t walk back up from the detailed cursor as cursors have no “parent” (and lexical and semantic are often both None)</li></ol><li class="">Even if I solve this, includes cause me problems</li><ol class=""><li class="">Although the AST walk includes everything imported from the include; I can’t find a way to get the includes Tokens - so I don’t even “see” comments in them</li><li class="">You get a INCLUSION_DIRECTIVE where the include starts; but theres no way to find what the include path actually resolved to (without relying on there being an AST symbol inside it)</li><li class="">You can’t get the scope of an include (from the AST) as the included file isn’t a TranslationUnit</li></ol></ol><div class=""><br class=""></div></div><div class="">Now, I’ve got a solution to all these - but it’s not pretty and I can’t help feeling I’m missing something obvious.</div><div class=""><br class=""></div><div class="">My solution works by:</div><div class=""><ul class="MailOutline"><li class="">for each ccmd in compile_command.json</li><ul class=""><li class="">Create translation unit with PARSE_DETAILED_PROCESSING_RECORD (a root_tu)</li><li class="">for each token in root_tu</li><ul class=""><li class="">save in a lookup table by it's location</li></ul><li class="">for each include in root_tu (from clang_getInclusions)</li><ul class=""><li class="">tweak the root_tu’s compiler arguments, replacing the compiled filename with the include path</li><li class="">Create an inc_tu with PARSE_INCOMPLETE</li><li class="">for each token in inc_tu</li><ul class=""><li class="">save in a lookup table by it's location</li></ul></ul><li class="">I walk the AST from root_tu (which goes into the included header)</li><ul class=""><li class="">for each token in the AST cursor</li><ul class=""><li class="">lookup the token by location and append the cursor to it’s scope</li></ul></ul></ul></ul><div class=""><br class=""></div></div><div class="">As I said, it appears to work … but it’s not pretty.</div><div class=""><br class=""></div><div class="">Am I missing some easy way to do this, particular when it comes to handling includes?</div><div class=""><br class=""></div><div class="">Thoughts?</div><div class=""><br class=""></div><div class="">Coops</div></body></html>