[cfe-dev] Source code documentation

[Douglas Gregor]

On Mar 9, 2010, at 4:55 PM, Stefan Seefeld wrote:

> On 03/09/2010 04:42 PM, Douglas Gregor wrote:
>>> Yes. At present the parser attaches comments to the next declaration it finds, from where Synopsis then picks it up to process it further (extract processing instructions, documentation, whatever).
>>>     
>> Okay. We don't really have this functionality in Clang yet. Comments are passed through to the AST consumer, and we have a hack that tries to find the comment associated with a declaration after the fact, but this will need work.
>>   
> 
> OK.
> 
>>> Also, in one mode of operation Synopsis wants to get a position-correct picture of the entire preprocessed source file, so it can generate a hyperlinked and otherwise styled version of it. Does CLang provide this level of detail ?
>>>     
>> 
>> Internally, yes. There isn't enough information exposed via the CIndex interface to do this (but I'd support extending CIndex in this direction).
>>   
> 
> OK. Without knowing CIndex, I'm not sure how useful it is to support such different levels of details through the same representation. For example, to generate a hyperlinked source tree, I'd operate on something close to the parse tree, i.e. individual tokens.

Check out clang_annotateTokens() at http://clang.llvm.org/doxygen/group__CINDEX__LEX.html . It maps from tokens (which you can get from clang_tokenize()) to the AST entities those tokens refer to. A "cursor" in CIndex parlance represents an AST element.

> But for the documentation, a much more high-level view is useful, such as a syntax tree or even a semantic graph.

Sure. Cursors point into the AST, which contains much semantic information.

> Do you think all of those will be represented by CIndex, eventually ?


I think so. The goal of CIndex is to support various tools (documentation generators, IDEs, syntax highlighters, whatever) without forcing those tools to deal with the ever-changing Clang ASTs directly. So if Synopsis needs something CIndex doesn't provide, it's probably a CIndex bug. We're not there yet, and it will probably be *more* work right now to use CIndex than it would to grok Clang ASTs directly, but the end result will be better if Synopsis can go through CIndex because many other tools will benefit from the CIndex improvements that Synopsis will need.

	- Doug