[cfe-dev] clang AST printer

[Philip Craig]

On Sat, Sep 22, 2012 at 7:00 AM, Philip Craig <philipjcraig at gmail.com> wrote:
> On Fri, Sep 21, 2012 at 11:21 PM, Alexander Kornienko <alexfh at google.com> wrote:
>>
>> On Fri, Sep 21, 2012 at 2:40 PM, Manuel Klimek <klimek at google.com> wrote:
>>>
>>> On Fri, Sep 21, 2012 at 12:32 PM, Philip Craig <philipjcraig at gmail.com>
>>> wrote:
>>> > On Fri, Sep 21, 2012 at 7:49 PM, Manuel Klimek <klimek at google.com>
>>> > wrote:
>>> >> Hi Philip,
>>> >>
>>> >> we had a discussion around the strategy for handling ast dumping
>>> >> fairly recently. I think the common agreement in the end was that
>>> >> instead of having an extra tool, we want to make clang's -ast-dump
>>> >> awesome.
>>> >
>>> > Was this discussion on the mailing list? Any chance you could point me
>>> > to it?
>>>
>>>
>>> http://lists.cs.uiuc.edu/pipermail/cfe-commits/Week-of-Mon-20120716/060831.html
>
> Thanks.
>
>>> >> I don't know what your ultimate goal is here, but from my
>>> >> side any work that goes towards making clang -ast-dump better would be
>>> >> highly appreciated :)
>>> >
>>> > My motivation was having some way of getting the structure of the AST
>>> > for a given piece of source code, so that it is easier to correctly
>>> > call the AST matchers to match that code. The issue with clang
>>> > -ast-dump is that it pretty prints decls and types. So would the goal
>>> > here be to add more command line options to clang to control how
>>> > -ast-dump prints the AST?
>>>
>>> No, I think the goal is to dump decls and types in a sensible way :)
>>
>>
>> More specifically, I think we agreed on changing -ast-dump from
>> pretty-printing declarations to outputting them in the same LISP-style
>> format, which is used for statements. I was going to implement that, but
>> never had time to do this. If you're going to continue work on your utility,
>> it would be much more valuable, if you instead improved clang's current
>> -ast-dump option. In that case you consider to go this way, here's where to
>> start:
>>
>> clang/lib/Frontend/ASTConsumers.cpp:120: ASTConsumer
>> *clang::CreateASTDumper(StringRef FilterString)
>>
>> is a common entry point, used by "clang -cc1 -ast-dump" and by "clang-check
>> -ast-dump". An additional benefit from this would be that after "-ast-dump"
>> starts outputting the AST for declarations in a structured form,
>> "-ast-dump-xml" will become useless and can be removed.
>
> I'll have a look into that and get an idea of what's involved.

I'm going to try working on this. Here's my thoughts on how to structure this.

Currently, it looks like this (I'm ignoring types for now):

CreateASTDumper calls either Decl::print() or Decl::dump().

Both Decl::print() and Decl::dump() use class DeclPrinter. The only
difference between these uses is that Decl::dump() passes a
PrintingPolicy that has DumpSourceManager set.

When DeclPrinter encounters a statement, it calls Stmt::printPretty().
If DumpSourceManger is set, then Stmt::printPretty() calls
Stmt::dump(), otherwise it uses class StmtPrinter.

Stmt::dump() uses class StmtDumper.


My proposed changes are:

I will change Stmt::printPretty() to never call Stmt::dump(). This
also removes the need for the DumpSourceManager field in
PrintingPolicy.

I will create a new DeclDumper class, which is called by Decl::dump(),
and which calls Stmt::dump() when it encounters a statement.

There will be some duplication of functionality and state between
StmtDumper and DeclDumper. So I will extract parts of StmtDumper out
into an ASTDumper class. This will result in a design that is very
similar to the ASTWriter for serialization.

My first patch will be to create the ASTDumper class. Later patches
will create DeclDumper. For the creation of the DeclDumper class,
should I do this as one big patch, or should I try to find a way to
break it into smaller patches?

>>
>> --
>> Regards,
>> Alex
>>
>>
>>>
>>> > Are there any other areas where you think -ast-dump needs to be better?
>>>
>>> As you said, decls and types need to be structured nicely. Also, a
>>> while ago Richard proposed a patch to add coloring, no idea how far he
>>> got (cc'ing him)
>>>
>>> Cheers,
>>> /Manuel
>>>
>>> >
>>> >>
>>> >> Cheers,
>>> >> /Manuel
>>> >>
>>> >> On Fri, Sep 21, 2012 at 1:17 AM, Philip Craig <philipjcraig at gmail.com>
>>> >> wrote:
>>> >>> Hi,
>>> >>>
>>> >>> I've been writing a tool to print a clang AST. You can find it at
>>> >>> https://github.com/philipc/clang-ast. I've been mostly writing this as
>>> >>> a learning exercise, but I would like to see if there is any wider
>>> >>> interest in the tool.
>>> >>>
>>> >>> I'm new to clang, and interested in working on tooling. For this task,
>>> >>> it is important to have an understanding of the AST. I've found the
>>> >>> documentation at http://clang.llvm.org/docs/InternalsManual.html and
>>> >>> http://clang.llvm.org/docs/IntroductionToTheClangAST.html (is there
>>> >>> any more?), but I think it would be helpful to be able to print the
>>> >>> AST for source code to see how it is used in practice. I've found two
>>> >>> ways to do this currently, which aren't quite what I wanted:
>>> >>>
>>> >>> clang --ast-dump
>>> >>> - pretty prints some parts, has too much internal info, more suited
>>> >>> for debugging use by clang developers
>>> >>>
>>> >>> clang --ast-dump-xml
>>> >>> - incomplete and XML is too verbose for this purpose
>>> >>>
>>> >>> Since RecursiveASTVisitor seems to be the API that will be used by
>>> >>> tool developers, I've been using RAV to print the AST. I've found a
>>> >>> few minor bugs in RAV as a result of this, which I'm in the process of
>>> >>> submitting.
>>> >>>
>>> >>> One limitation of RAV is that the Visit methods aren't given
>>> >>> information about their relationship with their parent, so the tool
>>> >>> can only list all the children of a node, without distinguishing
>>> >>> between the LHS and RHS of an operator, for example. Is there any
>>> >>> desire to extend RAV to be able to do this?
>>> >>>
>>> >>> I've been writing small code snippets to test printing the various
>>> >>> parts of the AST. If you want to see examples of the output of the
>>> >>> tool, this is included in the test cases. As a quick example, "void
>>> >>> foo() {}" is printed as:
>>> >>>
>>> >>>   FunctionDecl
>>> >>>     DeclarationName foo
>>> >>>     FunctionNoProtoType
>>> >>>       BuiltinType void
>>> >>>     CompoundStmt
>>> >>>
>>> >>> Finally, I've been trying to give a textual description of the AST
>>> >>> grammar in https://github.com/philipc/clang-ast/blob/master/ast.txt.
>>> >>> This is an attempt to give something similar to
>>> >>> http://docs.python.org/py3k/library/ast.html#abstract-grammar. I'm not
>>> >>> sure if it is turning out to be that useful though.
>>> >>>
>>> >>> Any comments welcome!
>>> >>>
>>> >>> Thanks,
>>> >>> Philip
>>> >>> _______________________________________________
>>> >>> cfe-dev mailing list
>>> >>> cfe-dev at cs.uiuc.edu
>>> >>> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>>
>>