[cfe-dev] Fwd: Patch: AST Matcher Framwork and an example tool

[Manuel Klimek]

+cfe-dev, -cfe-commits

As requested by Chris I have reverted the Tooling stuff, and am bumping this
thread on cfe-dev.

Please find the original mail proposing the patch sent to cfe-commits on May
23rd inlined below.

References to the original threads:
Proposal:
http://lists.cs.uiuc.edu/pipermail/cfe-commits/Week-of-Mon-20110523/042163.html
Commit:
http://lists.cs.uiuc.edu/pipermail/cfe-commits/Week-of-Mon-20110530/042375.html

Patches produced by the example tool sent out (has not got feedback yet):
Clang:
http://lists.cs.uiuc.edu/pipermail/cfe-commits/Week-of-Mon-20110523/042212.html
LLVM:
http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20110523/121387.html

More context on why this is useful: when writing code transformation tools
to do large scale changes on libraries with clang, we noticed that a lot of
the code we wrote was "unrolled" matching of tree structures on the AST,
which required a lot of dynamic casting and deeply nested control flow
structures.
The library we implemented based on those findings allows a very declarative
way of expressing interesting parts of the AST that you want to work on,
with a callback mechanism to have an imperative part that does the actual
transformation. The way this library uses templates to build up a reverse
hierarchy of the clang AST allows us to expand the library of matchers
without needing to manually copy the structure of the AST while still
providing full compile time type checking.
We have used this library for many internal transformations (one more
broadly applicable of them being included as an example tool), and Nico may
be able to explain more how he's using the infrastructure for Chromium.

The future work (as outlined to some extend in the original email) is to
build up code transformation tools (including the standard refactoring tools
we know from other languages) and libraries to make it easy for developers
to contribute both generally useful tools and make it easy to write one-off
tools for cleanup transformations on larger code bases.

I would love this to be part of clang mainline, as the expertise and
feedback of the community has been very helpful and valuable.

---------- Forwarded message ----------
From: Manuel Klimek <klimek at google.com>
Date: Mon, May 23, 2011 at 3:03 PM
Subject: Patch: AST Matcher Framwork and an example tool
To: cfe-commits at cs.uiuc.edu
Cc: Nico Weber <thakis at chromium.org>, Marcin Kowalczyk <qrczak at google.com>,
James Dennett <jdennett at google.com>


This patch implements an AST matching framework that allows to write tools
that match on the C++ ASTs. The main interface is in ASTMatchers.h, an
example implementation of a tool that removes redundant .c_str() calls is in
the example RemoveCStrCalls.cpp (patch to llvm/clang produced by running
this tool will be sent out shortly in an extra email).

Currently we have an in-language DSL that allows to write expressions such
as (taken from the .c_str() example):
ConstructorCall( HasDeclaration(Method(HasName(StringConstructor))),
ArgumentCountIs(2),
// The first argument must have the form x.c_str() or p->c_str() // where
the method is string::c_str(). We can use the copy // constructor of string
instead (or the compiler might share // the string object).
HasArgument( 0, Id("call",
Call( Callee(Id("member", MemberExpression())),
Callee(Method(HasName(StringCStrMethod))),
On(Id("arg", Expression()))))), // The second argument is the alloc object
which must not be // present explicitly. HasArgument( 1, DefaultArgument()))

The next steps will be to build up better support for in-process
refactorings based on the Rewriter, to build up higher-level matchers for
common patterns, and to extend the low-level matcher library.

(rietveld link: http://codereview.appspot.com/4552059/)

Cheers,
/Manuel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20110602/096e7397/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ast-matchers.patch
Type: text/x-patch
Size: 244373 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20110602/096e7397/attachment.bin>