[cfe-dev] Writing C++ refactoring tools in Common Lisp

[Samuel Benzaquen]

On Wed, Jan 8, 2014 at 2:44 AM, Manuel Klimek <klimek at google.com> wrote:

> +Sam, who has written the dynamic registry stuff for ideas...
>
>
> On Tue, Jan 7, 2014 at 7:50 PM, Christian Schafmeister <
> chris.schaf at verizon.net> wrote:
>
>>
>> On Jan 7, 2014, at 12:59 PM, Manuel Klimek <klimek at google.com> wrote:
>>
>> On Tue, Jan 7, 2014 at 6:52 PM, Christian Schafmeister <
>> chris.schaf at verizon.net> wrote:
>>
>>>
>>> On Jan 7, 2014, at 12:12 PM, Manuel Klimek <klimek at google.com> wrote:
>>>
>>> On Tue, Jan 7, 2014 at 5:16 PM, Christian Schafmeister <
>>> chris.schaf at verizon.net> wrote:
>>>
>>>> Hey folks,
>>>>
>>>>
>>>> I’ve written a new Common Lisp environment/compiler over the past two
>>>> years.
>>>> It uses LLVM as the back-end (by exposing the LLVM C++ library to
>>>> Common Lisp) and interoperates with C++ in that it has a template library
>>>> like boost::python for exposing arbitrary C++ classes/functions to Common
>>>> Lisp.
>>>>
>>>> I want to do some automated refactoring to improve my C++ code so I’m
>>>> exposing the Clang ASTMatcher and Refactoring libraries.
>>>>
>>>> I also think it would be really cool to write refactoring tools in a
>>>> dynamic language like Lisp - without all the exhausting boilerplate
>>>> required by the C++ approach.
>>>>
>>>
>>> C++ refactoring tools actually don't require that much boilerplate…
>>>
>>>
>>> Sorry, my statement sounded like a criticism - I never mean to criticize
>>> - these languages are what they are.
>>>
>>
>> No worries, I have no problems with criticism - and I didn't take it that
>> way. My remark was really just meant as that - I think writing a
>> refactoring tool has suprisingly little overhead. The main problem is
>> actually compilation, which is why I'd be very interested in seeing more
>> interpreted integrations.
>>
>>
>>> I was just speaking about my feelings from writing a few simple
>>> refactoring tools in C++ and my motivations for implementing a Common Lisp
>>> compiler and writing refactoring tools in Common Lisp.  I’m a new convert
>>> to Common Lisp - I just discovered it two years ago and most of my
>>> experience with it is writing a self-hosting compiler.
>>>
>>>
>>>
>>>
>>>> So I’m exposing the Clang AST library to this Common Lisp environment
>>>> with the goal of writing a general tool for writing C++ refactoring tools
>>>> in Common Lisp.
>>>>
>>>> My goal is to mimic C++ ASTMatchers like:
>>>>
>>>> *recordDecl(hasDescendant(*
>>>> *    ifStmt(hasTrueExpression(*
>>>> *        expr(hasDescendant(*
>>>> *            ifStmt()))))))*
>>>>
>>>> Using S-expressions: *(record-decl (has-descendent (if-stmt
>>>> (has-true-expression (expr (has-descendant( if-stmt))))))) *
>>>>
>>>> And the source-to-source translation code will be small lambda
>>>> functions that use the resulting MatchResults.
>>>>
>>>> That way I don’t have to write a lot of documentation :-).
>>>>
>>>> Currently I’m looking at calling the ASTMatcher/Dynamic/VariantValue
>>>> interface to build ASTMatchers from S-expressions - does that sound like a
>>>> good idea - or is there a better approach I should be exploring?
>>>>
>>>
>>> Well, if your boost shim can already match "arbitrary C++ classes /
>>> functions", then it should in principle be able to expose the AST matchers
>>> as is (without the need to go through the dynamic parsing).
>>> After all, the matchers are mostly just template functions :)
>>>
>>>
>>> I was trying to figure out how to do that at first but the ASTMatchers
>>> looked more exotic than just simple functions.
>>>
>>> For instance “namedDecl” has the type
>>> internal::VariadicDynCastAllOfMatcher<Decl,NamedDecl>
>>> VariadicDynCastAllOfMatcher is defined as:
>>>
>>> template <typename SourceT, typename TargetT>
>>> class VariadicDynCastAllOfMatcher
>>>     : public llvm::VariadicFunction<
>>>         BindableMatcher<SourceT>, Matcher<TargetT>,
>>>         makeDynCastAllOfComposite<SourceT, TargetT> > {
>>> public:
>>>   VariadicDynCastAllOfMatcher() {}
>>> };
>>>
>>> llvm::VariadicFunction has many templated overloads of "operator()"
>>> I haven’t written template code to automatically wrap variadic functions
>>> (as in functions that take … ellipses).
>>>
>>> I can wrap functions that take ArrayRef’s though - is there a function
>>> like namedDecl(ArrayRef<ASTMatcher>) for ASTMatchers?
>>>
>>> I’ve only been getting deep into C++ template programming for about two
>>> months so I’m still struggling with reading complex template code.
>>>
>>
>> Yes, I'd actually be suprised if this was possible without major
>> jump-through-hooping - the problem is that the compiler generates the
>> template code only when they get instantiated - so unless you have a
>> backend that takes the lisp code and generates C++ code from that that
>> integrates with the template code, I wouldn't know how to do it (and that
>> round-trip would seem like it kills a lot of the benefits of having lisp in
>> the first place - after all, you'd still get crappy C++ error messages :P)
>>
>>
>> Agreed.
>>
>> Maybe the best way to do this is to write a Registry class like the one
>> in Dynamic/Registry.h and define “constructMatcher” and
>> “constructBoundMatcher” that take Lisp symbols and assemble VariantMatchers
>> that way.
>>
>
Most of them are simple functions, but there are also a lot that are
"polymorphic". That is, they return a proxy type that allows late binding
of the real matcher type.
During the implementation of the dynamic matcher registry I changed all
these polymorphic/adaptative types to provide extra type information that
allowed the registry to instantiate all the required functions.

What about this?
Instead of wrapping the actual matchers the way it is done in C++, you
could make meta matchers that really only know their name and their
arguments.
Basically all matchers would be functions that return an object that has a
name() method and a list of their arguments.
Then you can generate the C++ matcher string and use the C++ matcher parser
to generate the actual matcher.
This way you only wrap a few [simple] functions from C++ and are shielded
from the template metaprogramming craziness.
Hadn't thought this through, so it might be too simple to work =P.

_Sam


>>
>>
>>
>>
>>
>>>
>>>
>>>
>>>
>>> Otherwise I think going the dynamic matcher route is a good approach.
>>>
>>> Cheers,
>>> /Manuel
>>>
>>>
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20140109/340f5ed9/attachment.html>