[cfe-dev] About AST rewriting / manipulation

[steve naroff]

Hi Simone,

A challenge with the rewriter is rewriting code that has already been  
rewritten. Rewriting an expression often involves synthesizing new  
expressions that don't have a SourceLocation (as a result, they can't  
be rewritten). The ObjC rewriter plays many "games" to work around  
this. The code for rewriting ObjCPropertyRefExpr's is particularly  
tricky.

Another limitation is you can't rewrite code that originates from a  
macro. If this is a problem, the workaround is to preprocess the code  
prior to rewriting it.

Another approach is to do all the transformations on the AST's and  
pretty print the result. The downside with this approach is: (a) The  
pretty printing is still incomplete (particularly for Decls).  
Expressions are in fairly good shape. (b) You will loose comments,  
macros.

The ObjC rewriter uses both rewriting and pretty printing (for  
expressions) to do it's job. It's currently the most complex usage of  
AST-based rewriting.

snaroff

On Feb 3, 2009, at 3:32 AM, Simone Pellegrini wrote:

> Yes, the Rewriter does the job!
> But the main problem with the Rewriter is that any change I apply to  
> the
> syntax tree (let's say... insert text or node replacement) are not  
> kept
> in the original syntax tree.
>
> To me the Rewriter seems like a buffer that keeps track of the  
> changes I
> made and when I say to print it back it merges the syntax tree with  
> the
> changes I have provided. Is that correct? So, I will explain my
> problem... I have a source file and I want to analyze it and then  
> apply
> (if it's worth doing it) a list of transformations considering a
> particular order! So... I want to be able to apply several
> transformations on the same AST. Using the rewriter it seems that I  
> can
> apply one transformation... then produce a source file... parse it
> again... analyze and apply another transformation... and so on!
> And that's it's real inefficient! :(
>
> I guess I could mix the usage of the TokenRewriter and the  
> prettyPrint()
> method to rewrite portions of the AST I have changed... or have a look
> at the Rewriter implementation because as now it's not clear to me how
> it works! :)
>
> regards, S. Pellegrini
>
> Douglas Gregor wrote:
>> On Feb 2, 2009, at 8:40 AM, Simone Pellegrini wrote:
>>> Thanks for the response,
>>> I've chosen the simpler way, I just create clang nodes from scratch
>>> and replace / insert these nodes with the old one in the syntax
>>> tree!  At the end it's not so complex as I thought... but now I have
>>> another problem! I want to write back the modified syntax tree but  
>>> no
>>> matter which kind of changes I made to the syntax tree... when I use
>>> the TokenRewriter:
>>>
>>> const LangOptions &LangOpts = Ctx.getLangOptions();
>>> TokenRewriter Rewriter(Ctx.getSourceManager().getMainFileID(),
>>> Ctx.getSourceManager(), LangOpts);
>>>
>>> // Print out the output.
>>> for (TokenRewriter::token_iterator I = Rewriter.token_begin(), E =
>>> Rewriter.token_end(); I != E; ++I)
>>>      out << pp.getSpelling(*I);
>>>
>>> I still get the original code!
>>
>> As I understand it, the TokenRewriter is only meant to be used to
>> insert tokens into the token stream as it's being rewritten. It's not
>> designed for AST-level rewrites.
>>
>>> How can I rewrite a syntax tree to Source file? Is there something  
>>> in
>>> the API that I am missing? Should I use the Rewriter? If yes, why  
>>> the
>>> rewriter doesn't provide an InsertStmt method? With the current  
>>> API I
>>> can add text... and replace statements... what about inserting
>>> statements? and... remove statements?
>>
>> I believe the Rewriter can do what you want, but it will take some
>> effort. The rewriter allows you to insert/remove/replace text in the
>> source file with any other text, using SourceLocations to specify
>> where in the source file the modifications will be made. That's one
>> approach to AST-based rewriting, since one can turn a
>> statement/expression/declaration into a string and then insert it at
>> the appropriate place in the source.
>>
>> Clang has some functionality for pretty-printing statements and
>> expressions as source code, but we haven't put a lot of effort into
>> making sure that the result is well-formed C++. Depending on how
>> complicated your statements and expressions are, this might take no
>> work or it might take a lot of work.
>>
>>    - Doug
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev