[cfe-dev] Source to source transformations questions in branches/tooling

[Manuel Klimek]

On Sun, Jun 24, 2012 at 4:53 PM, Philip Dunstan <phil at philipdunstan.com>wrote:

> Hi,
>
> I've been trying some source to source transformations using the
> matcher/callback framework in the tooling branch. They are mostly working
> but I have a couple of questions and would like some feedback on whether I
> am doing everything as intended. Will you please help me?
>
> There are two transformations I am trying to make: adding the override
> keyword to virtual functions where we can and replacing shared_ptr by
> value function parameters with const references. I have attached the two
> cpp files to this post. They are also available from
>
> https://bitbucket.org/phildunstan/clang_tools/src/5d63a291d377/add-override-specifier
>
> https://bitbucket.org/phildunstan/clang_tools/src/5d63a291d377/replace-shared_ptr-parameters
>
>
> add-override-specifier.cpp
> Adds the override specifier to functions that override virtual
> functions from a base class and don't already have the override specifier.
>
> I have written the following matcher to find candidates functions.
>        AST_MATCHER(clang::CXXMethodDecl, IsOverrideMissing)
>        {
>            // does this virtual function override a virtual function from
> a base class
>            return (Node.isVirtual() && (Node.size_overridden_methods() >
> 0) && !Node.hasAttr<OverrideAttr>());
>        }
>
> In my callback however I don't know how to find the right source location
> to insert the override keyword. If there is no body I can use the end of
> the function declaration. However if there is a body I don't know how to
> get the end of the declaration section before the body. For example in the
> following example I want to insert the keyword before the comment. How do I
> get the source location of this point?
>        void f() const // overrides a base class virtual function
>        {
>        }
>

I'll leave answering that to Daniel.

replace-shared_ptr-parameters.cpp
> Replace function parameters that specify a shared_ptr instance by
> value with shared_ptr by const reference.
>
> Should I be changing the type from shared_ptr<X> to const shared_ptr<X>& or
> is there a way to do this by modifying the AST to make the parameter const
> and by reference and then re-emit the source?
>

If you can match the location of the shared_ptr and the type, you can emit
a replacement to "const X&" (no need to go through the AST here). Is there
a part of the matching that's problematic?


> I have this working for normal functions, methods and template functions
> where instantiations of that template exist. What type of matcher should I
> write  to match a template function that doesn't have an instantiation.
> eg template <typename T> f(shared_ptr<T>);
>

Ugh. Templates that are never instantiated anywhere are really hard to
handle, as in your example all the types are dependent. -ast-dump-xml and
friends also don't really help here. I'll put it on my list to write a tool
that helps answer that kind of question.


> Additional Questions:
>
> It appears that I need to specify all of the system c++ include files when
> I run the tool on a file, but I don't need to provide them when I compile
> the with the clang compiler that I have built from the tooling branch. Is
> this expected behavior?
> replace-shared_ptr-parameters input.cpp -- -std=c++11 -Wall -g -O0
> -I/usr/include/c++/4.6 -I/usr/include/c++/4.6/x86_64-linux-gnu
> -I/usr/include/c++/4.6/backward -I/usr/lib/gcc/x86_64-linux-gnu/4.6/include
> -I/usr/local/include -I/usr/lib/gcc/x86_64-linux-gnu/4.6/include-fixed
> -I/usr/include/x86_64-linux-gnu -I/usr/include
>

Hm, this should work in principle. Can you add '-v' output to both the tool
run and the clang run from your self-built clang compiler and attach them?

A different solution would be to use the CompilationDatabase. If your
project uses cmake, it works out of the box; if not, the easiest thing is
to create a compilation database. See the docs here:
http://clang.llvm.org/docs/JSONCompilationDatabase.html


> I'd like to run these tools over our complete code base where the changes
> will affect both header and source files. What is the recommended way to
> limit changes to only my files and not system header files? I expect I can
> limit them to the main file specified by the source manager, however I
> assume that this would prevent changes to my header files.
>

I simply do match on the path / name of the files for that. Usually project
files have easily detectable paths.
Also note that what I do is key whether I want to replace something on the
file of the original declaration of the 'thing' I want to replace, so that
you don't get inconsistent states.


> How do I prevent a change from being made multiple times while in the same
> translation unit compilation? eg as a result of template or macro
> instantiations or multiple header includes. Is there a recommended way to
> solve this or is it up to the callback to be safe in the case that it is
> run twice on the same piece of code?
>

That's what the refactoring support in the tooling library is for:
http://llvm.org/viewvc/llvm-project/cfe/branches/tooling/include/clang/Tooling/Refactoring.h?view=markup

The idea is that you run over all TUs you want to refactor in the same
process. We're working on getting this multi-threaded so that it scales
better.

Another possibility is to output simple diffs from your tool, and use a
little python script to slurp in the changes, unique them and apply them
(that's what we currently do for really large scale MapReduce style
refactorings).


> Thanks for the excellent work you guys are doing,
>

Thanks :)

/Manuel


> Phil
> --
> Philip Dunstan
> phil at philipdunstan.com
> www.philipdunstan.com
>
>
> --
> View this message in context:
> http://clang-developers.42468.n3.nabble.com/Source-to-source-transformations-questions-in-branches-tooling-tp4024920.html
> Sent from the Clang Developers mailing list archive at Nabble.com.
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20120625/0809dc50/attachment.html>