Good point. I currently have a matcher that finds for loops that appear to have the right shape for conversion, and I do some extra checking along with the conversion work in the callback. I can completely handle array-style loops with a Preprocessor, which is used when generating the names of new variables. For iterator-style loops, I use Sema to determine the return type of operator*, which will be the type of the loop variable. Finally (though this isn't implemented yet), containers that are used as if they're arrays will require Sema to check if suitable begin() and end() functions exist.<div>
<br></div><div>For now, I can force the use of auto rather than an explicit type listing for iterator-based loops and get most of the work done without Sema. The question then becomes the correct way to avoid repeatedly creating a Preprocessor object, though if this is a cheap operation, I can create one in the callback.</div>
<div><br></div><div>Does this make the use clearer?</div><div>Thanks!</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jul 11, 2012 at 12:29 AM, Manuel Klimek <span dir="ltr"><<a href="mailto:klimek@google.com" target="_blank">klimek@google.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">On Wed, Jul 11, 2012 at 6:03 AM, Sam Panzer <<a href="mailto:panzer@google.com">panzer@google.com</a>> wrote:<br>
> I'm trying to port my code to take advantage of matchers, now that they are<br>
> in mainline. Some of the work I want to do involves semantic analysis of the<br>
> results (i.e. in the callback). What would be the best way to get a Sema or<br>
> CompilerInstance out of either RefactoringTool or MatchResult? I'm currently<br>
> playing with changing MatchASTConsumer to inherit from SemaConsumer, so that<br>
> MatchFinder can track a Sema object the same way it does an ASTContext.<br>
<br>
</div>I'd be interested in what the use cases for the semantic analysis are<br>
first. Often it seems like it would be better to have the information<br>
available in the AST instead of rerunning (potentially expensive)<br>
semantic analysis.<br>
<br>
Cheers,<br>
/Manuel<br>
<div class="HOEnZb"><div class="h5"><br>
><br>
> Thanks!<br>
><br>
><br>
> On Mon, Jul 2, 2012 at 7:22 AM, Manuel Klimek <<a href="mailto:klimek@google.com">klimek@google.com</a>> wrote:<br>
>><br>
>> On Mon, Jul 2, 2012 at 4:16 PM, Sam Panzer <<a href="mailto:panzer@google.com">panzer@google.com</a>> wrote:<br>
>>><br>
>>> On Sun, Jul 1, 2012 at 10:45 PM, Manuel Klimek <<a href="mailto:klimek@google.com">klimek@google.com</a>> wrote:<br>
>>>><br>
>>>> On Fri, Jun 29, 2012 at 8:17 PM, Sam Panzer <<a href="mailto:panzer@google.com">panzer@google.com</a>> wrote:<br>
>>>>><br>
>>>>> Thanks for the input!<br>
>>>>><br>
>>>>> Tooling/Refactoring is definitely the right way to go - dumping to<br>
>>>>> stdout was just a holdover from the example on LibTooling. I'll change it<br>
>>>>> once I figure out how it works - and a clean way to arrange the tests.<br>
>>>>><br>
>>>>> As for the use of matchers vs. visitors, I decided to use a visitor<br>
>>>>> because this is a relatively complex transformation. I would happily use<br>
>>>>> matchers if I thought I could - and I think that some other c++11 migrations<br>
>>>>> can easily be written with matchers - but I think the for loop<br>
>>>><br>
>>>><br>
>>>> I'm not claiming that the matchers needed to match those constructs are<br>
>>>> all already written - but if we write the questions you need to ask into<br>
>>>> matchers, other people who want to match similar things can reuse them, thus<br>
>>>> amplifying the impact of the code you write ;)<br>
>>>><br>
>>>>><br>
>>>>> checks need some features that matchers don't have (correct me if I'm<br>
>>>>> wrong!). For example, the check for statically allocated array-based loops<br>
>>>>> does this:<br>
>>>>> Given a for loop, determine if:<br>
>>>>> - The increment statement increments (via ++) exactly one integral<br>
>>>>> index variable, such that the variable is declared and initialized to zero<br>
>>>>> in the init portion of the loop, and that this variable's value is a<br>
>>>>> compared (via <, > or !=) to a nonnegative compile-time constant N in the<br>
>>>>> compare portion.<br>
>>>>> - The index variable is only ever used in an ArrayIndexExpession<br>
>>>>> indexing a single, statically allocated array A.<br>
>>>>> - The array A has exactly N elements.<br>
>>>>> - Additionally, if the ArrayIndexExpression A[index] is ever assigned,<br>
>>>>> passed to a function or copied as a non-const reference, or its address<br>
>>>>> taken with & (I still need to add a check for calls to non-const member<br>
>>>>> functions), the loop variable in the converted version needs to be a<br>
>>>>> non-const reference so that the value will be correctly updated (this step<br>
>>>>> adds the most complexity).<br>
>>>><br>
>>>><br>
>>>> ... and the matcher I would want to write for this looks something like<br>
>>>> that:<br>
>>>> ForLoop(<br>
>>>> HasInitialization(Declaration(Id("loopvar", HasType(IsIntegral())))),<br>
>>>> HasCondition(BinaryOperator(<br>
>>>> HasAnyOperand(DeclarationReference(Id("condref", To(Variable())))),<br>
>>>> HasAnyOperand(IntegerLiteral()))),<br>
>>>><br>
>>>> HasIncrement(UnaryOperator(HasUnaryOperand(DeclarationReference(Id("incref",<br>
>>>> To(Variable()))))), ...),<br>
>>>> )<br>
>>>><br>
>>>> In general, the complex stuff can stay complex, but the simple stuff<br>
>>>> shouldn't be lots of code.<br>
>>><br>
>>><br>
>>> Good point - and this is much easier to read than the equivalent code I<br>
>>> had written.<br>
>>><br>
>>>><br>
>>>><br>
>>>>><br>
>>>>> The other types of loop (iterator-based, and array-like container) are<br>
>>>>> more complicated to detect, since there are more permitted ways to define<br>
>>>>> and use the index/iterators. What makes this difficult to do entirely with<br>
>>>>> matchers is the number of back- and cross-references, as well as the<br>
>>>>> different local behaviors based on semantic properties. On the other hand,<br>
>>>>> if there were some kind of backreference-enabled matcher that<br>
>>>><br>
>>>><br>
>>>> The way to handle the back-refs is to bind the nodes you want, and then<br>
>>>> pull them out and compare them in the callback.<br>
>>><br>
>>><br>
>>> I see - make the matcher slightly more general, then filter the results,<br>
>>> perhaps with a visitor.<br>
>>><br>
>>>><br>
>>>><br>
>>>> Thoughts?<br>
>>>> /Manuel<br>
>>><br>
>>><br>
>>> This sounds like it would make at least half the work much easier, so I<br>
>>> think it would definitely be worth it to try switching to a matcher-based<br>
>>> solution. When are matchers supposed to hit mainline (or some extra<br>
>>> cloneable repo) :) ?<br>
>><br>
>><br>
>> Matchers are currently in<br>
>> ^cfe/branches/tooling/include/clang/ASTMatchers/...<br>
>><br>
>> I'm currently working on renaming them to camelCase from CamelCase;<br>
>> there's a Tool to help with the conversion though, so no problem in starting<br>
>> now ...<br>
>><br>
>> Cheers,<br>
>> /Manuel<br>
>><br>
>>><br>
>>><br>
>>> -Sam<br>
>>><br>
>>>><br>
>>>><br>
>>>>><br>
>>>>> allowed me to locate all matches in a given Stmt, it could be *much*<br>
>>>>> easier to express some parts of the logic, such as the first step in the<br>
>>>>> above list. I also suspect that a single-Stmt matcher would better way to<br>
>>>>> handle the last step; currently I track whether the visitor is looking at a<br>
>>>>> statement or expression which fits any of the const-disqualifying<br>
>>>>> conditions, and a note is made if I run into A[index].<br>
>>>>><br>
>>>>> Does this make the use case clearer? I don't really see a better way to<br>
>>>>> approach this problem, but you guys know the available toolkit far better<br>
>>>>> than I do.<br>
>>>>><br>
>>>>> On Fri, Jun 29, 2012 at 2:48 AM, Manuel Klimek <<a href="mailto:klimek@google.com">klimek@google.com</a>><br>
>>>>> wrote:<br>
>>>>>><br>
>>>>>> On Fri, Jun 29, 2012 at 11:45 AM, Chandler Carruth<br>
>>>>>> <<a href="mailto:chandlerc@google.com">chandlerc@google.com</a>> wrote:<br>
>>>>>>><br>
>>>>>>> I tend to agree that this should use the Tooling/Refactoring stuff.<br>
>>>>>>><br>
>>>>>>> However, I'm curious about the best way to structure the location of<br>
>>>>>>> migration candidates: AST matchers vs. visitor.<br>
>>>>>>><br>
>>>>>>> I can almost see the visitor pattern working really well here as each<br>
>>>>>>> different construct can have a pile of migration logic dropped in.... But if<br>
>>>>>>> there is a need to connect dots between more distant constructs, that<br>
>>>>>>> wouldn't work so well.... Not at all sure what would be best here.<br>
>>>>>><br>
>>>>>><br>
>>>>>> I've used a combination before - use matchers for the stuff where they<br>
>>>>>> work well, then write a very small easy-to-understand visitor if you need<br>
>>>>>> more. I think that brings down code size by quite a bit - obviously just a<br>
>>>>>> gut feeling here.<br>
>>>>>><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>> On Fri, Jun 29, 2012 at 1:37 AM, Manuel Klimek <<a href="mailto:klimek@google.com">klimek@google.com</a>><br>
>>>>>>> wrote:<br>
>>>>>>>><br>
>>>>>>>> On Fri, Jun 29, 2012 at 4:06 AM, Sam Panzer <<a href="mailto:panzer@google.com">panzer@google.com</a>><br>
>>>>>>>> wrote:<br>
>>>>>>>>><br>
>>>>>>>>> In case anyone wanted to take a look, the attached patch includes<br>
>>>>>>>>> the tool I've been working on. I create a new binary, c++migrate, which<br>
>>>>>>>>> attempts to convert for loops in the source files given to it. Most of my<br>
>>>>>>>>> focus has been on the FrontedAction, so I skirted all of the issues<br>
>>>>>>>>> mentioned above by keeping the frontend interaction minimal (i.e. I just<br>
>>>>>>>>> call Tooling::ClangTool::run), and the changes are just reported on standard<br>
>>>>>>>>> output, if there are any to be made.<br>
>>>>>>>>><br>
>>>>>>>>> The tool can currently convert for loops that range over (1)<br>
>>>>>>>>> statically allocated arrays, and (2) Clang-style iterator-based loops (with<br>
>>>>>>>>> begin and end iterators defined). All loop variables need to be declared<br>
>>>>>>>>> within the loop's initialization step in order for it to be converted,<br>
>>>>>>>>> though this requirement can potentially be eliminated. I'm working on<br>
>>>>>>>>> converting iterator-based loops that call someContainer.end() on each<br>
>>>>>>>>> iteration, since they're probably the common case in many codebases.<br>
>>>>>>>>><br>
>>>>>>>>> Just for fun, I ran the tool over the 41 .cpp files in lib/Sema,<br>
>>>>>>>>> and my tool found 71 convertible loops in 17 files. There is plenty more<br>
>>>>>>>>> work to go, because it clearly missed some easy ones.<br>
>>>>>>>>><br>
>>>>>>>>> Any input or feedback is welcome!<br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>>> High-level observations:<br>
>>>>>>>> 1. the handling of the rewrites; any reason not to use the<br>
>>>>>>>> Tooling/Refactoring stuff? Currently in the patch it looks to me like the<br>
>>>>>>>> files are not rewritten, but dumped to stdout<br>
>>>>>>>> 2. is the reason not to use the matchers here that they're not<br>
>>>>>>>> landed in mainline yet?<br>
>>>>>>>><br>
>>>>>>>> Cheers,<br>
>>>>>>>> /Manuel<br>
>>>>>>>><br>
>>>>>>>>><br>
>>>>>>>>><br>
>>>>>>>>> -Sam<br>
>>>>>>>>><br>
>>>>>>>>> On Thu, Jun 28, 2012 at 10:50 AM, Sam Panzer <<a href="mailto:panzer@google.com">panzer@google.com</a>><br>
>>>>>>>>> wrote:<br>
>>>>>>>>>><br>
>>>>>>>>>> I'm that intern :)<br>
>>>>>>>>>><br>
>>>>>>>>>> -Sam<br>
>>>>>>>>>><br>
>>>>>>>>>><br>
>>>>>>>>>> On Wed, Jun 27, 2012 at 9:48 PM, John Wiegley <<a href="mailto:johnw@boostpro.com">johnw@boostpro.com</a>><br>
>>>>>>>>>> wrote:<br>
>>>>>>>>>>><br>
>>>>>>>>>>> >>>>> Sam Panzer <<a href="mailto:panzer@google.com">panzer@google.com</a>> writes:<br>
>>>>>>>>>>><br>
>>>>>>>>>>> > In particular, I am working on a tool to convert existing C++<br>
>>>>>>>>>>> > for loops to<br>
>>>>>>>>>>> > take advantage of the new C++11 range-based syntax. I can<br>
>>>>>>>>>>> > imagine similar<br>
>>>>>>>>>>> > tools to replace const with constexpr, macro hacks with<br>
>>>>>>>>>>> > static_assert, and<br>
>>>>>>>>>>> > potentially other common refactorings.<br>
>>>>>>>>>>><br>
>>>>>>>>>>> > Thoughts? Suggestions?<br>
>>>>>>>>>>><br>
>>>>>>>>>>> You really must watch this presentation, if you haven't already:<br>
>>>>>>>>>>><br>
>>>>>>>>>>> <a href="http://www.youtube.com/watch?v=yuIOGfcOH0k" target="_blank">http://www.youtube.com/watch?v=yuIOGfcOH0k</a><br>
>>>>>>>>>>><br>
>>>>>>>>>>> --<br>
>>>>>>>>>>> John Wiegley<br>
>>>>>>>>>>> BoostPro Computing<br>
>>>>>>>>>>> <a href="http://www.boostpro.com" target="_blank">http://www.boostpro.com</a><br>
>>>>>>>>>>> _______________________________________________<br>
>>>>>>>>>>> cfe-dev mailing list<br>
>>>>>>>>>>> <a href="mailto:cfe-dev@cs.uiuc.edu">cfe-dev@cs.uiuc.edu</a><br>
>>>>>>>>>>> <a href="http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev</a><br>
>>>>>>>>>><br>
>>>>>>>>>><br>
>>>>>>>>><br>
>>>>>>>>><br>
>>>>>>>>> _______________________________________________<br>
>>>>>>>>> cfe-dev mailing list<br>
>>>>>>>>> <a href="mailto:cfe-dev@cs.uiuc.edu">cfe-dev@cs.uiuc.edu</a><br>
>>>>>>>>> <a href="http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev</a><br>
>>>>>>>>><br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>>> _______________________________________________<br>
>>>>>>>> cfe-dev mailing list<br>
>>>>>>>> <a href="mailto:cfe-dev@cs.uiuc.edu">cfe-dev@cs.uiuc.edu</a><br>
>>>>>>>> <a href="http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev</a><br>
>>>>>>>><br>
>>>>>>><br>
>>>>>><br>
>>>>><br>
>>>><br>
>>><br>
>><br>
><br>
><br>
> _______________________________________________<br>
> cfe-dev mailing list<br>
> <a href="mailto:cfe-dev@cs.uiuc.edu">cfe-dev@cs.uiuc.edu</a><br>
> <a href="http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev</a><br>
><br>
</div></div></blockquote></div><br></div>