<div dir="ltr">Ok, so if I understand you correctly, you're saying that we can abstract away differences in the AST that you'd previously have to handle yourself with common library functions. That definitely makes sense, cool!<br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jan 28, 2019 at 6:55 PM Yitzhak Mandelbaum <<a href="mailto:yitzhakm@google.com">yitzhakm@google.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div><span style="font-family:arial,helvetica,sans-serif"></span><font face="arial, helvetica, sans-serif">Manuel,</font></div><div><font face="arial, helvetica, sans-serif"><br></font></div><div><font face="arial, helvetica, sans-serif">Good question. I didn’t originally intend Stencil to be a stand-alone library, but as an integral part of Transformer. It is only as I’ve worked on it that I realized that it may in fact have value on its own. So, I’m still thinking this through. Here’s my logic:</font></div><div><font face="arial, helvetica, sans-serif"><br></font></div><div><font face="arial, helvetica, sans-serif">1. Abstraction level. The matchers work at the abstraction level of node identifiers. Stencil complements the matchers with string concatenation that operates as the same level. This allows users to write simple transformations without ever using the AST API. They need only understand the matchers and the string-concatenation model of Stencil.</font></div><div><font face="arial, helvetica, sans-serif"><br></font></div><div><font face="arial, helvetica, sans-serif">2. AST operations. For users that need more than simple transformations, Stencil also provides a (growing) assortment of AST operations. Ultimately, I’d like to collect all relevant AST codegen operations in the Stencil library. We could provide an API that works only on AST nodes, so we don’t need the string-concat functionality, but the combination makes the abstraction far more compelling, because you can do significant work without ever needing to learn how to extract pieces of source text with the AST API.</font></div><div><font face="arial, helvetica, sans-serif"><br></font></div><div><font face="arial, helvetica, sans-serif">3. clang-query integration. Stencil defines a mini-DSL, so we can naturally integrate it into clang-query. This will support writing (some) refactorings as clang-query scripts, as well as a richer tool for interactively exploring new refactoring ideas.</font></div><div><font face="arial, helvetica, sans-serif"><br></font></div><div><font face="arial, helvetica, sans-serif">Examples</font></div><div><font face="arial, helvetica, sans-serif"><br></font></div><div><font face="arial, helvetica, sans-serif">The tests are not great examples since they are designed as unit tests for single features rather than demos. That said, the test `MemberOpWithNameOp` combines two features in an interesting way:</font></div><div><font face="arial, helvetica, sans-serif"><br></font></div><div><font face="arial, helvetica, sans-serif">TEST_F(StencilTest, MemberOpWithNameOp) {</font></div><div><font face="arial, helvetica, sans-serif"> const std::string Snippet = R"cc(</font></div><div><font face="arial, helvetica, sans-serif"> int object;</font></div><div><font face="arial, helvetica, sans-serif"> int* method = &object;</font></div><div><font face="arial, helvetica, sans-serif"> (void)method;</font></div><div><font face="arial, helvetica, sans-serif"> return object;</font></div><div><font face="arial, helvetica, sans-serif"> )cc";</font></div><div><font face="arial, helvetica, sans-serif"> auto StmtMatch = matchStmt(</font></div><div><font face="arial, helvetica, sans-serif"> Snippet, declStmt(hasSingleDecl(</font></div><div><font face="arial, helvetica, sans-serif"> varDecl(hasInitializer(expr().bind("e"))).bind("d"))));</font></div><div><font face="arial, helvetica, sans-serif"> ASSERT_TRUE(StmtMatch);</font></div><div><font face="arial, helvetica, sans-serif"> auto Stencil = Stencil::cat(member("e", name("d")));</font></div><div><font face="arial, helvetica, sans-serif"> EXPECT_THAT(toOptional(Stencil.eval(StmtMatch->Result)),</font></div><div><font face="arial, helvetica, sans-serif"> IsSomething(Eq("object.method")));</font></div><div><font face="arial, helvetica, sans-serif"> EXPECT_THAT(Stencil.addedIncludes(), testing::IsEmpty());</font></div><div><font face="arial, helvetica, sans-serif"> EXPECT_THAT(Stencil.removedIncludes(), testing::IsEmpty());</font></div><div><font face="arial, helvetica, sans-serif">}</font></div><div><font face="arial, helvetica, sans-serif"><br></font></div><div><font face="arial, helvetica, sans-serif">A slightly more realistic example involves maps. For a map `s`, change `s.count(k)` in a boolean context to `s.contains(k)`:</font></div><div><font face="arial, helvetica, sans-serif"><br></font></div><div><font face="arial, helvetica, sans-serif">Matcher:</font></div><div><font face="arial, helvetica, sans-serif"> castExpr(hasCastKind(clang::CK_IntegralToBoolean),</font></div><div><font face="arial, helvetica, sans-serif"> hasSourceExpression(cxxMemberCallExpr(</font></div><div><font face="arial, helvetica, sans-serif"> on(expr().bind(“s”), </font></div><div><font face="arial, helvetica, sans-serif"> anyOf(hasType(FlatHashSetType), hasType(pointsTo(FlatHashSetType)))),</font></div><div><font face="arial, helvetica, sans-serif"> callee(cxxMethodDecl(hasName("count"))),</font></div><div><font face="arial, helvetica, sans-serif"> hasArgument(0, expr().bind("k")))))</font></div><div><font face="arial, helvetica, sans-serif"><br></font></div><div><font face="arial, helvetica, sans-serif">The code gen is:</font></div><div><font face="arial, helvetica, sans-serif"> Stencil::Cat(Member("s", "contains"), "(", Node("k"), ")")</font></div><div><font face="arial, helvetica, sans-serif">and this handles both the s.count and s->count cases.</font></div><div style="font-family:arial,helvetica,sans-serif"></div><div style="font-family:arial,helvetica,sans-serif"><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail-m_5722666651174684208gmail_attr">On Fri, Jan 25, 2019 at 5:13 AM Jonas Toth <<a href="mailto:development@jonas-toth.eu" target="_blank">development@jonas-toth.eu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Thanks for clarification. Having it directly in clang-tidy would
be nice though :)<br>
</p>
<div class="gmail-m_5722666651174684208gmail-m_-5850422716770053156moz-cite-prefix">Am 24.01.19 um 12:17 schrieb Manuel
Klimek:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div dir="ltr" class="gmail-m_5722666651174684208gmail-m_-5850422716770053156gmail_attr">On Thu, Jan 24, 2019 at
11:56 AM Jonas Toth <<a href="mailto:development@jonas-toth.eu" target="_blank">development@jonas-toth.eu</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi,</p>
<p>one question that came to my mind because of Joel's
Mail:</p>
<p>Is it possible to run in multiple passes?<br>
My use-case is adding `const` to variables that could be
'const' but aren't. If you declare multiple variables
like `int not_const, const_variable;`<br>
the transformation requires splitting these up, first
-> `int not_const; int const_variable;` and then do
the `const` transformation -> `int not_const; const
int const_variable;`.</p>
<p>I have implemented both transformations in clang-tidy
(only partially landed yet) but there is no easy way I
can run them in one check. The current workflow<br>
pretty much forces us to run clang-tidy multiple times
and converge to the final solution.</p>
<p>If your framework could give an improvement in this
place, would be awesome! And I think worth to consider
anyway If we change the way<br>
we do transformations.</p>
</div>
</blockquote>
<div><br>
</div>
<div>Generally, you can already do this, but it'd be outside
clang-tidy. You can write a clang tool that</div>
<div>a) runs over the codebase and stores the "const graph"</div>
<div>b) load the "const graph" into memory and generate all
the replacements (there's a way to create yaml replacements)</div>
<div>c) apply all the replacements</div>
<div><br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Best, Jonas<br>
</p>
<div class="gmail-m_5722666651174684208gmail-m_-5850422716770053156gmail-m_-7548564849762917817moz-cite-prefix">Am
16.11.18 um 16:22 schrieb Yitzhak Mandelbaum via
cfe-dev:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div><font face="arial, helvetica,
sans-serif">Hi all,</font></div>
<div><font face="arial, helvetica,
sans-serif"><br>
</font></div>
<div><font face="arial, helvetica,
sans-serif">I have a proposal for a framework
that makes it easier to write source to source
transformations with the clang::Tooling
libraries, including clang-tidy checks.</font></div>
<div><font face="arial, helvetica,
sans-serif"><br>
</font></div>
<div><font face="arial, helvetica,
sans-serif">The full proposal is in this doc:</font></div>
<div><font face="arial, helvetica,
sans-serif"><br>
</font></div>
<div><font face="arial, helvetica,
sans-serif"><a href="https://docs.google.com/document/d/1ppw0RhjwsrbBcHYhI85pe6ISDbA6r5d00ot3N8cQWeQ/edit?usp=sharing" target="_blank">https://docs.google.com/document/d/1ppw0RhjwsrbBcHYhI85pe6ISDbA6r5d00ot3N8cQWeQ/edit?usp=sharing</a></font><br>
</div>
<div><font face="arial, helvetica,
sans-serif"><br>
</font></div>
<div><font face="arial, helvetica,
sans-serif">From the doc:</font></div>
<div><font face="arial, helvetica,
sans-serif">Transformer is a framework that
aims to simplify development of clang-based
source-to-source transformations. It focuses
on the particular class of transformations
that act only locally — that is, use local
information about the code and make local
changes (like a syntax-aware
“find-and-replace”); and at scale — that is,
will be carried out on many source files. The
target audience is users that are comfortable
with, or willing to become comfortable with,
the clang AST matchers library.<br>
</font></div>
<div><font face="arial, helvetica,
sans-serif"><br>
</font></div>
<div><font face="arial, helvetica,
sans-serif">I have a working prototype of this
library which I've used on small examples
inside Google. I plan to put together a patch
for reference next week, although the doc
should stand on its own.</font></div>
<div><font face="arial, helvetica,
sans-serif"><br>
</font></div>
<div><font face="arial, helvetica,
sans-serif">Thanks!</font></div>
<div><font face="arial, helvetica,
sans-serif">Yitzhak Mandelbaum</font></div>
</div>
</div>
</div>
<br>
<fieldset class="gmail-m_5722666651174684208gmail-m_-5850422716770053156gmail-m_-7548564849762917817mimeAttachmentHeader"></fieldset>
<pre class="gmail-m_5722666651174684208gmail-m_-5850422716770053156gmail-m_-7548564849762917817moz-quote-pre">_______________________________________________
cfe-dev mailing list
<a class="gmail-m_5722666651174684208gmail-m_-5850422716770053156gmail-m_-7548564849762917817moz-txt-link-abbreviated" href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>
<a class="gmail-m_5722666651174684208gmail-m_-5850422716770053156gmail-m_-7548564849762917817moz-txt-link-freetext" href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a>
</pre>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</blockquote></div>
</blockquote></div></div>