[cfe-dev] [RFC] Easier source-to-source transformations with clang tooling
Yitzhak Mandelbaum via cfe-dev
cfe-dev at lists.llvm.org
Mon Jan 28 10:16:33 PST 2019
Kim,
Not a luddite question at all! I'm still exploring Stencil's relevance
outside the context of Transformer. I haven't written any AST traversals
myself, so I can't claim any special knowledge of that would fit with
Stencils, but I could imagine it might still be useful. Can I ask what
advantages you'd see in that context over string concatenation? I'm
thinking that without bound names, you lose the first and third selling
point I mentioned in my response to Manuel to this question, but point 2
still stands. We'd just to add a new operation to the stencil generators to
inject an ast node pointer/reference into the stencil. e.g. we could add
overloads to Node() for clang::Stmt, clang::Decl, etc. I'd love to get your
take, though.
On Mon, Jan 28, 2019 at 1:05 PM Yitzhak Mandelbaum <yitzhakm at google.com>
wrote:
> Jonas, Joel,
>
> I would like to eventually support easy pipelining of transformations.
> Jonas -- to your example, I would like for you to be able to express both
> transformations as Transformer rules and then ask Transformer to serialize
> them into one combined rule that can be used as a clang-tidy. As for how
> and when:
>
> * how: based on discussions with Sam McCall, it seems like we might be
> able to integrate with Clang’s “preamble” support (used by libclang/clangd)
> or (more forward looking) to integrate with the modules support. Either
> one should allows us to efficiently recompile from a change. A crazier
> idea would be to consider an alternative AST which is designed for
> efficient modification. However, this latter approach would involve a
> larger community discussion.
>
> * when: that very much depends on who's interested in working on the
> problem. On my own, I don't think I'll get to it before late March.
>
> On Mon, Jan 28, 2019 at 12:54 PM Yitzhak Mandelbaum <yitzhakm at google.com>
> wrote:
>
>> Manuel,
>>
>> Good question. I didn’t originally intend Stencil to be a stand-alone
>> library, but as an integral part of Transformer. It is only as I’ve worked
>> on it that I realized that it may in fact have value on its own. So, I’m
>> still thinking this through. Here’s my logic:
>>
>> 1. Abstraction level. The matchers work at the abstraction level of node
>> identifiers. Stencil complements the matchers with string
>> concatenation that operates as the same level. This allows users to write
>> simple transformations without ever using the AST API. They need only
>> understand the matchers and the string-concatenation model of Stencil.
>>
>> 2. AST operations. For users that need more than simple transformations,
>> Stencil also provides a (growing) assortment of AST operations.
>> Ultimately, I’d like to collect all relevant AST codegen operations in the
>> Stencil library. We could provide an API that works only on AST nodes, so
>> we don’t need the string-concat functionality, but the combination makes
>> the abstraction far more compelling, because you can do significant work
>> without ever needing to learn how to extract pieces of source text with the
>> AST API.
>>
>> 3. clang-query integration. Stencil defines a mini-DSL, so we can
>> naturally integrate it into clang-query. This will support writing (some)
>> refactorings as clang-query scripts, as well as a richer tool for
>> interactively exploring new refactoring ideas.
>>
>> Examples
>>
>> The tests are not great examples since they are designed as unit tests
>> for single features rather than demos. That said, the test
>> `MemberOpWithNameOp` combines two features in an interesting way:
>>
>> TEST_F(StencilTest, MemberOpWithNameOp) {
>> const std::string Snippet = R"cc(
>> int object;
>> int* method = &object;
>> (void)method;
>> return object;
>> )cc";
>> auto StmtMatch = matchStmt(
>> Snippet, declStmt(hasSingleDecl(
>> varDecl(hasInitializer(expr().bind("e"))).bind("d"))));
>> ASSERT_TRUE(StmtMatch);
>> auto Stencil = Stencil::cat(member("e", name("d")));
>> EXPECT_THAT(toOptional(Stencil.eval(StmtMatch->Result)),
>> IsSomething(Eq("object.method")));
>> EXPECT_THAT(Stencil.addedIncludes(), testing::IsEmpty());
>> EXPECT_THAT(Stencil.removedIncludes(), testing::IsEmpty());
>> }
>>
>> A slightly more realistic example involves maps. For a map `s`, change
>> `s.count(k)` in a boolean context to `s.contains(k)`:
>>
>> Matcher:
>> castExpr(hasCastKind(clang::CK_IntegralToBoolean),
>> hasSourceExpression(cxxMemberCallExpr(
>> on(expr().bind(“s”),
>> anyOf(hasType(FlatHashSetType),
>> hasType(pointsTo(FlatHashSetType)))),
>> callee(cxxMethodDecl(hasName("count"))),
>> hasArgument(0, expr().bind("k")))))
>>
>> The code gen is:
>> Stencil::Cat(Member("s", "contains"), "(", Node("k"), ")")
>> and this handles both the s.count and s->count cases.
>>
>>
>> On Fri, Jan 25, 2019 at 5:13 AM Jonas Toth <development at jonas-toth.eu>
>> wrote:
>>
>>> Thanks for clarification. Having it directly in clang-tidy would be nice
>>> though :)
>>> Am 24.01.19 um 12:17 schrieb Manuel Klimek:
>>>
>>> On Thu, Jan 24, 2019 at 11:56 AM Jonas Toth <development at jonas-toth.eu>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> one question that came to my mind because of Joel's Mail:
>>>>
>>>> Is it possible to run in multiple passes?
>>>> My use-case is adding `const` to variables that could be 'const' but
>>>> aren't. If you declare multiple variables like `int not_const,
>>>> const_variable;`
>>>> the transformation requires splitting these up, first -> `int
>>>> not_const; int const_variable;` and then do the `const` transformation ->
>>>> `int not_const; const int const_variable;`.
>>>>
>>>> I have implemented both transformations in clang-tidy (only partially
>>>> landed yet) but there is no easy way I can run them in one check. The
>>>> current workflow
>>>> pretty much forces us to run clang-tidy multiple times and converge to
>>>> the final solution.
>>>>
>>>> If your framework could give an improvement in this place, would be
>>>> awesome! And I think worth to consider anyway If we change the way
>>>> we do transformations.
>>>>
>>>
>>> Generally, you can already do this, but it'd be outside clang-tidy. You
>>> can write a clang tool that
>>> a) runs over the codebase and stores the "const graph"
>>> b) load the "const graph" into memory and generate all the replacements
>>> (there's a way to create yaml replacements)
>>> c) apply all the replacements
>>>
>>>
>>>
>>>> Best, Jonas
>>>> Am 16.11.18 um 16:22 schrieb Yitzhak Mandelbaum via cfe-dev:
>>>>
>>>> Hi all,
>>>>
>>>> I have a proposal for a framework that makes it easier to write source
>>>> to source transformations with the clang::Tooling libraries, including
>>>> clang-tidy checks.
>>>>
>>>> The full proposal is in this doc:
>>>>
>>>>
>>>> https://docs.google.com/document/d/1ppw0RhjwsrbBcHYhI85pe6ISDbA6r5d00ot3N8cQWeQ/edit?usp=sharing
>>>>
>>>> From the doc:
>>>> Transformer is a framework that aims to simplify development of
>>>> clang-based source-to-source transformations. It focuses on the particular
>>>> class of transformations that act only locally — that is, use local
>>>> information about the code and make local changes (like a syntax-aware
>>>> “find-and-replace”); and at scale — that is, will be carried out on many
>>>> source files. The target audience is users that are comfortable with, or
>>>> willing to become comfortable with, the clang AST matchers library.
>>>>
>>>> I have a working prototype of this library which I've used on small
>>>> examples inside Google. I plan to put together a patch for reference
>>>> next week, although the doc should stand on its own.
>>>>
>>>> Thanks!
>>>> Yitzhak Mandelbaum
>>>>
>>>> _______________________________________________
>>>> cfe-dev mailing listcfe-dev at lists.llvm.orghttp://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>>
>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20190128/691aa63b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4847 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20190128/691aa63b/attachment.bin>
More information about the cfe-dev
mailing list