[cfe-dev] [RFC] Easier source-to-source transformations with clang tooling
Yitzhak Mandelbaum via cfe-dev
cfe-dev at lists.llvm.org
Mon Jan 28 10:05:44 PST 2019
Jonas, Joel,
I would like to eventually support easy pipelining of transformations.
Jonas -- to your example, I would like for you to be able to express both
transformations as Transformer rules and then ask Transformer to serialize
them into one combined rule that can be used as a clang-tidy. As for how
and when:
* how: based on discussions with Sam McCall, it seems like we might be able
to integrate with Clang’s “preamble” support (used by libclang/clangd) or
(more forward looking) to integrate with the modules support. Either one
should allows us to efficiently recompile from a change. A crazier idea
would be to consider an alternative AST which is designed for efficient
modification. However, this latter approach would involve a larger
community discussion.
* when: that very much depends on who's interested in working on the
problem. On my own, I don't think I'll get to it before late March.
On Mon, Jan 28, 2019 at 12:54 PM Yitzhak Mandelbaum <yitzhakm at google.com>
wrote:
> Manuel,
>
> Good question. I didn’t originally intend Stencil to be a stand-alone
> library, but as an integral part of Transformer. It is only as I’ve worked
> on it that I realized that it may in fact have value on its own. So, I’m
> still thinking this through. Here’s my logic:
>
> 1. Abstraction level. The matchers work at the abstraction level of node
> identifiers. Stencil complements the matchers with string
> concatenation that operates as the same level. This allows users to write
> simple transformations without ever using the AST API. They need only
> understand the matchers and the string-concatenation model of Stencil.
>
> 2. AST operations. For users that need more than simple transformations,
> Stencil also provides a (growing) assortment of AST operations.
> Ultimately, I’d like to collect all relevant AST codegen operations in the
> Stencil library. We could provide an API that works only on AST nodes, so
> we don’t need the string-concat functionality, but the combination makes
> the abstraction far more compelling, because you can do significant work
> without ever needing to learn how to extract pieces of source text with the
> AST API.
>
> 3. clang-query integration. Stencil defines a mini-DSL, so we can
> naturally integrate it into clang-query. This will support writing (some)
> refactorings as clang-query scripts, as well as a richer tool for
> interactively exploring new refactoring ideas.
>
> Examples
>
> The tests are not great examples since they are designed as unit tests for
> single features rather than demos. That said, the test
> `MemberOpWithNameOp` combines two features in an interesting way:
>
> TEST_F(StencilTest, MemberOpWithNameOp) {
> const std::string Snippet = R"cc(
> int object;
> int* method = &object;
> (void)method;
> return object;
> )cc";
> auto StmtMatch = matchStmt(
> Snippet, declStmt(hasSingleDecl(
> varDecl(hasInitializer(expr().bind("e"))).bind("d"))));
> ASSERT_TRUE(StmtMatch);
> auto Stencil = Stencil::cat(member("e", name("d")));
> EXPECT_THAT(toOptional(Stencil.eval(StmtMatch->Result)),
> IsSomething(Eq("object.method")));
> EXPECT_THAT(Stencil.addedIncludes(), testing::IsEmpty());
> EXPECT_THAT(Stencil.removedIncludes(), testing::IsEmpty());
> }
>
> A slightly more realistic example involves maps. For a map `s`, change
> `s.count(k)` in a boolean context to `s.contains(k)`:
>
> Matcher:
> castExpr(hasCastKind(clang::CK_IntegralToBoolean),
> hasSourceExpression(cxxMemberCallExpr(
> on(expr().bind(“s”),
> anyOf(hasType(FlatHashSetType),
> hasType(pointsTo(FlatHashSetType)))),
> callee(cxxMethodDecl(hasName("count"))),
> hasArgument(0, expr().bind("k")))))
>
> The code gen is:
> Stencil::Cat(Member("s", "contains"), "(", Node("k"), ")")
> and this handles both the s.count and s->count cases.
>
>
> On Fri, Jan 25, 2019 at 5:13 AM Jonas Toth <development at jonas-toth.eu>
> wrote:
>
>> Thanks for clarification. Having it directly in clang-tidy would be nice
>> though :)
>> Am 24.01.19 um 12:17 schrieb Manuel Klimek:
>>
>> On Thu, Jan 24, 2019 at 11:56 AM Jonas Toth <development at jonas-toth.eu>
>> wrote:
>>
>>> Hi,
>>>
>>> one question that came to my mind because of Joel's Mail:
>>>
>>> Is it possible to run in multiple passes?
>>> My use-case is adding `const` to variables that could be 'const' but
>>> aren't. If you declare multiple variables like `int not_const,
>>> const_variable;`
>>> the transformation requires splitting these up, first -> `int not_const;
>>> int const_variable;` and then do the `const` transformation -> `int
>>> not_const; const int const_variable;`.
>>>
>>> I have implemented both transformations in clang-tidy (only partially
>>> landed yet) but there is no easy way I can run them in one check. The
>>> current workflow
>>> pretty much forces us to run clang-tidy multiple times and converge to
>>> the final solution.
>>>
>>> If your framework could give an improvement in this place, would be
>>> awesome! And I think worth to consider anyway If we change the way
>>> we do transformations.
>>>
>>
>> Generally, you can already do this, but it'd be outside clang-tidy. You
>> can write a clang tool that
>> a) runs over the codebase and stores the "const graph"
>> b) load the "const graph" into memory and generate all the replacements
>> (there's a way to create yaml replacements)
>> c) apply all the replacements
>>
>>
>>
>>> Best, Jonas
>>> Am 16.11.18 um 16:22 schrieb Yitzhak Mandelbaum via cfe-dev:
>>>
>>> Hi all,
>>>
>>> I have a proposal for a framework that makes it easier to write source
>>> to source transformations with the clang::Tooling libraries, including
>>> clang-tidy checks.
>>>
>>> The full proposal is in this doc:
>>>
>>>
>>> https://docs.google.com/document/d/1ppw0RhjwsrbBcHYhI85pe6ISDbA6r5d00ot3N8cQWeQ/edit?usp=sharing
>>>
>>> From the doc:
>>> Transformer is a framework that aims to simplify development of
>>> clang-based source-to-source transformations. It focuses on the particular
>>> class of transformations that act only locally — that is, use local
>>> information about the code and make local changes (like a syntax-aware
>>> “find-and-replace”); and at scale — that is, will be carried out on many
>>> source files. The target audience is users that are comfortable with, or
>>> willing to become comfortable with, the clang AST matchers library.
>>>
>>> I have a working prototype of this library which I've used on small
>>> examples inside Google. I plan to put together a patch for reference
>>> next week, although the doc should stand on its own.
>>>
>>> Thanks!
>>> Yitzhak Mandelbaum
>>>
>>> _______________________________________________
>>> cfe-dev mailing listcfe-dev at lists.llvm.orghttp://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20190128/099876c5/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4847 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20190128/099876c5/attachment.bin>
More information about the cfe-dev
mailing list