[cfe-dev] [RFC] Easier source-to-source transformations with clang tooling
Joel E. Denny via cfe-dev
cfe-dev at lists.llvm.org
Mon Jan 28 13:53:14 PST 2019
Hi Yitzhak,
On Mon, Jan 28, 2019 at 1:06 PM Yitzhak Mandelbaum via cfe-dev <
cfe-dev at lists.llvm.org> wrote:
> Jonas, Joel,
>
> I would like to eventually support easy pipelining of transformations.
> Jonas -- to your example, I would like for you to be able to express both
> transformations as Transformer rules and then ask Transformer to serialize
> them into one combined rule that can be used as a clang-tidy. As for how
> and when:
>
> * how: based on discussions with Sam McCall, it seems like we might be
> able to integrate with Clang’s “preamble” support (used by libclang/clangd)
> or (more forward looking) to integrate with the modules support. Either
> one should allows us to efficiently recompile from a change.
>
Interesting. I have only a passing familiarity with these representations.
I know that some parts of Sema build data structures while parsing a
particular scope and then destroy those data structures when leaving that
scope. That seems to mean that, without significant refactoring of Sema,
there is some non-trivial minimum granularity at which incremental
compilation could run because it has to rebuild such data structures. For
example, I'd be surprised if, in general, it's possible to compile part of
a function without recompiling the entire function.
Weren't the preamble and module representations originally designed for
precompiled headers? Does that mean their granularity is, at smallest, the
size of a file scope declaration? Actually, using those representations,
would you be able to recompile part of a file without recompiling the
remainder of the file?
It would be cool if this strategy turns out to be efficient enough for
practical source-to-source use cases.
>
> A crazier idea would be to consider an alternative AST which is designed
> for efficient modification. However, this latter approach would involve a
> larger community discussion.
>
Probably so. But that should offer finer-grained modifications.
Thanks!
Joel
>
> * when: that very much depends on who's interested in working on the
> problem. On my own, I don't think I'll get to it before late March.
>
> On Mon, Jan 28, 2019 at 12:54 PM Yitzhak Mandelbaum <yitzhakm at google.com>
> wrote:
>
>> Manuel,
>>
>> Good question. I didn’t originally intend Stencil to be a stand-alone
>> library, but as an integral part of Transformer. It is only as I’ve worked
>> on it that I realized that it may in fact have value on its own. So, I’m
>> still thinking this through. Here’s my logic:
>>
>> 1. Abstraction level. The matchers work at the abstraction level of node
>> identifiers. Stencil complements the matchers with string
>> concatenation that operates as the same level. This allows users to write
>> simple transformations without ever using the AST API. They need only
>> understand the matchers and the string-concatenation model of Stencil.
>>
>> 2. AST operations. For users that need more than simple transformations,
>> Stencil also provides a (growing) assortment of AST operations.
>> Ultimately, I’d like to collect all relevant AST codegen operations in the
>> Stencil library. We could provide an API that works only on AST nodes, so
>> we don’t need the string-concat functionality, but the combination makes
>> the abstraction far more compelling, because you can do significant work
>> without ever needing to learn how to extract pieces of source text with the
>> AST API.
>>
>> 3. clang-query integration. Stencil defines a mini-DSL, so we can
>> naturally integrate it into clang-query. This will support writing (some)
>> refactorings as clang-query scripts, as well as a richer tool for
>> interactively exploring new refactoring ideas.
>>
>> Examples
>>
>> The tests are not great examples since they are designed as unit tests
>> for single features rather than demos. That said, the test
>> `MemberOpWithNameOp` combines two features in an interesting way:
>>
>> TEST_F(StencilTest, MemberOpWithNameOp) {
>> const std::string Snippet = R"cc(
>> int object;
>> int* method = &object;
>> (void)method;
>> return object;
>> )cc";
>> auto StmtMatch = matchStmt(
>> Snippet, declStmt(hasSingleDecl(
>> varDecl(hasInitializer(expr().bind("e"))).bind("d"))));
>> ASSERT_TRUE(StmtMatch);
>> auto Stencil = Stencil::cat(member("e", name("d")));
>> EXPECT_THAT(toOptional(Stencil.eval(StmtMatch->Result)),
>> IsSomething(Eq("object.method")));
>> EXPECT_THAT(Stencil.addedIncludes(), testing::IsEmpty());
>> EXPECT_THAT(Stencil.removedIncludes(), testing::IsEmpty());
>> }
>>
>> A slightly more realistic example involves maps. For a map `s`, change
>> `s.count(k)` in a boolean context to `s.contains(k)`:
>>
>> Matcher:
>> castExpr(hasCastKind(clang::CK_IntegralToBoolean),
>> hasSourceExpression(cxxMemberCallExpr(
>> on(expr().bind(“s”),
>> anyOf(hasType(FlatHashSetType),
>> hasType(pointsTo(FlatHashSetType)))),
>> callee(cxxMethodDecl(hasName("count"))),
>> hasArgument(0, expr().bind("k")))))
>>
>> The code gen is:
>> Stencil::Cat(Member("s", "contains"), "(", Node("k"), ")")
>> and this handles both the s.count and s->count cases.
>>
>>
>> On Fri, Jan 25, 2019 at 5:13 AM Jonas Toth <development at jonas-toth.eu>
>> wrote:
>>
>>> Thanks for clarification. Having it directly in clang-tidy would be nice
>>> though :)
>>> Am 24.01.19 um 12:17 schrieb Manuel Klimek:
>>>
>>> On Thu, Jan 24, 2019 at 11:56 AM Jonas Toth <development at jonas-toth.eu>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> one question that came to my mind because of Joel's Mail:
>>>>
>>>> Is it possible to run in multiple passes?
>>>> My use-case is adding `const` to variables that could be 'const' but
>>>> aren't. If you declare multiple variables like `int not_const,
>>>> const_variable;`
>>>> the transformation requires splitting these up, first -> `int
>>>> not_const; int const_variable;` and then do the `const` transformation ->
>>>> `int not_const; const int const_variable;`.
>>>>
>>>> I have implemented both transformations in clang-tidy (only partially
>>>> landed yet) but there is no easy way I can run them in one check. The
>>>> current workflow
>>>> pretty much forces us to run clang-tidy multiple times and converge to
>>>> the final solution.
>>>>
>>>> If your framework could give an improvement in this place, would be
>>>> awesome! And I think worth to consider anyway If we change the way
>>>> we do transformations.
>>>>
>>>
>>> Generally, you can already do this, but it'd be outside clang-tidy. You
>>> can write a clang tool that
>>> a) runs over the codebase and stores the "const graph"
>>> b) load the "const graph" into memory and generate all the replacements
>>> (there's a way to create yaml replacements)
>>> c) apply all the replacements
>>>
>>>
>>>
>>>> Best, Jonas
>>>> Am 16.11.18 um 16:22 schrieb Yitzhak Mandelbaum via cfe-dev:
>>>>
>>>> Hi all,
>>>>
>>>> I have a proposal for a framework that makes it easier to write source
>>>> to source transformations with the clang::Tooling libraries, including
>>>> clang-tidy checks.
>>>>
>>>> The full proposal is in this doc:
>>>>
>>>>
>>>> https://docs.google.com/document/d/1ppw0RhjwsrbBcHYhI85pe6ISDbA6r5d00ot3N8cQWeQ/edit?usp=sharing
>>>>
>>>> From the doc:
>>>> Transformer is a framework that aims to simplify development of
>>>> clang-based source-to-source transformations. It focuses on the particular
>>>> class of transformations that act only locally — that is, use local
>>>> information about the code and make local changes (like a syntax-aware
>>>> “find-and-replace”); and at scale — that is, will be carried out on many
>>>> source files. The target audience is users that are comfortable with, or
>>>> willing to become comfortable with, the clang AST matchers library.
>>>>
>>>> I have a working prototype of this library which I've used on small
>>>> examples inside Google. I plan to put together a patch for reference
>>>> next week, although the doc should stand on its own.
>>>>
>>>> Thanks!
>>>> Yitzhak Mandelbaum
>>>>
>>>> _______________________________________________
>>>> cfe-dev mailing listcfe-dev at lists.llvm.orghttp://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>>
>>>> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20190128/576e50dc/attachment.html>
More information about the cfe-dev
mailing list