[cfe-dev] AST Writer
Matthieu Brucher via cfe-dev
cfe-dev at lists.llvm.org
Wed Jul 18 03:23:09 PDT 2018
Thanks. Hopefully, it won't be too ugly to tear out a basic work piece out
of it. None of the tutorials online tackle this aspect 😩
Cheers
Matthieu
Le mer. 18 juil. 2018 à 11:20, mats petersson <mats at planetcatfish.com> a
écrit :
> Try having a look at an OpenCL implementation - pocl is the one that comes
> to mind. OpenCL relies on taking a string and outputting code, all in
> memory [the spec doesn't precisely say you can't generate a file and
> compile that through a standalone executable, but that's not exactly a
> "nice" solution].
>
> I work on ARM's OpenCL solution, so I'm not familiar with the details of
> the pocl, but I'm 100% sure that they do something similar to what we do -
> build/take a string, call various parts of clang functions, and produce a
> binary executable in memory.
>
> It may not be 100% like what you want to do, but it should give you
> something to start from.
>
> --
> Mats
>
> On 17 July 2018 at 19:08, Matthieu Brucher via cfe-dev <
> cfe-dev at lists.llvm.org> wrote:
>
>> Hi,
>>
>> I can easily add source code to a file. It's a no brainer, I'm not going
>> to use clang for this, it's overkill.
>> What doesn't work, as stated in my previous example, is getting a module
>> out of clang, a module that can be used inside llvm. When executing the
>> code below, I get a write error. That's a problem because there are no
>> resources online on this issue. The api changes too quickly for this, and
>> even lib clang doesn't help because then the triple is not set (and then
>> llvm breaks).
>>
>> Cheers
>> Matthieu
>>
>> Le mar. 17 juil. 2018 à 18:25, Firat Kasmis <firatkasmis at gmail.com> a
>> écrit :
>>
>>> Matthieu, try https://github.com/firolino/clang-tool as getting started
>>> and change the transformer to your needs to insert code/text at a given
>>> location. Hope it helps.
>>>
>>> Best,
>>> Firat
>>>
>>> Am Di., 17. Juli 2018 um 09:00 Uhr schrieb Matthieu Brucher via cfe-dev <
>>> cfe-dev at lists.llvm.org>:
>>>
>>>> Indeed, that's what I'm now aiming at. Unfortunately, it seems that
>>>> there are no examples as how to use FrontEndAction properly with clang
>>>> 6.0.0. I can use libeling with runToolOnCode to generate a module, but the
>>>> triple is not set up properly in that case when I want to use the JIT. And
>>>> it seems to be a problem with clang, as if I do this:
>>>>
>>>> clang::DiagnosticOptions diagnosticOptions;
>>>> std::unique_ptr<clang::TextDiagnosticPrinter> textDiagnosticPrinter =
>>>> std::make_unique<clang::TextDiagnosticPrinter>(llvm::outs(),
>>>> &diagnosticOptions);
>>>> llvm::IntrusiveRefCntPtr<clang::DiagnosticIDs> diagIDs;
>>>>
>>>> std::unique_ptr<clang::DiagnosticsEngine> diagnosticsEngine =
>>>> std::make_unique<clang::DiagnosticsEngine>(diagIDs, &diagnosticOptions, textDiagnosticPrinter.get());
>>>>
>>>> clang::LangOptions languageOptions;
>>>> clang::FileSystemOptions fileSystemOptions;
>>>> clang::FileManager fileManager(fileSystemOptions);
>>>> clang::SourceManager sourceManager(*diagnosticsEngine,
>>>> fileManager);
>>>> std::shared_ptr<clang::HeaderSearchOptions> headerSearchOptions(new clang::HeaderSearchOptions());
>>>>
>>>> const std::shared_ptr<clang::TargetOptions> targetOptions = std::make_shared<clang::TargetOptions>();
>>>> targetOptions->Triple = llvm::sys::getDefaultTargetTriple();
>>>>
>>>> std::unique_ptr<clang::TargetInfo> targetInfo(
>>>> clang::TargetInfo::CreateTargetInfo(*diagnosticsEngine, targetOptions));
>>>>
>>>> clang::HeaderSearch headerSearch(headerSearchOptions,
>>>> sourceManager,
>>>> *diagnosticsEngine,
>>>> languageOptions,
>>>> targetInfo.get());
>>>> clang::MemoryBufferCache PCMCache;
>>>> clang::CompilerInstance compInst;
>>>>
>>>> std::shared_ptr<clang::PreprocessorOptions> opts(std::make_shared<clang::PreprocessorOptions>());
>>>> clang::Preprocessor preprocessor(opts,
>>>> *diagnosticsEngine,
>>>> languageOptions,
>>>> sourceManager,
>>>> PCMCache,
>>>> headerSearch,
>>>> compInst);
>>>> preprocessor.Initialize(*targetInfo);
>>>>
>>>> auto filter = llvm::MemoryBuffer::getMemBufferCopy(fullfile);
>>>>
>>>> sourceManager.setMainFileID(sourceManager.createFileID(std::move(filter)));
>>>>
>>>> clang::IdentifierTable identifierTable(languageOptions);
>>>> clang::SelectorTable selectorTable;
>>>>
>>>> clang::Builtin::Context builtinContext;
>>>> builtinContext.InitializeTarget(*targetInfo, nullptr);
>>>> clang::ASTContext astContext(languageOptions,
>>>> sourceManager,
>>>> identifierTable,
>>>> selectorTable,
>>>> builtinContext);
>>>> astContext.InitBuiltinTypes(*targetInfo);
>>>> compInst.setTarget(targetInfo.get());
>>>>
>>>> llvm::LLVMContext context;
>>>> std::unique_ptr<clang::CodeGenAction> action = std::make_unique<clang::EmitLLVMAction>(&context);
>>>>
>>>> textDiagnosticPrinter->BeginSourceFile(languageOptions, &preprocessor);
>>>>
>>>> compInst.ExecuteAction(*action);
>>>>
>>>>
>>>> Then inside the action, even if I created the TargetInfo myself, clang
>>>> tries something nasty:
>>>>
>>>> ASAN:DEADLYSIGNAL
>>>>
>>>> =================================================================
>>>>
>>>> ==25220==ERROR: AddressSanitizer: SEGV on unknown address
>>>> 0x000000000120 (pc 0x00010e786a2b bp 0x7ffee23ffda0 sp 0x7ffee23ffcc0 T0)
>>>>
>>>> ==25220==The signal is caused by a WRITE memory access.
>>>>
>>>> ==25220==Hint: address points to the zero page.
>>>>
>>>> #0 0x10e786a2a in
>>>> clang::TargetInfo::CreateTargetInfo(clang::DiagnosticsEngine&,
>>>> std::__1::shared_ptr<clang::TargetOptions> const&)
>>>> (libATKModelling.dylib:x86_64+0xf19a2a)
>>>>
>>>> #1 0x10ea7559b in
>>>> clang::CompilerInstance::ExecuteAction(clang::FrontendAction&)
>>>> (libATKModelling.dylib:x86_64+0x120859b)
>>>>
>>>> Cheers,
>>>>
>>>> Matthieu
>>>>
>>>> Le lun. 16 juil. 2018 à 18:00, David Blaikie <dblaikie at gmail.com> a
>>>> écrit :
>>>>
>>>>> I guess a few layers:
>>>>>
>>>>> If you're going source-to-source and want users to see/modify the new
>>>>> source, then making text edits based on source locations found in the AST
>>>>> (but not modifying the AST itself) is generally the suggested idea. If you
>>>>> simultaneously want to produce that source and compile it - yeah, probably
>>>>> easier to write it out, then compile it from that source on the filesystem.
>>>>>
>>>>> (there are probably some ways to compile from source in memory - but
>>>>> I'm not sure of the details, it might involve using the virtual filesystem
>>>>> layers - I think they were implemented for continuous compilation in IDEs
>>>>> (compiling from the edited source buffers open in the editor without having
>>>>> to write them to disk first))
>>>>>
>>>>> On Fri, Jul 13, 2018 at 3:01 PM Matthieu Brucher <
>>>>> matthieu.brucher at gmail.com> wrote:
>>>>>
>>>>>> My domain would be electrical schema modeling. Some people would like
>>>>>> to have the generated code, but then change one model of a component to
>>>>>> something else. Or remove the Newton Raphson algorithm for another one. Or
>>>>>> remove entries in the Jacobian matrix to check for terms that don't bring
>>>>>> much to the result but could enhance performance.
>>>>>> I could write the code in memory and then pass it to clang, but it
>>>>>> feels... odd. But maybe that what I need to do in the end? In there an
>>>>>> example of getting code from a string?
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Matthieu
>>>>>>
>>>>>> Le mar. 10 juil. 2018 à 23:17, David Blaikie <dblaikie at gmail.com> a
>>>>>> écrit :
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Jul 10, 2018 at 2:49 PM Matthieu Brucher <
>>>>>>> matthieu.brucher at gmail.com> wrote:
>>>>>>>
>>>>>>>> That's my use case, it's different than the OP, probably.
>>>>>>>>
>>>>>>>> In my case, I want to generate a first pass, with a JIT (the code
>>>>>>>> is generated from another description), but the generated code could be
>>>>>>>> changed by the user in a subsequent pass.
>>>>>>>>
>>>>>>>
>>>>>>> Curious. As much as possible, I'd encourage you to find ways to not
>>>>>>> have users work with generated code (by abstracting that generated code
>>>>>>> away from them - giving them a higher level representation to write, places
>>>>>>> where the generated code calls back into the user code, etc). But I don't
>>>>>>> know your domain, etc, and wouldn't suggest what is or isn't right for you
>>>>>>> and your users.
>>>>>>>
>>>>>>> But the main takeaway is that modifying the AST and generating code
>>>>>>> from that is discouraged in favor of generating source code edits.
>>>>>>>
>>>>>>>
>>>>>>>> Modifying directly the AST is not an option, try generating
>>>>>>>> equations with thousands of parameters that are solved in real time. Just
>>>>>>>> no way someone can write them efficiently in IR (that's why you have the
>>>>>>>> AST to IR generator!).
>>>>>>>>
>>>>>>>> I don't understand your last paragraph. If clang-format can cleanup
>>>>>>>> rewrites, why can't it reformat code from the AST? If the AST printer
>>>>>>>> writes any kind of code, why couldn't clang-format reformat it?
>>>>>>>>
>>>>>>>
>>>>>>> clang-format could format AST generated source too - I was
>>>>>>> commenting on that in answer to your question "Easier to generate correctly
>>>>>>> formatted code from the AST?" - that it's not easier to generate correctly
>>>>>>> formatted code from the AST than it is from a textual edit. In both cases
>>>>>>> you'd use something like clang-format to tidy up the result. The AST itself
>>>>>>> doesn't have fancy formatting support so it's no better than a textual edit
>>>>>>> in terms of getting nicely formatted results.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Le mar. 10 juil. 2018 à 22:41, David Blaikie <dblaikie at gmail.com>
>>>>>>>> a écrit :
>>>>>>>>
>>>>>>>>> Hmm, not sure I follow.
>>>>>>>>>
>>>>>>>>> Did the user write this source code? Are they going to want to
>>>>>>>>> change it later? Does it make sense for them to see the edits you're
>>>>>>>>> suggesting, or are those edits really compiler
>>>>>>>>> optimizations/transformations? If they're more the latter, then perhaps
>>>>>>>>> caching the LLVM IR (with these optimizations/transformations applied)
>>>>>>>>> rather than modifying the source would be more suitable.
>>>>>>>>>
>>>>>>>>> Easier to generate correctly formatted code from the AST? Not
>>>>>>>>> really - the AST printing doesn't have any particularly nuanced formatted
>>>>>>>>> printing. That's what clang-format is for (it was specifically built for
>>>>>>>>> doing code rewrites based on ASTs - where the rewrite is expressed as a
>>>>>>>>> textual change to the original source (not an AST modification) & that
>>>>>>>>> change is applied, then clang-format is used to tidy it up).
>>>>>>>>>
>>>>>>>>> On Tue, Jul 10, 2018 at 2:11 PM Matthieu Brucher <
>>>>>>>>> matthieu.brucher at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> It's odd though, because generating code on the fly would be
>>>>>>>>>> easier on the AST than on the IR tree, if the goal is JIT and also saving
>>>>>>>>>> the code at the same time.
>>>>>>>>>> It's probably also easier also to generate properly formatted
>>>>>>>>>> code?
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>> Matthieu
>>>>>>>>>>
>>>>>>>>>> Le mar. 10 juil. 2018 à 16:21, David Blaikie via cfe-dev <
>>>>>>>>>> cfe-dev at lists.llvm.org> a écrit :
>>>>>>>>>>
>>>>>>>>>>> It's generally considered that the AST invariants are too
>>>>>>>>>>> subtle/complex to use AST modification and AST->source conversion reliably.
>>>>>>>>>>> Refactoring/source code modification is generally encouraged to be done via
>>>>>>>>>>> textual edits generated from source location information in the AST.
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jul 9, 2018 at 8:36 PM Ridwan Shariffdeen via cfe-dev <
>>>>>>>>>>> cfe-dev at lists.llvm.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I am trying to build a tool which can insert new AST nodes to a
>>>>>>>>>>>> AST tree obtained from a source code and generate the modified source code.
>>>>>>>>>>>> For example add an if condition to a given location.
>>>>>>>>>>>>
>>>>>>>>>>>> I have seen examples on ReWriter which can insert text, but I
>>>>>>>>>>>> want to insert a proper AST node and generate the source code from the
>>>>>>>>>>>> modified AST.
>>>>>>>>>>>>
>>>>>>>>>>>> For this purpose, I think I should be using ASTWriter and not
>>>>>>>>>>>> ReWriter. Is there any documentation I can refer on how to implement this?
>>>>>>>>>>>>
>>>>>>>>>>>> Any help in this regard is highly appreciated.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks!
>>>>>>>>>>>> Ridwan
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> cfe-dev mailing list
>>>>>>>>>>>> cfe-dev at lists.llvm.org
>>>>>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> cfe-dev mailing list
>>>>>>>>>>> cfe-dev at lists.llvm.org
>>>>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Quantitative analyst, Ph.D.
>>>>>>>>>> Blog: http://blog.audio-tk.com/
>>>>>>>>>> LinkedIn: http://www.linkedin.com/in/matthieubrucher
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Quantitative analyst, Ph.D.
>>>>>>>> Blog: http://blog.audio-tk.com/
>>>>>>>> LinkedIn: http://www.linkedin.com/in/matthieubrucher
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Quantitative analyst, Ph.D.
>>>>>> Blog: http://blog.audio-tk.com/
>>>>>> LinkedIn: http://www.linkedin.com/in/matthieubrucher
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Quantitative analyst, Ph.D.
>>>> Blog: http://blog.audio-tk.com/
>>>> LinkedIn: http://www.linkedin.com/in/matthieubrucher
>>>> _______________________________________________
>>>> cfe-dev mailing list
>>>> cfe-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>>
>>>
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20180718/cb3ee794/attachment.html>
More information about the cfe-dev
mailing list