<div dir="auto">Hi,<div dir="auto"><br></div><div dir="auto">I can easily add source code to a file. It's a no brainer, I'm not going to use clang for this, it's overkill. </div><div dir="auto">What doesn't work, as stated in my previous example, is getting a module out of clang, a module that can be used inside llvm. When executing the code below, I get a write error. That's a problem because there are no resources online on this issue. The api changes too quickly for this, and even lib clang doesn't help because then the triple is not set (and then llvm breaks). </div><div dir="auto"><br></div><div dir="auto">Cheers </div><div dir="auto">Matthieu </div></div><br><div class="gmail_quote"><div dir="ltr">Le mar. 17 juil. 2018 à 18:25, Firat Kasmis <<a href="mailto:firatkasmis@gmail.com">firatkasmis@gmail.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Matthieu, try <a href="https://github.com/firolino/clang-tool" target="_blank" rel="noreferrer">https://github.com/firolino/clang-tool</a> as getting started and change the transformer to your needs to insert code/text at a given location. Hope it helps.<div><br></div><div>Best,</div><div>Firat</div></div><br><div class="gmail_quote"><div dir="ltr">Am Di., 17. Juli 2018 um 09:00 Uhr schrieb Matthieu Brucher via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank" rel="noreferrer">cfe-dev@lists.llvm.org</a>>:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Indeed, that's what I'm now aiming at. Unfortunately, it seems that there are no examples as how to use FrontEndAction properly with clang 6.0.0. I can use libeling with runToolOnCode to generate a module, but the triple is not set up properly in that case when I want to use the JIT. And it seems to be a problem with clang, as if I do this:<div><pre style="color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap"> clang::DiagnosticOptions diagnosticOptions;
std::unique_ptr<clang::TextDiagnosticPrinter> textDiagnosticPrinter =
std::make_unique<clang::TextDiagnosticPrinter>(llvm::outs(),
&diagnosticOptions);
llvm::IntrusiveRefCntPtr<clang::DiagnosticIDs> diagIDs;
std::unique_ptr<clang::DiagnosticsEngine> diagnosticsEngine =
std::make_unique<clang::DiagnosticsEngine>(diagIDs, &diagnosticOptions, textDiagnosticPrinter.get());
clang::LangOptions languageOptions;
clang::FileSystemOptions fileSystemOptions;
clang::FileManager fileManager(fileSystemOptions);
clang::SourceManager sourceManager(*diagnosticsEngine,
fileManager);
std::shared_ptr<clang::HeaderSearchOptions> headerSearchOptions(new clang::HeaderSearchOptions());
const std::shared_ptr<clang::TargetOptions> targetOptions = std::make_shared<clang::TargetOptions>();
targetOptions->Triple = llvm::sys::getDefaultTargetTriple();
std::unique_ptr<clang::TargetInfo> targetInfo(
clang::TargetInfo::CreateTargetInfo(*diagnosticsEngine, targetOptions));
clang::HeaderSearch headerSearch(headerSearchOptions,
sourceManager,
*diagnosticsEngine,
languageOptions,
targetInfo.get());
clang::MemoryBufferCache PCMCache;
clang::CompilerInstance compInst;
std::shared_ptr<clang::PreprocessorOptions> opts(std::make_shared<clang::PreprocessorOptions>());
clang::Preprocessor preprocessor(opts,
*diagnosticsEngine,
languageOptions,
sourceManager,
PCMCache,
headerSearch,
compInst);
preprocessor.Initialize(*targetInfo);
auto filter = llvm::MemoryBuffer::getMemBufferCopy(fullfile);
sourceManager.setMainFileID(sourceManager.createFileID(std::move(filter)));
clang::IdentifierTable identifierTable(languageOptions);
clang::SelectorTable selectorTable;
clang::Builtin::Context builtinContext;
builtinContext.InitializeTarget(*targetInfo, nullptr);
clang::ASTContext astContext(languageOptions,
sourceManager,
identifierTable,
selectorTable,
builtinContext);
astContext.InitBuiltinTypes(*targetInfo);
compInst.setTarget(targetInfo.get());
llvm::LLVMContext context;
std::unique_ptr<clang::CodeGenAction> action = std::make_unique<clang::EmitLLVMAction>(&context);
textDiagnosticPrinter->BeginSourceFile(languageOptions, &preprocessor);
compInst.ExecuteAction(*action);
</pre><br class="m_-6011694467046257201m_-6653598594885440323gmail-Apple-interchange-newline" style="color:rgb(0,0,0);font-family:-webkit-standard"><div>Then inside the action, even if I created the TargetInfo myself, clang tries something nasty:</div></div><div><p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures">ASAN:DEADLYSIGNAL</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures">=================================================================</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(195,55,32);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures">==25220==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000120 (pc 0x00010e786a2b bp 0x7ffee23ffda0 sp 0x7ffee23ffcc0 T0)</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures">==25220==The signal is caused by a WRITE memory access.</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures">==25220==Hint: address points to the zero page.</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures"> #0 0x10e786a2a in clang::TargetInfo::CreateTargetInfo(clang::DiagnosticsEngine&, std::__1::shared_ptr<clang::TargetOptions> const&) (libATKModelling.dylib:x86_64+0xf19a2a)</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures"> #1 0x10ea7559b in clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (libATKModelling.dylib:x86_64+0x120859b)</span></p></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div>Cheers,</div><div><br></div><div>Matthieu</div></div><br><div class="gmail_quote"><div dir="ltr">Le lun. 16 juil. 2018 à 18:00, David Blaikie <<a href="mailto:dblaikie@gmail.com" target="_blank" rel="noreferrer">dblaikie@gmail.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I guess a few layers:<br><br>If you're going source-to-source and want users to see/modify the new source, then making text edits based on source locations found in the AST (but not modifying the AST itself) is generally the suggested idea. If you simultaneously want to produce that source and compile it - yeah, probably easier to write it out, then compile it from that source on the filesystem.<br><br>(there are probably some ways to compile from source in memory - but I'm not sure of the details, it might involve using the virtual filesystem layers - I think they were implemented for continuous compilation in IDEs (compiling from the edited source buffers open in the editor without having to write them to disk first))<br></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Jul 13, 2018 at 3:01 PM Matthieu Brucher <<a href="mailto:matthieu.brucher@gmail.com" target="_blank" rel="noreferrer">matthieu.brucher@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">My domain would be electrical schema modeling. Some people would like to have the generated code, but then change one model of a component to something else. Or remove the Newton Raphson algorithm for another one. Or remove entries in the Jacobian matrix to check for terms that don't bring much to the result but could enhance performance.<div>I could write the code in memory and then pass it to clang, but it feels... odd. But maybe that what I need to do in the end? In there an example of getting code from a string?</div><div><br></div><div>Cheers,</div><div><br></div><div>Matthieu</div></div><br><div class="gmail_quote"><div dir="ltr">Le mar. 10 juil. 2018 à 23:17, David Blaikie <<a href="mailto:dblaikie@gmail.com" target="_blank" rel="noreferrer">dblaikie@gmail.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr">On Tue, Jul 10, 2018 at 2:49 PM Matthieu Brucher <<a href="mailto:matthieu.brucher@gmail.com" target="_blank" rel="noreferrer">matthieu.brucher@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">That's my use case, it's different than the OP, probably.<div><br></div><div>In my case, I want to generate a first pass, with a JIT (the code is generated from another description), but the generated code could be changed by the user in a subsequent pass.</div></div></blockquote><div><br></div><div>Curious. As much as possible, I'd encourage you to find ways to not have users work with generated code (by abstracting that generated code away from them - giving them a higher level representation to write, places where the generated code calls back into the user code, etc). But I don't know your domain, etc, and wouldn't suggest what is or isn't right for you and your users.<br><br>But the main takeaway is that modifying the AST and generating code from that is discouraged in favor of generating source code edits.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Modifying directly the AST is not an option, try generating equations with thousands of parameters that are solved in real time. Just no way someone can write them efficiently in IR (that's why you have the AST to IR generator!).</div><div><br></div><div>I don't understand your last paragraph. If clang-format can cleanup rewrites, why can't it reformat code from the AST? If the AST printer writes any kind of code, why couldn't clang-format reformat it?</div></div></blockquote><div><br>clang-format could format AST generated source too - I was commenting on that in answer to your question "Easier to generate correctly formatted code from the AST?" - that it's not easier to generate correctly formatted code from the AST than it is from a textual edit. In both cases you'd use something like clang-format to tidy up the result. The AST itself doesn't have fancy formatting support so it's no better than a textual edit in terms of getting nicely formatted results.<br> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">Le mar. 10 juil. 2018 à 22:41, David Blaikie <<a href="mailto:dblaikie@gmail.com" target="_blank" rel="noreferrer">dblaikie@gmail.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hmm, not sure I follow.<br><br>Did the user write this source code? Are they going to want to change it later? Does it make sense for them to see the edits you're suggesting, or are those edits really compiler optimizations/transformations? If they're more the latter, then perhaps caching the LLVM IR (with these optimizations/transformations applied) rather than modifying the source would be more suitable.<br><br>Easier to generate correctly formatted code from the AST? Not really - the AST printing doesn't have any particularly nuanced formatted printing. That's what clang-format is for (it was specifically built for doing code rewrites based on ASTs - where the rewrite is expressed as a textual change to the original source (not an AST modification) & that change is applied, then clang-format is used to tidy it up).</div><br><div class="gmail_quote"><div dir="ltr">On Tue, Jul 10, 2018 at 2:11 PM Matthieu Brucher <<a href="mailto:matthieu.brucher@gmail.com" target="_blank" rel="noreferrer">matthieu.brucher@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">It's odd though, because generating code on the fly would be easier on the AST than on the IR tree, if the goal is JIT and also saving the code at the same time.<div>It's probably also easier also to generate properly formatted code?</div><div><br></div><div>Regards,</div><div><br></div><div>Matthieu</div></div><br><div class="gmail_quote"><div dir="ltr">Le mar. 10 juil. 2018 à 16:21, David Blaikie via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank" rel="noreferrer">cfe-dev@lists.llvm.org</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">It's generally considered that the AST invariants are too subtle/complex to use AST modification and AST->source conversion reliably. Refactoring/source code modification is generally encouraged to be done via textual edits generated from source location information in the AST.<br><br><div class="gmail_quote"><div dir="ltr">On Mon, Jul 9, 2018 at 8:36 PM Ridwan Shariffdeen via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank" rel="noreferrer">cfe-dev@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:garamond,serif;font-size:small">Hi,</div><div class="gmail_default" style="font-family:garamond,serif;font-size:small"><br></div><div class="gmail_default" style="font-family:garamond,serif;font-size:small">I am trying to build a tool which can insert new AST nodes to a AST tree obtained from a source code and generate the modified source code. For example add an if condition to a given location. <br><br>I have seen examples on ReWriter which can insert text, but I want to insert a proper AST node and generate the source code from the modified AST. </div><div class="gmail_default" style="font-family:garamond,serif;font-size:small"><br></div><div class="gmail_default" style="font-family:garamond,serif;font-size:small">For this purpose, I think I should be using ASTWriter and not ReWriter. Is there any documentation I can refer on how to implement this?</div><div class="gmail_default" style="font-family:garamond,serif;font-size:small"><br></div><div class="gmail_default" style="font-family:garamond,serif;font-size:small">Any help in this regard is highly appreciated. </div><div class="gmail_default" style="font-family:garamond,serif;font-size:small"><br></div><div class="gmail_default" style="font-family:garamond,serif;font-size:small">Thanks!</div><div class="gmail_default" style="font-family:garamond,serif;font-size:small">Ridwan</div></div>
_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank" rel="noreferrer">cfe-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br>
</blockquote></div></div>
_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank" rel="noreferrer">cfe-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_-6011694467046257201m_-6653598594885440323m_-239466046332143829m_5490588545301335289m_-7209265378770982007m_-6944897690394873246m_5750601677052724679m_-4642569081504176905gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Quantitative analyst, Ph.D.<br>Blog: <a href="http://blog.audio-tk.com/" target="_blank" rel="noreferrer">http://blog.audio-tk.com/</a><br>LinkedIn: <a href="http://www.linkedin.com/in/matthieubrucher" target="_blank" rel="noreferrer">http://www.linkedin.com/in/matthieubrucher</a></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_-6011694467046257201m_-6653598594885440323m_-239466046332143829m_5490588545301335289m_-7209265378770982007m_-6944897690394873246gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Quantitative analyst, Ph.D.<br>Blog: <a href="http://blog.audio-tk.com/" target="_blank" rel="noreferrer">http://blog.audio-tk.com/</a><br>LinkedIn: <a href="http://www.linkedin.com/in/matthieubrucher" target="_blank" rel="noreferrer">http://www.linkedin.com/in/matthieubrucher</a></div></div></div></div></div>
</blockquote></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_-6011694467046257201m_-6653598594885440323m_-239466046332143829m_5490588545301335289gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Quantitative analyst, Ph.D.<br>Blog: <a href="http://blog.audio-tk.com/" target="_blank" rel="noreferrer">http://blog.audio-tk.com/</a><br>LinkedIn: <a href="http://www.linkedin.com/in/matthieubrucher" target="_blank" rel="noreferrer">http://www.linkedin.com/in/matthieubrucher</a></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_-6011694467046257201m_-6653598594885440323gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Quantitative analyst, Ph.D.<br>Blog: <a href="http://blog.audio-tk.com/" target="_blank" rel="noreferrer">http://blog.audio-tk.com/</a><br>LinkedIn: <a href="http://www.linkedin.com/in/matthieubrucher" target="_blank" rel="noreferrer">http://www.linkedin.com/in/matthieubrucher</a></div></div></div></div></div>
_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank" rel="noreferrer">cfe-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br>
</blockquote></div>
</blockquote></div>