<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Lang, </div><div><br></div>I added the add/get global mapping to my kaleidoscope JIT, but I think perhaps these would make more sense if they were added to the object linking layer as they would be generally usable there. <div><br></div><div><br><div><div>On Aug 19, 2015, at 11:19 PM, Andy Somogyi wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hey Lang, <div><br></div><div>I've added this to my Kaleidoscope JIT, and it seems to work just fine, basically I copied the global mapping functionality out of execution engine and hooked up the lambda resolver to reference them. </div><div><br></div><div>Here's the code if you want it:</div><div><br></div><div><a href="https://github.com/AndySomogyi/cayman/blob/master/src/llvm_orc_initial.cpp">https://github.com/AndySomogyi/cayman/blob/master/src/llvm_orc_initial.cpp</a></div><div><br></div><div><br><div><div>On Aug 19, 2015, at 9:14 PM, Lang Hames wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr">Hi Andy,<div><br></div><div>I think that makes sense. I'm currently rewriting the core Kaleidoscope tutorials - I'll look at adding support for this.</div><div><br></div><div>- Lang.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Aug 14, 2015 at 7:38 AM, Andy Somogyi <span dir="ltr"><<a href="mailto:andy.somogyi@gmail.com" target="_blank">andy.somogyi@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">After some fiddling with it, it does in fact look like it works as you describe Lang. <div><br></div><div>The trick was you had to call </div><div><br></div><div><div style="margin:0px;font-size:11px;font-family:Menlo;color:rgb(61,29,129)"><span style="color:#703daa">llvm</span><span style="color:#000000">::</span><span style="color:#703daa">sys</span><span style="color:#000000">::</span><span style="color:#703daa">DynamicLibrary</span><span style="color:#000000">::</span>LoadLibraryPermanently<span style="color:#000000">(</span><span style="color:#bb2ca2">nullptr</span><span style="color:#000000">);</span></div></div><div style="margin:0px;font-size:11px;font-family:Menlo;color:rgb(61,29,129)"><span style="color:#000000"><br></span></div><div style="margin:0px;font-size:11px;font-family:Menlo;color:rgb(61,29,129)"><span style="color:rgb(0,0,0);font-family:Courier;font-size:12px">to add the currently running process before calling </span></div><div style="margin:0px;font-size:11px;font-family:Menlo;color:rgb(61,29,129)"><span style="font-family:Monaco;color:rgb(0,0,0)"><br></span></div><div style="margin:0px;font-size:11px;font-family:Menlo;color:rgb(61,29,129)"><span style="font-family:Monaco;color:rgb(0,0,0)">llvm::</span><span style="font-family:Monaco;color:rgb(0,97,65)">RTDyldMemoryManager</span><span style="font-family:Monaco;color:rgb(0,0,0)">::</span><span style="color:rgb(121,61,147);font-family:Monaco">getSymbolAddressInProcess</span><span style="font-family:Monaco;color:rgb(0,0,0)">(name);</span><span style="color:rgb(0,0,0);font-family:Courier;font-size:12px">.</span></div><div style="margin:0px;font-size:11px;font-family:Menlo;color:rgb(61,29,129)"><span style="color:rgb(0,0,0);font-family:Courier;font-size:12px"><br></span></div><div style="margin:0px">Also, all of the function needs to be declared as extern C to avoid name mangling. </div><div style="margin:0px"><br></div><div style="margin:0px">I however think that adding the addGlobalMapping/getPointerToGlobalMapping functions to the ORC kaleidoscope examples is very useful, as for example one wants a set of private symbols made available to JITed code. </div><div><div class="h5"><div style="margin:0px"><br></div><div style="margin:0px"> </div><div><br><div><blockquote type="cite"><div>On Aug 13, 2015, at 7:41 PM, Lang Hames <<a href="mailto:lhames@gmail.com" target="_blank">lhames@gmail.com</a>> wrote:</div><br><div><div dir="ltr">Hi Andy,<div><br></div><div>I haven't tested this on Linux, but on MacOS the RuntimeDyldMemorManager::getSymbolAddressInProcess method should find symbol addresses in the host program, including symbols from static archives linked into the program. However, one gotcha is that the symbol has to be reachable from main, otherwise the linker may strip it from the final executable.</div><div><br></div><div>Do you have a test-case that I could try to reproduce the issue with?</div><div><br></div><div>Alternatively, if you run the code under a debugger, do you see symbols that the JIT failed to find? If the symbols are visible in the debugger but invisible to the JIT that sounds like a bug in the JIT. If the symbols are invisible in both that would suggest that the linker is stripping them out.</div><div><br></div><div>- Lang.</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Aug 13, 2015 at 1:12 PM, David Blaikie <span dir="ltr"><<a href="mailto:dblaikie@gmail.com" target="_blank">dblaikie@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">+"Lang Hames, Linker of Linkers"<br><br>(Lang developed the ORC JIT you seem to be using & can provide more context than I have)</div><div class="gmail_extra"><br><div class="gmail_quote"><div><div>On Thu, Aug 13, 2015 at 12:43 PM, Andy Somogyi via llvm-dev <span dir="ltr"><<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr"><span style="font-family:Courier;font-size:12px">Hi</span><div style="font-family:Courier;font-size:12px"><br></div><div style="font-family:Courier;font-size:12px">I’ve previously used the ExecutionEngine::addGlobalMapping to make existing functions available to my JITed code. </div><div style="font-family:Courier;font-size:12px"><br></div><div style="font-family:Courier;font-size:12px">I’m currently using ORC, as MCJIT does not appear to be maintained any longer (the kaleidoscope examples have not worked for some time with MCJIT). </div><div style="font-family:Courier;font-size:12px"><br></div><div style="font-family:Courier;font-size:12px">I’m using just the basic ORC CompileLayer directly. </div><div style="font-family:Courier;font-size:12px"><br></div><div style="font-family:Courier;font-size:12px">So, I’ve essentially copied the ExecutionEngine::addGlobalMapping related function to my JIT context, and I create a lambda resolver as such:</div><div style="font-family:Courier;font-size:12px"><br></div><div style="font-family:Courier;font-size:12px">JITContext::addModule(…) {</div><div style="font-family:Courier;font-size:12px"><br></div><div style="font-family:Courier;font-size:12px"><div style="margin:0px"><span style="color:rgb(147,26,104)">auto</span> Resolver = createLambdaResolver(</div><div style="margin:0px"><span style="white-space:pre-wrap"> </span>[&](<span style="color:rgb(147,26,104)">const</span> std::<span style="color:rgb(0,97,65)">string</span> &name) {</div><div style="margin:0px"><br></div><div style="margin:0px;min-height:15px"><span style="white-space:pre-wrap"> </span><span style="color:rgb(78,144,114)">// look up first in JIT'ed code</span></div><div style="margin:0px"><span style="white-space:pre-wrap"> </span><span style="color:rgb(147,26,104)">if</span> (<span style="color:rgb(147,26,104)">auto</span> sym = findMangledSymbol(name)) {</div><div style="margin:0px"><span style="white-space:pre-wrap"> </span><span style="color:rgb(147,26,104)">return</span> <span style="color:rgb(0,97,65)">RuntimeDyld</span>::<span style="color:rgb(0,97,65)">SymbolInfo</span>(sym.getAddress(),</div><div style="margin:0px"><span style="white-space:pre-wrap"> </span>sym.getFlags());</div><div style="margin:0px;color:rgb(0,97,65)"><span><span style="white-space:pre-wrap"> </span></span><span style="color:rgb(147,26,104)">return</span><span> </span>RuntimeDyld<span>::</span>SymbolInfo<span>(</span><span style="color:rgb(147,26,104)">nullptr</span><span>);</span></div><div style="margin:0px"><span style="white-space:pre-wrap"> </span>}</div><div style="margin:0px;min-height:15px"><br></div><div style="margin:0px;color:rgb(78,144,114)"><span><span style="white-space:pre-wrap"> </span></span>// look up in added globals</div><div style="margin:0px"><span style="white-space:pre-wrap"> </span><span style="color:rgb(147,26,104)">if</span> (<span style="color:rgb(147,26,104)">auto</span> addr = getPointerToGlobalMapping(name)) {</div><div style="margin:0px"><span style="white-space:pre-wrap"> </span><span style="color:rgb(147,26,104)">return</span> <span style="color:rgb(0,97,65)">RuntimeDyld</span>::<span style="color:rgb(0,97,65)">SymbolInfo</span>(addr, JITSymbolFlags::<span style="color:rgb(3,38,204)">Exported</span>);</div><div style="margin:0px"><span style="white-space:pre-wrap"> </span>}</div><div style="margin:0px;min-height:15px"><br></div><div style="margin:0px;color:rgb(78,144,114)"><span><span style="white-space:pre-wrap"> </span></span>// finally try to look up existing process symbols, note</div><div style="margin:0px;color:rgb(78,144,114)"><span><span style="white-space:pre-wrap"> </span></span>// this works for symbols loaded in shared libraries, but</div><div style="margin:0px;color:rgb(78,144,114)"><span><span style="white-space:pre-wrap"> </span></span>// does NOT seem to find symbols declared in the executable.</div><div style="margin:0px"><span style="white-space:pre-wrap"> </span><span style="color:rgb(147,26,104)">if</span> (<span style="color:rgb(147,26,104)">auto</span> Addr =</div><div style="margin:0px;color:rgb(121,61,147)"><span><span style="white-space:pre-wrap"> </span></span><span style="color:rgb(0,97,65)">RTDyldMemoryManager</span><span>::</span>getSymbolAddressInProcess<span>(name)) {</span></div><div style="margin:0px"><span style="white-space:pre-wrap"> </span><span style="color:rgb(147,26,104)">return</span> <span style="color:rgb(0,97,65)">RuntimeDyld</span>::<span style="color:rgb(0,97,65)">SymbolInfo</span>(Addr, JITSymbolFlags::<span style="color:rgb(3,38,204)">Exported</span>);</div><div style="margin:0px"><span style="white-space:pre-wrap"> </span>}</div><div style="margin:0px">},</div><div style="margin:0px">[](<span style="color:rgb(147,26,104)">const</span> std::<span style="color:rgb(0,97,65)">string</span> &S) { <span style="color:rgb(147,26,104)">return</span> <span style="color:rgb(147,26,104)">nullptr</span>; }</div><div style="margin:0px">);</div></div><div style="font-family:Courier;font-size:12px;margin:0px">}</div><div style="font-family:Courier;font-size:12px;margin:0px"><br></div><div style="font-family:Courier;font-size:12px;margin:0px">Here the getPointerToGlobalMapping function looks in a uint64 StringMap into which values are added via the addGlobalMapping functions. </div><div style="font-family:Courier;font-size:12px;margin:0px"><br></div><div style="font-family:Courier;font-size:12px;margin:0px"><br></div><div style="font-family:Courier;font-size:12px;margin:0px">This approach seems to be working, but my question is do you suppose there any are issues with such an approach? </div><div style="font-family:Courier;font-size:12px;margin:0px"><br></div><div style="font-family:Courier;font-size:12px;margin:0px">The troubling thing is why doesn’t <span style="color:rgb(0,97,65)">RTDyldMemoryManager</span>::<span style="color:rgb(121,61,147)">getSymbolAddressInProcess</span>(name)) return an address for a symbol that is defined in either a static library, or in the executable itself. </div><div style="font-family:Courier;font-size:12px;margin:0px"><br></div><div style="font-family:Courier;font-size:12px;margin:0px">If this approach is correct, in adding the global values to the context, and looking them up the lambda resolver, in addition to looking up external symbols, and considering that the ORC kaleidoscope examples do in fact allow external function calls (which are broken currently), should they be fixed with this approach?</div></div>
<br></div></div>_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a> <a href="http://llvm.cs.uiuc.edu/" rel="noreferrer" target="_blank">http://llvm.cs.uiuc.edu</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br></div>
</div></blockquote></div><br></div></div></div></div></blockquote></div><br></div>
</blockquote></div><br></div></div></blockquote></div><br></div></body></html>