<div dir="ltr">Hi Dave, Andy,<div><br></div><div>Apologies for a very tardy response. I think Andy covered most of this already, but I wanted to comment on the address-fixup issue: If you make your references symbolic then MCJIT should be able to relocate the code for you. I.e. you want "call void @foo", rather than "call 0xcafef00d". MCJIT will call you back when it finalizes the object to find out the address of "foo" in the remote process.</div><div><br></div><div>LLDB used to use constant addresses in the IR, but they have been moving to symbolic references, at least for some languages. We now JIT a lot of code with symbolic references, at least with MachO on x86-64, ARM and AArch64. We still occasionally hit bugs but this is reasonably robust now, at least for the code patterns that LLDB is producing.</div><div><br></div><div>Cheers,</div><div>Lang.</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 27, 2015 at 9:39 AM, Kaylor, Andrew <span dir="ltr"><<a href="mailto:andrew.kaylor@intel.com" target="_blank">andrew.kaylor@intel.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">My first thought was that you’d want the code that finds the address to be part of the generated object, but I suppose that’s not always possible/preferable.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">As a point of information, LLDB uses MCJIT to execute small fragments of code in the inferior process for expression evaluation. This was originally implemented
before MCJIT’s remote execution support was complete so it had to resolve addresses in a way very similar to what you describe. I don’t know if it still does that, but it definitely used to. At the very least, this is a validation that what you are suggesting
can be done.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">It occurs to me now that there may be a limitation in MCJIT that prevents you from calling mapSectionAddress again after a Module has been prepared for execution.
I could be wrong about this. I know a lot of work has been done in the past year to clean up that part of the interface. I think it would be pretty obvious if that happens. If it does, report back and we can talk about what can be done.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">-Andy<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Dave Pitsbawn [mailto:<a href="mailto:dpitsbawn@gmail.com" target="_blank">dpitsbawn@gmail.com</a>]
<br>
<b>Sent:</b> Thursday, March 26, 2015 8:31 PM</span></p><div><div class="h5"><br>
<b>To:</b> Kaylor, Andrew<br>
<b>Cc:</b> LLVM Developers Mailing List<br>
<b>Subject:</b> Re: [LLVMdev] MCJIT finalizeObject output to use in external process<u></u><u></u></div></div><p></p><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">Thanks, Andy.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I'm now thinking about the case where the generated code has references to memory addresses in the IR (the only case I suppose is call remoteProcessADDR). I know this is not LLVM specific but what happens when that ADDR changes due to process
restart or different machine. I know this is the clients responsibility to put the right address in the call instruction, but generally how will one achieve this? Do they iterate over the assembly (generated by MCJIT) and fixup calls after the fact outside
of their LLVM/MCJIT infrastructure?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Or another approach could be they store the optimized IR and just re-do it every single time for every single time they need an update.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">It seems like holding on the final output and then visiting all calls, and find/replacing them (:-)) with a new address seems most efficient<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thoughts and ideas welcome.<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Thu, Mar 26, 2015 at 12:52 PM, Kaylor, Andrew <<a href="mailto:andrew.kaylor@intel.com" target="_blank">andrew.kaylor@intel.com</a>> wrote:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">You don’t need to put all the sections in the same memory block or maintain the same offsets that
had on the host. What I meant was that each section must be in a contiguous block of memory.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Depending on the memory model you specify when compiling the code you made need to guarantee that
all sections are within 2GB of one another, but how you meet this requirement is at your discretion. Apart from that, MCJIT will take care of applying relocations to account for the relative location of each section in the remote process according to the
addresses you provided in calls to mapSectionAddress.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Another thing your custom memory manager may need to do that I forgot to mention is handle linking
of external functions needed by the generated code. For in-process execution MCJIT will automatically link to things like standard library functions that it recognizes using the implementation of those functions in the host process. For out-of-process execution,
your memory manager will need to perform this task. MCJIT will call your memory manager’s getSymbolAddress function if it encounters a call to an external function.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">-Andy</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Dave Pitsbawn [mailto:<a href="mailto:dpitsbawn@gmail.com" target="_blank">dpitsbawn@gmail.com</a>]
<br>
<b>Sent:</b> Wednesday, March 25, 2015 7:42 PM</span><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><br>
<b>To:</b> Kaylor, Andrew<br>
<b>Cc:</b> LLVM Developers Mailing List<br>
<b>Subject:</b> Re: [LLVMdev] MCJIT finalizeObject output to use in external process<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">No, I was asking how to extract the code from MCJIT, and you said it use a custom memory manager.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">When you say that I must treat each section as a block, do you mean that there is inter-block relative offsets need to be maintained? Or that when I get a section, I must copy it
to target process memory as a one-shot contiguous block. If it's second, I think we're ok.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">On Wed, Mar 25, 2015 at 5:39 PM, Kaylor, Andrew <<a href="mailto:andrew.kaylor@intel.com" target="_blank">andrew.kaylor@intel.com</a>> wrote:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Are you asking about the actual mechanism for transferring the bits into the remote process or how
you locate the generated code in memory?</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">The mechanism for transferring the bits is outside the scope of MCJIT. The easiest way to locate
the generate code is to use a custom memory manager as lli does. MCJIT will call the memory manager to allocate memory on a section-by-section basis. You should treat each section as a monolithic block when copying to the remote system because the generated
code may depend on relative offsets staying fixed.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">-Andy</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Dave Pitsbawn [mailto:<a href="mailto:dpitsbawn@gmail.com" target="_blank">dpitsbawn@gmail.com</a>]
<br>
<b>Sent:</b> Wednesday, March 25, 2015 5:00 PM<br>
<b>To:</b> Kaylor, Andrew<br>
<b>Cc:</b> LLVM Developers Mailing List<br>
<b>Subject:</b> Re: [LLVMdev] MCJIT finalizeObject output to use in external process</span><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">Aha. Thanks.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Seems like I need to call mapSectionAddress with the target address. But how I copy the code? What function would I call?<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">On Wed, Mar 25, 2015 at 4:32 PM, Kaylor, Andrew <<a href="mailto:andrew.kaylor@intel.com" target="_blank">andrew.kaylor@intel.com</a>> wrote:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Yes, that is one of the intended use models for MCJIT.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">If you look at the source for ‘lli’ you’ll find an option called “remote-mcjit” which does exactly
this (for testing purposes).</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">The key function (which in the lli case is called from lli’s RemoteMemoryManager::notifyObjectLoaded
method) is ExecutionEngine::mapSectionAddress. This function tells MCJIT to reapply relocations to the loaded object as if it were loaded at a different address in memory than it actually is. The client is responsible for the particulars of allocating memory
in the remote process, determining the address of the memory in the remote process’ address space and (after calling mapSectionAddress) copying the JIT’d object to the remote process memory and setting the memory attributes as needed for execution.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">-Andy</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
<a href="mailto:llvmdev-bounces@cs.uiuc.edu" target="_blank">llvmdev-bounces@cs.uiuc.edu</a> [mailto:<a href="mailto:llvmdev-bounces@cs.uiuc.edu" target="_blank">llvmdev-bounces@cs.uiuc.edu</a>]
<b>On Behalf Of </b>Dave Pitsbawn<br>
<b>Sent:</b> Wednesday, March 25, 2015 4:07 PM<br>
<b>To:</b> LLVM Developers Mailing List<br>
<b>Subject:</b> [LLVMdev] MCJIT finalizeObject output to use in external process</span><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">A need has arisen to generate code using MCJIT but not in the target process instead in a different process (and possibly even different machine though not in the scope).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Reading through the tutorials and MCJIT design document, it seems like this is possible or was kept in mind during design of MCJIT.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">How do I achieve this? Are there examples?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Dave<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div></div></div>
</div>
<br>_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:LLVMdev@cs.uiuc.edu">LLVMdev@cs.uiuc.edu</a> <a href="http://llvm.cs.uiuc.edu" target="_blank">http://llvm.cs.uiuc.edu</a><br>
<a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev</a><br>
<br></blockquote></div><br></div>