[llvm-dev] Eager compilation and relocatable dynamic linkable code-generation

[Praveen Velliengiri via llvm-dev]

Hi Rajesh,
Quick Note : If you don't want to use Lazy Compilation, don't use
CompileOnDemand Layer, instead just add Module to CompileLayer and do a
lookup.

On Fri, 19 Jul 2019 at 00:20, Rajesh S R via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> Thanks Stefan!
>
> On Thu, Jul 18, 2019 at 9:59 AM Stefan Gränitz <stefan.graenitz at gmail.com>
> wrote:
>
>> Hi Rajesh
>>
>> On 1.: ORC defers compilation of the module and on first request,
>> compiles the entire module. This will trigger recursive lookups for
>> external symbols. Eager compilation means that the modules which provide
>> these symbols will now be compiled too, while lazy compilation should
>> instead insert stubs that will trigger compilation once they are reached by
>> execution.
>>
>
> ha I see! So looking up any symbol in a module will trigger compilation of
> a module. So one trick is to insert a dummy function that does nothing and
> look it up, so that the entire module will be compiled?
>
> Is there some API/mode to turn off lazy compilation entirely and just
> compile each module on addModule call? My understanding is ORC is the new
> API which supports both lazy and eager computation as the old APIs have
> been made "legacy" APIs, so there must be some mode with ORC API to default
> to full eager compilation? Is that correct?
>
>
>> On 2.: I am maybe not perfectly up-to-date, but what I would expect to
>> happen is this. First, your remote process allocates memory for the
>> executable code and sends the base address to the compiler process. Then
>> the compiler process compiles and links with this base address and sends
>> the (position-dependent) code back. You may also manage to go with
>> position-independent code, but then your remote needs a dynamic linker. Not
>> sure if there's an example for it.
>>
>
> hmm, interesting! Although for this work you need some lower bound on code
> size generated. My case though needs position-independent code as i want to
> just compile my module once and use it across multiple processes (same idea
> as shared library) even across different machines.
>
> Yeah a dynamic linker should be fine for this case. Just curious, the
> tutorial uses this linker
> <https://llvm.org/doxygen/classllvm_1_1orc_1_1RTDyldObjectLinkingLayer.html>.
> Is this dynamic linking resolving any unknown symbol at first execution?
> In this case the symbols in the generated code will fall into 2 category:
> 1. Symbols are defined in the host process. This is common case.
> 2. Symbols are defined in another module which is similarly compiled into
> position-independent code. This is not likely and i don't see a use-case
> currently, but may have some use-case in the future.
>
> My understanding is that the symbol resolver does exactly that:
>
>  Expected<JITEvaluatedSymbol> lookup(StringRef Name) {
>     return ES.lookup({&ES.getMainJITDylib()}, Mangle(Name.str()));
>   }
>
>
> with symbols from hostporcess already exported to execution session.
>
> In particular none of them needs to open a separate file and load the
> symbol definition from it.
> So, may be that "linker
> <https://llvm.org/doxygen/classllvm_1_1orc_1_1RTDyldObjectLinkingLayer.html>"
> is sufficient for my use-case?
>
> So the main open question is: For compiling to position independent code,
> wondering if there are some CompileLayer compiler which can be used to emit
> position-independent code?
>
>
>> Best
>> Stefan
>>
>> On 18/07/2019 03:08, Rajesh S R via llvm-dev wrote:
>>
>> Hi all,
>> I am following the LLVM JIT tutorial API based on:
>> https://llvm.org/docs/tutorial/BuildingAJIT1.html
>>
>> I have 2 independent questions:
>> 1. The tutorial mentions that "*will defer compilation of the module
>> until any of its definitions is looked up*".
>>
>> Does that mean to force eager compilation, i have to keep track of all
>> functions added to the Module and do a lookup? I want the module to be
>> compiled fully after I do add module to compile_layer. What is the best way
>> to achieve this? Or should I not use ORC API at all?
>>
>> 2. I want to compile on one process and want to ship the machine code to
>> another process and load it for execution (with no lazy compilation where
>> an uncompiled function will trigger compilation to remote process which as
>> I understand is what the tutorial talks about). This means we need to
>> generate position-independent code and may need dynamic linking to resolve
>> any unknown function call based on symbol name. What is the best way to
>> achieve this? Is there some code pointers?
>>
>> Thanks!
>>
>> Rajesh S R
>>
>> _______________________________________________
>> LLVM Developers mailing listllvm-dev at lists.llvm.orghttps://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>> -- https://flowcrypt.com/pub/stefan.graenitz@gmail.com
>>
>> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190719/11e3038f/attachment.html>