[llvm-dev] OrcV1 removal

[Lang Hames via llvm-dev]

Hi Andres,

Cool!I assume this works on "non-native" jitlink platforms as well? Or
> just mach?


This framework is totally materializer agnostic -- It works for
ObjectLinkingLayer (JITLink), RTDyldObjectLinkingLayer (RuntimeDyld), and
even materializers that aren't emitted via a linker (e.g. stubs and
callbacks).

 Looks like there's not yet a C API yet - not a problem, just checking.


No C API yet, but I'll add one as soon as the C++ API settles down
(probably in a week or so).

Currently it looks like only the main jit dylib is exposed via C, right?
We can't really do the same for resource trackers, if I understand this
correctly. Sketching out what I'd need to do to test postgres, so I
don't waste time going in the wrong direction:

Currently it looks like only the main jit dylib is exposed via C, right?


That was true -- I just hadn't gotten around to it. Your question seems as
good an excuse as any to take a minute out and do that, so I've added them
in 9a0d1b66730. :)

We can't really do the same for resource trackers, if I understand this
> correctly. Sketching out what I'd need to do to test postgres, so I
> don't waste time going in the wrong direction:


1) Add API for management (create, destroy) a non-default resource tracker
>    (i.e. JITDylib::createResourceTracker)
> 3) Add JITDylib::clear() wrapper
> 2) LLVMOrcLLJITAdd{LLVMIRModule,ObjectFile} would need a new argument
>    (defaulting to NULL for the default resource tracker?) to specify the
>    resource tracker
> 4) LLJIT would need to grow the underlying support methods


That's all spot-on. It should not require much code either: Both the C API
and LLJIT itself are pretty thin wrappers around the core APIs -- each of
these wrappers should just be a few lines.

I assume it is not legal to use the same resource tracker with two
> different JITDylib's?


That's correct -- ResourceTrackers are bound to the JITDylib they're
created from. They should assert if you use them with the wrong JITDylib,
though I'm not sure how good the coverage on those asserts are yet.

I think this system could be generalized to enable cross-dylib tracking at
the cost of a few dozen extra bytes per tracker. Do you have a use-case for
this?

Regards,
Lang.

On Wed, Sep 16, 2020 at 12:48 PM Andres Freund <andres at anarazel.de> wrote:

> Hi,
>
> On 2020-09-16 11:52:11 -0700, Lang Hames wrote:
> > I've updated the orcv1 removal branch (
> > https://github.com/lhames/llvm-project/tree/orcv1-removal) with an
> initial
> > patch for removable code. If anyone wants to follow along with the
> > development or share thoughts on the design you're very welcome to.
>
> Cool!I assume this works on "non-native" jitlink platforms as well? Or
> just mach?
>
> Looks like there's not yet a C API yet - not a problem, just checking.
>
> Currently it looks like only the main jit dylib is exposed via C, right?
> We can't really do the same for resource trackers, if I understand this
> correctly. Sketching out what I'd need to do to test postgres, so I
> don't waste time going in the wrong direction:
>
> 1) Add API for management (create, destroy) a non-default resource tracker
>    (i.e. JITDylib::createResourceTracker)
> 3) Add JITDylib::clear() wrapper
> 2) LLVMOrcLLJITAdd{LLVMIRModule,ObjectFile} would need a new argument
>    (defaulting to NULL for the default resource tracker?) to specify the
>    resource tracker
> 4) LLJIT would need to grow the underlying support methods
>
> Does that sound roughly right?
>
> Independent questions about the current C API:
> - Would be nice to document [non-]mangling behaviour of
>   LLVMOrcLLJITLookup()
> - Do I understand correctly that there's no way to keep a memory buffer
>   with an object file alive after adding it with
>   LLVMOrcLLJITAddObjectFile()? But that it's fine to create multiple
>   buffers for the same memory with LLVMCreateMemoryBufferWithMemoryRange()?
>
>
> > You can call ResourceTracker::remove at any time to remove all symbols
> and
> > resources associated with a tracker. Any active compiles associated with
> > the tracker will receive an error when they try to update the JIT state
> via
> > their MaterializationResponsibility, and will not be able to associate
> > resources with the tracker's associated ResourceKey.
> >
> > You can call ResourceTracker::transferTo at any time. This will transfer
> > tracking of all associated symbols and resources to the destination
> > tracker. Any active compiles associated with the tracker will be
> > reassociated with the destination tracker, and all future resources will
> be
> > associated with the destination tracker. Merging trackers can reduce
> > administrative overhead, especially when merging onto the default tracker
> > for the JITDylib, which has a more compact representation for ownership
> of
> > symbols.
> >
> > Calling JITDylib::clear() will call remove on all trackers created by the
> > JITDylib (including the default one).
> >
> > ResourceTrackers have shared ownership, but the ExecutionSession and
> > JITDylib do not retain ownership (except for the default tracker). If you
> > release all pointers to a tracker its resources will be automatically
> > transferred (via transferTo) to the default tracker for the JITDylib.
>
> I assume it is not legal to use the same resource tracker with two
> different JITDylib's?
>
>
> Greetings,
>
> Andres Freund
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200916/802f2e7a/attachment.html>