[llvm-dev] OrcV1 removal

[Lang Hames via llvm-dev]

Hi Andres,

I've just realised that we're going to need a change to the definition
generator API in the long term: Right now it is called under the session
lock, but we want to shift to calling it outside the lock and passing a
lookup-continuation. This would allow definition discovery to take an
arbitrarily long time without blocking forward progress on other JIT work.

I'm guessing your objective for now is to minimize churn, right? If that's
the case I'll focus on moving to the continuation passing API on the
orcv1-removal branch before I update the C API.

Alternatively if you don't mind some minor churn before LLVM 12 I'll focus
on landing the C API in its current form, then landing orcv1-removal on the
mainline, then I can work on the continuation passing update in the
mainline.

-- Lang.

On Tue, Sep 29, 2020 at 5:26 PM Andres Freund <andres at anarazel.de> wrote:

> Hi,
>
> On 2020-09-29 13:58:35 -0700, Lang Hames wrote:
> > > Have the basics working. Should I open a PR to your branch for what I
> > > got so far?
>
> > Sounds good to me.
>
> Done, https://github.com/lhames/llvm-project/pull/2 .
>
>
> > The first is to expose more of symbol resolution. Postgres can be
> > > extended via dynamically loaded libraries, and we need to be careful to
> > > look up symbols in the correct library. In the generated JIT IR
> > > references to functions in such libraries use a distinguishable symbol
> > > name (see example below), indicating that they should not be looked up
> > > in the global namespace.
> >
> >
> > From the example code I guess the namespacing for symbols from external
> > libraries is something like: "<modulename>.<symbolname>" ?
>
> It's currently pgextern.<path_to_library>.<symbolname>
>
>
> > This is what definition generators are for. We can add an API for
> attaching
> > generators and provide facilities to define a generator in C.
> > Here you'd want a generator with a map of module names to dylib
> > handles.
> > First you'd search for a handle for <modulename> (loading it if it didn't
> > already exist) then you'd dlsym <symbolname> in that handle.
>
> Ok, cool.
>
>
> > Related to that, when hitting this case I currently get, on stderr:
> > > JIT session error: Symbols not found: [
> > >
> pgextern./home/andres/build/postgres/dev-assert/vpath/src/test/regress/regress.so.pt_in_widget
> > > ]
> > > but currently that error is not bubbled up to the C API. There I just
> > > get:
> > > ERROR:  XX000: failed to look up symbol "evalexpr_0_0": Failed to
> > > materialize symbols: { (main, { evalexpr_0_0 }) }
> >
> > (obviously the first part is postgres code)
> > > Do you have thoughts on what that should look like?
> >
> >
> > We need an API to install an error reporter on the session: That's where
> > the SymbolsNotFound error will be delivered. I'll add that in a minute.
>
> Sounds good.
>
>
> > > The other part that I am still missing is replacment for
> > > LLVMCreateGDBRegistrationListener();
> > > LLVMCreatePerfJITEventListener();
> > > LLVMOrcRegisterJITEventListener(llvm_opt0_orc, l);
> > > I haven't yet looked at whether there is replacement infrastructure for
> > > this already. Is there?  Will look after the next few meetings...
>
> > You can continue to use LLVMCreateGDBRegistrationListener and
> > LLVMCreatePerfJITEventListener from ExecutionEngine.h for now. We just
> need
> > to add an OrcV2 equivalent of LLVMOrcRegisterJITEventListener. I'll do
> that
> > today.
>
> Cool.
>
> Greetings,
>
> Andres Freund
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200930/d14df4c0/attachment.html>