<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi Andres,<div><br></div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">Have the basics working. Should I open a PR to your branch for what I<br>got so far?</blockquote></div><div><br></div><div>Sounds good to me.</div><div><br></div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">There's currently two things I am still missing from the C API.<br></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"> </blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">The first is to expose more of symbol resolution. Postgres can be<br>extended via dynamically loaded libraries, and we need to be careful to<br>look up symbols in the correct library. In the generated JIT IR<br>references to functions in such libraries use a distinguishable symbol<br>name (see example below), indicating that they should not be looked up<br>in the global namespace.</blockquote></div><div><br></div><div>From the example code I guess the namespacing for symbols from external libraries is something like: "<modulename>.<symbolname>" ?</div><div><br></div><div>This is what definition generators are for. We can add an API for attaching generators and provide facilities to define a generator in C.</div><div><br></div><div>Here you'd want a generator with a map of module names to dylib handles. First you'd search for a handle for <modulename> (loading it if it didn't already exist) then you'd dlsym <symbolname> in that handle.</div><div><br></div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">Related to that, when hitting this case I currently get, on stderr:<br>JIT session error: Symbols not found: [ pgextern./home/andres/build/postgres/dev-assert/vpath/src/test/regress/regress.so.pt_in_widget ]<br>but currently that error is not bubbled up to the C API. There I just<br>get:<br>ERROR: XX000: failed to look up symbol "evalexpr_0_0": Failed to materialize symbols: { (main, { evalexpr_0_0 }) }</blockquote></div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">(obviously the first part is postgres code)<br>Do you have thoughts on what that should look like?</blockquote></div><div><br></div><div>We need an API to install an error reporter on the session: That's where the SymbolsNotFound error will be delivered. I'll add that in a minute.</div><div><br></div><div><div>The other part that I am still missing is replacment for</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">LLVMCreateGDBRegistrationListener();<br>LLVMCreatePerfJITEventListener();<br>LLVMOrcRegisterJITEventListener(llvm_opt0_orc, l);<br>I haven't yet looked at whether there is replacement infrastructure for<br>this already. Is there? Will look after the next few meetings...</blockquote></div><div><br></div><div>You can continue to use LLVMCreateGDBRegistrationListener and LLVMCreatePerfJITEventListener from ExecutionEngine.h for now. We just need to add an OrcV2 equivalent of LLVMOrcRegisterJITEventListener. I'll do that today.</div><div><br></div><div>-- Lang.</div><div><br></div></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Sep 29, 2020 at 11:58 AM Andres Freund <<a href="mailto:andres@anarazel.de">andres@anarazel.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
On 2020-09-25 16:38:41 -0700, Andres Freund via llvm-dev wrote:<br>
> On 2020-09-24 16:34:30 -0700, Lang Hames wrote:<br>
> > If anyone wants to check out the OrcV1 removal branch and provide feedback<br>
> > now is the time. Otherwise I will aim to land the work in the mainline<br>
> > early next week.<br>
> <br>
> I'm trying to get it to work with postgres. Unfortunately this week was a bit less<br>
> productive than I had hoped... Hope to get there Mon/Tues.<br>
<br>
Have the basics working. Should I open a PR to your branch for what I<br>
got so far?<br>
<br>
<br>
There's currently two things I am still missing from the C API.<br>
<br>
The first is to expose more of symbol resolution. Postgres can be<br>
extended via dynamically loaded libraries, and we need to be careful to<br>
look up symbols in the correct library. In the generated JIT IR<br>
references to functions in such libraries use a distinguishable symbol<br>
name (see example below), indicating that they should not be looked up<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
in the global namespace.</blockquote>
<br>
Previously we used the symbol resolution callback to<br>
LLVMOrcAddEagerlyCompiledIR to implement this. If a symbol name was<br>
scoped to a library, we only searched inside that, rather than globally:<br>
<br>
/*<br>
* Attempt to resolve symbol, so LLVM can emit a reference to it.<br>
*/<br>
static uint64_t<br>
llvm_resolve_symbol(const char *symname, void *ctx)<br>
{<br>
...<br>
llvm_split_symbol_name(symname, &modname, &funcname);<br>
...<br>
<br>
if (modname)<br>
addr = (uintptr_t) load_external_function(modname, funcname,<br>
true, NULL);<br>
else<br>
addr = (uintptr_t) LLVMSearchForAddressOfSymbol(symname);<br>
...<br>
return (uint64_t) addr;<br>
}<br>
<br>
I don't think that can be implemented with the current C API.<br>
<br>
<br>
Related to that, when hitting this case I currently get, on stderr:<br>
JIT session error: Symbols not found: [ pgextern./home/andres/build/postgres/dev-assert/vpath/src/test/regress/regress.so.pt_in_widget ]<br>
<br>
but currently that error is not bubbled up to the C API. There I just<br>
get:<br>
ERROR: XX000: failed to look up symbol "evalexpr_0_0": Failed to materialize symbols: { (main, { evalexpr_0_0 }) }<br>
<br>
(obviously the first part is postgres code)<br>
<br>
Do you have thoughts on what that should look like?<br>
<br>
<br>
The other part that I am still missing is replacment for<br>
<br>
LLVMCreateGDBRegistrationListener();<br>
LLVMCreatePerfJITEventListener();<br>
LLVMOrcRegisterJITEventListener(llvm_opt0_orc, l);<br>
<br>
I haven't yet looked at whether there is replacement infrastructure for<br>
this already. Is there? Will look after the next few meetings...<br>
<br>
Greetings,<br>
<br>
Andres Freund<br>
</blockquote></div></div></div></div></div></div></div></div></div>