[LLVMdev] Migration from JIT to MCJIT

Thu Apr 11 00:40:12 PDT 2013

Andrew,

I've attached a small reproduction of the issue.
Reproduce by:
$ /usr/bin/g++ `llvm-config --cxxflags` -g -m32 -c mcjit_external_symbol.cpp
$ /usr/bin/g++ `llvm-config --ldflags` -g -m32 -o mcjit_external_symbol mcjit_external_symbol.o `llvm-config --libs all`
$ ./mcjit_external_symbol
verifying...
LLVM ERROR: Program used external function '_external' which could not be resolved!

Note that using JIT the code works fine, in case it might help.

Eran.

From: "Kaylor, Andrew" <andrew.kaylor at intel.com<mailto:andrew.kaylor at intel.com>>
Date: Wed, 10 Apr 2013 20:50:14 -0400
To: Eran Weiss <eran.weiss at emc.com<mailto:eran.weiss at emc.com>>
Cc: "llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>" <llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>>, Jim Grosbach <grosbach at apple.com<mailto:grosbach at apple.com>>, Jiong Wang <jiwang at tilera.com<mailto:jiwang at tilera.com>>
Subject: RE: [LLVMdev] Migration from JIT to MCJIT

Eran,

Is there any chance you could boil this down to a small reproducer?

I’ve got a mid-to-long-term goal of getting rid of the ugliness in LLDB that Jim mentioned, and fixing your problem would be a good first step.

Thanks,
Andy

From: Jim Grosbach [mailto:grosbach at apple.com]
Sent: Wednesday, April 10, 2013 4:19 PM
To: Kaylor, Andrew; Jiong Wang
Cc: Weiss, Eran; llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>
Subject: Re: [LLVMdev] Migration from JIT to MCJIT

Existing clients (LLDB) deal with externals by resolving them to constant function pointers that are referenced in the IR. That’s obviously ugly as hell, but it gets things done.

The old JIT was able to simplify things because it assumed the JITed code was running in the same process as the JIT compiler. The MCJIT doesn’t assume that, so it has to handle more possibilities. For example, that the invoked function may be in a dylib which hasn’t yet been loaded into the target address space. I suggest having a look at the ld64, lld and dyld implementations to see what these relocation really imply. You won’t have to solve all of the potential scenarios to get anything done, but you will want to make sure to carefully check which use-case you’re in and error out when it’s one you haven’t handled yet. The worst case is if the code makes overly broad assumptions about possible inputs and resolves a relocation incorrectly in some subtle way.

-Jim

On Apr 10, 2013, at 11:49 AM, "Kaylor, Andrew" <andrew.kaylor at intel.com<mailto:andrew.kaylor at intel.com>> wrote:

The MachO handling isn’t always straightforward in that it has a lot of bit fields to handle while it’s also trying to accommodate the format abstractions baked into the interfaces.  The actual relocation type gets extracted in the RuntimeDyldMachO::resolveRelocation function (RuntimeDyldMachO.cpp:32) before it is passed on to the architecture-specific handlers.  So the mask you mentioned is probably OK.

The fact that this relocation type isn’t handled is, of course, the root of your problem.

MCJIT has grown on an as-needed basis up to now, with only the relocation types we were actually seeing being implemented.  Unfortunately, I don’t know enough about MachO to help with this one.  Maybe Jim Grosbach can help?

-Andy

From: llvmdev-bounces at cs.uiuc.edu<mailto:llvmdev-bounces at cs.uiuc.edu> [mailto:llvmdev-bounces at cs.uiuc.edu] On Behalf Of Weiss, Eran
Sent: Tuesday, April 09, 2013 11:51 PM
To: Jiong Wang
Cc: llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>
Subject: Re: [LLVMdev] Migration from JIT to MCJIT

Thank you for the help.

The relocation type value is anded with 0xffffffffL. (RuntimeDyldMachO.cpp:214)
Maybe this mask should be different?
Anyway, it seems like this relocation isn't implemented. (RuntimeDyldMachO.cpp:104)

From: Jiong Wang <jiwang at tilera.com<mailto:jiwang at tilera.com>>
Date: Tue, 9 Apr 2013 09:42:03 -0400
To: Eran Weiss <eran.weiss at emc.com<mailto:eran.weiss at emc.com>>
Cc: "llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>" <llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>>
Subject: Re: [LLVMdev] Migration from JIT to MCJIT

于 2013/4/9 21:08, Weiss, Eran 写道:
Hi,

I'm migrating my code (running on mac) from using JIT to MCJIT. My code generates in memory, mostly using the llvm-c api, and then runs the generated code.
When I try to use MCJIT I encounter a problem with relocations of external symbols – functions compiled statically beforehand with gcc.

I get the following error:

Invalid relocation type!
UNREACHABLE executed at /Users/weisse4/dev/llvm/llvm/lib/ExecutionEngine/RuntimeDyld/RuntimeDyldMachO.cpp:89

While debugging, I see that the relocation type read in  is 218103811, which seems corrupt to me.

Hi Weiss,

    I do not have any experience on Mach binary format, but the hex value of 218103811 is 0xd000003  (maybe the relocation type is RIT_Generic_PreboundLazyPointer = 3),  something looks like a relocation entry composed of "symbol index" + "relocation type".
    maybe something is wrong, that the relocation entry is not anded with a mask to get the final relocation type.

---
Regards,
Jiong

Did someone encounter a similar error? Or can direct me to changes that I need to do while migrating from JIT to MCJIT?

Thanks.

_______________________________________________

LLVM Developers mailing list

LLVMdev at cs.uiuc.edu<mailto:LLVMdev at cs.uiuc.edu>         http://llvm.cs.uiuc.edu<http://llvm.cs.uiuc.edu/>http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130411/28d4700a/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mcjit_external_symbol.cpp
Type: application/octet-stream
Size: 2680 bytes
Desc: mcjit_external_symbol.cpp
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130411/28d4700a/attachment.obj>