[llvm-dev] Is it possible to execute Objective-C code via LLVM JIT?

Fri Apr 6 12:19:09 PDT 2018

Hi again,

I had tried to follow David's suggestion to take a step back and look
into codegen instead of hacking on RuntimeDyld but then I quickly
realized that I don't understand what exactly needs to be done to
fully register Objective-C runtime. I decided to iterate on JIT code
again and somehow I found that I can hook into SectionMemory by
subclassing it and working with its allocateDataSection method:

1) I collect pointers to objc-related sections for which the memory is
allocated. Before SectionMemoryManager::finalizeMemory() method is
called I register the ObjC runtime classes.
2) I iterate over __objc_selrefs sections and fixup selectors. This
does fix the original crash of this thread.
3) I iterate over __objc_classlist sections and register the new
classes using objc_allocateClassPair function, register the properties
and ivars to these new classes, run objc_registerClassPair to complete
the registration. (I still have to implement the protocol
registration)
4) I iterate over __objc_classrefs and __objc_superrefs and fix up the
class pointers with the new classes created at step 2.
5) I iterate over __objc_classlist and fix up its classes with the new
classes created at step 2.

The very basic Objective-C code seems to work now without any issues,
however when I switch to mixed Objective-C/Swift code I start getting
some crashes which I don't fully understand. In particular I am trying
to run a simple XCTestCase test written in Swift and my code crashes
when I access the property of this Swift's ObjC-based class.

My questions are:

1) Should I do anything else to properly register Objective-C besides
what I am doing at steps 1-5?

2) Do I have to do anything additionally to make run Swift code with
LLVM JIT? On of the attached files show that the object I am working
with has sections like: __swift3_* or __swift2_*. Should I do anything
about this sections or they are irrelevant?

3) Anything that I am missing / should know about if I want to run
mixed Objective-C/Swift code via LLVM JIT?

My very hacky code is located here [1]. If needed I can also share the
code of my latest attempts to run the combined ObjC/Swift code.

https://github.com/mull-project/mull-jit-lab/tree/master/lab-jit-objc/llvm-jit-lab/src

Any help very much appreciated.

Stanislav

On Thu, Feb 15, 2018 at 2:33 AM, Lang Hames <lhames at gmail.com> wrote:
> Hi David, Stanislav,
>
> Sorry for the delayed reply.
>
> Short version: There hasn't been any progress on this just yet, as I have
> been busy with an overhaul of the underlying ORC APIs.
>
>> 1) Hack up something in RuntimeDyldMachO to handle the data structures
>> currently generated by clang.  This is fragile, because the interface
>> between the compiler and the runtime is not documented, and is unique to
>> each runtime.  This code will be different on i386 and ARM, for example.
>>
>>
>>
>> 2) Create a new CGObjCRuntime subclass that creates a module init function
>> that constructs all of the classes using the public APIs, by adding
>> something like -fobjc-runtime=jit to the clang flags.  This is not
>> particularly difficult and means that the same code can be used with any
>> Objective-C runtime.
>
>
> (1) is the preferred long-term solution as we want to minimize differences
> between generated code in the JIT'd and non-JIT'd cases. (2) seems like a
> reasonable interim solution if it is easier to implement.
>
> Steven -- what is the status of the ObjC parsing code these days?
>
> -- Lang.
>
>
> On Wed, Feb 14, 2018 at 3:08 AM, David Chisnall via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>>
>>
>> > On 13 Feb 2018, at 17:42, Stanislav Pankevich <s.pankevich at gmail.com>
>> > wrote:
>> >
>> > On Tue, Feb 13, 2018 at 12:18 PM, David Chisnall
>> > <David.Chisnall at cl.cam.ac.uk> wrote:
>> >> On 12 Feb 2018, at 22:31, Stanislav Pankevich via llvm-dev
>> >> <llvm-dev at lists.llvm.org> wrote:
>> >>>
>> >>> Specifically I explored the latest objc4-723
>> >>> from Apple Open Source and it looks like all of the APIs that allow
>> >>> the registration of Objective-C classes, selectors, etc. are all very
>> >>> private.
>> >>
>> >> The Objective-C runtime provides public APIs for doing all of this.
>> >> They’re even documented.  They are also more or less standard and so work
>> >> with all runtime implementations, not just the Apple one.  I was using them
>> >> for JIT’d code on macOS and FreeBSD 10 years ago.
>> >
>> > Which methods are you referring to? For example of class registration,
>> > do you mean objc_allocateClassPair/objc_registerClassPair or something
>> > else?
>>
>> Yes, those set of APIs.  They provide an interface for building classes,
>> protocols, and so on.
>>
>> >>> One year ago you said you could help anyone interested in working on
>> >>> this. Let me check here again as a volunteer (if this work can ever be
>> >>> accomplished by someone outside Apple).
>> >>
>> >> As I said in the earlier thread, the best way of doing this is to add a
>> >> new subclass of CGObjCRuntime that generates the code using the public APIs.
>> >
>> > Let me get this right. What clang::CodeGen:: CGObjCRuntime has to do
>> > with this? My understanding of Lang's hint was that one has to extend
>> > llvm's classes like RuntimeDyldMachO to parse Mach-O, find classes,
>> > selectors, categories etc and register them all manually. Are you
>> > saying that something has to be be added to CodeGen/*?
>>
>> You have two options:
>>
>> 1) Hack up something in RuntimeDyldMachO to handle the data structures
>> currently generated by clang.  This is fragile, because the interface
>> between the compiler and the runtime is not documented, and is unique to
>> each runtime.  This code will be different on i386 and ARM, for example.
>>
>> 2) Create a new CGObjCRuntime subclass that creates a module init function
>> that constructs all of the classes using the public APIs, by adding
>> something like -fobjc-runtime=jit to the clang flags.  This is not
>> particularly difficult and means that the same code can be used with any
>> Objective-C runtime.
>>
>> >> If you’re running in the same process as the JIT, you could register
>> >> the selectors in the host environment and just inject the values as symbols
>> >> (this is what I did).  I’d be happy to help out someone who wants to do
>> >> this.
>> >
>> > It would be nice to get this working without embedding any of
>> > Objective-C to the host process this is
>>
>> It’s an optimisation, not a compulsory part of the process.
>>
>> > why I am particularly
>> > interested in knowing how to do the work that objc4 does in the
>> > methods such as: objc4/_objc_init, objc4/map_images_nolock and
>> > objc4/_read_images.
>> >
>> > My understanding of the goal is to make the lli example from this
>> > thread working:
>> >
>> > https://stackoverflow.com/questions/10375324/all-selectors-unrecognised-when-invoking-objective-c-methods-using-the-llvm-exec.
>> >
>> > I would be happy to get a hint on which functions of Objective-C
>> > Runtime's public API should I use to get that simple example working
>> > in a quick and dirty way.
>>
>> You seem to have decided that you want to use unmodified IR from a
>> specific version of Apple's Objective-C implementation.  I can’t help you
>> there.
>>
>> David
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2018-04-06 at 21.06.12.png
Type: image/png
Size: 83531 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180406/ee5c31fe/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2018-04-06 at 21.05.59.png
Type: image/png
Size: 1172508 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180406/ee5c31fe/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2018-04-06 at 21.14.01.png
Type: image/png
Size: 508841 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180406/ee5c31fe/attachment-0005.png>