[llvm-dev] Is it possible to execute Objective-C code via LLVM JIT?

Tue May 1 09:40:23 PDT 2018

I continue looking into the Swift part of this topic.

I am missing some knowledge and skills to debug all of this faster so
without a guidance I am doing it using a trial and error method, which
is according to Wikipedia is "a fundamental method of problem solving"
so hopefully I will get it working sooner or later.

On a serious note, current issue is that Swift seems to use some
static offsets in the binary and they do not work for the classes I
create with the public APIs of Objective-C Runtime.

I have asked about this on forums.swift.org, hopefully I will get a
hint here or there. Otherwise I am going to continue debugging and
learn some more assembly :)

https://forums.swift.org/t/how-to-run-mixed-objective-c-swift-code-with-llvm-jit/12295

Thanks.

On Fri, Apr 6, 2018 at 9:19 PM, Stanislav Pankevich
<s.pankevich at gmail.com> wrote:
> Hi again,
>
> I had tried to follow David's suggestion to take a step back and look
> into codegen instead of hacking on RuntimeDyld but then I quickly
> realized that I don't understand what exactly needs to be done to
> fully register Objective-C runtime. I decided to iterate on JIT code
> again and somehow I found that I can hook into SectionMemory by
> subclassing it and working with its allocateDataSection method:
>
> 1) I collect pointers to objc-related sections for which the memory is
> allocated. Before SectionMemoryManager::finalizeMemory() method is
> called I register the ObjC runtime classes.
> 2) I iterate over __objc_selrefs sections and fixup selectors. This
> does fix the original crash of this thread.
> 3) I iterate over __objc_classlist sections and register the new
> classes using objc_allocateClassPair function, register the properties
> and ivars to these new classes, run objc_registerClassPair to complete
> the registration. (I still have to implement the protocol
> registration)
> 4) I iterate over __objc_classrefs and __objc_superrefs and fix up the
> class pointers with the new classes created at step 2.
> 5) I iterate over __objc_classlist and fix up its classes with the new
> classes created at step 2.
>
> The very basic Objective-C code seems to work now without any issues,
> however when I switch to mixed Objective-C/Swift code I start getting
> some crashes which I don't fully understand. In particular I am trying
> to run a simple XCTestCase test written in Swift and my code crashes
> when I access the property of this Swift's ObjC-based class.
>
> My questions are:
>
> 1) Should I do anything else to properly register Objective-C besides
> what I am doing at steps 1-5?
>
> 2) Do I have to do anything additionally to make run Swift code with
> LLVM JIT? On of the attached files show that the object I am working
> with has sections like: __swift3_* or __swift2_*. Should I do anything
> about this sections or they are irrelevant?
>
> 3) Anything that I am missing / should know about if I want to run
> mixed Objective-C/Swift code via LLVM JIT?
>
> My very hacky code is located here [1]. If needed I can also share the
> code of my latest attempts to run the combined ObjC/Swift code.
>
> https://github.com/mull-project/mull-jit-lab/tree/master/lab-jit-objc/llvm-jit-lab/src
>
> Any help very much appreciated.
>
> Stanislav
>
>
> On Thu, Feb 15, 2018 at 2:33 AM, Lang Hames <lhames at gmail.com> wrote:
>> Hi David, Stanislav,
>>
>> Sorry for the delayed reply.
>>
>> Short version: There hasn't been any progress on this just yet, as I have
>> been busy with an overhaul of the underlying ORC APIs.
>>
>>> 1) Hack up something in RuntimeDyldMachO to handle the data structures
>>> currently generated by clang.  This is fragile, because the interface
>>> between the compiler and the runtime is not documented, and is unique to
>>> each runtime.  This code will be different on i386 and ARM, for example.
>>>
>>>
>>>
>>> 2) Create a new CGObjCRuntime subclass that creates a module init function
>>> that constructs all of the classes using the public APIs, by adding
>>> something like -fobjc-runtime=jit to the clang flags.  This is not
>>> particularly difficult and means that the same code can be used with any
>>> Objective-C runtime.
>>
>>
>> (1) is the preferred long-term solution as we want to minimize differences
>> between generated code in the JIT'd and non-JIT'd cases. (2) seems like a
>> reasonable interim solution if it is easier to implement.
>>
>> Steven -- what is the status of the ObjC parsing code these days?
>>
>> -- Lang.
>>
>>
>> On Wed, Feb 14, 2018 at 3:08 AM, David Chisnall via llvm-dev
>> <llvm-dev at lists.llvm.org> wrote:
>>>
>>>
>>> > On 13 Feb 2018, at 17:42, Stanislav Pankevich <s.pankevich at gmail.com>
>>> > wrote:
>>> >
>>> > On Tue, Feb 13, 2018 at 12:18 PM, David Chisnall
>>> > <David.Chisnall at cl.cam.ac.uk> wrote:
>>> >> On 12 Feb 2018, at 22:31, Stanislav Pankevich via llvm-dev
>>> >> <llvm-dev at lists.llvm.org> wrote:
>>> >>>
>>> >>> Specifically I explored the latest objc4-723
>>> >>> from Apple Open Source and it looks like all of the APIs that allow
>>> >>> the registration of Objective-C classes, selectors, etc. are all very
>>> >>> private.
>>> >>
>>> >> The Objective-C runtime provides public APIs for doing all of this.
>>> >> They’re even documented.  They are also more or less standard and so work
>>> >> with all runtime implementations, not just the Apple one.  I was using them
>>> >> for JIT’d code on macOS and FreeBSD 10 years ago.
>>> >
>>> > Which methods are you referring to? For example of class registration,
>>> > do you mean objc_allocateClassPair/objc_registerClassPair or something
>>> > else?
>>>
>>> Yes, those set of APIs.  They provide an interface for building classes,
>>> protocols, and so on.
>>>
>>> >>> One year ago you said you could help anyone interested in working on
>>> >>> this. Let me check here again as a volunteer (if this work can ever be
>>> >>> accomplished by someone outside Apple).
>>> >>
>>> >> As I said in the earlier thread, the best way of doing this is to add a
>>> >> new subclass of CGObjCRuntime that generates the code using the public APIs.
>>> >
>>> > Let me get this right. What clang::CodeGen:: CGObjCRuntime has to do
>>> > with this? My understanding of Lang's hint was that one has to extend
>>> > llvm's classes like RuntimeDyldMachO to parse Mach-O, find classes,
>>> > selectors, categories etc and register them all manually. Are you
>>> > saying that something has to be be added to CodeGen/*?
>>>
>>> You have two options:
>>>
>>> 1) Hack up something in RuntimeDyldMachO to handle the data structures
>>> currently generated by clang.  This is fragile, because the interface
>>> between the compiler and the runtime is not documented, and is unique to
>>> each runtime.  This code will be different on i386 and ARM, for example.
>>>
>>> 2) Create a new CGObjCRuntime subclass that creates a module init function
>>> that constructs all of the classes using the public APIs, by adding
>>> something like -fobjc-runtime=jit to the clang flags.  This is not
>>> particularly difficult and means that the same code can be used with any
>>> Objective-C runtime.
>>>
>>> >> If you’re running in the same process as the JIT, you could register
>>> >> the selectors in the host environment and just inject the values as symbols
>>> >> (this is what I did).  I’d be happy to help out someone who wants to do
>>> >> this.
>>> >
>>> > It would be nice to get this working without embedding any of
>>> > Objective-C to the host process this is
>>>
>>> It’s an optimisation, not a compulsory part of the process.
>>>
>>> > why I am particularly
>>> > interested in knowing how to do the work that objc4 does in the
>>> > methods such as: objc4/_objc_init, objc4/map_images_nolock and
>>> > objc4/_read_images.
>>> >
>>> > My understanding of the goal is to make the lli example from this
>>> > thread working:
>>> >
>>> > https://stackoverflow.com/questions/10375324/all-selectors-unrecognised-when-invoking-objective-c-methods-using-the-llvm-exec.
>>> >
>>> > I would be happy to get a hint on which functions of Objective-C
>>> > Runtime's public API should I use to get that simple example working
>>> > in a quick and dirty way.
>>>
>>> You seem to have decided that you want to use unmodified IR from a
>>> specific version of Apple's Objective-C implementation.  I can’t help you
>>> there.
>>>
>>> David
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>