<div dir="ltr">Hi Stanislav,<div><br></div><div>Sorry -- I am not much help here. I would like to get ObjC runtime support for the JIT, but have not had time to look in to it closely. When I last tested the idea (a couple of years ago now) we used selector registration and objc_readClassPair to get basic test cases working as you have, but ran into (possibly similar) failures on non-trivial test cases. I think the interpretation (and possibly the layout?) of some runtime metadata varies between relocatable objects and linked images. The advice we got at the time was that we would be better off parsing the metadata in the object file and calling the runtime registration methods in objc/runtime.h, rather than fixing up the metadata ourselves. That might be worth investigating.</div><div><br></div><div>If you are not worried about performance at the moment, another option may be to ditch RuntimeDyld entirely. ORC is now decoupled from RuntimeDyld, so you are free to substitute your own linking layer. Last year I hacked up an alternative linking layer that dumps objects to disk and links them into dylibs, then dlopens them. The performance is obviously terrible, but this punts all the ObjC/Swift metadata parsing to ld64 and dyld (and you get debug info for free!) so it is handy for prototyping. Let me know if you would like me to share the code for that.</div><div><br></div><div>Cheers,</div><div>Lang.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Apr 6, 2018 at 12:19 PM, Stanislav Pankevich <span dir="ltr"><<a href="mailto:s.pankevich@gmail.com" target="_blank">s.pankevich@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi again,<br>
<br>
I had tried to follow David's suggestion to take a step back and look<br>
into codegen instead of hacking on RuntimeDyld but then I quickly<br>
realized that I don't understand what exactly needs to be done to<br>
fully register Objective-C runtime. I decided to iterate on JIT code<br>
again and somehow I found that I can hook into SectionMemory by<br>
subclassing it and working with its allocateDataSection method:<br>
<br>
1) I collect pointers to objc-related sections for which the memory is<br>
allocated. Before SectionMemoryManager::<wbr>finalizeMemory() method is<br>
called I register the ObjC runtime classes.<br>
2) I iterate over __objc_selrefs sections and fixup selectors. This<br>
does fix the original crash of this thread.<br>
3) I iterate over __objc_classlist sections and register the new<br>
classes using objc_allocateClassPair function, register the properties<br>
and ivars to these new classes, run objc_registerClassPair to complete<br>
the registration. (I still have to implement the protocol<br>
registration)<br>
4) I iterate over __objc_classrefs and __objc_superrefs and fix up the<br>
class pointers with the new classes created at step 2.<br>
5) I iterate over __objc_classlist and fix up its classes with the new<br>
classes created at step 2.<br>
<br>
The very basic Objective-C code seems to work now without any issues,<br>
however when I switch to mixed Objective-C/Swift code I start getting<br>
some crashes which I don't fully understand. In particular I am trying<br>
to run a simple XCTestCase test written in Swift and my code crashes<br>
when I access the property of this Swift's ObjC-based class.<br>
<br>
My questions are:<br>
<br>
1) Should I do anything else to properly register Objective-C besides<br>
what I am doing at steps 1-5?<br>
<br>
2) Do I have to do anything additionally to make run Swift code with<br>
LLVM JIT? On of the attached files show that the object I am working<br>
with has sections like: __swift3_* or __swift2_*. Should I do anything<br>
about this sections or they are irrelevant?<br>
<br>
3) Anything that I am missing / should know about if I want to run<br>
mixed Objective-C/Swift code via LLVM JIT?<br>
<br>
My very hacky code is located here [1]. If needed I can also share the<br>
code of my latest attempts to run the combined ObjC/Swift code.<br>
<br>
<a href="https://github.com/mull-project/mull-jit-lab/tree/master/lab-jit-objc/llvm-jit-lab/src" rel="noreferrer" target="_blank">https://github.com/mull-<wbr>project/mull-jit-lab/tree/<wbr>master/lab-jit-objc/llvm-jit-<wbr>lab/src</a><br>
<br>
Any help very much appreciated.<br>
<span class="HOEnZb"><font color="#888888"><br>
Stanislav<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
<br>
On Thu, Feb 15, 2018 at 2:33 AM, Lang Hames <<a href="mailto:lhames@gmail.com">lhames@gmail.com</a>> wrote:<br>
> Hi David, Stanislav,<br>
><br>
> Sorry for the delayed reply.<br>
><br>
> Short version: There hasn't been any progress on this just yet, as I have<br>
> been busy with an overhaul of the underlying ORC APIs.<br>
><br>
>> 1) Hack up something in RuntimeDyldMachO to handle the data structures<br>
>> currently generated by clang. This is fragile, because the interface<br>
>> between the compiler and the runtime is not documented, and is unique to<br>
>> each runtime. This code will be different on i386 and ARM, for example.<br>
>><br>
>><br>
>><br>
>> 2) Create a new CGObjCRuntime subclass that creates a module init function<br>
>> that constructs all of the classes using the public APIs, by adding<br>
>> something like -fobjc-runtime=jit to the clang flags. This is not<br>
>> particularly difficult and means that the same code can be used with any<br>
>> Objective-C runtime.<br>
><br>
><br>
> (1) is the preferred long-term solution as we want to minimize differences<br>
> between generated code in the JIT'd and non-JIT'd cases. (2) seems like a<br>
> reasonable interim solution if it is easier to implement.<br>
><br>
> Steven -- what is the status of the ObjC parsing code these days?<br>
><br>
> -- Lang.<br>
><br>
><br>
> On Wed, Feb 14, 2018 at 3:08 AM, David Chisnall via llvm-dev<br>
> <<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a>> wrote:<br>
>><br>
>><br>
>> > On 13 Feb 2018, at 17:42, Stanislav Pankevich <<a href="mailto:s.pankevich@gmail.com">s.pankevich@gmail.com</a>><br>
>> > wrote:<br>
>> ><br>
>> > On Tue, Feb 13, 2018 at 12:18 PM, David Chisnall<br>
>> > <<a href="mailto:David.Chisnall@cl.cam.ac.uk">David.Chisnall@cl.cam.ac.uk</a>> wrote:<br>
>> >> On 12 Feb 2018, at 22:31, Stanislav Pankevich via llvm-dev<br>
>> >> <<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a>> wrote:<br>
>> >>><br>
>> >>> Specifically I explored the latest objc4-723<br>
>> >>> from Apple Open Source and it looks like all of the APIs that allow<br>
>> >>> the registration of Objective-C classes, selectors, etc. are all very<br>
>> >>> private.<br>
>> >><br>
>> >> The Objective-C runtime provides public APIs for doing all of this.<br>
>> >> They’re even documented. They are also more or less standard and so work<br>
>> >> with all runtime implementations, not just the Apple one. I was using them<br>
>> >> for JIT’d code on macOS and FreeBSD 10 years ago.<br>
>> ><br>
>> > Which methods are you referring to? For example of class registration,<br>
>> > do you mean objc_allocateClassPair/objc_<wbr>registerClassPair or something<br>
>> > else?<br>
>><br>
>> Yes, those set of APIs. They provide an interface for building classes,<br>
>> protocols, and so on.<br>
>><br>
>> >>> One year ago you said you could help anyone interested in working on<br>
>> >>> this. Let me check here again as a volunteer (if this work can ever be<br>
>> >>> accomplished by someone outside Apple).<br>
>> >><br>
>> >> As I said in the earlier thread, the best way of doing this is to add a<br>
>> >> new subclass of CGObjCRuntime that generates the code using the public APIs.<br>
>> ><br>
>> > Let me get this right. What clang::CodeGen:: CGObjCRuntime has to do<br>
>> > with this? My understanding of Lang's hint was that one has to extend<br>
>> > llvm's classes like RuntimeDyldMachO to parse Mach-O, find classes,<br>
>> > selectors, categories etc and register them all manually. Are you<br>
>> > saying that something has to be be added to CodeGen/*?<br>
>><br>
>> You have two options:<br>
>><br>
>> 1) Hack up something in RuntimeDyldMachO to handle the data structures<br>
>> currently generated by clang. This is fragile, because the interface<br>
>> between the compiler and the runtime is not documented, and is unique to<br>
>> each runtime. This code will be different on i386 and ARM, for example.<br>
>><br>
>> 2) Create a new CGObjCRuntime subclass that creates a module init function<br>
>> that constructs all of the classes using the public APIs, by adding<br>
>> something like -fobjc-runtime=jit to the clang flags. This is not<br>
>> particularly difficult and means that the same code can be used with any<br>
>> Objective-C runtime.<br>
>><br>
>> >> If you’re running in the same process as the JIT, you could register<br>
>> >> the selectors in the host environment and just inject the values as symbols<br>
>> >> (this is what I did). I’d be happy to help out someone who wants to do<br>
>> >> this.<br>
>> ><br>
>> > It would be nice to get this working without embedding any of<br>
>> > Objective-C to the host process this is<br>
>><br>
>> It’s an optimisation, not a compulsory part of the process.<br>
>><br>
>> > why I am particularly<br>
>> > interested in knowing how to do the work that objc4 does in the<br>
>> > methods such as: objc4/_objc_init, objc4/map_images_nolock and<br>
>> > objc4/_read_images.<br>
>> ><br>
>> > My understanding of the goal is to make the lli example from this<br>
>> > thread working:<br>
>> ><br>
>> > <a href="https://stackoverflow.com/questions/10375324/all-selectors-unrecognised-when-invoking-objective-c-methods-using-the-llvm-exec" rel="noreferrer" target="_blank">https://stackoverflow.com/<wbr>questions/10375324/all-<wbr>selectors-unrecognised-when-<wbr>invoking-objective-c-methods-<wbr>using-the-llvm-exec</a>.<br>
>> ><br>
>> > I would be happy to get a hint on which functions of Objective-C<br>
>> > Runtime's public API should I use to get that simple example working<br>
>> > in a quick and dirty way.<br>
>><br>
>> You seem to have decided that you want to use unmodified IR from a<br>
>> specific version of Apple's Objective-C implementation. I can’t help you<br>
>> there.<br>
>><br>
>> David<br>
>><br>
>> ______________________________<wbr>_________________<br>
>> LLVM Developers mailing list<br>
>> <a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a><br>
>> <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/<wbr>mailman/listinfo/llvm-dev</a><br>
><br>
><br>
</div></div></blockquote></div><br></div>