[compiler-rt] [orc][mach-o] Unlock the JITDylib state mutex during +load (PR #105333)

[Ben Langmuir via llvm-commits]

https://github.com/benlangmuir created https://github.com/llvm/llvm-project/pull/105333

Similar to what was already done for static initializers, we need to unlock the state mutext when calling out to libobjc to run +load methods in case they cause us to reenter the runtime, which was previously deadlocking. No test for now, because we don't have any code paths in llvm-jitlink itself that could lead to this deadlock. If we interpose calls to dlopen to go back to the JIT in the future then calling dlopen from a +load is the easiest way to reproduce this.

rdar://133430490

>From 0d75abba98bb796bcd514c1d6d26c71c27d0c76f Mon Sep 17 00:00:00 2001
From: Ben Langmuir <blangmuir at apple.com>
Date: Tue, 20 Aug 2024 11:34:04 -0700
Subject: [PATCH] [orc][mach-o] Unlock the JITDylib state mutex during +load

Similar to what was already done for static initializers, we need to
unlock the state mutext when calling out to libobjc to run +load methods
in case they cause us to reenter the runtime, which was previously
deadlocking. No test for now, because we don't have any code paths in
llvm-jitlink itself that could lead to this deadlock. If we interpose
calls to dlopen to go back to the JIT in the future then calling dlopen
from a +load is the easiest way to reproduce this.

rdar://133430490
---
 compiler-rt/lib/orc/macho_platform.cpp | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/compiler-rt/lib/orc/macho_platform.cpp b/compiler-rt/lib/orc/macho_platform.cpp
index 340846f5f9001..9b4f6e5fd417c 100644
--- a/compiler-rt/lib/orc/macho_platform.cpp
+++ b/compiler-rt/lib/orc/macho_platform.cpp
@@ -367,7 +367,9 @@ class MachOPlatformRuntimeState {
   static Error registerEHFrames(span<const char> EHFrameSection);
   static Error deregisterEHFrames(span<const char> EHFrameSection);
 
-  static Error registerObjCRegistrationObjects(JITDylibState &JDS);
+  static Error
+  registerObjCRegistrationObjects(std::unique_lock<std::mutex> &JDStatesLock,
+                                  JITDylibState &JDS);
   static Error runModInits(std::unique_lock<std::mutex> &JDStatesLock,
                            JITDylibState &JDS);
 
@@ -1059,7 +1061,7 @@ Error MachOPlatformRuntimeState::deregisterEHFrames(
 }
 
 Error MachOPlatformRuntimeState::registerObjCRegistrationObjects(
-    JITDylibState &JDS) {
+    std::unique_lock<std::mutex> &JDStatesLock, JITDylibState &JDS) {
   ORC_RT_DEBUG(printdbg("Registering Objective-C / Swift metadata.\n"));
 
   std::vector<char *> RegObjBases;
@@ -1074,6 +1076,9 @@ Error MachOPlatformRuntimeState::registerObjCRegistrationObjects(
         "Could not register Objective-C / Swift metadata: _objc_map_images / "
         "_objc_load_image not found");
 
+  // Release the lock while calling out to libobjc in case +load methods cause
+  // reentering the orc runtime.
+  JDStatesLock.unlock();
   std::vector<char *> Paths;
   Paths.resize(RegObjBases.size());
   _objc_map_images(RegObjBases.size(), Paths.data(),
@@ -1081,6 +1086,7 @@ Error MachOPlatformRuntimeState::registerObjCRegistrationObjects(
 
   for (void *RegObjBase : RegObjBases)
     _objc_load_image(nullptr, reinterpret_cast<mach_header *>(RegObjBase));
+  JDStatesLock.lock();
 
   return Error::success();
 }
@@ -1218,7 +1224,7 @@ Error MachOPlatformRuntimeState::dlopenInitialize(
   }
 
   // Initialize this JITDylib.
-  if (auto Err = registerObjCRegistrationObjects(JDS))
+  if (auto Err = registerObjCRegistrationObjects(JDStatesLock, JDS))
     return Err;
   if (auto Err = runModInits(JDStatesLock, JDS))
     return Err;