[Mlir-commits] [mlir] [mlir][python] fix symbol resolution on MacOS with multiple packages (PR #174057)
llvmlistbot at llvm.org
llvmlistbot at llvm.org
Wed Dec 31 00:14:35 PST 2025
llvmbot wrote:
<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-mlir
Author: Maksim Levental (makslevental)
<details>
<summary>Changes</summary>
# Problem:
There are two build system bugs on MacOS in the case where one intends to use multiple bindings packages simultaneously (same Python interpreter session):
1. The nanobind modules are built with [`-Wl,-flat_namespace`](https://github.com/llvm/llvm-project/blob/8518d2c4057d9aa4249b8466a4d77771e4f1bf4f/llvm/cmake/modules/HandleLLVMOptions.cmake#L268) thereby leading to ambiguous symbols across multiple whatever dylibs;
2. Intra-library symbol resolution (within the C API aggregate dylib) fails to resolve symbols correctly unless _everything_ is built with `-DCMAKE_C_VISIBILITY_PRESET=hidden -DCMAKE_CXX_VISIBILITY_PRESET=hidden -DCMAKE_VISIBILITY_INLINES_HIDDEN=ON`
# Verification:
On a Mac:
1. Comment out the added `target_link_options(${libname} PRIVATE "LINKER:-twolevel_namespace")` CMake and run `LIT_FILTER=test.toy ninja check-mlir` (assuming you have `-DLLVM_BUILD_EXAMPLES=ON -DLLVM_INCLUDE_EXAMPLES=ON`) and you will see:
```
LLVM ERROR: can't create Attribute 'mlir::StringAttr' because storage uniquer isn't initialized: the dialect was likely not loaded, or the attribute wasn't added with addAttributes<...>() in the Dialect::initialize() method.
```
2. Uncomment `# linalg` in `smoketest.py` and run the same lit test and you will see:
```
LLVM ERROR: Attempting to attach an interface to an unregistered operation builtin.unrealized_conversion_cast.
```
# Fix
We can only fix the first of these (by adding the `twolevel_namespace` link flag) because just adding something like
```
set_target_properties(MLIRPythonCAPI PROPERTIES
C_VISIBILITY_PRESET hidden
CXX_VISIBILITY_PRESET hidden
VISIBILITY_INLINES_HIDDEN YES)
```
won't actually affect the visibilities set on the constituent lib targets (and of course we cannot mandate that these visibilities should be applied to all constituent lib targets e.g., within `mlir_add_library`).
# Why is this not happening on Linux
Quoting [`Linux man dlopen`](https://man7.org/linux/man-pages/man3/dlopen.3.html ):
> RTLD_LOCAL: This is ... **the default if neither flag is specified**. Symbols defined in this shared object are not made available to resolve references in subsequently loaded shared objects.
vs. [`MacOs man dlopen`](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/dlopen.3.html):
> If neither RTLD_GLOBAL nor RTLD_LOCAL is specified, **the default is RTLD_GLOBAL**.
Note, these scopes are transitive:
> The dlmopen() function differs from dlopen() primarily in that it accepts an additional argument, lmid, that specifies the link-map list (also referred to as a namespace) in which the shared object should be loaded. **By comparison, dlopen() adds the dynamically loaded shared object to the same namespace as the shared object from which the dlopen() call is made** (cf Linux man page).
---
Full diff: https://github.com/llvm/llvm-project/pull/174057.diff
6 Files Affected:
- (modified) mlir/cmake/modules/AddMLIRPython.cmake (+6)
- (modified) mlir/examples/standalone/CMakeLists.txt (+3-1)
- (modified) mlir/examples/standalone/test/lit.cfg.py (+5-7)
- (modified) mlir/examples/standalone/test/python/smoketest.py (+46-7)
- (modified) mlir/test/Examples/standalone/test.toy (+2)
- (modified) mlir/test/Examples/standalone/test.wheel.toy (+12-2)
``````````diff
diff --git a/mlir/cmake/modules/AddMLIRPython.cmake b/mlir/cmake/modules/AddMLIRPython.cmake
index ca90151e76268..8c301faf0941a 100644
--- a/mlir/cmake/modules/AddMLIRPython.cmake
+++ b/mlir/cmake/modules/AddMLIRPython.cmake
@@ -766,6 +766,12 @@ function(add_mlir_python_extension libname extname)
FREE_THREADED
${ARG_SOURCES}
)
+ if(APPLE)
+ # In llvm/cmake/modules/HandleLLVMOptions.cmake:268 we set -Wl,-flat_namespace which breaks
+ # the default name spacing on MacOS and causes "cross-wired" symbol resolution when multiple
+ # bindings packages are loaded.
+ target_link_options(${libname} PRIVATE "LINKER:-twolevel_namespace")
+ endif()
if (NOT MLIR_DISABLE_CONFIGURE_PYTHON_DEV_PACKAGES
AND (LLVM_COMPILER_IS_GCC_COMPATIBLE OR CLANG_CL))
diff --git a/mlir/examples/standalone/CMakeLists.txt b/mlir/examples/standalone/CMakeLists.txt
index c6c49fde12d2e..955c9ec7a7b4c 100644
--- a/mlir/examples/standalone/CMakeLists.txt
+++ b/mlir/examples/standalone/CMakeLists.txt
@@ -71,7 +71,9 @@ if(MLIR_ENABLE_BINDINGS_PYTHON)
endif()
add_subdirectory(python)
endif()
-add_subdirectory(test)
+if(MLIR_INCLUDE_TESTS)
+ add_subdirectory(test)
+endif()
add_subdirectory(standalone-opt)
if(NOT WIN32)
add_subdirectory(standalone-plugin)
diff --git a/mlir/examples/standalone/test/lit.cfg.py b/mlir/examples/standalone/test/lit.cfg.py
index e27dddd7fb0b9..89cdd6889a1f2 100644
--- a/mlir/examples/standalone/test/lit.cfg.py
+++ b/mlir/examples/standalone/test/lit.cfg.py
@@ -61,10 +61,8 @@
llvm_config.add_tool_substitutions(tools, tool_dirs)
-llvm_config.with_environment(
- "PYTHONPATH",
- [
- os.path.join(config.mlir_obj_dir, "python_packages", "standalone"),
- ],
- append_path=True,
-)
+python_path = [os.path.join(config.mlir_obj_dir, "python_packages", "standalone")]
+if "PYTHONPATH" in os.environ:
+ python_path += [os.environ["PYTHONPATH"]]
+
+llvm_config.with_environment("PYTHONPATH", python_path, append_path=True)
diff --git a/mlir/examples/standalone/test/python/smoketest.py b/mlir/examples/standalone/test/python/smoketest.py
index f8819841fac45..addd767f53592 100644
--- a/mlir/examples/standalone/test/python/smoketest.py
+++ b/mlir/examples/standalone/test/python/smoketest.py
@@ -1,16 +1,55 @@
-# RUN: %python %s nanobind | FileCheck %s
+# RUN: %python %s 2>&1 | FileCheck %s
+import sys
-from mlir_standalone.ir import *
+# CHECK: Testing mlir_standalone package
+print("Testing mlir_standalone package", file=sys.stderr)
+
+import mlir_standalone.ir
from mlir_standalone.dialects import standalone_nanobind as standalone_d
-with Context():
+with mlir_standalone.ir.Context():
standalone_d.register_dialects()
- module = Module.parse(
+ standalone_module = mlir_standalone.ir.Module.parse(
"""
%0 = arith.constant 2 : i32
%1 = standalone.foo %0 : i32
"""
)
- # CHECK: %[[C:.*]] = arith.constant 2 : i32
- # CHECK: standalone.foo %[[C]] : i32
- print(str(module))
+ # CHECK: %[[C2:.*]] = arith.constant 2 : i32
+ # CHECK: standalone.foo %[[C2]] : i32
+ print(str(standalone_module), file=sys.stderr)
+
+
+# CHECK: Testing mlir package
+print("Testing mlir package", file=sys.stderr)
+
+import mlir.ir
+from mlir.dialects import (
+ amdgpu,
+ gpu,
+ irdl,
+ llvm,
+ nvgpu,
+ pdl,
+ quant,
+ smt,
+ sparse_tensor,
+ transform,
+ # Note: uncommenting linalg below will cause
+ # LLVM ERROR: Attempting to attach an interface to an unregistered operation builtin.unrealized_conversion_cast.
+ # unless you have built both mlir and mlir_standalone with
+ # -DCMAKE_C_VISIBILITY_PRESET=hidden -DCMAKE_CXX_VISIBILITY_PRESET=hidden -DCMAKE_VISIBILITY_INLINES_HIDDEN=ON
+ # which is highly recommended.
+ # linalg,
+)
+
+# CHECK-NOT: RuntimeWarning: nanobind: type '{{.*}}' was already registered!
+
+with mlir.ir.Context():
+ mlir_module = mlir.ir.Module.parse(
+ """
+ %0 = arith.constant 3 : i32
+ """
+ )
+ # CHECK: %[[C3:.*]] = arith.constant 3 : i32
+ print(str(mlir_module), file=sys.stderr)
diff --git a/mlir/test/Examples/standalone/test.toy b/mlir/test/Examples/standalone/test.toy
index a88c115ebf197..dc3c17f3da3d9 100644
--- a/mlir/test/Examples/standalone/test.toy
+++ b/mlir/test/Examples/standalone/test.toy
@@ -4,8 +4,10 @@
# RUN: -DLLVM_ENABLE_LIBCXX=%enable_libcxx -DMLIR_DIR=%mlir_cmake_dir \
# RUN: -DLLVM_USE_LINKER=%llvm_use_linker \
# RUN: -DMLIR_PYTHON_PACKAGE_PREFIX=mlir_standalone \
+# RUN: -DMLIR_INCLUDE_TESTS=ON \
# RUN: -DPython3_EXECUTABLE=%python \
# RUN: -DPython_EXECUTABLE=%python
+# RUN: export PYTHONPATH="%mlir_obj_root/python_packages/mlir_core"
# RUN: "%cmake_exe" --build . --target check-standalone | tee %t
# RUN: FileCheck --input-file=%t %s
diff --git a/mlir/test/Examples/standalone/test.wheel.toy b/mlir/test/Examples/standalone/test.wheel.toy
index c8d188a3cacd0..e9232e3f16098 100644
--- a/mlir/test/Examples/standalone/test.wheel.toy
+++ b/mlir/test/Examples/standalone/test.wheel.toy
@@ -14,21 +14,31 @@
# RUN: export CMAKE_GENERATOR=%cmake_generator
# RUN: export LLVM_USE_LINKER=%llvm_use_linker
# RUN: export MLIR_DIR="%mlir_cmake_dir"
+# RUN: export MLIR_INCLUDE_TESTS=ON
# RUN: %python -m pip wheel "%mlir_src_root/examples/standalone" -w "%mlir_obj_root/wheelhouse" -v | tee %t
# RUN: rm -rf "%mlir_obj_root/standalone-python-bindings-install"
# RUN: %python -m pip install standalone_python_bindings -f "%mlir_obj_root/wheelhouse" --target "%mlir_obj_root/standalone-python-bindings-install" -v | tee -a %t
-# RUN: export PYTHONPATH="%mlir_obj_root/standalone-python-bindings-install"
-# RUN: %python "%mlir_src_root/examples/standalone/test/python/smoketest.py" nanobind | tee -a %t
+# RUN: export PYTHONPATH="%mlir_obj_root/standalone-python-bindings-install:%mlir_obj_root/python_packages/mlir_core"
+# RUN: %python "%mlir_src_root/examples/standalone/test/python/smoketest.py" 2>&1 | tee -a %t
# RUN: FileCheck --input-file=%t %s
# CHECK: Successfully built standalone-python-bindings
+# CHECK: Testing mlir_standalone package
+
# CHECK: module {
# CHECK: %[[C2:.*]] = arith.constant 2 : i32
# CHECK: %[[V0:.*]] = standalone.foo %[[C2]] : i32
# CHECK: }
+# CHECK: Testing mlir package
+
+# CHECK-NOT: RuntimeWarning: nanobind: type '{{.*}}' was already registered!
+
+# CHECK: module {
+# CHECK: %[[C3:.*]] = arith.constant 3 : i32
+# CHECK: }
``````````
</details>
https://github.com/llvm/llvm-project/pull/174057
More information about the Mlir-commits
mailing list