[Mlir-commits] [mlir] e4af5b1 - [mlir][python] fix symbol resolution on MacOS with multiple packages (#174057)
llvmlistbot at llvm.org
llvmlistbot at llvm.org
Fri Jan 2 10:54:01 PST 2026
Author: Maksim Levental
Date: 2026-01-02T18:53:57Z
New Revision: e4af5b102b4de999ba43837ee580db0de3432331
URL: https://github.com/llvm/llvm-project/commit/e4af5b102b4de999ba43837ee580db0de3432331
DIFF: https://github.com/llvm/llvm-project/commit/e4af5b102b4de999ba43837ee580db0de3432331.diff
LOG: [mlir][python] fix symbol resolution on MacOS with multiple packages (#174057)
# Problem:
There are two build system bugs on MacOS in the case where one intends
to use multiple bindings packages simultaneously (same Python
interpreter session):
1. The nanobind modules are built with
[`-Wl,-flat_namespace`](https://github.com/llvm/llvm-project/blob/8518d2c4057d9aa4249b8466a4d77771e4f1bf4f/llvm/cmake/modules/HandleLLVMOptions.cmake#L268)
thereby leading to ambiguous symbols across multiple whatever dylibs;
2. Intra-library symbol resolution (within the C API aggregate dylib)
fails to resolve symbols correctly unless things are built with
`-DCMAKE_C_VISIBILITY_PRESET=hidden -DCMAKE_CXX_VISIBILITY_PRESET=hidden
-DCMAKE_VISIBILITY_INLINES_HIDDEN=ON`.
# Repro:
On a Mac (with this patch applied):
1. Build without `twolevel_namespace` and without hidden vis properties
and run `LIT_FILTER=test.toy ninja check-mlir` (assuming you have
`-DLLVM_BUILD_EXAMPLES=ON -DLLVM_INCLUDE_EXAMPLES=ON`) and you will see:
```
LLVM ERROR: can't create Attribute 'mlir::StringAttr' because storage
uniquer isn't initialized: the dialect was likely not loaded, or the
attribute wasn't added with addAttributes<...>() in the
Dialect::initialize() method.
```
2. Build with `twolevel_namespace` but not hidden vis and run the same
lit test and you will see:
```
LLVM ERROR: Attempting to attach an interface to an unregistered
operation builtin.unrealized_conversion_cast.
```
# Fix
We only do a partial fix here (adding `twolevel_namespace` to Python
bindings modules) because a full fix requires adding visibility
attributes to all object files. I added docs discussing this.
# Why is this not happening on Linux
Using `DYLD_PRINT_BINDINGS=1` I observe that for the checked-in/updated
test (without the fix) `libMLIRPythonCAPI` resolves many of its symbols
to `libStandalonePythonCAPI`:
```
dyld[98449]: looking for weak-def symbol '__ZN4mlir6TypeID3getINS_13AffineMapAttrEEES0_v':
dyld[98449]: found __ZN4mlir6TypeID3getINS_13AffineMapAttrEEES0_v in map, using impl from /Users/maksimlevental/dev_projects/llvm-project/cmake-build-debug/tools/mlir/test/Examples/standalone/python_packages/standalone/mlir_standalone/_mlir_libs/libStandalonePythonCAPI.dylib
dyld[98449]: <libMLIRPythonCAPI.dylib/bind#22> -> 0x11348fa9c <libStandalonePythonCAPI.dylib/__ZN4mlir6TypeID3getINS_13AffineMapAttrEEES0_v>)
dyld[98449]: looking for weak-def symbol '__ZN4mlir6TypeID3getINS_9ArrayAttrEEES0_v':
dyld[98449]: found __ZN4mlir6TypeID3getINS_9ArrayAttrEEES0_v in map, using impl from /Users/maksimlevental/dev_projects/llvm-project/cmake-build-debug/tools/mlir/test/Examples/standalone/python_packages/standalone/mlir_standalone/_mlir_libs/libStandalonePythonCAPI.dylib
dyld[98449]: <libMLIRPythonCAPI.dylib/bind#23> -> 0x11348f990 <libStandalonePythonCAPI.dylib/__ZN4mlir6TypeID3getINS_9ArrayAttrEEES0_v>)
dyld[98449]: looking for weak-def symbol '__ZN4mlir6TypeID3getINS_14DictionaryAttrEEES0_v':
dyld[98449]: found __ZN4mlir6TypeID3getINS_14DictionaryAttrEEES0_v in map, using impl from /Users/maksimlevental/dev_projects/llvm-project/cmake-build-debug/tools/mlir/test/Examples/standalone/python_packages/standalone/mlir_standalone/_mlir_libs/libStandalonePythonCAPI.dylib
dyld[98449]: <libMLIRPythonCAPI.dylib/bind#24> -> 0x11348eec0 <libStandalonePythonCAPI.dylib/__ZN4mlir6TypeID3getINS_14DictionaryAttrEEES0_v>)
```
Turns out this is "expected" behavior:
> It appears on macOS, when a static library is compiled without
-fvisibility=hidden, its C++ template instantiations could lead to
leftover weak symbols that are resolved and bound at runtime
https://joyeecheung.github.io/blog/2025/01/11/executable-loading-and-startup-performance-on-macos/
🤷
Added:
Modified:
mlir/cmake/modules/AddMLIRPython.cmake
mlir/docs/Bindings/Python.md
mlir/examples/standalone/CMakeLists.txt
mlir/examples/standalone/pyproject.toml
mlir/examples/standalone/test/lit.cfg.py
mlir/examples/standalone/test/python/smoketest.py
mlir/test/Examples/standalone/test.toy
mlir/test/Examples/standalone/test.wheel.toy
Removed:
################################################################################
diff --git a/mlir/cmake/modules/AddMLIRPython.cmake b/mlir/cmake/modules/AddMLIRPython.cmake
index ca90151e76268..8c301faf0941a 100644
--- a/mlir/cmake/modules/AddMLIRPython.cmake
+++ b/mlir/cmake/modules/AddMLIRPython.cmake
@@ -766,6 +766,12 @@ function(add_mlir_python_extension libname extname)
FREE_THREADED
${ARG_SOURCES}
)
+ if(APPLE)
+ # In llvm/cmake/modules/HandleLLVMOptions.cmake:268 we set -Wl,-flat_namespace which breaks
+ # the default name spacing on MacOS and causes "cross-wired" symbol resolution when multiple
+ # bindings packages are loaded.
+ target_link_options(${libname} PRIVATE "LINKER:-twolevel_namespace")
+ endif()
if (NOT MLIR_DISABLE_CONFIGURE_PYTHON_DEV_PACKAGES
AND (LLVM_COMPILER_IS_GCC_COMPATIBLE OR CLANG_CL))
diff --git a/mlir/docs/Bindings/Python.md b/mlir/docs/Bindings/Python.md
index 877ae5170d68c..4f4f531f7723c 100644
--- a/mlir/docs/Bindings/Python.md
+++ b/mlir/docs/Bindings/Python.md
@@ -25,6 +25,18 @@
multiple Python implementations, setting this explicitly to the preferred
`python3` executable is strongly recommended.
+* **`CMAKE_C_VISIBILITY_PRESET`**: `STRING`
+* **`CMAKE_CXX_VISIBILITY_PRESET`**: `STRING`
+* **`CMAKE_VISIBILITY_INLINES_HIDDEN`**: `BOOL`
+
+ It is **highly** recommended these are set to `hidden`, `hidden`, and `ON` (respectively) if the final built package
+ is intended to be used in a context/use-case where multiple bindings packages will be used simultaneously
+ (i.e., multiple bindings packages loaded in the same Python interpreter session). Failing to do so can lead
+ to incorrect/ambiguous symbol resolution; the symptom of this is an `LLVM ERROR` like:
+ ```
+ LLVM ERROR: ... unregistered/uninitialized dialect/type/pass ...`
+ ```
+
### Recommended development practices
It is recommended to use a Python virtual environment. Many ways exist for this,
diff --git a/mlir/examples/standalone/CMakeLists.txt b/mlir/examples/standalone/CMakeLists.txt
index c6c49fde12d2e..17d712f6a1064 100644
--- a/mlir/examples/standalone/CMakeLists.txt
+++ b/mlir/examples/standalone/CMakeLists.txt
@@ -66,12 +66,17 @@ if(MLIR_ENABLE_BINDINGS_PYTHON)
if(NOT MLIR_PYTHON_PACKAGE_PREFIX)
set(MLIR_PYTHON_PACKAGE_PREFIX "mlir_standalone" CACHE STRING "" FORCE)
endif()
+ if(NOT MLIR_BINDINGS_PYTHON_NB_DOMAIN)
+ set(MLIR_BINDINGS_PYTHON_NB_DOMAIN "mlir_standalone" CACHE STRING "" FORCE)
+ endif()
if(NOT MLIR_BINDINGS_PYTHON_INSTALL_PREFIX)
set(MLIR_BINDINGS_PYTHON_INSTALL_PREFIX "python_packages/standalone/${MLIR_PYTHON_PACKAGE_PREFIX}" CACHE STRING "" FORCE)
endif()
add_subdirectory(python)
endif()
-add_subdirectory(test)
+if(MLIR_INCLUDE_TESTS)
+ add_subdirectory(test)
+endif()
add_subdirectory(standalone-opt)
if(NOT WIN32)
add_subdirectory(standalone-plugin)
diff --git a/mlir/examples/standalone/pyproject.toml b/mlir/examples/standalone/pyproject.toml
index c4194153743ef..4b9627a3ca582 100644
--- a/mlir/examples/standalone/pyproject.toml
+++ b/mlir/examples/standalone/pyproject.toml
@@ -6,7 +6,7 @@
[project]
name = "standalone-python-bindings"
dynamic = ["version"]
-requires-python = ">=3.8,<=3.14"
+requires-python = ">=3.8"
dependencies = [
"numpy>=1.19.5, <=2.1.2",
"PyYAML>=5.4.0, <=6.0.1",
@@ -56,9 +56,14 @@ MLIR_DIR = { env = "MLIR_DIR", default = "" }
# Non-optional
CMAKE_BUILD_TYPE = { env = "CMAKE_BUILD_TYPE", default = "Release" }
MLIR_ENABLE_BINDINGS_PYTHON = "ON"
+
# Effectively non-optional (any downstream project should specify this).
-MLIR_PYTHON_PACKAGE_PREFIX = "mlir_standalone"
+MLIR_BINDINGS_PYTHON_NB_DOMAIN = { env = "MLIR_BINDINGS_PYTHON_NB_DOMAIN", default = "mlir_standalone" }
+MLIR_PYTHON_PACKAGE_PREFIX = { env = "MLIR_PYTHON_PACKAGE_PREFIX", default = "mlir_standalone" }
+
# This specifies the directory in the install directory (i.e., /tmp/pip-wheel/platlib) where _mlir_libs, dialects, etc.
# are installed. Thus, this will be the package location (and the name of the package) that pip assumes is
# the root package.
-MLIR_BINDINGS_PYTHON_INSTALL_PREFIX = "mlir_standalone"
+MLIR_BINDINGS_PYTHON_INSTALL_PREFIX = { env = "MLIR_BINDINGS_PYTHON_INSTALL_PREFIX", default = "mlir_standalone" }
+# Optional
+MLIR_INCLUDE_TESTS = { env = "MLIR_INCLUDE_TESTS", default = "ON" }
diff --git a/mlir/examples/standalone/test/lit.cfg.py b/mlir/examples/standalone/test/lit.cfg.py
index e27dddd7fb0b9..89cdd6889a1f2 100644
--- a/mlir/examples/standalone/test/lit.cfg.py
+++ b/mlir/examples/standalone/test/lit.cfg.py
@@ -61,10 +61,8 @@
llvm_config.add_tool_substitutions(tools, tool_dirs)
-llvm_config.with_environment(
- "PYTHONPATH",
- [
- os.path.join(config.mlir_obj_dir, "python_packages", "standalone"),
- ],
- append_path=True,
-)
+python_path = [os.path.join(config.mlir_obj_dir, "python_packages", "standalone")]
+if "PYTHONPATH" in os.environ:
+ python_path += [os.environ["PYTHONPATH"]]
+
+llvm_config.with_environment("PYTHONPATH", python_path, append_path=True)
diff --git a/mlir/examples/standalone/test/python/smoketest.py b/mlir/examples/standalone/test/python/smoketest.py
index f8819841fac45..09040eb2f45dc 100644
--- a/mlir/examples/standalone/test/python/smoketest.py
+++ b/mlir/examples/standalone/test/python/smoketest.py
@@ -1,16 +1,28 @@
-# RUN: %python %s nanobind | FileCheck %s
+# RUN: %python %s 2>&1 | FileCheck %s
+import sys
-from mlir_standalone.ir import *
+# CHECK: Testing mlir_standalone package
+print("Testing mlir_standalone package", file=sys.stderr)
+
+import mlir_standalone.ir
from mlir_standalone.dialects import standalone_nanobind as standalone_d
-with Context():
+with mlir_standalone.ir.Context():
standalone_d.register_dialects()
- module = Module.parse(
+ standalone_module = mlir_standalone.ir.Module.parse(
"""
%0 = arith.constant 2 : i32
%1 = standalone.foo %0 : i32
"""
)
- # CHECK: %[[C:.*]] = arith.constant 2 : i32
- # CHECK: standalone.foo %[[C]] : i32
- print(str(module))
+ # CHECK: %[[C2:.*]] = arith.constant 2 : i32
+ # CHECK: standalone.foo %[[C2]] : i32
+ print(str(standalone_module), file=sys.stderr)
+
+
+# CHECK: Testing mlir package
+print("Testing mlir package", file=sys.stderr)
+
+from mlir.ir import *
+
+# CHECK-NOT: RuntimeWarning: nanobind: type '{{.*}}' was already registered!
diff --git a/mlir/test/Examples/standalone/test.toy b/mlir/test/Examples/standalone/test.toy
index a88c115ebf197..8836adde72ed8 100644
--- a/mlir/test/Examples/standalone/test.toy
+++ b/mlir/test/Examples/standalone/test.toy
@@ -4,8 +4,11 @@
# RUN: -DLLVM_ENABLE_LIBCXX=%enable_libcxx -DMLIR_DIR=%mlir_cmake_dir \
# RUN: -DLLVM_USE_LINKER=%llvm_use_linker \
# RUN: -DMLIR_PYTHON_PACKAGE_PREFIX=mlir_standalone \
+# RUN: -DMLIR_BINDINGS_PYTHON_NB_DOMAIN=mlir_standalone \
+# RUN: -DMLIR_INCLUDE_TESTS=ON \
# RUN: -DPython3_EXECUTABLE=%python \
# RUN: -DPython_EXECUTABLE=%python
+# RUN: export PYTHONPATH="%mlir_obj_root/python_packages/mlir_core"
# RUN: "%cmake_exe" --build . --target check-standalone | tee %t
# RUN: FileCheck --input-file=%t %s
diff --git a/mlir/test/Examples/standalone/test.wheel.toy b/mlir/test/Examples/standalone/test.wheel.toy
index c8d188a3cacd0..b60347ba687d0 100644
--- a/mlir/test/Examples/standalone/test.wheel.toy
+++ b/mlir/test/Examples/standalone/test.wheel.toy
@@ -14,21 +14,30 @@
# RUN: export CMAKE_GENERATOR=%cmake_generator
# RUN: export LLVM_USE_LINKER=%llvm_use_linker
# RUN: export MLIR_DIR="%mlir_cmake_dir"
+# RUN: export MLIR_INCLUDE_TESTS=ON
+# RUN: export MLIR_PYTHON_PACKAGE_PREFIX=mlir_standalone
+# RUN: export MLIR_BINDINGS_PYTHON_NB_DOMAIN=mlir_standalone
# RUN: %python -m pip wheel "%mlir_src_root/examples/standalone" -w "%mlir_obj_root/wheelhouse" -v | tee %t
# RUN: rm -rf "%mlir_obj_root/standalone-python-bindings-install"
# RUN: %python -m pip install standalone_python_bindings -f "%mlir_obj_root/wheelhouse" --target "%mlir_obj_root/standalone-python-bindings-install" -v | tee -a %t
-# RUN: export PYTHONPATH="%mlir_obj_root/standalone-python-bindings-install"
-# RUN: %python "%mlir_src_root/examples/standalone/test/python/smoketest.py" nanobind | tee -a %t
+# RUN: export PYTHONPATH="%mlir_obj_root/standalone-python-bindings-install:%mlir_obj_root/python_packages/mlir_core"
+# RUN: %python "%mlir_src_root/examples/standalone/test/python/smoketest.py" 2>&1 | tee -a %t
# RUN: FileCheck --input-file=%t %s
# CHECK: Successfully built standalone-python-bindings
+# CHECK: Testing mlir_standalone package
+
# CHECK: module {
# CHECK: %[[C2:.*]] = arith.constant 2 : i32
# CHECK: %[[V0:.*]] = standalone.foo %[[C2]] : i32
# CHECK: }
+# CHECK: Testing mlir package
+
+# CHECK-NOT: RuntimeWarning: nanobind: type '{{.*}}' was already registered!
+
More information about the Mlir-commits
mailing list