[Mlir-commits] [mlir] [MLIR][ExecutionEngine] Introduce shared library (PR #87067)

[Maksim Levental]

makslevental wrote:

I recently started building my mlir wheels with `-DLLVM_BUILD_LLVM_DYLIB=ON` and today I had a downstream python bindings test fail (on Mac):

```
Could not load /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/mlir/_mlir_libs/libmlir_async_runtime.dylib:
tests/test_async.py 
  dlopen(/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/mlir/_mlir_libs/libmlir_async_runtime.dylib, 0x0009): Library not loaded: '@loader_path/libLLVM.dylib'
  Referenced from: '/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/mlir/_mlir_libs/libmlir_async_runtime.dylib'
  Reason: tried: '/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/mlir/_mlir_libs/libLLVM.dylib' (no such file), '/usr/local/lib/libLLVM.dylib' (no such file), '/usr/lib/libLLVM.dylib' (no such file) 
```

Sanity checking 

```shell
(numba-mlir) mlevental at maksims-mbp llvm-project % otool -L mlir/_mlir_libs/libmlir_async_runtime.dylib
mlir/_mlir_libs/libmlir_async_runtime.dylib:
        @rpath/libmlir_async_runtime.dylib (compatibility version 0.0.0, current version 0.0.0)
        @loader_path/libLLVM.dylib (compatibility version 1.0.0, current version 19.0.0)
        /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1300.36.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.0.0)
```

so that's unfortunate. I'm guessing if I do a `ldd libmlir_async_runtime.so` on a linux box I'll see the same thing and the runtime paths are being handled more permissively (`auditwheel`/`delocate-wheel` does inject `libLLVM.dylib` into the package but maybe Mac doesn't consider where it's injected as part of the `@loader_path` :person_shrugging:). I'll double check that tomorrow.

Went and checked the CMake and found this PR. Note sure how `MLIRExecutionEngineShared` connects to `libmlir_async_runtime`. Anyway not sure what we should do - for the moment I'm going to patch out that target in my builds so it's not a big deal for me personally but nonetheless it seems like a bug. 

Btw futzing with the paths (eg by moving `libLLVM.dylib` to a place where Mac runtime linker can find it) is a no-go because of some `cl::opt` collision:

```
test_async.py::test_simple_parfor 
Skipped: only check latest
LLVM ERROR: Option 'basic' already exists!
Fatal Python error: Aborted
```





https://github.com/llvm/llvm-project/pull/87067