[Mlir-commits] [mlir] Added free-threading CPython mode support in MLIR Python bindings (PR #107103)

[Peter Hawkins]

================
@@ -0,0 +1,526 @@
+# RUN: %PYTHON %s
+"""
+This script generates multi-threaded tests to check free-threading mode using CPython compiled with TSAN.
+Tests can be run using pytest:
+```bash
+python3.13t -mpytest -vvv multithreaded_tests.py
+```
+
+IMPORTANT. Running tests are not checking the correctness, but just the execution of the tests in multi-threaded context
+and passing if no warnings reported by TSAN and failing otherwise.
+
+
+Details on the generated tests and execution:
+1) Multi-threaded execution: all generated tests are executed independently by
+a pool of threads, running each test multiple times, see @multi_threaded for details
+
+2) Tests generation: we use existing tests: test/python/ir/*.py,
+test/python/dialects/*.py, etc to generate multi-threaded tests.
+In details, we perform the following:
+a) we define a list of source tests to be used to generate multi-threaded tests, see `TEST_MODULES`.
+b) we define `TestAllMultiThreaded` class and add existing tests to the class. See `add_existing_tests` method.
+c) for each test file, we copy and modify it: test/python/ir/affine_expr.py -> /tmp/ir/affine_expr.py.
+In order to import the test file as python module, we remove all executing functions, like
+`@run` or `run(testMethod)`. See `copy_and_update` and `add_existing_tests` methods for details.
+
+
+Observed warnings reported by TSAN.
+
+CPython and free-threading known data-races:
+1) ctypes, libffi, dlmalloc
+```
+WARNING: ThreadSanitizer: data race (pid=59410)
+  Read of size 8 at 0x7ffff618b6d0 by thread T2:
+    #0 dlmalloc /project/libffi/x86_64-pc-linux-gnu/../src/dlmalloc.c:4158:8 (libffi.so.8+0x58a5) (BuildId: 7b5c105b8eca23b7c54bb5eeab8b4ecf28fc7756)
+    #1 ffi_closure_alloc /project/libffi/x86_64-pc-linux-gnu/../src/closures.c:998:9 (libffi.so.8+0x56ec) (BuildId: 7b5c105b8eca23b7c54bb5eeab8b4ecf28fc7756)
+    #2 _ctypes_alloc_callback /project/cpython/./Modules/_ctypes/callbacks.c:367:20 (_ctypes.cpython-313t-x86_64-linux-gnu.so+0x1b435) (BuildId: e9fba41204b37973fe60945270803301c7df9615)
+    #3 PyCFuncPtr_new /project/cpython/./Modules/_ctypes/_ctypes.c:3955:13 (_ctypes.cpython-313t-x86_64-linux-gnu.so+0x18ba4) (BuildId: e9fba41204b37973fe60945270803301c7df9615)
+```
+
+2) LLVM related data-races, llvm::raw_ostream is not thread-safe
----------------
hawkinsp wrote:

Well, we should *not* observe these, right? The test should only test things that don't exhibit them?

(This is a great reason not to run those test cases!)

https://github.com/llvm/llvm-project/pull/107103