[Mlir-commits] [mlir] [MLIR, Python] Support converting boolean numpy arrays to and from mlir attributes (PR #113064)
Dmitri Gribenko
llvmlistbot at llvm.org
Tue Nov 5 06:47:26 PST 2024
gribozavr wrote:
I'm afraid there is an actual ASan issue here. While the above one is a flake, I am getting an ASan failure that is real.
<details>
ASan report
```
==8566==ERROR: AddressSanitizer: heap-use-after-free on address 0x7b47396a8dd0 at pc 0x5647f3be84fc bp 0x7fff90c4d900 sp 0x7fff90c4d8f8
READ of size 1 at 0x7b47396a8dd0 thread T0
#0 0x5647f3be84fb in _aligned_strided_to_contig_size1_srcstride0 <redacted>/numpy/_core/src/multiarray/lowlevel_strided_loops.c.src:227:17
#1 0x5647f3a9dd61 in raw_array_assign_array <redacted>/numpy/_core/src/multiarray/array_assign_array.c:150:13
#2 0x5647f3a9f4a2 in PyArray_AssignArray <redacted>/numpy/_core/src/multiarray/array_assign_array.c:444:13
#3 0x5647f3acf4ce in PyArray_CopyInto <redacted>/numpy/_core/src/multiarray/ctors.c:2895:12
#4 0x5647f3acf4ce in PyArray_FromArray <redacted>/numpy/_core/src/multiarray/ctors.c:1981:23
#5 0x5647f3acee59 in PyArray_FromAny_int <redacted>/numpy/_core/src/multiarray/ctors.c:1620:25
#6 0x5647f3acfa05 in PyArray_CheckFromAny_int <redacted>/numpy/_core/src/multiarray/ctors.c:1848:11
#7 0x5647f3b49b6a in _array_fromobject_generic <redacted>/numpy/_core/src/multiarray/multiarraymodule.c:1681:28
#8 0x5647f3b3fec4 in array_array <redacted>/numpy/_core/src/multiarray/multiarraymodule.c:1747:21
#9 0x5647f53af212 in cfunction_vectorcall_FASTCALL_KEYWORDS python_runtime/<redacted>/v3_11/Objects/methodobject.c:443:24
#10 0x5647f535cbf5 in _PyObject_VectorcallTstate python_runtime/<redacted>/v3_11/Include/internal/pycore_call.h:92:11
#11 0x5647f535cbf5 in PyObject_Vectorcall python_runtime/<redacted>/v3_11/Objects/call.c:299:12
#12 0x5647f54a1662 in _PyEval_EvalFrameDefault python_runtime/<redacted>/v3_11/Python/ceval.c
#13 0x5647f549871f in _PyEval_EvalFrame python_runtime/<redacted>/v3_11/Include/internal/pycore_ceval.h:94:16
#14 0x5647f549871f in _PyEval_Vector python_runtime/<redacted>/v3_11/Python/ceval.c:6442:24
#15 0x5647f54985ce in PyEval_EvalCode python_runtime/<redacted>/v3_11/Python/ceval.c:1149:21
#16 0x5647f5500d0f in run_eval_code_obj python_runtime/<redacted>/v3_11/Python/pythonrun.c:1741:9
#17 0x5647f54ff126 in run_mod python_runtime/<redacted>/v3_11/Python/pythonrun.c:1762:19
#18 0x5647f54fd45c in pyrun_file python_runtime/<redacted>/v3_11/Python/pythonrun.c:1657:15
#19 0x5647f54fd45c in _PyRun_SimpleFileObject python_runtime/<redacted>/v3_11/Python/pythonrun.c:440:13
#20 0x5647f54fcf1d in _PyRun_AnyFileObject python_runtime/<redacted>/v3_11/Python/pythonrun.c:79:15
#21 0x5647f533c7d1 in pymain_run_file_obj python_runtime/<redacted>/v3_11/Modules/main.c:360:15
#22 0x5647f533c7d1 in pymain_run_file python_runtime/<redacted>/v3_11/Modules/main.c:379:15
#23 0x5647f533bf3d in pymain_run_python python_runtime/<redacted>/v3_11/Modules/main.c:601:21
#24 0x5647f533bf3d in Py_RunMain python_runtime/<redacted>/v3_11/Modules/main.c:680:5
#25 0x5647f533c1f1 in pymain_main python_runtime/<redacted>/v3_11/Modules/main.c:710:12
#26 0x5647f533c24a in Py_BytesMain python_runtime/<redacted>/v3_11/Modules/main.c:734:12
#27 0x5647e919845e in main devtools/python/launcher/python_interpreter.cc:67:10
#28 0x7f27397923d3 in __libc_start_main (/usr/grte/v5/lib64/libc.so.6+0x613d3) (BuildId: 9a996398ce14a94560b0c642eb4f6e94)
#29 0x5647e90be029 in _start /usr/grte/v5/debug-src/src/csu/../sysdeps/x86_64/start.S:120
0x7b47396a8dd0 is located 0 bytes inside of 8-byte region [0x7b47396a8dd0,0x7b47396a8dd8)
freed by thread T0 here:
#0 0x5647e915dd06 in free <redacted>/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:51:3
#1 0x5647f3a9c89b in _npy_free_cache <redacted>/numpy/_core/src/multiarray/alloc.c:145:5
#2 0x5647f3a9c89b in default_free <redacted>/numpy/_core/src/multiarray/alloc.c:331:5
#3 0x5647f3a9c9f4 in PyDataMem_UserFREE <redacted>/numpy/_core/src/multiarray/alloc.c:395:5
#4 0x5647f3aac5ac in array_dealloc <redacted>/numpy/_core/src/multiarray/arrayobject.c:444:13
#5 0x5647f53b5ca4 in _Py_Dealloc python_runtime/<redacted>/v3_11/Objects/object.c:2390:5
#6 0x5647eb2ee09f in Py_DECREF python_runtime/<redacted>/v3_11/Include/object.h:538:9
#7 0x5647eb2ee09f in Py_XDECREF python_runtime/<redacted>/v3_11/Include/object.h:602:9
#8 0x5647eb2ee09f in dec_ref pybind11/<redacted>/include/pybind11/pytypes.h:284:9
#9 0x5647eb2ee09f in ~object pybind11/<redacted>/include/pybind11/pytypes.h:380:17
#10 0x5647eb2ee09f in (anonymous namespace)::PyDenseElementsAttribute::getBooleanBufferFromBitpackedAttribute() <redacted>/llvm-project/mlir/lib/Bindings/Python/IRAttributes.cpp:1095:3
#11 0x5647eb2e09e8 in (anonymous namespace)::PyDenseElementsAttribute::accessBuffer() <redacted>/llvm-project/mlir/lib/Bindings/Python/IRAttributes.cpp:880:14
#12 0x5647eb2e8d43 in operator() pybind11/<redacted>/include/pybind11/pybind11.h:1975:54
#13 0x5647eb2e8d43 in operator() pybind11/<redacted>/include/pybind11/pybind11.h:1962:40
#14 0x5647eb2e8d43 in pybind11::class_<(anonymous namespace)::PyDenseElementsAttribute, mlir::python::PyAttribute>& pybind11::class_<(anonymous namespace)::PyDenseElementsAttribute, mlir::python::PyAttribute>::def_buffer<pybind11::class_<(anonymous namespace)::PyDenseElementsAttribute, mlir::python::PyAttribute>& pybind11::class_<(anonymous namespace)::PyDenseElementsAttribute, mlir::python::PyAttribute>::def_buffer<pybind11::buffer_info, (anonymous namespace)::PyDenseElementsAttribute>(pybind11::buffer_info ((anonymous namespace)::PyDenseElementsAttribute::*)())::'lambda'((anonymous namespace)::PyDenseElementsAttribute&)>(pybind11::buffer_info&&)::'lambda'(_object*, void*)::__invoke(_object*, void*) pybind11/<redacted>/include/pybind11/pybind11.h:1957:13
#15 0x5647eb227b6e in pybind11_getbuffer pybind11/<redacted>/include/pybind11/detail/class.h:592:16
#16 0x5647f533dfb7 in PyObject_GetBuffer python_runtime/<redacted>/v3_11/Objects/abstract.c:390:15
#17 0x5647f53a9733 in _PyManagedBuffer_FromObject python_runtime/<redacted>/v3_11/Objects/memoryobject.c:96:9
#18 0x5647f53a95d2 in PyMemoryView_FromObject python_runtime/<redacted>/v3_11/Objects/memoryobject.c:797:42
#19 0x5647f3accd58 in _array_from_array_like <redacted>/numpy/_core/src/multiarray/ctors.c:1443:32
#20 0x5647f3aa2951 in PyArray_DiscoverDTypeAndShape_Recursive <redacted>/numpy/_core/src/multiarray/array_coercion.c:1041:32
#21 0x5647f3aa3585 in PyArray_DiscoverDTypeAndShape <redacted>/numpy/_core/src/multiarray/array_coercion.c:1303:16
#22 0x5647f3acebf5 in PyArray_FromAny_int <redacted>/numpy/_core/src/multiarray/ctors.c:1578:12
#23 0x5647f3acfa05 in PyArray_CheckFromAny_int <redacted>/numpy/_core/src/multiarray/ctors.c:1848:11
#24 0x5647f3b49b6a in _array_fromobject_generic <redacted>/numpy/_core/src/multiarray/multiarraymodule.c:1681:28
#25 0x5647f3b3fec4 in array_array <redacted>/numpy/_core/src/multiarray/multiarraymodule.c:1747:21
#26 0x5647f53af212 in cfunction_vectorcall_FASTCALL_KEYWORDS python_runtime/<redacted>/v3_11/Objects/methodobject.c:443:24
#27 0x5647f535cbf5 in _PyObject_VectorcallTstate python_runtime/<redacted>/v3_11/Include/internal/pycore_call.h:92:11
#28 0x5647f535cbf5 in PyObject_Vectorcall python_runtime/<redacted>/v3_11/Objects/call.c:299:12
#29 0x5647f54a1662 in _PyEval_EvalFrameDefault python_runtime/<redacted>/v3_11/Python/ceval.c
#30 0x5647f549871f in _PyEval_EvalFrame python_runtime/<redacted>/v3_11/Include/internal/pycore_ceval.h:94:16
#31 0x5647f549871f in _PyEval_Vector python_runtime/<redacted>/v3_11/Python/ceval.c:6442:24
#32 0x5647f54985ce in PyEval_EvalCode python_runtime/<redacted>/v3_11/Python/ceval.c:1149:21
#33 0x5647f5500d0f in run_eval_code_obj python_runtime/<redacted>/v3_11/Python/pythonrun.c:1741:9
#34 0x5647f54ff126 in run_mod python_runtime/<redacted>/v3_11/Python/pythonrun.c:1762:19
#35 0x5647f54fd45c in pyrun_file python_runtime/<redacted>/v3_11/Python/pythonrun.c:1657:15
#36 0x5647f54fd45c in _PyRun_SimpleFileObject python_runtime/<redacted>/v3_11/Python/pythonrun.c:440:13
#37 0x5647f54fcf1d in _PyRun_AnyFileObject python_runtime/<redacted>/v3_11/Python/pythonrun.c:79:15
#38 0x5647f533c7d1 in pymain_run_file_obj python_runtime/<redacted>/v3_11/Modules/main.c:360:15
#39 0x5647f533c7d1 in pymain_run_file python_runtime/<redacted>/v3_11/Modules/main.c:379:15
#40 0x5647f533bf3d in pymain_run_python python_runtime/<redacted>/v3_11/Modules/main.c:601:21
#41 0x5647f533bf3d in Py_RunMain python_runtime/<redacted>/v3_11/Modules/main.c:680:5
previously allocated by thread T0 here:
#0 0x5647e915dfa4 in malloc <redacted>/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:67:3
#1 0x5647f3a9c780 in _npy_alloc_cache <redacted>/numpy/_core/src/multiarray/alloc.c:106:9
#2 0x5647f3a9c780 in default_malloc <redacted>/numpy/_core/src/multiarray/alloc.c:291:12
#3 0x5647f3a9c8ee in PyDataMem_UserNEW <redacted>/numpy/_core/src/multiarray/alloc.c:364:14
#4 0x5647f3acb59f in PyArray_NewFromDescr_int <redacted>/numpy/_core/src/multiarray/ctors.c:879:20
#5 0x5647f3acbb3d in PyArray_NewFromDescrAndBase <redacted>/numpy/_core/src/multiarray/ctors.c:1037:12
#6 0x5647f3acbb3d in PyArray_NewFromDescr <redacted>/numpy/_core/src/multiarray/ctors.c:1022:12
#7 0x5647f3abb3e8 in unpack_bits <redacted>/numpy/_core/src/multiarray/compiled_base.c:1866:28
#8 0x5647f3abb3e8 in io_unpack <redacted>/numpy/_core/src/multiarray/compiled_base.c:2042:12
#9 0x5647f53afa6a in cfunction_call python_runtime/<redacted>/v3_11/Objects/methodobject.c:542:18
#10 0x5647f535c5ca in _PyObject_MakeTpCall python_runtime/<redacted>/v3_11/Objects/call.c:214:18
#11 0x5647f535cc1c in _PyObject_VectorcallTstate python_runtime/<redacted>/v3_11/Include/internal/pycore_call.h:90:16
#12 0x5647f535cc1c in PyObject_Vectorcall python_runtime/<redacted>/v3_11/Objects/call.c:299:12
#13 0x5647f3aa970b in dispatcher_vectorcall <redacted>/numpy/_core/src/multiarray/arrayfunction_override.c:567:18
#14 0x5647f535cd64 in _PyVectorcall_Call python_runtime/<redacted>/v3_11/Objects/call.c:257:24
#15 0x5647f535cd64 in _PyObject_Call python_runtime/<redacted>/v3_11/Objects/call.c:328:16
#16 0x5647f535ce8f in PyObject_Call python_runtime/<redacted>/v3_11/Objects/call.c:355:12
#17 0x5647eb2e307e in call pybind11/<redacted>/include/pybind11/cast.h:1974:28
#18 0x5647eb2e307e in pybind11::object pybind11::detail::object_api<pybind11::handle>::operator()<(pybind11::return_value_policy)1, pybind11::array_t<unsigned char, 16>&, pybind11::arg_v>(pybind11::array_t<unsigned char, 16>&, pybind11::arg_v&&) const pybind11/<redacted>/include/pybind11/cast.h:2110:75
#19 0x5647eb2ed942 in (anonymous namespace)::PyDenseElementsAttribute::getBooleanBufferFromBitpackedAttribute() <redacted>/llvm-project/mlir/lib/Bindings/Python/IRAttributes.cpp:1089:9
#20 0x5647eb2e09e8 in (anonymous namespace)::PyDenseElementsAttribute::accessBuffer() <redacted>/llvm-project/mlir/lib/Bindings/Python/IRAttributes.cpp:880:14
#21 0x5647eb2e8d43 in operator() pybind11/<redacted>/include/pybind11/pybind11.h:1975:54
#22 0x5647eb2e8d43 in operator() pybind11/<redacted>/include/pybind11/pybind11.h:1962:40
#23 0x5647eb2e8d43 in pybind11::class_<(anonymous namespace)::PyDenseElementsAttribute, mlir::python::PyAttribute>& pybind11::class_<(anonymous namespace)::PyDenseElementsAttribute, mlir::python::PyAttribute>::def_buffer<pybind11::class_<(anonymous namespace)::PyDenseElementsAttribute, mlir::python::PyAttribute>& pybind11::class_<(anonymous namespace)::PyDenseElementsAttribute, mlir::python::PyAttribute>::def_buffer<pybind11::buffer_info, (anonymous namespace)::PyDenseElementsAttribute>(pybind11::buffer_info ((anonymous namespace)::PyDenseElementsAttribute::*)())::'lambda'((anonymous namespace)::PyDenseElementsAttribute&)>(pybind11::buffer_info&&)::'lambda'(_object*, void*)::__invoke(_object*, void*) pybind11/<redacted>/include/pybind11/pybind11.h:1957:13
#24 0x5647eb227b6e in pybind11_getbuffer pybind11/<redacted>/include/pybind11/detail/class.h:592:16
#25 0x5647f533dfb7 in PyObject_GetBuffer python_runtime/<redacted>/v3_11/Objects/abstract.c:390:15
#26 0x5647f53a9733 in _PyManagedBuffer_FromObject python_runtime/<redacted>/v3_11/Objects/memoryobject.c:96:9
#27 0x5647f53a95d2 in PyMemoryView_FromObject python_runtime/<redacted>/v3_11/Objects/memoryobject.c:797:42
#28 0x5647f3accd58 in _array_from_array_like <redacted>/numpy/_core/src/multiarray/ctors.c:1443:32
#29 0x5647f3aa2951 in PyArray_DiscoverDTypeAndShape_Recursive <redacted>/numpy/_core/src/multiarray/array_coercion.c:1041:32
#30 0x5647f3aa3585 in PyArray_DiscoverDTypeAndShape <redacted>/numpy/_core/src/multiarray/array_coercion.c:1303:16
#31 0x5647f3acebf5 in PyArray_FromAny_int <redacted>/numpy/_core/src/multiarray/ctors.c:1578:12
#32 0x5647f3acfa05 in PyArray_CheckFromAny_int <redacted>/numpy/_core/src/multiarray/ctors.c:1848:11
#33 0x5647f3b49b6a in _array_fromobject_generic <redacted>/numpy/_core/src/multiarray/multiarraymodule.c:1681:28
#34 0x5647f3b3fec4 in array_array <redacted>/numpy/_core/src/multiarray/multiarraymodule.c:1747:21
#35 0x5647f53af212 in cfunction_vectorcall_FASTCALL_KEYWORDS python_runtime/<redacted>/v3_11/Objects/methodobject.c:443:24
#36 0x5647f535cbf5 in _PyObject_VectorcallTstate python_runtime/<redacted>/v3_11/Include/internal/pycore_call.h:92:11
#37 0x5647f535cbf5 in PyObject_Vectorcall python_runtime/<redacted>/v3_11/Objects/call.c:299:12
#38 0x5647f54a1662 in _PyEval_EvalFrameDefault python_runtime/<redacted>/v3_11/Python/ceval.c
SUMMARY: AddressSanitizer: heap-use-after-free <redacted>/numpy/_core/src/multiarray/lowlevel_strided_loops.c.src:227:17 in _aligned_strided_to_contig_size1_srcstride0
Shadow bytes around the buggy address:
0x7b47396a8b00: fa fa fd fa fa fa 00 04 fa fa fd fd fa fa fd fd
0x7b47396a8b80: fa fa fd fa fa fa fd fa fa fa fd fd fa fa fd fa
0x7b47396a8c00: fa fa fd fd fa fa 00 04 fa fa fd fa fa fa fd fa
0x7b47396a8c80: fa fa fd fd fa fa fd fa fa fa 00 04 fa fa fd fd
0x7b47396a8d00: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa
=>0x7b47396a8d80: fa fa fd fa fa fa fd fa fa fa[fd]fd fa fa fd fa
0x7b47396a8e00: fa fa fd fd fa fa 00 04 fa fa 00 fa fa fa 00 fa
0x7b47396a8e80: fa fa 00 00 fa fa 00 00 fa fa 01 fa fa fa 00 fa
0x7b47396a8f00: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
0x7b47396a8f80: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
0x7b47396a9000: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==8566==ABORTING
FileCheck error: '<stdin>' is empty.
```
</details>
IIUC what's happening here, there is a bug in `getBooleanBufferFromBitpackedAttribute`. It allocates a buffer on line 1089 as a local variable:
```c++
py::object unpacked_booleans =
unpackbits_func(packedArray, "bitorder"_a = "little");
```
then extracts a pointer into it:
```c++
py::buffer_info pythonBuffer =
unpacked_booleans.cast<py::buffer>().request();
```
finally, the pointer is returned from the function, but the buffer is then immediately freed as the function runs the destructors for local variables at `}`:
```c++
return bufferInfo<bool>(shapedType, (bool *)pythonBuffer.ptr, "?");
}
```
I'm going to revert the patch now.
https://github.com/llvm/llvm-project/pull/113064
More information about the Mlir-commits
mailing list