<table border="1" cellspacing="0" cellpadding="8">
<tr>
<th>Issue</th>
<td>
<a href=https://github.com/llvm/llvm-project/issues/152459>152459</a>
</td>
</tr>
<tr>
<th>Summary</th>
<td>
Very slow symbolizer invocation in libfuzzer with Go c-archive library
</td>
</tr>
<tr>
<th>Labels</th>
<td>
compiler-rt,
compiler-rt:fuzzer
</td>
</tr>
<tr>
<th>Assignees</th>
<td>
</td>
</tr>
<tr>
<th>Reporter</th>
<td>
chfast
</td>
</tr>
</table>
<pre>
I have a relatively simple fuzzing tool using libfuzzer. It is fuzzing C++ and Go code. The Go code is built to the "c-archive" with `go build -buildmode=c-archive -tags=libfuzzer -gcflags=-d=libfuzzer`. All seems to work correctly, except the moment the libfuzzer needs the function names (e.g. to print a stack trace or to focus a function or to report new functions covered) there is a big minutes long pause.
My investigation lead me to the `llvm-symbolizer` subprocess invocation. The below is libfuzzer output with UBSAN verbosity increased. The second module with 2 PCs seems to be the Go code.
```
export UBSAN_OPTIONS=verbosity=4
fuzz ../corpus -use_value_profile=1 -focus_function=FUNC -runs=0
==145033==Using llvm-symbolizer found at: /usr/bin/llvm-symbolizer-21
==145033==Installed the sigaction for signal 11
==145033==Installed the sigaction for signal 7
==145033==Installed the sigaction for signal 8
INFO: Seed: 32135115
INFO: Loaded 2 modules (11211 inline 8-bit counters): 11209 [0x5f4829450bf2, 0x5f48294537bb), 2 [0x5f4829e18e00, 0x5f4829e18e02),
INFO: Loaded 2 PC tables (11211 PCs): 11209 [0x5f48294537c0,0x5f482947f450), 2 [0xc00001a120,0xc00001a140),
INFO: 224 files found in ../corpus
==145033==Launching Symbolizer process: /usr/bin/llvm-symbolizer-21 --demangle --inlines --default-arch=x86_64
(PAUSE)
ERROR: Failed to set focus function. Make sure the function name is valid (FUNC) and symbolization is enabled.
```
However, I'm not sure the `llvm-symbolizer` is slow. Some perf tracing suggest this is the subprocess management. According to `perf record` mostly of the time is spent on closing file descriptors.
```
94.97% 19.76% fuzz fuzz [.] __sanitizer::internal_close(int)
|
|--89.99%--__sanitizer::internal_close(int)
| |
| |--32.39%--entry_SYSCALL_64_after_hwframe
| | |
| | |--26.76%--do_syscall_64
| | | |
| | | |--12.98%--x64_sys_call
| | | | |
| | | | |--10.60%--__x64_sys_close
| | | | | |
| | | | | |--6.06%--_raw_spin_lock
| | | | | |
| | | | | |--2.44%--file_close_fd
| | | | | | |
| | | | | | --0.52%--file_close_fd_locked
| | | | | |
| | | | | |--0.90%--file_close_fd_locked
| | | | | |
| | | | | --0.58%--_raw_spin_unlock
| | | | |
| | | | --0.53%--file_close_fd
| | | |
| | | |--7.25%--syscall_exit_to_user_mode
| | | | |
| | | | |--3.34%--arch_exit_to_user_mode_prepare.isra.0
| | | | | |
| | | | | --0.66%--fpregs_assert_state_consistent
| | | | |
| | | | |--1.26%--syscall_exit_to_user_mode_prepare
| | | | |
| | | | --0.65%--fpregs_assert_state_consistent
| | | |
| | | |--1.16%--arch_exit_to_user_mode_prepare.isra.0
| | | |
| | | --0.58%--syscall_exit_to_user_mode_prepare
| | |
| | --0.71%--syscall_exit_to_user_mode
| |
| |--28.62%--entry_SYSCALL_64
| |
| |--5.53%--entry_SYSRETQ_unsafe_stack
| |
| |--1.71%--syscall_return_via_sysret
| |
| |--1.36%--entry_SYSCALL_64_safe_stack
| |
| --0.68%--do_syscall_64
|
--4.99%--__sanitizer::StartSubprocess(char const*, char const* const*, char const* const*, int, int, int)
```
The `llvm-symbolizer` run is only 2% if I interpret the perf report correctly.
```
1.97% 0.00% llvm-symbolizer llvm-symbolizer [.] 0x0000564172f90db5
|
---0x564172f90db5
__libc_start_main@@GLIBC_2.34
__libc_start_call_main
0x564172f977c3
|
--1.86%--0x564172f927e6
0x564172f941a6
0x564172f95622
llvm::symbolize::LLVMSymbolizer::symbolizeInlinedCode(llvm::StringRef, llvm::object::SectionedAddress)
llvm::Expected<llvm::DIInliningInfo> llvm::symbolize::LLVMSymbolizer::symbolizeInlinedCodeCommon<llvm::StringRef>(llvm::StringRef const&, llvm::object::SectionedAddress)
|
--1.82%--llvm::symbolize::SymbolizableObjectFile::symbolizeInlinedCode(llvm::object::SectionedAddress, llvm::DILineInfoSpecifier, bool) const
llvm::DWARFContext::getInliningInfoForAddress(llvm::object::SectionedAddress, llvm::DILineInfoSpecifier)
|
--1.61%--llvm::DWARFUnit::getInlinedChainForAddress(unsigned long, llvm::SmallVectorImpl<llvm::DWARFDie>&)
llvm::DWARFUnit::getSubroutineForAddress(unsigned long)
|
|--1.03%--llvm::DWARFUnit::tryExtractDIEsIfNeeded(bool)
| |
| --1.01%--llvm::DWARFUnit::extractDIEsToVector(bool, bool, std::vector<llvm::DWARFDebugInfoEntry, std::allocator<llvm::DWARFDebugInfoEntry> >&) const
|
--0.58%--llvm::DWARFUnit::updateAddressDieMap(llvm::DWARFDie)
llvm::DWARFUnit::updateAddressDieMap(llvm::DWARFDie)
|
--0.51%--llvm::DWARFUnit::updateAddressDieMap(llvm::DWARFDie)
```
</pre>
<img width="1" height="1" alt="" src="http://email.email.llvm.org/o/eJzMWU1v47qS_TXMhpBAUZZsLbJw7PiOgfTHtLvvYFYCJZVsTlOkQFJOcn_9gJS_7XYnaeO9ZwSIRBVZh6dOlUSSGcOXEuAeJQ8omd6xzq6Uvi9XNTP2rlDV6_0cr9gaMMMaBLN8DeIVG960AnDd_fMPl0tslRK4M-5S8MK1gg7x3GJudjYTRB8QfcBMVvgvhUtVQYi_r2B742yLjguLrcJ2BRhRWgZMlyu-BkQpfuZ2hVFKlsrbVTjw_xpVAYqnO1McWLY0KJ7ukOBgWdaibwyqwycoJSEeC4ENQGOc42elf-JSaQ2lFa-ITjC8lNBaj6hRDcj-cj-4BKiMb6s7WVquJJasAYMRHUG4DN2orebSYoaNZeVPbDUrASvtntSq7Axm-759s4ZWaYslPO-eGFyqNWioEM2cO-0ZY7jgS9xw2VkwWCi5xC3rDISIjBEZf3rFXK7BWL5kfngBrMIN7DhOiRDrJjCvTaEE7ynBpitarUowxvVWpe_aB6sAoZ6d4z0BqrNtZ_vw_HhYjD_jNehCGW6d81IDM1D1vQ2USla4UVUnoO9B8deJ2fNfgMe1FUg_C5SSzR8Zw4tnxjvKv3z9Pv_yeYHi6c4liqcDRMYOGw5DRGel0m1ncNAZyNdMdJC3WtVcONFEOPAByLcko3g6-_F5ggPdSScXgp37eOpsBwmJ4_76Ry_1Y-ZwrTpZYWZRPMaIzjqjEZ0VXCI6OzENaHRx3Lk0lgkBlSfB8CXrRVEr7e4kEzj6cM_hRzuOEBnPP8--uHktACr3P6ZRnERRcvDoSbEKKkw34TUYuxSIIhpFmEvBJeBRUHCLS9VJC9ogmrl-UURJhlHyQF6SejCi2SAhRU1d7u1b4mFROHs6wfTQFqIREHJo61voxvYSvK8TbFkh-gzt4X2dXAETD0vnYNcwrAcJOcZSEkJIxCLaG25vB-QcBqUD7NRnNnLh8lCmF0P0xDpZrpziFnuxbRL0TVrDQVBBw-RSAA6CPhbGN9asE9ZXThRPX0ZpnrrkQXT0dfxj8ejQk_Hjt29fvjk_M8a9ThQ2YDeVa5s4If7EfgI2nYbzWujqxZoJXjnGXX65EubeA1uUfW3iBoN0kanCk6RHZPxf6hnWoB2bc0SHDZbK7t1drmPcYCPUc4gXqgHcgq597XVMmm65BONqOTfOzot_X_YaJtkSXLUP8bgsla7615xz5MfR4Bqdk0YZK16xqv0YlvfTNa17UyiJS6F8sXAxxxWYUvPWKm0ulDacDcJsiGiCMcZRFg5Tf-1L2eHvrMH_UPIQomSK89wwya3nIB6jeMxdskkmcgcFEB1xafvIHnUfTg5uyNg1BMEoC7MM0SQIbjLs0KfCtedBENMw7l2CtPo1X_zvYjJ-esrTQc5qCzpfPdeaNbDBeMXV0XSuYnJ-adoTHgSVys2rKZkQfTpc7fvhmV7jIKJhNvJYXtKBA5M7NO-Z8VUuPjhIEEQkTMlGDjtkPvw3gXYjnD3WNCR9OHPNnnPTcpkLVf78g3h-WF9vHzMIaDgYeNiuYPTk5nX1Lg8fpRvfYJjhBAcBCRN6PgdPP7xvKm8FdDN6HPiMvAf8TZR-gzF73kcnku_kVvT_tozsgcWXRX0jwo5uzmM6DGni_W_rOrxwm1uVdwZ07leQ7xzzRtBOcMZh3Ce_-yI7B5m3GlqmIeRGs5D8ZxVdF-W0r7h1q2FpcmYMaJsbyyzkpZKGGwvS3ipZ_2SYIIhCml6XxJbtW-X8h6XRU5u8kdp_TUZFYZS-T6kfpPA9yA5L4I3D-hsYzvEw-rMSc-TswofBKEzpxe_iP_7kTrbleTfyt8fv_5130rAacr9p9RtlXeTSqeSEFA220zJfc-a-HTW8oxhcJCUK4_TyWuEY-jm4q-D3P596ozctC06acBAMfrl4Wlim7WK33kR0VK6Yxi6TLaJjt8o9anjjE78EO_6XnS-lv_9ywaw7vwpXUrxipzXMazzHfqXXauh3PjfLX78Lt9sovbSadb9ov6DFJCTEX5_um501bJey5IUQQpJ0EA1pnZGqSM5pP2kJgoC8nPbIc8GL0qlB27xhXKIBQQPy19P8YZJT9849NfJR9pYnQcV4P_xwWMbnzy9JwQOLwlGv1f0IdAips923DCKWXuh9yX2SUupMHXu9qnYU9rdPT39_2u8YnZjM_S5QNXGFiY72Yyys5nL5DWqnoH2zKv4PSrsxAb-1A9W4qrRXb3aM4_GlhdJCheLJvnE69y65XM5lrVD8-OfAJ6pplDzysocfP16e1zZh0ndP8Je_bch9jPsa_cu5befFCgFfvM-Z34t-U3CugzycznT-xCU4phctlLzm_dZZoZRANNtwcBS06f-Mv80mSlp42XhYgj0M2UzpnavbYbrO7C9ITqMTkj34H5KfIIdqsmJcHiHvpD_xqvxhyTHARcOE-BtKq_S8acWxfJ2HKQevq_RM8ucAFl2hVWe5hGvus_P32nbvLQpJfH2aVr8-vljNSjudP5p5_RmgggrR0SbMv39p998uUUh-Qyjs3XxXPUM7N1tVTbCxVW--7i3O-YOi81p6dK_roy5MCFWyt_SKH_EuCHshn03t4Fvwl9Pq2opZ2ARnyuETa4-0vQv6b8L9znEuY_1NCN7nY_cmvqvu4yqLM3YH99EwGSRJOsySu9U9y2jNGKF1QeLRkA3TtIyHUVYWQGk8GBV3_J4SmpARGZJREkdJOKpikpGMkqxIyyqL0YBAw7gIHYhQ6eUdN6aD-yihgyS7E6wAYfzBMqWlalouQAfaIkoRnZy0xePNiSylKJne6Xv_TVB0S4MGRHBjzd6L5VbA_d-gX_32Pj74ctgfV2IuD84o_VHjXwrvT4kFLzTTr3edFvcra1vjWKQzRGdLblddEZaq2RymbM9UWq18qaMzP0-D6Gwz1fU9_f8AAAD__zR8WUQ">