<table border="1" cellspacing="0" cellpadding="8">
    <tr>
        <th>Issue</th>
        <td>
            <a href=https://github.com/llvm/llvm-project/issues/208182>208182</a>
        </td>
    </tr>

    <tr>
        <th>Summary</th>
        <td>
            [llvm-cov] `llvm-cov report` intermittently crashes with `-num-threads > 1`
        </td>
    </tr>

    <tr>
      <th>Labels</th>
      <td>
      </td>
    </tr>

    <tr>
      <th>Assignees</th>
      <td>
      </td>
    </tr>

    <tr>
      <th>Reporter</th>
      <td>
          LucaCiucci
      </td>
    </tr>
</table>

<pre>
    `llvm-cov-20 report` intermittently crashes when generating a summary from multiple instrumented executables and an indexed profile. The crash only appears when more than one worker thread is used. `-num-threads=1` seems to be a reliable workaround.

## Environment

I have currently tested this behavior on Ubuntu 24.04 with LLVM 20.1.8 and 22.1.8.

## Reproducer

The attached `reproducer` directory contains:

- `Overall.profdata`
- four instrumented test executables with debug information
- `crash.sh`, which repeatedly runs `llvm-cov-20 report`

Run:

```sh
cd reproducer
./crash.sh
```

The minimized command is:

```sh
llvm-cov-22 report \
  ut_Date_Time \
 -object ut_Unit_Conversion \
  -object ut_Isolette \
  -object ut_Room_Air \
  -instr-profile=Overall.profdata \
  -dump \
 -num-threads=8 \
  > /dev/null
```

The crash is nondeterministic. On my machine it usually happens within a few iterations with `-num-threads=$(nproc)`.

## Expected behavior

`llvm-cov report` should either complete successfully or report a normal diagnostic. It should not crash or corrupt heap state.

## Actual behavior

The failure is nondeterministic and the exact symptom varies. I have observed heap corruption diagnostics, segmentation faults, aborts, and in one run the LLVM crash-report message was printed but the process did not exit at all.

A few representative crashes are shown below.
<details>
<summary>Example 1</summary>

```plain
+ llvm-cov-22 report ut_Date_Time -object ut_Unit_Conversion -object ut_Isolette -object ut_Room_Air -instr-profile=Overall.profdata -dump -num-threads=4
double free or corruption (!prev)
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace and instructions to reproduce the bug.
 #0 0x00007588094dd17d llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/home/luca/micromamba/envs/clang-22/bin/../lib/libLLVM.so.22.1+0xcdd17d)
 #1 0x00007588094da120 SignalHandler(int, siginfo_t*, void*) Signals.cpp:0:0
 #2 0x0000758808045330 (/lib/x86_64-linux-gnu/libc.so.6+0x45330)
 #3 0x000075880809eb2c __pthread_kill_implementation ./nptl/pthread_kill.c:44:76
 #4 0x000075880809eb2c __pthread_kill_internal ./nptl/pthread_kill.c:78:10
 #5 0x000075880809eb2c pthread_kill ./nptl/pthread_kill.c:89:10
 #6 0x000075880804527e raise ./signal/../sysdeps/posix/raise.c:27:6
 #7 0x00007588080288ff abort ./stdlib/abort.c:81:7
 #8 0x00007588080297b6 _IO_acquire_lock_fct ./libio/libioP.h:979:6
 #9 0x00007588080297b6 _IO_peekc_locked ./libio/peekc.c:37:3
#10 0x00007588080a8ff5 ./malloc/malloc.c:5775:3
#11 0x00007588080ab12c _int_free_merge_chunk ./malloc/malloc.c:4679:5
#12 0x00007588080ab529 _int_free ./malloc/malloc.c:4648:5
#13 0x00007588080addce cfree ./malloc/malloc.c:3401:3
#14 0x00007588094adae7 llvm::raw_ostream::SetBuffered() (/home/luca/micromamba/envs/clang-22/bin/../lib/libLLVM.so.22.1+0xcadae7)
#15 0x00007588094ae537 llvm::raw_ostream::write(char const*, unsigned long) (/home/luca/micromamba/envs/clang-22/bin/../lib/libLLVM.so.22.1+0xcae537)
#16 0x000058ab3f596306 llvm::CoverageReport::prepareSingleFileReport(llvm::StringRef, llvm::coverage::CoverageMapping const*, llvm::CoverageViewOptions const&, unsigned int, llvm::FileCoverageSummary*, llvm::CoverageFilter const*) (/home/luca/micromamba/envs/clang-22/bin/llvm-cov-22+0x48306)
#17 0x000058ab3f594f1d void llvm::detail::UniqueFunctionBase<void>::CallImpl<std::_Bind<void (* (llvm::StringRef, llvm::coverage::CoverageMapping const*, llvm::CoverageViewOptions, unsigned int, llvm::FileCoverageSummary*, llvm::CoverageFilter const*))(llvm::StringRef, llvm::coverage::CoverageMapping const*, llvm::CoverageViewOptions const&, unsigned int, llvm::FileCoverageSummary*, llvm::CoverageFilter const*)>>(void*) (/home/luca/micromamba/envs/clang-22/bin/llvm-cov-22+0x46f1d)
#18 0x000058ab3f5652e7 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<llvm::unique_function<void ()>>>, void>>::_M_invoke(std::_Any_data const&) (/home/luca/micromamba/envs/clang-22/bin/llvm-cov-22+0x172e7)
#19 0x000058ab3f567249 std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) (/home/luca/micromamba/envs/clang-22/bin/llvm-cov-22+0x19249)
#20 0x00007588080a1ed3 __pthread_once_slow ./nptl/pthread_once.c:118:7
#21 0x000058ab3f56719c std::__future_base::_Deferred_state<std::thread::_Invoker<std::tuple<llvm::unique_function<void ()>>>, void>::_M_complete_async() (/home/luca/micromamba/envs/clang-22/bin/llvm-cov-22+0x1919c)
#22 0x000058ab3f5664ec void llvm::detail::UniqueFunctionBase<void>::CallImpl<std::shared_future<void> llvm::ThreadPoolInterface::asyncImpl<void>(llvm::unique_function<void ()>, llvm::ThreadPoolTaskGroup*)::'lambda'()>(void*) (/home/luca/micromamba/envs/clang-22/bin/llvm-cov-22+0x184ec)
#23 0x000075880944d639 llvm::StdThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) (/home/luca/micromamba/envs/clang-22/bin/../lib/libLLVM.so.22.1+0xc4d639)
#24 0x000075880944e2e7 void* llvm::thread::ThreadProxy<std::tuple<llvm::StdThreadPool::grow(int)::'lambda'()>>(void*) ThreadPool.cpp:0:0
#25 0x000075880809caa4 start_thread ./nptl/pthread_create.c:447:8
#26 0x0000758808129c6c clone3 ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:80:0
./crash.sh: line 20: 214489 Aborted                 llvm-cov-22 report "${overall_objects[@]}" -instr-profile=Overall.profdata -dump "-num-threads=4" > /dev/null
```
</details>

<details>
<summary>Example 2</summary>

```plain
double free or corruption (out)
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace and instructions to reproduce the bug.
malloc(): unaligned tcache chunk detected
./crash.sh: line 20: 217872 Aborted llvm-cov-22 report "${overall_objects[@]}" -instr-profile=Overall.profdata -dump "-num-threads=4" > /dev/null
```
</details>

<details>
<summary>Example 3</summary>

```plain
+ llvm-cov-22 report ut_Date_Time -object ut_Unit_Conversion -object ut_Isolette -object ut_Room_Air -instr-profile=Overall.profdata -dump -num-threads=4
LLVM ERROR: IO failure on output stream: Bad address
```
</details>

<details>
<summary>Example 4</summary>

```plain
corrupted size vs. prev_size
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  libLLVM.so.20.1 0x000078adcac6343f llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 63
1  libLLVM.so.20.1 0x000078adcac61149 llvm::sys::RunSignalHandlers() + 89
2  libLLVM.so.20.1 0x000078adcac63b50
3  libc.so.6       0x000078adc9602330
4  libc.so.6 0x000078adc965bb2c pthread_kill + 284
5  libc.so.6       0x000078adc960227e gsignal + 30
6  libc.so.6       0x000078adc95e58ff abort + 223
7  libc.so.6 0x000078adc95e67b6
8  libc.so.6       0x000078adc9665ff5
9  libc.so.6 0x000078adc9666b96
10 libc.so.6       0x000078adc96680e3
11 libc.so.6 0x000078adc9668529
12 libc.so.6       0x000078adc966adce __libc_free + 126
13 libLLVM.so.20.1 0x000078adcac4256b llvm::raw_ostream::SetBuffered() + 91
14 libLLVM.so.20.1 0x000078adcac432b7 llvm::raw_ostream::write(char const*, unsigned long) + 167
15 llvm-cov-20     0x000058550904c6cf
16 llvm-cov-20     0x000058550904f50a
17 llvm-cov-20     0x000058550903a10f
18 llvm-cov-20     0x000058550903a060
19 libc.so.6       0x000078adc965eed3
20 llvm-cov-20     0x000058550903a54e
21 llvm-cov-20     0x000058550903a5fe
22 libLLVM.so.20.1 0x000078adcac044df llvm::StdThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) + 655
23 libLLVM.so.20.1 0x000078adcac05aa8
24 libc.so.6       0x000078adc9659aa4
25 libc.so.6       0x000078adc96e6a64 __clone + 68
./test_coverage.sh: line 164: 10466 Aborted                 (core dumped) llvm-cov-20 report "${overall_objects[@]}" -instr-profile=Overall.profdata --show-mcdc-summary "${common_opt[@]}" -dump > "$output_txt"
```
</details>

## Workaround

Passing `-num-threads=1` avoids the crash in my testing.

[llvm-cov-reproducer.zip](https://github.com/user-attachments/files/29794220/llvm-cov-reproducer.zip)
</pre>
<img width="1" height="1" alt="" src="http://email.email.llvm.org/o/eJzcOtty47hyXwO_oKwiwfuDH-SLclw1WzNlz24qeWGBYFNCTAI8ACjb-_WpBimJ0tjynMzMJtkt70gCGo2-d6MBbq1cK4ArklyT5PaCD26jzdWnQfAbOQghLypdv16RNGjbbXcp9PaSBdRAr40jaUClcmA66Rwo175SYbjdgKXPG1B0DQoMd1KtKad26DpuXmljdEe7oXWyb4FKZZ0ZOlAOagovIAbHqxYs5aqmXFGpaniBmvZGN7KFBf26gXETqlX7SnnfAzfTfp02QN2GK6oV0GdtnsBQtzHAayotHSzUC0rS4FIN3eU4bkl0GyIfFqCz1GlaAeXUQCuRDo-EGz2oekGCJf6xiLCI3qmtNFoh4eP4Pd3wLVAxGDNKwoFFntxGWlrBhm-lNlQr-ns1KDdQFi-CmD5Lt6GfPv3xG2XBIlzknm3G8Ovxfg_QG10PAsw4jGLgznGxgRpZMof5NKC1NCCcNq9UaOW4VJZEy3HhJUJ_3oLhbbtAqdbccZIGfqrRgzlWCXJxpBdPcQ3VsKZSNdp03Emtdoi9ZhZ2gwjZDX3eSLFBYwHuoG5fqRmUpe_Z0kjgw6D2xOKg_7MbEixFTc1cDAvCVvsNZ8AHCXVSyU7-CTUVuutQttK-g_xAEZsooiS5IcGS0sGVt9xB-VV2sBu81NV_gXA497uSrrzRagvGSq32y2Yg91a34By8NfegdVcupTnMeQVcThZPottTbR0g66Hr9xQdG3W-hyLRHSVsVcOWsJUa2vZtWY1OJS1VWtXgnVpJ66RY0M-Kdq-042IjFVDp6GAH3ravdIPup0ajkIpy2sAzlc47vZ7G33A4wmLCctUbLQgrSBqc-NZLDwKNb-c2e33tlDSLP3ajh7amIN0GDKq5b8EBtYMQYG0zIJna7FTKqUKTbWkt-Vrpkb17t8OitNsFF8RlzNA7ugHeU-u4g2Myl8INvD0hEiXZcNkOBt6SpfdvtwEKL1w4al-73umObrmRYBd0CiK6smC2UI9bT3SgaR2otuheFtbopl7YtOFD6_wwr7SZvqHFj9HQDMpv7ION5_FykkkH1vI10GduaW-kd_tqcB4aVQTW0lqOwoEX6Sh3FM1xZHjpdY5-CXakZQv7LMANoGifFa2g1c9-SXRTg-OytSS6G39PmYFEd3cvHPVHQxLdELY6TJx4bN9yqbwmrukbjnvksGc89S0PfcszP_bI0RNPzDwmwbLWA6aRxgDMbMqHCZYTFvYG3bIgwfLLp7vl4x21Q9WhjClG2Ikfp-nGud6HLrYibLWWbjNUC6E7wlYogOkDSUTqCVtJawewhK0mIxDtUIPX6WjgFRdPznAB0zzGfDF6rdOHOOtXVMMaVUcJiwIavARBEGRJngdFXNdhVnsVIG3R0r7a8csXNKRHx8XTV9yEsPwAZPhzqa0zwDvCUrRTqRxhxSiS1UZ3gPwMghO26qQwuuNdhT9AbZEl0XK1vmSMsFUlFWGrBaaCVlbjv2jiC6sXmEcJuw5ehCdzFDMyEZ4wwUMW0Ee5Vrz9B1d1C4aw3NN0Q61cY54rHWFL_L3VsvZfi2mFXYi-J9Ey8P-PG7D5BnkQJ1EUTNyNVL7kaZnGl61Uw8vlWg3jhECqU0-yX3KgODpGWEDFBC3LfjS28km2bSnRdQ7xAEWietcStpqDLQSJlnFMomWWTsjj70GOZZ7i7Vm0WU6iZbgTQvIW2vmas7jyYo4rPRUoy4AaLi14JNZrYmcI9tXW0KOh9NrKF8JWHtKjZRmJljvGs2OsLM-bZoyeI1ZXj9ryQyNVIbI5Lc9PlhdZldLy_nPJxT8HaaBstXgqGzFia2Ul9e7zy2JDomWRFTNqivfQ9QBPwiOD-giVn_B0RchWNOamMDhGxPOmSfy6jrctptzpi1-ZZFkyWxuerK1CtAWpXIkhrOzArKEUm0E9vYsxTj1byYSRnWJMWHHAeAZLnM-wnDgAr2sBVJzFEMVBOOMsPnZ6XnPI6NtByQ88grsemgYM1N5zf2F48rSMzo6UJieUQhKdpfTZSIcRVmw4Zhhld7FqUP5sV9NWq_Wv5QBpPHCwc9ck51XUJEUaBemMgxuN-XMND2MV58d6Az038CjVuoWVbHeT88Tx6IxU6wdokLnDsJjQHSP_jfc9njznAvmWhD8kPH_ux7w3QaZHopvSwGEl0rZb_TgVKO9hX8nWwVwnP6CCWZ0zZog8CtKDyLMTkcdNWPtkNSNrLLzG778r-c8BVoPySf-aWyDRjU9u0d3EBG_b-65vsUJz9ThWXktVT4AjKxi6_jIV_UrN-L-_l7F5Vd4Rls-Klp9nf2kT1gf7y4_tL00YZPRgNztDKzdTgTUzqsGbYtm7o9GybAY3GCgrb5t-6AHs0LpxBGuz74UdR24Bj4ZmPA_nXt8f4PjK7VNp8VzwQwTPPOtHqD5aPNZLE8y92uqnYxrd0ONR5eZgJxPVzaSJIy_emcq4yT4MTJGg_K2UfgfC8gP1S_Va-sPPwZR_onmFGZvnxOLEvDIWF-cl-Ygndj_wB9uzUWtU55yNmTj-Dxgkyt-7d6V1-7M9NixYXOxFyk7LxBDqaFb2ayWgtK1-fqtMx0lfZIVhPlXEiDI81VJYiPNCuYUGjIG69A2WIwP-Kwx8Motd26jk9lWJHy33vpV7WIiD3NmJkNIYxM_P1HbDUayjxA8LZnt89QL-onV7j4e7Bo_pfsILYcK3j1v5vyDno_R02Aaj6b8ZPfS77LT0DY2s5V1Vc8KyGYZfkrLCPIaZIqLjOjuu06ig8wKgPtC-K1J9OwwZsUcieZfJX1dye3IPzJwcb2LA_DsJccbU3Ksmoo1-eT3vWG9IYm30865JclaTp8o8IDrpmyATpy0DwXlMrePGldM1zhuxSBjgDqa-Bp6D8wnbcdMgZIVIBRWtVhCNh0Zpxbc9g0H5loF9tVsvejW87Bs2Xk-4fvGI--xIP7qIiJa0lQoow1nKwjjOC7qstHFQ09P_3rp1QBuISXatxyZjOTYkLUmuSRyQ5JZkt4Sx725KEsa-6Usy9vGtgO_AHrdqv7N7y76ze3u2N6oH9_-lL7prPowmv6SD4u1YzzvBBWL2_ZIanL_W-Mhgsjxje4P5WxpI9Ldp7_vblLuHh88PqLv7z_u7H62oHlw_OLrv1NBrXlNe1was_XmyjL9TlpN3QU2t_BPo1i5ob2Bb4q__RS9D__HXBNSL-Fm6jR78xVilW6p4BxajASiLUn3Vw3hD5rcbgeSfYKhUOGfol-XXf2AssTDeX8Hhmp5uuaEkDVBl5eN__Hb9-dP9f949lLiEpAGy2WupPL9yl9WCZUDpPP0Gi32HNOe14CKN4qj5iTcg1zSNSLAMP9o2DOPirW0fBnV0k2H31ew1zTGesg8ZqhK0ycjDjVcSU7aawRVpwKII4eI53BFEUn3T8UcqWI6ek3yInmVA12Nn36_zu6XnlyWQHHr4fjOG0szeozGBNKtSEizzD8hJk6ZJSLAs3uU2TasCMYXBB5jyALyGw3cx5YnPfCH7ABOvBdCyRKCxmY4ch8xTEZ3XcsyStPpX29_XtAgRd_wB7ohVP61hfU3DFI-XYULnj0YOwkjyJAmKIBapaBAu_QCuSQKOcNl5uIiHgceXfwQX-BAeFueVlQDUqHYWfIQviTEgs_BDuMbDsfPKCOK4bn7Bqeaapgk6BPvA0IKEcyzIWfyBfArOMS6w5DwcpDyNaVn6OnykI58qKgfWlbv-7LyyCtMYv4dBnKbvluJoj9qAz0No8wX99pHSz6y9Lu1GP192ohaXu6dxe-xCd51Wpe7dN4jHks3XZgg8Fhmle3H4-7uqivH1yr_vH7aNw1-4tVKt33sex_EIZ2cZXPo3QShyqda7hzHJ9V5mh5daiz9ljwyw_EwZMVgwl-ODNkzWWDqg5PCTFVkRMxbMD_Qn2FlxUV9FdREV_AKuwiyPkiCKo-xic5WGLGiCqmJBHEIcAGRJJaI8CKNEhHGWXMgrFrA0yPC8GWVhthAZb4qM1XmQQVbkIYkD6LhsF7j7Qpv1ha9urliQhzm7aHkFrd09nzRXnsZqWFsSB620zh7WOela_9ByxwdJbunbz5ree1b5xoMqbw2opIvBtFf_81Jt4md7xf47AAD__-TN87o">