<table border="1" cellspacing="0" cellpadding="8">
    <tr>
        <th>Issue</th>
        <td>
            <a href=https://github.com/llvm/llvm-project/issues/72006>72006</a>
        </td>
    </tr>

    <tr>
        <th>Summary</th>
        <td>
            [bug] clang incorrectly uses coroutine frame for scratch space after suspending (take 3)
        </td>
    </tr>

    <tr>
      <th>Labels</th>
      <td>
            clang
      </td>
    </tr>

    <tr>
      <th>Assignees</th>
      <td>
      </td>
    </tr>

    <tr>
      <th>Reporter</th>
      <td>
          jacobsa
      </td>
    </tr>
</table>

<pre>
    This issue is related to #65018 and #65054, where clang incorrectly uses a coroutine frame for scratch space after the coroutine suspended. To recap: **clang must not insert code that accesses the coroutine frame after it suspends, because this may introduce a data race**. Another thread may already have resumed or destroyed the coroutine by the time this thread gets around to accessing its frame.

The immediate symptoms in those two issues were fixed by @ChuanqiXu9, by manipulating whether clang will inline `std::coroutine_handle::address`. But as far as I'm aware there was no systematic fix to teach clang that it's not allowed to access the frame after suspending. I think the new issue here is a consequence of this.

---

With a large piece of internal Google code we've found that AddressSanitizer sometimes reports an incorrect use-after-free error in coroutine code, claiming that the frame is being written to after it was already destroyed by another thread that ran concurrently after it suspended.

I don't have a nicely packaged version of the original code to share, but I believe the reproducer from #65018 also shows this issue with the appropriate build settings. Here is the code again, containing a function that uses a foreign `await_suspend` method with symmetric transfer of control to await each element of an array of 32 integers:

```c++
#include <coroutine>

// A simple awaiter type with an await_suspend method that can't be
// inlined.
struct Awaiter {
  const int& x;

  bool await_ready() { return false; }
  std::coroutine_handle<> await_suspend(const std::coroutine_handle<> h);
  void await_resume() {}
};

struct MyTask {
  // A lazy promise with an await_transform method that supports awaiting
  // integer references using the Awaiter struct above.
  struct promise_type {
    MyTask get_return_object() { return {}; }
    std::suspend_always initial_suspend() { return {}; }
    std::suspend_always final_suspend() noexcept { return {}; }
    void unhandled_exception();

    auto await_transform(const int& x) { return Awaiter{x}; }
  };
};

// A global array of integers.
int g_array[32];

// A coroutine that awaits each integer in the global array.
MyTask FooBar() {
  for (const int& x : g_array) {
    co_await x;
  }
}
```

If you build this on armv8-a clang trunk with `-std=c++20 -fno-exceptions -fsanitize=address -fsanitize=undefined -fno-sanitize=nonnull-attribute -fno-sanitize=vptr -fno-sanitize=function -fno-sanitize=enum -fno-sanitize-trap=all -O0 -emit-llvm` ([Compiler Explorer](https://godbolt.org/z/zW9x8Gh48)), you get this assembly as part of the resume function for `FooBar` ([full dump](https://gist.github.com/jacobsa/a92ec465a01d2f2d697587415db23f66)):

```asm
 bl Awaiter::await_suspend(std::__n4861::coroutine_handle<void>)
        ldr x9, [x19, #536]                 // 8-byte Folded Reload
        str x0, [x19, #240]                 // 8-byte Folded Spill
        str     x9, [x19, #248]                 // 8-byte Folded Spill
        mov     x8, #68719476736                // =0x1000000000
        add     x8, x8, x9, lsr #3
        ldrb    w8, [x8]
        cbz     w8, .LBB16_52
 b       .LBB16_51
.LBB16_51:
        ldr     x0, [x19, #248] // 8-byte Folded Reload
        bl __asan_report_store8
```

This shows that after suspension asan is being told that there is a store to an address loaded from `x19+536`, which is something earlier spilled to the stack. What's going on is clearer with `-emit-llvm` ([full dump](https://gist.github.com/jacobsa/55d7871d7f86fec76d9fba639817318d), [just `FooBar.resume`](https://gist.github.com/jacobsa/fa4955a60a891879c8e80bcb2a5d2026)):

```llvm
 %call29 = call i64 @Awaiter::await_suspend(std::__n4861::coroutine_handle<void>)(ptr noundef nonnull align 8 dereferenceable(8) %ref.tmp11.reload.addr, i64 %coerce.val.pi28), !dbg !329
  %coerce.dive30 = getelementptr inbounds %"struct.std::__n4861::coroutine_handle", ptr %ref.tmp18.reload.addr, i32 0, i32 0, !dbg !329
  %coerce.val.ip31 = inttoptr i64 %call29 to ptr, !dbg !329
  %144 = ptrtoint ptr %coerce.dive30 to i64, !dbg !329
  %145 = lshr i64 %144, 3, !dbg !329
  %146 = add i64 %145, 68719476736, !dbg !329
  %147 = inttoptr i64 %146 to ptr, !dbg !329
  %148 = load i8, ptr %147, align 1, !dbg !329
  %149 = icmp ne i8 %148, 0, !dbg !329
  br i1 %149, label %150, label %151, !dbg !329

150:
  call void @__asan_report_store8(i64 %144) #10, !dbg !329
  unreachable, !dbg !329

151:
  store ptr %coerce.val.ip31, ptr %coerce.dive30, align 8, !dbg !329
```

There is a store to `%coerce.dive30`, which is a pointer to a `std::coroutine_handle` on the coroutine frame:

```llvm
%FooBar().Frame = type { ptr, ptr, %"struct.MyTask::promise_type", i2, %"struct.std::__n4861::suspend_always", %"struct.std::__n4861::suspend_always", ptr, ptr, ptr, ptr, %struct.Awaiter, %"struct.std::__n4861::coroutine_handle", %"struct.std::__n4861::coroutine_handle.0" }
[...]
 %ref.tmp18.reload.addr = getelementptr inbounds %FooBar().Frame, ptr %0, i32 0, i32 11, !dbg !320
```

In other words, **with these build settings clang still incorrectly dumps a coroutine handle to the coroutine frame after it suspends**.
</pre>
<img width="1px" height="1px" alt="" src="http://email.email.llvm.org/o/eJysWU1v4zjS_jXMpWBDoixZPuQQdybzNvAuFthpYPZmUFLJ5jRFakgqjvvXL4qUbSlO0tmZNbpjWybr86mnipJwTu414j3Ltyx_vBODPxh7_4eoTeXEXWWa0_23g3QgnRsQpAOLSnhswBtgPCvyJC1B6Gb8kq8Y_wLHA1qEWgm9B6lrYy3WXp1gcOhAQG2sGbzUCK0VHUJrLLjaCl8fwPWiRhCtRwv-gJO1bnA96gabJXwzYLEWPcsegPEHxh-irm5wHrTxILVD66E2DYI_CA-irtGR9rnMqD9qk_6swpEPFdZicLRdOujECaT21jQDWQeN8AKsqDFqX8KDNv4QTLYomrBeKPp4goN4RrDohg4bMBYadN6aE4VwZkt1Che87Ealo6w9egfCmkGHoEdPJEXWu-jAkiWPLHmIf78dEGTXYSOFR3CnrvemcyA1-IMhf44mJtPBkbLUyhdsSDlbJV8Og9B_yn8PmxCBE3RCy35QwpO-4wGDjzHWR6kUSK3IdFYkzjcse2DZw8Wh3UHoRmG8KprGonOsSJawHTwIB62w9PaV8XUH4igsuU0WHYUDbcCdnMdOeFmTjeS6R1EfRvUhq9IzvnYh40Ipc8RJhEIwp_kdkyv1fglfKcL6e1ij8TiiO2iXEaHa4Z8D6hrBtCEdsyAvFovp19-lP4AAJeweoZcYd0nt0Wqh4Fdj9gojGo_I-PqZMB_ySV48xNj8JrT08gdZajokGFCx9cZS-vW1jqiKFsGlRWsRAa01lvJ7hRJpogzWSshOnqN1DYh0UGFIqZXeow5ROxcBhf-M3StWqxOIOcaDTCtIr64Ha1FTib-uJWxmgfsKjdGMr30sCwFa1qhO0Iv6u9hjA89onTQ6Rh3BWLmXFMNYygbcQdjgWzV4-AoVKonPATkUrFigFlprugk9KUcbzdHFworZPlLSaJ_oe2t6G-qlGqRqwKEnxLsl_N8IiViqDYLYC6lDbI32QmqKooB20LUns0NQRpprjUW511Qd4iik340hYUUCHfqDaaIN7tR16K2swVuhXYuWvCf51qiQGtoNAfyosEPtaYHQIKwVJ_qc8YC2PVpH9TYJOCuS-K9mfEv_4lWeSV2roUFg2ZcLcFj2y2wvf2L8CR7Aya5XGO0gAJz6MXxkw9S1s18hDLWIma5wJi5yxhkWztuh9vAwimbr0UIINUhM7hkv4IVl26lpAJUxalQesMp4yfiGBIBFP1gNrVAOWbYFtn487_qApr6w7Je5N4yX0Yif7TowvrkYCPBsZHMxjXj_atvFFPowd2mMxD9O34T7Pg3EJQ1K_DhBb00n3ev4R-QY280y4IZ-5A9aJPX-lcgRM2CxRUtk52BwkS_wkpLRLlGZ53OngfPF0ZhdQMTEZDi7sUeKAaVjZ6o_sPa3aRqjMs_TJFNjMnZCHcWJGpn0UqhJjv6evJbo5ZU0bfClxt5_QmzI9aAjHJpd3CeJ48opKK4bxHCu6GvSLkC7oH3u0pgKtt6-3NowQ9INqi7g2StTCXWljDNfjCmV2sN-F35l-TbjLH9X0rXPxNGKjHORns6ACtMGznSOekZcPBmzFXZSF6MzNAreRANozjsbN19PNLGL_PgyqcBZmc15cNaNWjiZYST90BoMkWr3XC7Eec6wg_4ei40VySKg6HHkUp7AotVmcUm6g0XrxjbOssdx6JlfHHSDLRFg3Dv5RRutB6UWwnsrq8HjzYrn3tubi5fe8_oH1EM3v7jwlobmR6EULP6ZwAI76RdKPXfUkygb-faL6Xqp0MIvL70yFi0hgZcH7_vQWwIK9qapjPJLY_eMP_2g_79vXspfD6uAeh7mRwrtHn0MrHAOu4oGBAe9sP7c4CNBXhtoAECRjPi4WNUOSkEzdP2b1kjnl3vpD0O1rE3H-NN4fmH8SWw41qsiF0na8JY3xWadl-tVmjcVz9qiGM19p2kK142QqtSlCuNA-6pRXOhlt9Orskjf7RdEGNRn-eaKYXqpxsJLiBvLty9p_MSzPCtY_givX2Mxlovq5BGejGqwgX-hMqKZi3XewktyI5avkk-L_a2XSt1KpdcbBvNV-Xckd-Y5Si5HeUW5TjerdbHOindksuwxeUmT82suTzTNRN74N9iqHFFNlt2koaL3Y3n2jNyZr6mrH3Bds_z_7TYtdjk_I2Vcdb6exuvXr9nDbeaDiW-lKQTz09muFOx2wgm9iyeHnfPGYvkBBYbD_Xk2JjafnJXCIE7SrgcGb1RzOU2cT0tBSRhUNZw5j2zDZhzEiyS4tCUwF0m8QyCpXbh41jmQaBRWSdJNoIgnOSII50X9fQm_H0Q87O0NLTbBplqhsGiv9PwGn_1l5sjzZl2u02bdlkWL9bpoNm0limxTpussLZuR5Vi-_WNw_kpay3HmK5L_VmMrVps8F0Uiyk1arjd1iWVS1RUXecMT_hOuCm5HNDCe10IpapXZI9BHkMWKTvj_cw7jJfUkbUJbg7GFgVB08CmhwctoKSpFc3Bs-Dy32C5916fp0gYcLwk4FM9gKM9rg7bG5bNQy17GWSqURNpUe3rLrvx5Xd3IZ8yS4PQe_XhWIvukrshCR2sZ53F6XX7SX85JNYmZ2F3e2J1xSGYffmIsuSb7LA3mSu29CZaO7sf0eUN6PxKWrlZBQO-tNzTEjXbOI-INCf5YTh7kKHe4GJGuwpbs431F2Ec8e9mW044JcX8sYP1mBEjwZ9wvo9lGNCDLSaLS1Zq-RSSmH8uIdSLrrgeNIMtRMm16P5OVBZmO-0M_ERWq8D1PXn1_T338S-uvPSFUazhVsFXyJpnzcpofqqYsfd_MQVsazWP5fWjFtDNFSp9j6QzYSZBnKLuGu3xP09sd6LaRhC7xSvqrxiGgN-EOW2g8H9-CLBIw-q1bvz_nUsbz6Vll-RRuoBFczmfeM0QvSJ0wTDztRKumZ-WRUyS_2fE2J83PrOPuv7pvbu-N9aPAc6f4nKJ3SPMv7FwmjPPJAS7fLpfL6wj2HgX_hPRvczhBcXLD3-lNxSYfHSM1xDujR2Pjw4P4WOB8h9G9vq84ni2dj_fQrw9IaE6ZPyGJQTkPQ594chGeR9w191mzyTbiDu_TYrMpCp6tirvD_UpkedVmmzZdr_JNWxXFOmk3ZbWusxTFpriT9zzhWZqmaZLwfFUs6yrLc562Jce8KdcVWyXYCamWVCN0ALwLt1Pv1zxJirtAei48TeI8eElIyB_v7D2tX1TD3rFVoqTz7irBS6_CI6hq2NO8-86To88-N7re6acZ0IvvSD1sczdYdf96GpsMYqHo49uityberXqKz0oYfwoO_icAAP__mZhM1A">