[llvm-dev] [RFC] LLVM Coroutines

[Gor Nishanov via llvm-dev]

Hi David:

>>   has anyone in WG21 yet come up with a sane proposal about how coroutines interact with thread-local storage?

There is a discussion in SG1 subgroup (Concurrency and Parallelism) on
the meaning of TLS in the lightweight thread of execution. I don't
think there is a resolution yet. Whether to think of a coroutine of
this kind as a thread of execution or just an "unusual" function is
not decided either.

>> Does this mean that, for correctness, we must regard these suspension points as barriers that prevent loads or stores of thread-local variables and prevent caching of loads of addresses of thread-local variables?

You are right! P0057 implies (probably need to make a note about it)
that when a coroutine is executing, you get thread_local data for the
current thread. For your first point (caching TLS loads), I think I am
getting it for free. coro.suspend intrinsic, from llvm standpoint, may
read and write any memory it can get access to, therefore, if a value
was loaded from TLS before coro.suspend, it would have to be reloaded
after a call to coro.suspend.

With respect to caching of addresses of TLS variables.

1) if a user does the caching, we should let him:

  thread_local int a;
  task f() {
     auto x = &a; // captures the address of a in Thread 1
     co_await some_async_call();
     *x = *x + 1; // updates the value in Thread1, whatever the
current thread is
  }

2) If llvm is does it, than, absolutely, we should prevent "prevent
caching of loads of addresses across suspend points". I poked around a
little and was not able to come up with the case where it would, but,
I'll keep looking. If you know of such a case, let me know.
Coroutines are split into normal functions in the middle optimizer, so
if there is any caching done during lowering, coroutines won't be
affected.

>> Do we have to treat every function call as a suspension point, as no function can know whether it is executing in
>> a coroutine or whether a function that it calls may trigger a thread migration?  This sounds as if
>> it would cause noticeable performance regressions in code that makes heavy use of thread-local variables.

That should not be necessary. The proposed coroutines are "stackless",
they can only get suspended when control is in the body of the
coroutine at explicitly marked suspend points. Thus thread migration
can only happen at suspend point boundaries.

With stackful coroutines (aka Fibers, Green Threads, User-Mode
threads, etc), it is a big concern, since none of the functions that
runs on a fiber are aware that they run on a fiber and potentially can
get a thread switch at every call. Yep. TLS is pretty much busted
there.

Thank you very much for your comments!!!

On Thu, Jun 9, 2016 at 1:25 AM, David Chisnall
<David.Chisnall at cl.cam.ac.uk> wrote:
> On 9 Jun 2016, at 06:57, Gor Nishanov via llvm-dev <llvm-dev at lists.llvm.org> wrote:
>>
>> Though this
>> proposal is motivated primarily by the desire to support C++ Coroutines [1],
>> the llvm representation is language neutral and can be used to support
>> coroutines in other languages as well.
>
> I’m sorry if this is off-topic, but has anyone in WG21 yet come up with a sane proposal about how coroutines interact with thread-local storage?  In LLVM-compiled code, it is possible that the address of a thread-local variable will be cached to avoid repeated lookups.  In an environment that contains both coroutines and TLS, it is possible that the coroutine will be migrated to a different thread between suspension and resumption.  Does this mean that, for correctness, we must regard these suspension points as barriers that prevent loads or stores of thread-local variables and prevent caching of loads of addresses of thread-local variables?  Do we have to treat every function call as a suspension point, as no function can know whether it is executing in a coroutine or whether a function that it calls may trigger a thread migration?  This sounds as if it would cause noticeable performance regressions in code that makes heavy use of thread-local variables.
>
> David
>