[llvm] [Coroutines][LazyCallGraph] resumes are not really SCC (PR #116285)

[Tyler Nowicki via llvm-commits]

================
@@ -1720,6 +1720,7 @@ void LazyCallGraph::addSplitRefRecursiveFunctions(
   for (Function *NewFunction : NewFunctions) {
     Node &NewN = initNode(*NewFunction);
 
+    // Make the original function reference each new function
----------------
TylerNowicki wrote:

What you describe differs from what is happening right now (without my patch). Here are some more details on how it works right now. First for the coroutine function we create a single/unified return block for each suspend point and create a phi in the return BB that takes a reference to a continuation function. We create one empty continuation function for each suspend point. Then for each continuation function we clone the contents of the coroutine function in its entirety and fix-up the entry point. Notice that the return BB and its phi are retained in each continuation function. Then we call addSplitRefRecursiveFunctions and then we do simplify-cfg. The simplify-cfg removes all of the unnecessary blocks and as a result it will remove some or all of the edges in the phi we added to the return BB.

For example, if we have a coroutine like
```
foo() {
 // part A
 suspend1:
 // part B
 suspend2:
 // part C
 exit:
}
```

Then we first generate
```
foo() {
 // part A
 suspend1:
 // part B
 suspend2:
 // part C
 exit:v
 returnBB:
 phi = [continuationB suspend1, continuationC, suspend2]
}

continuationB () {
 // part A
 suspend1:
 // part B  <-- entry point
 suspend2:
 // part C
 exit:
 returnBB:
 phi = [continuationB suspend1, continuationC, suspend2]
}

continuationC () {
 // part A
 suspend1:
 // part B
 suspend2:
 // part C <-- entry point
 exit:
 returnBB:
 phi = [continuationB suspend1, continuationC, suspend2]
}
```

At this point we call addSplitRefRecursiveFunctions, adding the new continuation functions to the call graph and they will form a (Ref)SCC.

However, then we do simplify-cfg and the resulting functions look like
```
foo() {
 // part A
 returnBB:
 phi = [continuationB suspend1]
}

continuationB () {
 // part B  <-- entry point
 returnBB
 phi = [continuationC, suspend2]
}

continuationC () {
 // part C <-- entry point
 exit:
}
```

So, the question is, what is the correct call-graph? Based on previous comments I assumed we actually wanted to keep the current behavior and keep the continuation functions in the same (Ref)SCC. This is partly because there are actually 2 different methods for creating continuations and what I described above is only one of those methods.  See an earlier comment by John.

>  adding a ref from every funclet to every non-initial funclet is definitely the conservatively correct way to handle all of the lowerings that use non-unified continuations (which is everything but the switch lowering).

The Async method and the returned-continuation method both use addSplitRefRecursiveFunctions to add their new continuation functions to the call graph. As John said we don't need to worry about the switch method because it uses 
addSplitFunction to add its funclets to the call graph.

https://github.com/llvm/llvm-project/pull/116285