[llvm] [AMDGPU] Add islands to StructurizeCFG (PR #201675)

[Robert Imschweiler via llvm-commits]

================
@@ -1188,17 +1309,231 @@ void StructurizeCFG::wireFlow(bool ExitUseAllowed,
     addPhiValues(Flow, Entry);
     DT->changeImmediateDominator(Entry, Flow);
 
-    PrevNode = Node;
+    Tail = Node;
     while (!Order.empty() && !Visited.count(LoopEnd) &&
            dominatesPredicates(Entry, Order.back())) {
       handleLoops(false, LoopEnd);
     }
 
-    changeExit(PrevNode, Next, false);
+    if (Tail.isRewritable())
+      changeExit(Tail, Next, false);
     setPrevNode(Next);
   }
 }
 
+/// Handle an "island", i.e. a block whose terminator is not rewritable (e.g., a
+/// callbr) in this context and considered immutable from this pass's
+/// perspective.
+///
+/// The island is left untouched; its edges are split into forwarders that
+/// converge at a new Flow block (ExitFlow). The callbr's runtime target choice
+/// is recovered as a per-target i1 "sel" phi and re-dispatched by a structured
+/// ladder of 2-way branches:
+///
+///                  callbr
+///                /   |   \
+///            fwd_0 fwd_1 fwd_2      forwarders (intermediate target blocks)
+///                \   |   /
+///             ExitFlow (Flow_0) -- sel_0? --> real_0
+///                   |  else
+///                 Flow_1 --------- sel_1? --> real_1
+///                   |  else
+///              real_2
+///            or Flow_2 (if real_1 == real_2) ------> real_2   (shared last
+///            target)
+///
+void StructurizeCFG::handleIsland(BasicBlock *IslandBB) {
+  DomTreeUpdater DTU(DT, DomTreeUpdater::UpdateStrategy::Eager);
+
+  // 1. Split the island's edges into forwarders, so the island's targets are no
+  // longer direct successors of the (immutable) callbr terminator. This also
+  // turns a self-loop back-edge into a *mutable* forwarder, so later steps
+  // never rewrite the island edge itself.
+  //
+  // Reuse a target as-is only if it is already a trivial forwarder *inside this
+  // region*. A trivial target that is the region's exit block is still split:
+  // the ladder consumes its forwarders, and consuming the exit block would move
+  // the region's SESE boundary, which the precomputed RegionInfo cannot
+  // represent.
+  SmallVector<BasicBlock *, 4> Targets;
+  // Several island edges to the same block are indistinguishable, so each
+  // distinct target is given a single forwarder (one ladder arm, one sel
+  // value).
+  SmallDenseMap<BasicBlock *, BasicBlock *> ForwarderFor;
+
+  if (CallBrInst *CallBr = dyn_cast<CallBrInst>(IslandBB->getTerminator())) {
+    for (unsigned I = 0; I < CallBr->getNumSuccessors(); ++I) {
+      BasicBlock *Target = CallBr->getSuccessor(I);
+      // Leave an unreachable target's callbr edge direct and keep the dead lane
+      // out of the dispatch sel phi, which SILowerI1Copies cannot lower. It
+      // stays a direct predecessor of its successor and is not part of the
+      // ladder.
+      // However, that gives the region exit a predecessor createFlow's
+      // single-pred exit wiring does not account for. Flag that the exit's
+      // immediate dominator needs to be fixed.
+      if (isUnreachableTarget(*Target)) {
+        NeedFixRegionExitDom = true;
+        continue;
+      }
+      // Already handled this target: route this edge to the same forwarder.
+      if (BasicBlock *Fwd = ForwarderFor.lookup(Target)) {
+        CallBr->setSuccessor(I, Fwd);
----------------
ro-i wrote:

rightfully so, there was a bug. But this is covered by SplitMultiBrEdge now, anyway

https://github.com/llvm/llvm-project/pull/201675