[llvm] [VPlan] Sink recipes from the vector loop region in licm. (PR #168031)

[Mel Chen via llvm-commits]

================
@@ -2545,6 +2545,58 @@ static void licm(VPlan &Plan) {
       R.moveBefore(*Preheader, Preheader->end());
     }
   }
+
+  VPDominatorTree VPDT(Plan);
+  // Sink recipes with no users inside the vector loop region into a dedicated
+  // exit block.
+  // TODO: Sinking to non-dedicated exits is not supported yet, as it would
+  // require splitting the edge to create a dedicated exit block. Without this,
+  // the sunk instruction would incorrectly execute on paths entering the exit
+  // block from other predecessors.
+  // TODO: Extend to sink recipes from inner loops.
+  for (VPBasicBlock *VPBB : VPBlockUtils::blocksOnly<VPBasicBlock>(
+           vp_post_order_shallow(LoopRegion->getEntry()))) {
+    for (VPRecipeBase &R : make_early_inc_range(reverse(*VPBB))) {
+      if (cannotHoistOrSinkRecipe(R))
+        continue;
+
+      // TODO: Use R.definedValues() instead of casting to VPSingleDefRecipe to
+      // support recipes with multiple defined values (e.g., interleaved loads),
+      auto *Def = cast<VPSingleDefRecipe>(&R);
+      if (Def->getNumUsers() == 0)
+        continue;
+
+      SmallPtrSet<VPBasicBlock *, 2> UserBBs;
+      // Cannot sink the recipe if any user is defined in the same loop or in
+      // any nested inner loop region.
+      if (any_of(Def->users(), [&UserBBs](VPUser *U) {
+            auto *UserR = cast<VPRecipeBase>(U);
+            VPBasicBlock *Parent = UserR->getParent();
+            // TODO: If the user is a PHI node, we should check the block of
+            // incoming value. Support PHI node users if needed.
+            if (UserR->isPhi() || Parent->getEnclosingLoopRegion())
+              return true;
+            // Collect the basic block of users.
+            UserBBs.insert(Parent);
+            return false;
+          }))
+        continue;
+
+      // TODO: Support sinking when users are in multiple blocks.
+      if (UserBBs.size() != 1)
+        continue;
+
+      VPBasicBlock *SinkBB = *UserBBs.begin();
+      // Only sink to dedicated exit blocks.
+      if (SinkBB->getSinglePredecessor() != LoopRegion)
+        continue;
----------------
Mel-Chen wrote:

Pre-commit test case is https://github.com/llvm/llvm-project/pull/177954
We can sink recipes to exit blocks other than the middle block. 196ea93d37b0336ec5b3227742914b0cf99f1a58

https://github.com/llvm/llvm-project/pull/168031