[llvm] [OpenMPOpt] Fix incorrect end-of-kernel barrier removal (PR #65670)

[Daniel Woodworth via llvm-commits]

================
@@ -2676,17 +2676,19 @@ struct AAExecutionDomainFunction : public AAExecutionDomain {
       if (!ED.EncounteredAssumes.empty() && !A.isModulePass())
         return;
 
-      // We can remove this barrier, if it is one, or all aligned barriers
-      // reaching the kernel end. In the latter case we can transitively work
-      // our way back until we find a barrier that guards a side-effect if we
-      // are dealing with the kernel end here.
+      // We can remove this barrier, if it is one, or aligned barriers reaching
+      // the kernel end (if CB is nullptr). Aligned barriers reaching the kernel
+      // end may have other successors besides the kernel end (especially if
+      // they're in loops) with non-local side-effects, so those barriers can
+      // only be removed if they also only reach the kernel end. If those
+      // barriers have other barriers reaching them, those can be transitively
+      // removed as well.
       if (CB) {
         DeletedBarriers.insert(CB);
         A.deleteAfterManifest(*CB);
         ++NumBarriersEliminated;
         Changed = ChangeStatus::CHANGED;
       } else if (!ED.AlignedBarriers.empty()) {
----------------
dwoodwor-intel wrote:

I've added the requested testcases in my update.

With this new updated the three barriers in `multiple_blocks_functions_kernel_effects_0` are back to being removed, but is this the correct behavior? It seems to me that we should preserve at least one of the barriers between the `barrier_then_write0` calls and the `write_then_barrier0` call.

https://github.com/llvm/llvm-project/pull/65670