[llvm] [AMDGPU] Add waterfall intrinsics (PR #192409)

[via llvm-commits]

================
@@ -2073,11 +2081,46 @@ bool AMDGPUCodeGenPrepareImpl::visitIntrinsicInst(IntrinsicInst &I) {
   case Intrinsic::uadd_sat:
   case Intrinsic::sadd_sat:
     return visitSaturatingAdd(I);
+  case Intrinsic::amdgcn_waterfall_begin:
+    CurrentWaterfall = nullptr;
+    return false;
+  case Intrinsic::amdgcn_waterfall_end:
+    CurrentWaterfall = &I;
+    return false;
   default:
     return false;
   }
 }
 
+bool AMDGPUCodeGenPrepareImpl::visitGetElementPtrInst(GetElementPtrInst &I) {
+  if (!CurrentWaterfall || UA.isUniformAtDef(&I))
+    return false;
+  if (I.getParent() != CurrentWaterfall->getParent())
+    return false;
+
+  // Divergent GEP within a waterfall region will introduce a nested waterfall.
----------------
gretay-amd wrote:

GEPs by themselves won't introduce waterfall loops, but the pointer formation might be sunk into a divergent loop by an earlier compiler pass. Uniformity analysis won't be able to prove that a value is uniform, even if that value is invariant in the loop. The backend will introduce a waterfall loop around the GEP. If the GEP was inside a waterfall region specified by the frontend, the nested waterfall loop causes an assertion failure during AMDGPUInsertWaterfall pass (because the frontend's waterfall intrinsics not in the same basic block any more).

This happened in a real-world use case and there was a discussion about it involving @jayfoad  @perlfu @dstutt about a year ago (internally).  The correct fix might require extending the uniformity analysis or allowing the backend to generate a waterfall loop that is nested inside a frontend-generated waterfall loop.

I can split this out into a separate PR, but my concern is that the intrinsics won't be usable downstream without it, and we are repeating work that has already been put into this fix downstream. How do you feel about keeping it with an appropriate TODO and open a separate ticket/issue to look into a proper fix?  



https://github.com/llvm/llvm-project/pull/192409