[llvm] [AMDGPU] Add waterfall intrinsics (PR #192409)

[Sebastian Neubauer via llvm-commits]

================
@@ -1776,6 +1776,59 @@ The AMDGPU backend implements the following LLVM IR intrinsics.
                                                    * :ref:`Synchronization Scope<amdgpu-intrinsics-syncscope-metadata-operand>`.
                                                      Note that the scope used must ensure that the L2 cache will be hit.
 
+  llvm.amdgcn.waterfall.begin                      Marks the beginning of a waterfall region of code.
+
+                                                   The compiler generates a waterfall loop around the region.
+                                                   A waterfall loop handles the case where an operation that requires
+                                                   a uniform operand (e.g., held in an SGPR) is applied to a divergent operand
+                                                   (held in a VGPR, with values varying per lane).
+                                                   Each iteration of the waterfall loop activates a subset of lanes that
+                                                   share the same value of the VGPR (the value in the first active lane).
+                                                   The operation is then executed using that value as the uniform operand.
+                                                   If the VGPR is already uniform, the waterfall loop executes only once.
+                                                   The worst case for a waterfall loop is one iteration per lane (all lanes
+                                                   have different values of the VGPR), but it is not common in practice.
+
+                                                   The intrinsic takes a previous token
+                                                   (``i32``; use a null/zero value if this is the first ``waterfall.begin`` in
+                                                   a waterfall group) and a VGPR.
+
+                                                   The intrinsic returns a new token that must be threaded through the
+                                                   corresponding ``waterfall.readfirstlane`` and ``waterfall.end`` or
+                                                   ``waterfall.last_use`` intrinsics, forming a waterfall group of intrinsics
+                                                   that together define a waterfall region.
+
+                                                   All intrinsics in a waterfall group must reside in the same basic block.
+
+                                                   Multiple ``waterfall.begin`` intrinsics can be chained by
----------------
Flakebi wrote:

chained -> combined

Chained sounds like they are executed after each other but that doesn’t seem to be the intention in the rest of the paragraph.

https://github.com/llvm/llvm-project/pull/192409