[llvm] [AMDGPU] Add waterfall intrinsics (PR #192409)

[Sebastian Neubauer via llvm-commits]

================
@@ -1776,6 +1776,59 @@ The AMDGPU backend implements the following LLVM IR intrinsics.
                                                    * :ref:`Synchronization Scope<amdgpu-intrinsics-syncscope-metadata-operand>`.
                                                      Note that the scope used must ensure that the L2 cache will be hit.
 
+  llvm.amdgcn.waterfall.begin                      Marks the beginning of a waterfall region of code.
+
+                                                   The compiler generates a waterfall loop around the region.
+                                                   A waterfall loop handles the case where an operation that requires
+                                                   a uniform operand (e.g., held in an SGPR) is applied to a divergent operand
+                                                   (held in a VGPR, with values varying per lane).
+                                                   Each iteration of the waterfall loop activates a subset of lanes that
+                                                   share the same value of the VGPR (the value in the first active lane).
+                                                   The operation is then executed using that value as the uniform operand.
+                                                   If the VGPR is already uniform, the waterfall loop executes only once.
+                                                   The worst case for a waterfall loop is one iteration per lane (all lanes
+                                                   have different values of the VGPR), but it is not common in practice.
+
+                                                   The intrinsic takes a previous token
+                                                   (``i32``; use a null/zero value if this is the first ``waterfall.begin`` in
+                                                   a waterfall group) and a VGPR.
+
+                                                   The intrinsic returns a new token that must be threaded through the
+                                                   corresponding ``waterfall.readfirstlane`` and ``waterfall.end`` or
+                                                   ``waterfall.last_use`` intrinsics, forming a waterfall group of intrinsics
+                                                   that together define a waterfall region.
+
+                                                   All intrinsics in a waterfall group must reside in the same basic block.
+
+                                                   Multiple ``waterfall.begin`` intrinsics can be chained by
+                                                   passing the token of the preceding ``waterfall.begin`` as the first argument.
+                                                   This allows a front-end to create one waterfall loop for
+                                                   multiple non-uniform values.
+                                                   Later compiler passes may remove values determined as uniform.
+                                                   The final token is used for other waterfall intrinsics in the same group.
+
+  llvm.amdgcn.waterfall.readfirstlane              Reads the first active lane's value of the VGPR and returns it as
+                                                   an SGPR for use within a waterfall region.
+
+                                                   Takes the ``i32`` token from the final ``waterfall.begin`` in the waterfall
+                                                   group and the VGPR. Returns the uniform (SGPR) result.
----------------
Flakebi wrote:

It seems somewhat unfitting to me to speak about VGPRs and SGPRs here as IR does not know them. It definitely makes sense to mention this in the motivation, but the definitions of the intrinsics should talk about divergent and uniform values.

Not sure how other people feel about it.

https://github.com/llvm/llvm-project/pull/192409