[llvm] [AMDGPU] Add waterfall intrinsics (PR #192409)

[Sebastian Neubauer via llvm-commits]

================
@@ -1776,6 +1776,59 @@ The AMDGPU backend implements the following LLVM IR intrinsics.
                                                    * :ref:`Synchronization Scope<amdgpu-intrinsics-syncscope-metadata-operand>`.
                                                      Note that the scope used must ensure that the L2 cache will be hit.
 
+  llvm.amdgcn.waterfall.begin                      Marks the beginning of a waterfall region of code.
+
+                                                   The compiler generates a waterfall loop around the region.
+                                                   A waterfall loop handles the case where an operation that requires
+                                                   a uniform operand (e.g., held in an SGPR) is applied to a divergent operand
+                                                   (held in a VGPR, with values varying per lane).
+                                                   Each iteration of the waterfall loop activates a subset of lanes that
+                                                   share the same value of the VGPR (the value in the first active lane).
+                                                   The operation is then executed using that value as the uniform operand.
+                                                   If the VGPR is already uniform, the waterfall loop executes only once.
+                                                   The worst case for a waterfall loop is one iteration per lane (all lanes
+                                                   have different values of the VGPR), but it is not common in practice.
+
+                                                   The intrinsic takes a previous token
+                                                   (``i32``; use a null/zero value if this is the first ``waterfall.begin`` in
+                                                   a waterfall group) and a VGPR.
+
+                                                   The intrinsic returns a new token that must be threaded through the
+                                                   corresponding ``waterfall.readfirstlane`` and ``waterfall.end`` or
+                                                   ``waterfall.last_use`` intrinsics, forming a waterfall group of intrinsics
+                                                   that together define a waterfall region.
+
+                                                   All intrinsics in a waterfall group must reside in the same basic block.
+
+                                                   Multiple ``waterfall.begin`` intrinsics can be chained by
+                                                   passing the token of the preceding ``waterfall.begin`` as the first argument.
+                                                   This allows a front-end to create one waterfall loop for
+                                                   multiple non-uniform values.
+                                                   Later compiler passes may remove values determined as uniform.
+                                                   The final token is used for other waterfall intrinsics in the same group.
+
+  llvm.amdgcn.waterfall.readfirstlane              Reads the first active lane's value of the VGPR and returns it as
+                                                   an SGPR for use within a waterfall region.
+
+                                                   Takes the ``i32`` token from the final ``waterfall.begin`` in the waterfall
+                                                   group and the VGPR. Returns the uniform (SGPR) result.
+
+                                                   If the VGPR is determined to be uniform at compile time, this intrinsic
+                                                   is optimized away (the input VGPR value is used directly).
----------------
Flakebi wrote:

is optimized -> can be optimized

I think this is an optional optimization that we do not want to guarantee.

https://github.com/llvm/llvm-project/pull/192409