[llvm] [AMDGPU] Add waterfall intrinsics (PR #192409)

[Jay Foad via llvm-commits]

jayfoad wrote:

Thank you for working on this! I have not read the patch yet, only the description.

> Waterfall loops can already be generated during instruction selection. This PR introduces LLVM IR level intrinsics to explicitly mark waterfall regions. These intrinsics provide a suitable level of abstraction for a frontend to describe waterfall regions and enable later passes to correctly transform and optimize waterfall regions.

Right. I think the main benefit of this is that it provides a practical way for a frontend to group multiple waterfall-able operations into a single waterfall loop.

> These intrinsics have been used in production for a long time as part of an internal graphics compilation pipeline.

It's not just "internal", it is open sourced at https://github.com/GPUOpen-Drivers/llvm-project

> A waterfall loop iterates over an index of values for active lanes.

Nit: "index" does not seem like quite the right word, here and in the description below, but I don't have a better suggestion yet.

u * `llvm.amdgcn.waterfall.end`
>   pseudo-instruction to mark the end of a waterfall region.
> * `llvm.amdgcn.waterfall.last_use`
>   a variant of `waterfall.end` to mark the last use of a uniform value in  the waterfall region, for uniform values used in non-defining operations   such as stores.
> * `llvm.amdgcn.waterfall.last_use_vgpr`
>   a variant of `last_use` for uniform values that remain in a VGPR.
> * `llvm.amdgcn.waterfall.loop_end`
>   pseudo-instruction inserted late to mark loop end for special handling such as SCC clobbers.

Maybe not worth trying to document the intrinsics here in the description? They are rather confusing and this documentation is not at all complete. Probably better just to document them in AMDGPUUsage, and make that documentation as clear as possible,

>   A  waterfall group must contain at least one `waterfall.begin`, at least one     `waterfall.readfirstlane`, and at least one of `waterfall.end` or `waterfall.loop_end` intrinsic.

Must it be _exactly_ one "end" or "loop_end"?

> * Each waterfall group must be contained within a single basic block. Extending to multiple blocks in non-trivial.

Typo "in" for "is".

> * Divergent GEPs inside waterfall regions cause nested waterfall loops to    be generated. The compiler identifies this situation and prints a warning.

Not GEPs specifically; this applies to any instruction that takes a uniform-only operand, e.g. most BUFFER and IMAGE instructions that take uniform descriptors. Also, printing a warning in the backend is not a good long term solution!

> * A new pass `SIInsertWaterfall` replaces the waterfall intrinsic markers with  waterfall loop code. The pass operates at the MachineIR level rather than LLVM IR level because waterfall loops require direct manipulation of the    EXEC mask.

Can we please call it "AMDGPUInsertWAterfall"? The "SI" prefix is anachronistic and really should not be used for new passes.

https://github.com/llvm/llvm-project/pull/192409