[clang] [llvm] [AMDGPU] Implement AsyncMark Stages (PR #220442)

[Krzysztof Drewniak via cfe-commits]

================
@@ -16,31 +16,66 @@ internally by the compiler. A thread that initiates one or more async operations
 An *asyncmark* created by a thread can be used to track async operations
 initiated by that thread.
 
+### Stages
+
+Every asyncmark belongs to a *stage*, which names a kind of async operation.
+Each async operation has an *own stage*, determined by the instruction that
+initiates it, and *belongs to* that stage and to the stage `ALL`.
+
+The stages are:
+
+| Value | Stage | Async operations |
+|---|---|---|
+| 0 | `TENSOR` | tensor loads and stores |
+| 1 | `GLOBAL_LOAD_ASYNC_TO_LDS` | global loads async to LDS |
+| 2 | `GLOBAL_LOAD_ASYNC_TO_LDS_MCAST` | multicast (cluster) global loads async to LDS |
+| 3 | `ASYNC_LDS_STORE` | async stores from LDS |
+| 5 | `UNFORMATTED_BUFFER_GLOBAL_LOAD` | unformatted buffer and global loads to LDS |
+| 16 | `ALL` | all of the above |
+
+The values not listed above are reserved for future async operations.
+Using one is an error.
+
+Which async operations a given subtarget actually has is described in
+{ref}`AMDGPU DMA Operations <amdgpu-dma-operations>`. A stage is valid on every
+subtarget that supports asyncmarks, whether or not that subtarget has any
+operation belonging to it; marking an empty stage is simply a no-op.
+
+Stage `ALL` is the catch-all: since every async operation belongs to it, an
+asyncmark in `ALL` tracks all async operations whatever their own stage.
+
 ### Current Sequence
 
-The abstract machine maintains a sequence of asyncmarks during the execution of
-a function body, which excludes any asyncmarks produced by calls to other
-functions encountered in the currently executing function. The state of this
-sequence at each program point in the function is called the *current sequence*.
+The abstract machine maintains a separate sequence of asyncmarks *for each
+stage* during the execution of a function body, which excludes any asyncmarks
+produced by calls to other functions encountered in the currently executing
+function. The state of the sequence for a stage `S` at each program point in
+the function is called the *current sequence of* `S`.
----------------
krzysz00 wrote:

I agree that the wording is tricky here

Perhaps the phrasing is that there's still only one queue, but each mark in the queue is associated with a mask of the stages it ignores, and waits ignore marks that have [the relevant phrasing around overlapping] masks when traversing the queue

https://github.com/llvm/llvm-project/pull/220442