[all-commits] [llvm/llvm-project] 69ddc4: [AMDGPU] Add MemoryPipeline scheduling to Coexec s...

[Jeffrey Byrnes via All-commits]

  Branch: refs/heads/main
  Home:   https://github.com/llvm/llvm-project
  Commit: 69ddc4658fe6742c023043a4760194c530bc86c8
      https://github.com/llvm/llvm-project/commit/69ddc4658fe6742c023043a4760194c530bc86c8
  Author: Jeffrey Byrnes <jeffrey.byrnes at amd.com>
  Date:   2026-09-21 (Mon, 21 Sep 2026)

  Changed paths:
    M llvm/lib/Target/AMDGPU/AMDGPUCoExecSchedStrategy.cpp
    M llvm/lib/Target/AMDGPU/AMDGPUCoExecSchedStrategy.h
    M llvm/test/CodeGen/AMDGPU/coexec-block-carried-latency.mir
    M llvm/test/CodeGen/AMDGPU/coexec-sched-ds-fifo.mir
    M llvm/test/CodeGen/AMDGPU/coexec-sched-effective-stall.mir

  Log Message:
  -----------
  [AMDGPU] Add MemoryPipeline scheduling to Coexec sched (#192325)

This adds two pieces to the new scheduler:

Stalls between ds_load and fence
tryAsyncPipe
For both of these pieces we are making some assumptions about the
pipelining of memory instructions in the incoming IR. Namely, that the
long latency memory operations are pipelined and the shorter latency
memory operations (e.g. ds_loads) aren't. Moreover, this assumes that
any user added waitcnts appear directly before fences. These assumptions
are compensating for the fact that we don't have any analysis of the
pipeline available during scheduling.

Stalls between ds_load and fence
The main concept here is that when legalizing the fences we will insert
wait_dscnts for any outstanding ds_load instructions. By teaching the
scheduler about the stall between these instructions, we can hide
latency between ds_load -> wait_dscnt. A common idiom is:

wait_loadcnt
fence
signal
fence

Since waitcnts are often included with fences, we also add stalls for
these waitcnt instructions.

tryAsyncPipe
This helps us to acheive good memory pipelining without having good
memory pipeline modelling. This assumes the pipeline has been well
designed and amdgpu-barrier-signal-wait-latency is set appropriately.
Then, this heuristic will schedule memory pipeline components as soon as
the stalls are cleared. It will, for example, keep waitcnt and fence
instructions together, and will schedule load instructions as soon as
registers from previous iterations have been consumed (ensuring max
latency between load -> consumer). It errs on the side of hiding too
much latency for long latency loads.

---------

Co-authored-by: Austin Kerbow <Austin.Kerbow at amd.com>



To unsubscribe from these emails, change your notification settings at https://github.com/llvm/llvm-project/settings/notifications