[all-commits] [llvm/llvm-project] 9a402f: [AMDGPU] Form VOPD3 pairs with pair-local literal ...

[Shilei Tian via All-commits]

  Branch: refs/heads/users/shiltian/improve-vopd3-imm-folding
  Home:   https://github.com/llvm/llvm-project
  Commit: 9a402fb977c573e2e6efcca984a2e0dca52b8b34
      https://github.com/llvm/llvm-project/commit/9a402fb977c573e2e6efcca984a2e0dca52b8b34
  Author: Shilei Tian <i at tianshilei.me>
  Date:   2026-09-14 (Mon, 14 Sep 2026)

  Changed paths:
    M llvm/lib/Target/AMDGPU/GCNCreateVOPD.cpp
    M llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
    M llvm/lib/Target/AMDGPU/GCNVOPDUtils.h
    M llvm/test/CodeGen/AMDGPU/GlobalISel/fadd.bf16.ll
    M llvm/test/CodeGen/AMDGPU/GlobalISel/fcanonicalize.bf16.ll
    M llvm/test/CodeGen/AMDGPU/GlobalISel/fma.bf16.ll
    M llvm/test/CodeGen/AMDGPU/GlobalISel/fmul.bf16.ll
    M llvm/test/CodeGen/AMDGPU/GlobalISel/fptrunc.bf16.ll
    M llvm/test/CodeGen/AMDGPU/GlobalISel/fptrunc.ll
    M llvm/test/CodeGen/AMDGPU/atomic_optimizations_buffer.ll
    M llvm/test/CodeGen/AMDGPU/atomic_optimizations_raw_buffer.ll
    M llvm/test/CodeGen/AMDGPU/atomic_optimizations_struct_buffer.ll
    M llvm/test/CodeGen/AMDGPU/bf16.ll
    M llvm/test/CodeGen/AMDGPU/branch-relaxation-gfx1250.ll
    M llvm/test/CodeGen/AMDGPU/calling-conventions.ll
    M llvm/test/CodeGen/AMDGPU/carryout-selection.ll
    M llvm/test/CodeGen/AMDGPU/ds_read2-gfx1250.ll
    M llvm/test/CodeGen/AMDGPU/ds_write2.ll
    M llvm/test/CodeGen/AMDGPU/fcanonicalize.bf16.ll
    M llvm/test/CodeGen/AMDGPU/fcanonicalize.ll
    M llvm/test/CodeGen/AMDGPU/flat-saddr-load.ll
    M llvm/test/CodeGen/AMDGPU/float-to-arbitrary-fp-fp8-f16-hw.ll
    M llvm/test/CodeGen/AMDGPU/float-to-arbitrary-fp-fp8-hw.ll
    M llvm/test/CodeGen/AMDGPU/float-to-arbitrary-fp-widen.ll
    M llvm/test/CodeGen/AMDGPU/integer-mad-patterns.ll
    M llvm/test/CodeGen/AMDGPU/llvm.amdgcn.av.load.b128.ll
    M llvm/test/CodeGen/AMDGPU/llvm.amdgcn.permlane.ll
    M llvm/test/CodeGen/AMDGPU/llvm.amdgcn.ptr.s.buffer.load-gfx12.ll
    M llvm/test/CodeGen/AMDGPU/llvm.amdgcn.raw.atomic.buffer.load.ll
    M llvm/test/CodeGen/AMDGPU/llvm.amdgcn.raw.ptr.atomic.buffer.load.ll
    M llvm/test/CodeGen/AMDGPU/llvm.amdgcn.rcp.bf16.ll
    M llvm/test/CodeGen/AMDGPU/llvm.amdgcn.sched.group.barrier.gfx12.ll
    M llvm/test/CodeGen/AMDGPU/llvm.amdgcn.struct.atomic.buffer.load.ll
    M llvm/test/CodeGen/AMDGPU/llvm.amdgcn.struct.ptr.atomic.buffer.load.ll
    M llvm/test/CodeGen/AMDGPU/llvm.sqrt.bf16.ll
    M llvm/test/CodeGen/AMDGPU/load-constant-i1.ll
    M llvm/test/CodeGen/AMDGPU/loop-prefetch-data.ll
    M llvm/test/CodeGen/AMDGPU/mad-mix-bf16.ll
    M llvm/test/CodeGen/AMDGPU/mad-mix-lo-bf16.ll
    M llvm/test/CodeGen/AMDGPU/mul.ll
    M llvm/test/CodeGen/AMDGPU/packed-fp32.ll
    M llvm/test/CodeGen/AMDGPU/packed-fp64.ll
    M llvm/test/CodeGen/AMDGPU/packed-u64.ll
    M llvm/test/CodeGen/AMDGPU/promote-constOffset-to-imm-gfx12.ll
    M llvm/test/CodeGen/AMDGPU/promote-constOffset-to-imm-gfx12.mir
    M llvm/test/CodeGen/AMDGPU/reassoc-mul-add-1-to-mad.ll
    M llvm/test/CodeGen/AMDGPU/v_ashr_pk.ll
    M llvm/test/CodeGen/AMDGPU/vopd-combine-gfx1250.mir
    M llvm/test/CodeGen/AMDGPU/vopd-combine.mir
    A llvm/test/CodeGen/AMDGPU/vopd3-imm-fold.ll
    A llvm/test/CodeGen/AMDGPU/vopd3-imm-fold.mir
    M llvm/test/CodeGen/AMDGPU/waitcnt-loop-ds-prefetch-flushed.ll
    M llvm/test/CodeGen/AMDGPU/waitcnt-loop-ds-prefetch-pattern.ll

  Log Message:
  -----------
  [AMDGPU] Form VOPD3 pairs with pair-local literal moves

VOPD3 cannot encode literal operands, but its src0 can read scalar registers.
In this PR, we try to allow to form a pair with one distinct non-inline constant
value by moving that value to a free SGPR. If both components use the same value,
one move serves both, but we reject pairs that need two different values, since
that doesn't buy us anything. Ffunctions without tracked liveness, and functions
optimized for size are also rejected.

With this change, `GCNCreateVOPD` checks every adjacent pair before it selects a
greedy non-overlapping pair. This lets an infeasible pair fall through to the
overlapping next one. We use one reverse liveness walk to find an SGPR that is
free over each pair-local range. Disjoint selected pairs can reuse the same SGPR.
Each accepted pair adds at most one `S_MOV_B32` for the one instruction removed
by fusion.

The post-RA scheduler uses the same matcher policy, so it will not cluster a
two-value pair that the create pass cannot build.



To unsubscribe from these emails, change your notification settings at https://github.com/llvm/llvm-project/settings/notifications