[all-commits] [llvm/llvm-project] 291e37: [AMDGPU] Form VOPD3 pairs with pair-local literal ...

Mon Sep 14 13:37:20 PDT 2026

  Branch: refs/heads/users/shiltian/improve-vopd3-imm-folding
  Home:   https://github.com/llvm/llvm-project
  Commit: 291e376e6b081eeb090d9c4710bae3c89ae20fbe
      https://github.com/llvm/llvm-project/commit/291e376e6b081eeb090d9c4710bae3c89ae20fbe
  Author: Shilei Tian <i at tianshilei.me>
  Date:   2026-09-14 (Mon, 14 Sep 2026)

  Changed paths:
    M llvm/lib/Target/AMDGPU/GCNCreateVOPD.cpp
    M llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
    M llvm/lib/Target/AMDGPU/GCNVOPDUtils.h
    M llvm/test/CodeGen/AMDGPU/GlobalISel/fadd.bf16.ll
    M llvm/test/CodeGen/AMDGPU/GlobalISel/fcanonicalize.bf16.ll
    M llvm/test/CodeGen/AMDGPU/GlobalISel/fma.bf16.ll
    M llvm/test/CodeGen/AMDGPU/GlobalISel/fmul.bf16.ll
    M llvm/test/CodeGen/AMDGPU/GlobalISel/fptrunc.bf16.ll
    M llvm/test/CodeGen/AMDGPU/GlobalISel/fptrunc.ll
    M llvm/test/CodeGen/AMDGPU/atomic_optimizations_buffer.ll
    M llvm/test/CodeGen/AMDGPU/atomic_optimizations_raw_buffer.ll
    M llvm/test/CodeGen/AMDGPU/atomic_optimizations_struct_buffer.ll
    M llvm/test/CodeGen/AMDGPU/bf16.ll
    M llvm/test/CodeGen/AMDGPU/branch-relaxation-gfx1250.ll
    M llvm/test/CodeGen/AMDGPU/calling-conventions.ll
    M llvm/test/CodeGen/AMDGPU/carryout-selection.ll
    M llvm/test/CodeGen/AMDGPU/ds_read2-gfx1250.ll
    M llvm/test/CodeGen/AMDGPU/ds_write2.ll
    M llvm/test/CodeGen/AMDGPU/fcanonicalize.bf16.ll
    M llvm/test/CodeGen/AMDGPU/fcanonicalize.ll
    M llvm/test/CodeGen/AMDGPU/flat-saddr-load.ll
    M llvm/test/CodeGen/AMDGPU/float-to-arbitrary-fp-fp8-f16-hw.ll
    M llvm/test/CodeGen/AMDGPU/float-to-arbitrary-fp-fp8-hw.ll
    M llvm/test/CodeGen/AMDGPU/float-to-arbitrary-fp-widen.ll
    M llvm/test/CodeGen/AMDGPU/integer-mad-patterns.ll
    M llvm/test/CodeGen/AMDGPU/llvm.amdgcn.av.load.b128.ll
    M llvm/test/CodeGen/AMDGPU/llvm.amdgcn.permlane.ll
    M llvm/test/CodeGen/AMDGPU/llvm.amdgcn.ptr.s.buffer.load-gfx12.ll
    M llvm/test/CodeGen/AMDGPU/llvm.amdgcn.raw.atomic.buffer.load.ll
    M llvm/test/CodeGen/AMDGPU/llvm.amdgcn.raw.ptr.atomic.buffer.load.ll
    M llvm/test/CodeGen/AMDGPU/llvm.amdgcn.rcp.bf16.ll
    M llvm/test/CodeGen/AMDGPU/llvm.amdgcn.sched.group.barrier.gfx12.ll
    M llvm/test/CodeGen/AMDGPU/llvm.amdgcn.struct.atomic.buffer.load.ll
    M llvm/test/CodeGen/AMDGPU/llvm.amdgcn.struct.ptr.atomic.buffer.load.ll
    M llvm/test/CodeGen/AMDGPU/llvm.sqrt.bf16.ll
    M llvm/test/CodeGen/AMDGPU/load-constant-i1.ll
    M llvm/test/CodeGen/AMDGPU/loop-prefetch-data.ll
    M llvm/test/CodeGen/AMDGPU/mad-mix-bf16.ll
    M llvm/test/CodeGen/AMDGPU/mad-mix-lo-bf16.ll
    M llvm/test/CodeGen/AMDGPU/mul.ll
    M llvm/test/CodeGen/AMDGPU/packed-fp32.ll
    M llvm/test/CodeGen/AMDGPU/packed-fp64.ll
    M llvm/test/CodeGen/AMDGPU/packed-u64.ll
    M llvm/test/CodeGen/AMDGPU/promote-constOffset-to-imm-gfx12.ll
    M llvm/test/CodeGen/AMDGPU/promote-constOffset-to-imm-gfx12.mir
    M llvm/test/CodeGen/AMDGPU/reassoc-mul-add-1-to-mad.ll
    M llvm/test/CodeGen/AMDGPU/v_ashr_pk.ll
    M llvm/test/CodeGen/AMDGPU/vopd-combine-gfx1250.mir
    M llvm/test/CodeGen/AMDGPU/vopd-combine.mir
    A llvm/test/CodeGen/AMDGPU/vopd3-imm-fold.ll
    A llvm/test/CodeGen/AMDGPU/vopd3-imm-fold.mir
    M llvm/test/CodeGen/AMDGPU/waitcnt-loop-ds-prefetch-flushed.ll
    M llvm/test/CodeGen/AMDGPU/waitcnt-loop-ds-prefetch-pattern.ll

  Log Message:
  -----------
  [AMDGPU] Form VOPD3 pairs with pair-local literal moves

This PR lets GCNCreateVOPD form a VOPD3 pair when its components use one
distinct non-inline constant. VOPD3 cannot encode literal operands, but src0
can read scalar registers, so we move the value to a free SGPR. If both
components use the same value, one move serves both.

We reject pairs that need two values because two moves add more instructions
than one fusion removes. We also reject functions without tracked liveness and
functions optimized for size.

We use one reverse liveness walk to find an SGPR that is free over each
pair-local range. We exclude reserved registers and VCC. Disjoint selected
pairs can reuse the same SGPR, and each accepted pair adds at most one
S_MOV_B32 for the one instruction removed by fusion.

When overlapping candidates form the same number of pairs, we prefer the set
that needs fewer scalar moves. We keep pair count as the primary objective.

The post-RA scheduler uses the same matcher policy, so it does not cluster a
two-value pair that the create pass cannot build.

To unsubscribe from these emails, change your notification settings at https://github.com/llvm/llvm-project/settings/notifications