[all-commits] [llvm/llvm-project] 40ba09: [AMDGPU][SILowerSGPRSpills] Spill SGPRs to virtual...

Fri Dec 16 22:27:02 PST 2022

  Branch: refs/heads/main
  Home:   https://github.com/llvm/llvm-project
  Commit: 40ba0942e2ab1107f83aa5a0ee5ae2980bf47b1a
      https://github.com/llvm/llvm-project/commit/40ba0942e2ab1107f83aa5a0ee5ae2980bf47b1a
  Author: Christudasan Devadasan <Christudasan.Devadasan at amd.com>
  Date:   2022-12-17 (Sat, 17 Dec 2022)

  Changed paths:
    M llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp
    M llvm/lib/Target/AMDGPU/SIDefines.h
    M llvm/lib/Target/AMDGPU/SIFrameLowering.cpp
    M llvm/lib/Target/AMDGPU/SIFrameLowering.h
    M llvm/lib/Target/AMDGPU/SIISelLowering.cpp
    M llvm/lib/Target/AMDGPU/SIInstrInfo.cpp
    M llvm/lib/Target/AMDGPU/SIInstrInfo.h
    M llvm/lib/Target/AMDGPU/SIInstructions.td
    M llvm/lib/Target/AMDGPU/SILowerSGPRSpills.cpp
    M llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.cpp
    M llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.h
    M llvm/lib/Target/AMDGPU/SIRegisterInfo.cpp
    M llvm/test/CodeGen/AMDGPU/GlobalISel/assert-align.ll
    M llvm/test/CodeGen/AMDGPU/GlobalISel/call-outgoing-stack-args.ll
    M llvm/test/CodeGen/AMDGPU/GlobalISel/image-waterfall-loop-O0.ll
    M llvm/test/CodeGen/AMDGPU/GlobalISel/localizer.ll
    M llvm/test/CodeGen/AMDGPU/abi-attribute-hints-undefined-behavior.ll
    M llvm/test/CodeGen/AMDGPU/bf16.ll
    M llvm/test/CodeGen/AMDGPU/branch-relax-spill.ll
    M llvm/test/CodeGen/AMDGPU/call-alias-register-usage-agpr.ll
    M llvm/test/CodeGen/AMDGPU/call-alias-register-usage1.ll
    M llvm/test/CodeGen/AMDGPU/callee-frame-setup.ll
    M llvm/test/CodeGen/AMDGPU/cf-loop-on-constant.ll
    M llvm/test/CodeGen/AMDGPU/collapse-endcf.ll
    M llvm/test/CodeGen/AMDGPU/control-flow-fastregalloc.ll
    M llvm/test/CodeGen/AMDGPU/cross-block-use-is-not-abi-copy.ll
    M llvm/test/CodeGen/AMDGPU/csr-sgpr-spill-live-ins.mir
    M llvm/test/CodeGen/AMDGPU/dwarf-multi-register-use-crash.ll
    M llvm/test/CodeGen/AMDGPU/fix-frame-reg-in-custom-csr-spills.ll
    M llvm/test/CodeGen/AMDGPU/flat-scratch-init.ll
    M llvm/test/CodeGen/AMDGPU/fold-reload-into-exec.mir
    M llvm/test/CodeGen/AMDGPU/fold-reload-into-m0.mir
    M llvm/test/CodeGen/AMDGPU/frame-setup-without-sgpr-to-vgpr-spills.ll
    M llvm/test/CodeGen/AMDGPU/gfx-call-non-gfx-func.ll
    M llvm/test/CodeGen/AMDGPU/gfx-callable-argument-types.ll
    M llvm/test/CodeGen/AMDGPU/gfx-callable-preserved-registers.ll
    M llvm/test/CodeGen/AMDGPU/gfx-callable-return-types.ll
    M llvm/test/CodeGen/AMDGPU/indirect-call.ll
    M llvm/test/CodeGen/AMDGPU/kernel-vgpr-spill-mubuf-with-voffset.ll
    M llvm/test/CodeGen/AMDGPU/load-constant-i16.ll
    M llvm/test/CodeGen/AMDGPU/mubuf-legalize-operands.ll
    M llvm/test/CodeGen/AMDGPU/mul24-pass-ordering.ll
    M llvm/test/CodeGen/AMDGPU/need-fp-from-vgpr-spills.ll
    M llvm/test/CodeGen/AMDGPU/no-source-locations-in-prologue.ll
    M llvm/test/CodeGen/AMDGPU/partial-sgpr-to-vgpr-spills.ll
    M llvm/test/CodeGen/AMDGPU/scc-clobbered-sgpr-to-vmem-spill.ll
    M llvm/test/CodeGen/AMDGPU/sgpr-spill-dead-frame-in-dbg-value.mir
    M llvm/test/CodeGen/AMDGPU/sgpr-spill-fi-skip-processing-stack-arg-dbg-value.mir
    M llvm/test/CodeGen/AMDGPU/sgpr-spill-no-vgprs.ll
    M llvm/test/CodeGen/AMDGPU/sgpr-spill-partially-undef.mir
    M llvm/test/CodeGen/AMDGPU/sgpr-spill-update-only-slot-indexes.ll
    M llvm/test/CodeGen/AMDGPU/sgpr-spill-vmem-large-frame.mir
    M llvm/test/CodeGen/AMDGPU/sgpr-spills-split-regalloc.ll
    M llvm/test/CodeGen/AMDGPU/si-spill-sgpr-stack.ll
    M llvm/test/CodeGen/AMDGPU/sibling-call.ll
    M llvm/test/CodeGen/AMDGPU/spill-csr-frame-ptr-reg-copy.ll
    M llvm/test/CodeGen/AMDGPU/spill-offset-calculation.ll
    M llvm/test/CodeGen/AMDGPU/spill-reg-tuple-super-reg-use.mir
    M llvm/test/CodeGen/AMDGPU/spill-scavenge-offset.ll
    M llvm/test/CodeGen/AMDGPU/spill-sgpr-csr-live-ins.mir
    M llvm/test/CodeGen/AMDGPU/spill-sgpr-stack-no-sgpr.ll
    A llvm/test/CodeGen/AMDGPU/spill-sgpr-to-virtual-vgpr.mir
    A llvm/test/CodeGen/AMDGPU/spill-vgpr-to-agpr-update-regscavenger.ll
    M llvm/test/CodeGen/AMDGPU/spill-writelane-vgprs.ll
    M llvm/test/CodeGen/AMDGPU/spill192.mir
    M llvm/test/CodeGen/AMDGPU/spill224.mir
    M llvm/test/CodeGen/AMDGPU/spill288.mir
    M llvm/test/CodeGen/AMDGPU/spill320.mir
    M llvm/test/CodeGen/AMDGPU/spill352.mir
    M llvm/test/CodeGen/AMDGPU/spill384.mir
    M llvm/test/CodeGen/AMDGPU/tail-call-amdgpu-gfx.ll
    M llvm/test/CodeGen/AMDGPU/tuple-allocation-failure.ll
    M llvm/test/CodeGen/AMDGPU/unstructured-cfg-def-use-issue.ll
    M llvm/test/CodeGen/AMDGPU/vgpr-tuple-allocation.ll
    M llvm/test/CodeGen/AMDGPU/vgpr_constant_to_sgpr.ll
    A llvm/test/CodeGen/AMDGPU/wwm-register-spill-during-regalloc.ll
    M llvm/test/CodeGen/AMDGPU/wwm-reserved-spill.ll
    M llvm/test/CodeGen/MIR/AMDGPU/machine-function-info-after-pei.ll
    M llvm/test/CodeGen/MIR/AMDGPU/machine-function-info-no-ir.mir
    M llvm/test/CodeGen/MIR/AMDGPU/machine-function-info.ll
    A llvm/test/CodeGen/MIR/AMDGPU/sgpr-for-exec-copy-invalid-reg.mir
    M llvm/test/CodeGen/MIR/AMDGPU/stack-id-assert.mir

  Log Message:
  -----------
  [AMDGPU][SILowerSGPRSpills] Spill SGPRs to virtual VGPRs

Currently, the custom SGPR spill lowering pass spills
SGPRs into physical VGPR lanes and the remaining VGPRs
are used by regalloc for vector regclass allocation.
This imposes many restrictions that we ended up with
unsuccessful SGPR spilling when there won't be enough
VGPRs and we are forced to spill the leftover into
memory during PEI. The custom spill handling during PEI
has many edge cases and often breaks the compiler time
to time.

This patch implements spilling SGPRs into virtual VGPR
lanes. Since we now split the register allocation for
SGPRs and VGPRs, the virtual registers introduced for
the spill lanes would get allocated automatically in
the subsequent regalloc invocation for VGPRs.

Spill to virtual registers will always be successful,
even in the high-pressure situations, and hence it avoids
most of the edge cases during PEI. We are now left with
only the custom SGPR spills during PEI for special registers
like the frame pointer which isn an unproblematic case.

This patch also implements the whole wave spills which
might occur if RA spills any live range of virtual registers
involved in the whole wave operations. Earlier, we had
been hand-picking registers for such machine operands.
But now with SGPR spills into virtual VGPR lanes, we are
exposing them to the allocator.

Reviewed By: arsenm

Differential Revision: https://reviews.llvm.org/D124196