[PATCH] D146761: [AMDGPU] Make buffer intrinsics pointer-valued

Thu Mar 23 14:38:34 PDT 2023

krzysz00 created this revision.
Herald added subscribers: kosarev, foad, kerbowa, steven.zhang, javed.absar, hiraditya, arichardson, tpr, dstuttard, yaxunl, jvesely, kzhuravl, arsenm, qcolombet.
Herald added a project: All.
krzysz00 requested review of this revision.
Herald added subscribers: llvm-commits, wdng.
Herald added a project: LLVM.

( See
https://discourse.llvm.org/t/representing-buffer-descriptors-in-the-amdgpu-target-call-for-suggestions/68798
for some of the context )

Currently, the intrinsics for operating on AMDGPU's buffer descriptors
take the descriptor as a <4 x i32> value. This is problematic, in that
these intrinsics cannot be analyzed like the load/store/atomic
operations they are, which weakens our ability to perform compiler
optimizations and blocks the address space 7 work done elsewhere.
Therefore, this commit:

1. Changes all intrinsics that take a 128-bit buffer resource (V#) to

take a ptr addrspace(8) instead.

2. Updates the definitions of those intrinsics to indicate that they

only operate on memory related to their arguments.

3. Updates Global ISel and SelectionDAG to, early in their operation,

lower these address space 8 pointers (which are 128-bit scalars) to
<4 x i32> vectors. This must be done because the definitions of the
resource instructions themselves (and all the code for reasoning about
them) expects a <4 x i32> value, and this cannot be changed because
assigning a register class to i128 values would cause breakages in our
SelectionDAG usage.

  - Specifically, in GlobalISel, we use G_UNMERGE/G_BUIlD_VECTOR and VECTOR_EXTRACT_ELEMENT/G_MERGE in the non-vector case, and G_{INTTOPTR,PTRTOINT} and G_BITCAST in the complex case, as we have found this produces better codegen
  - In SelectionDAG, we simply use a bitcast We have found that these produce better-looking codegen.
1. Update some of the creation of MachineMemOperands to keep a

reference to the original buffer descriptor. Designing a
PseudoSourceValue that also holds the original index/offset/... values
is future work.

6. While here, update amdgcn.s.buffer.load to have the correct memory

effects (namely, a read from its argument).

7. Define auto-upgrade from the old intrinscic forms to the new ones

that performs the transformation

%v = call T @llvm.amdgcn.*buffer*(...a, <4 x i32> %rsrc, ...b)

>
-

%.rsrc.int = bitcast <4 x i32> %rsrc to i128
%.rsrc.ptr = inttoptr i128 %.rsrc.int to ptr addrspace(8)
%v = call T @llvm.amdgcn.*buffer*(...a, ptr addrspace(8) %.rsrc.ptr, ...b)

Tests have been updated to the new intrinsic forms.
The test changes, generally, fall into

1. Changing the declarations of buffer intrinsics to their new forms
2. Updating MIR tests to account for the new types or for additional

COPY instructions that are temporarily introduced by the new
legalizations

3. Occasional drifts of a movk to the other side of a buffer operation
4. Changes to the behavior of loading a ptr addrspace(8) from an undef

location, which can now act like [???0, ???1, ???0, ???1] : <4 x i32>
instead of [??? 0, ???1, ???2, ???3] : <4 x i32>. These were often
avoided by removing such loads and replacing them with loads from the
relevant null pointer.

One of the unrelated changes, caused by the legalizaiton not being
idempotent, is llvm/test/CodeGen/AMDGPU/dagcombine-fma-fmad.ll 's lack
of a mad, which is, per discussion, fine.

This patch does not appear to affect the final generated code, except
for the introduction of additional moves at O0 in some cases.

Depends on D145441 <https://reviews.llvm.org/D145441>

Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D146761

Files:
  llvm/include/llvm/IR/IntrinsicsAMDGPU.td
  llvm/lib/IR/AutoUpgrade.cpp
  llvm/lib/Target/AMDGPU/AMDGPUISelLowering.cpp
  llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp
  llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.h
  llvm/lib/Target/AMDGPU/AMDGPURegisterBankInfo.cpp
  llvm/lib/Target/AMDGPU/SIISelLowering.cpp
  llvm/lib/Target/AMDGPU/SIISelLowering.h
  llvm/lib/Target/AMDGPU/SIInstrInfo.td
  llvm/test/Analysis/DivergenceAnalysis/AMDGPU/llvm.amdgcn.buffer.atomic.ll
  llvm/test/Analysis/LegacyDivergenceAnalysis/AMDGPU/llvm.amdgcn.buffer.atomic.ll
  llvm/test/Bitcode/upgrade-amdgpu-amdgcn-buffer-intrinsics.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/atomic_optimizations_mul_one.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/buffer-atomic-fadd.f32-no-rtn.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/buffer-atomic-fadd.f32-rtn.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/buffer-atomic-fadd.f64.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/buffer-atomic-fadd.v2f16-no-rtn.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/buffer-atomic-fadd.v2f16-rtn.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/buffer-schedule.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/fp64-atomics-gfx90a.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/legalize-llvm.amdgcn.s.buffer.load.mir
  llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.raw.buffer.atomic.add.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.raw.buffer.atomic.cmpswap.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.raw.buffer.atomic.fadd-with-ret.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.raw.buffer.atomic.fadd.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.raw.buffer.load.format.f16.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.raw.buffer.load.format.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.raw.buffer.load.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.raw.buffer.store.format.f16.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.raw.buffer.store.format.f32.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.raw.buffer.store.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.raw.tbuffer.load.f16.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.raw.tbuffer.load.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.raw.tbuffer.store.f16.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.raw.tbuffer.store.i8.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.raw.tbuffer.store.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.s.buffer.load.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.set.inactive.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.struct.buffer.atomic.add.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.struct.buffer.atomic.cmpswap.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.struct.buffer.atomic.fadd-with-ret.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.struct.buffer.atomic.fadd.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.struct.buffer.load.format.f16.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.struct.buffer.load.format.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.struct.buffer.load.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.struct.buffer.store.format.f16.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.struct.buffer.store.format.f32.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.struct.buffer.store.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.struct.tbuffer.load.f16.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.struct.tbuffer.load.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/merge-buffer-stores.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/regbankselect-amdgcn-s-buffer-load.mir
  llvm/test/CodeGen/AMDGPU/GlobalISel/regbankselect-amdgcn.raw.buffer.load.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/regbankselect-amdgcn.s.buffer.load.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/regbankselect-amdgcn.s.buffer.load.mir
  llvm/test/CodeGen/AMDGPU/GlobalISel/regbankselect-amdgcn.struct.buffer.load.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/regbankselect-amdgcn.struct.buffer.store.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/unsupported-load.ll
  llvm/test/CodeGen/AMDGPU/GlobalISel/unsupported-ptr-add.ll
  llvm/test/CodeGen/AMDGPU/amdgcn-load-offset-from-reg.ll
  llvm/test/CodeGen/AMDGPU/amdgcn.bitcast.ll
  llvm/test/CodeGen/AMDGPU/amdpal.ll
  llvm/test/CodeGen/AMDGPU/atomic-optimizer-strict-wqm.ll
  llvm/test/CodeGen/AMDGPU/atomic_optimizations_buffer.ll
  llvm/test/CodeGen/AMDGPU/atomic_optimizations_pixelshader.ll
  llvm/test/CodeGen/AMDGPU/atomic_optimizations_raw_buffer.ll
  llvm/test/CodeGen/AMDGPU/atomic_optimizations_struct_buffer.ll
  llvm/test/CodeGen/AMDGPU/bitcast-v4f16-v4i16.ll
  llvm/test/CodeGen/AMDGPU/buffer-atomic-fadd.f32-no-rtn.ll
  llvm/test/CodeGen/AMDGPU/buffer-atomic-fadd.f32-rtn.ll
  llvm/test/CodeGen/AMDGPU/buffer-atomic-fadd.f64.ll
  llvm/test/CodeGen/AMDGPU/buffer-atomic-fadd.v2f16-no-rtn.ll
  llvm/test/CodeGen/AMDGPU/buffer-atomic-fadd.v2f16-rtn.ll
  llvm/test/CodeGen/AMDGPU/buffer-intrinsics-mmo-offsets.ll
  llvm/test/CodeGen/AMDGPU/buffer-rsrc-ptr-ops.ll
  llvm/test/CodeGen/AMDGPU/buffer-schedule.ll
  llvm/test/CodeGen/AMDGPU/bug-vopc-commute.ll
  llvm/test/CodeGen/AMDGPU/cc-sgpr-limit.ll
  llvm/test/CodeGen/AMDGPU/cc-sgpr-over-limit.ll
  llvm/test/CodeGen/AMDGPU/combine-add-zext-xor.ll
  llvm/test/CodeGen/AMDGPU/constant-address-space-32bit.ll
  llvm/test/CodeGen/AMDGPU/copy_to_scc.ll
  llvm/test/CodeGen/AMDGPU/dag-divergence-atomic.ll
  llvm/test/CodeGen/AMDGPU/dagcombine-fma-fmad.ll
  llvm/test/CodeGen/AMDGPU/else.ll
  llvm/test/CodeGen/AMDGPU/extract_subvector_vec4_vec3.ll
  llvm/test/CodeGen/AMDGPU/fail-select-buffer-atomic-fadd.ll
  llvm/test/CodeGen/AMDGPU/fix-wwm-vgpr-copy.ll
  llvm/test/CodeGen/AMDGPU/fp-min-max-buffer-atomics.ll
  llvm/test/CodeGen/AMDGPU/fp64-atomics-gfx90a.ll
  llvm/test/CodeGen/AMDGPU/fp64-min-max-buffer-atomics.ll
  llvm/test/CodeGen/AMDGPU/gfx90a-enc.ll
  llvm/test/CodeGen/AMDGPU/i1-copy-from-loop.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.atomic.csub.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.atomic.fadd.gfx90a.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.atomic.fadd.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.buffer.atomic.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.buffer.load.dwordx3.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.buffer.load.format.d16.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.buffer.load.format.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.buffer.load.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.buffer.store.dwordx3.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.buffer.store.format.d16.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.buffer.store.format.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.buffer.store.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.exp.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.lds.direct.load.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.lds.param.load.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.raw.buffer.atomic.fadd.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.raw.buffer.atomic.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.raw.buffer.load.format.d16.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.raw.buffer.load.format.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.raw.buffer.load.lds.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.raw.buffer.load.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.raw.buffer.store.format.d16.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.raw.buffer.store.format.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.raw.buffer.store.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.raw.tbuffer.load.d16.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.raw.tbuffer.load.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.raw.tbuffer.store.d16.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.raw.tbuffer.store.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.s.buffer.load.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.set.inactive.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.softwqm.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.struct.buffer.atomic.fadd.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.struct.buffer.atomic.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.struct.buffer.load.format.d16.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.struct.buffer.load.format.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.struct.buffer.load.format.v3f16.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.struct.buffer.load.lds.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.struct.buffer.load.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.struct.buffer.store.format.d16.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.struct.buffer.store.format.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.struct.buffer.store.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.struct.tbuffer.load.d16.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.struct.tbuffer.load.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.struct.tbuffer.store.d16.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.struct.tbuffer.store.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.tbuffer.load.d16.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.tbuffer.load.dwordx3.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.tbuffer.load.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.tbuffer.store.d16.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.tbuffer.store.dwordx3.ll
  llvm/test/CodeGen/AMDGPU/llvm.amdgcn.tbuffer.store.ll
  llvm/test/CodeGen/AMDGPU/load-local-redundant-copies.ll
  llvm/test/CodeGen/AMDGPU/loop_exit_with_xor.ll
  llvm/test/CodeGen/AMDGPU/lower-work-group-id-intrinsics.ll
  llvm/test/CodeGen/AMDGPU/merge-store-crash.ll
  llvm/test/CodeGen/AMDGPU/merge-store-usedef.ll
  llvm/test/CodeGen/AMDGPU/mubuf-legalize-operands.ll
  llvm/test/CodeGen/AMDGPU/mubuf-shader-vgpr.ll
  llvm/test/CodeGen/AMDGPU/mubuf.ll
  llvm/test/CodeGen/AMDGPU/scheduler-subrange-crash.ll
  llvm/test/CodeGen/AMDGPU/set-inactive-wwm-overwrite.ll
  llvm/test/CodeGen/AMDGPU/set-wave-priority.ll
  llvm/test/CodeGen/AMDGPU/sgpr-copy.ll
  llvm/test/CodeGen/AMDGPU/si-annotate-cf-kill.ll
  llvm/test/CodeGen/AMDGPU/si-scheduler-exports.ll
  llvm/test/CodeGen/AMDGPU/si-scheduler.ll
  llvm/test/CodeGen/AMDGPU/si-sgpr-spill.ll
  llvm/test/CodeGen/AMDGPU/si-spill-cf.ll
  llvm/test/CodeGen/AMDGPU/si-triv-disjoint-mem-access.ll
  llvm/test/CodeGen/AMDGPU/smrd-gfx10.ll
  llvm/test/CodeGen/AMDGPU/smrd.ll
  llvm/test/CodeGen/AMDGPU/split-smrd.ll
  llvm/test/CodeGen/AMDGPU/splitkit-getsubrangeformask.ll
  llvm/test/CodeGen/AMDGPU/uniform-branch-intrinsic-cond.ll
  llvm/test/CodeGen/AMDGPU/vgpr-descriptor-waterfall-loop-idom-update.ll
  llvm/test/CodeGen/AMDGPU/vgpr-spill-emergency-stack-slot.ll
  llvm/test/CodeGen/AMDGPU/vopc_dpp.ll
  llvm/test/CodeGen/AMDGPU/wait.ll
  llvm/test/CodeGen/AMDGPU/wave32.ll
  llvm/test/CodeGen/AMDGPU/wqm.ll
  llvm/test/CodeGen/AMDGPU/wwm-reserved-spill.ll
  llvm/test/CodeGen/AMDGPU/wwm-reserved.ll
  llvm/test/CodeGen/MIR/AMDGPU/custom-pseudo-source-values.ll
  llvm/test/Transforms/EarlyCSE/AMDGPU/intrinsics.ll
  llvm/test/Transforms/InstCombine/AMDGPU/amdgcn-demanded-vector-elts-inseltpoison.ll
  llvm/test/Transforms/InstCombine/AMDGPU/amdgcn-demanded-vector-elts.ll
  llvm/test/Transforms/SROA/sroa-common-type-fail-promotion.ll
  llvm/test/Transforms/StructurizeCFG/rebuild-ssa-infinite-loop-inseltpoison.ll
  llvm/test/Transforms/StructurizeCFG/rebuild-ssa-infinite-loop.ll
  llvm/test/Verifier/AMDGPU/intrinsic-immarg.ll