[llvm] [AMDGPU] Re-fold masked hi16 pack into v_or_b32_sdwa (PR #210735)

[via llvm-commits]

michaelselehov wrote:

# [AMDGPU] Re-fold the masked high-half 16-bit pack into `v_or_b32_sdwa`

Follow-up to #206058, which added an isel pattern that selects the high-half
16-bit packing idiom `(hi << 16) | (z & 0xffff)` directly to the fused
`v_lshl_or_b32`. That pattern is a win when `z`'s upper 16 bits are already
known zero, but it left a small codegen-quality/perf hole in the common case
where they are not. This PR restores the lost fold without giving up the
#206058 win.

## The regression

When `z`'s high 16 bits are not known zero, isel first has to clear them:

```
v_and_b32   z, 0xffff, z
v_lshl_or_b32 d, hi, 16, z
```

Before #206058 this idiom stayed as a plain shift + `or`, and `SIPeepholeSDWA`
folded it into a single instruction:

```
v_or_b32_sdwa d, hi<<16, z src1_sel:WORD_0
```

The `WORD_0` source selector reads only the low 16 bits of `z`, so the `0xffff`
mask is implicit and free. `v_lshl_or_b32` has no SDWA form
(`getSDWAOp() == -1`), so once isel emits it the SDWA peephole can no longer
match the idiom: the fold is lost and the explicit `v_and` stays. In an
SDWA-heavy, VALU-bound kernel these extra ops are not hidden — a
`uavReadSpeed<char4>` microbenchmark on gfx942 regressed ~5%.

A coarse isel gate (suppressing the fused pattern) is not the right fix: for
the mask-free case `v_lshl_or_b32` is strictly one op and #206058's win is
real. The fix must recover SDWA only for the masked form.

## The fix

Add an additive peephole to `SIPeepholeSDWA` (`splitLshlOrForSDWA`), run
per-basic-block just before the existing SDWA matching. It recognizes

```
%m = V_AND_B32 0xffff, %z        ; single use
%d = V_LSHL_OR_B32 %hi, 16, %m
```

and rewrites it to

```
%s = V_LSHLREV_B32 16, %hi
%d = V_OR_B32_sdwa %s, %z src1_sel:WORD_0
```

erasing the now-dead `V_AND`.

Properties:

- **Fires only on the masked form** (`V_AND 0xffff`, single use), so the
  mask-free #206058 case is untouched — no regression there.
- **Never increases instruction count.** It is built atomically (2 ops -> 2 ops
  in isolation). The subsequent `MachineCSE` shares the `%hi << 16` shift when
  one already exists, and the generic SDWA matcher folds any source select of
  `%z` into the new `V_OR_B32_sdwa`, recovering the pre-#206058 codegen. In
  practice most sites shrink (the mask, and after CSE the shift, disappear).
- Done atomically rather than "split and hope the matcher re-folds": leaving the
  `V_AND` in place lets a competing fold win and strand an extra `V_LSHLREV`,
  which would grow the count.

Example (gfx9, from `amdgpu-codegenprepare-idiv.ll`):

```
; before
v_and_b32_e32  v1, 0xffff, v1
v_lshl_or_b32  v1, v4, 16, v1
; after
v_or_b32_sdwa  v1, v3, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_0
```

## Testing

- New MIR test `sdwa-peephole-lshl-or-hi16pack.mir`: positive cases (mask on
  either `V_AND` operand, `e32`/`e64`) and negative cases (multi-use `V_AND`,
  non-`0xffff` mask, non-16 shift).
- `check-llvm` is clean. The AMDGPU CodeGen tests affected by the change were
  regenerated; every delta is neutral or an improvement (fewer/equal VALU ops),
  with no instruction-count regression. Since the pass returns early when the
  subtarget has no SDWA, gfx11+/gfx12 codegen is unchanged.


https://github.com/llvm/llvm-project/pull/210735