[all-commits] [llvm/llvm-project] 004d59: [X86][CodeGen] Support partial-reduce dot products...

[Zihao Wang via All-commits]

  Branch: refs/heads/main
  Home:   https://github.com/llvm/llvm-project
  Commit: 004d5960e8e187b1e2b4a55e6221399aa076c258
      https://github.com/llvm/llvm-project/commit/004d5960e8e187b1e2b4a55e6221399aa076c258
  Author: Zihao Wang <rekind133 at outlook.com>
  Date:   2026-09-14 (Mon, 14 Sep 2026)

  Changed paths:
    M llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp
    M llvm/lib/Target/X86/X86ISelLowering.cpp
    M llvm/lib/Target/X86/X86TargetTransformInfo.cpp
    M llvm/lib/Target/X86/X86TargetTransformInfo.h
    A llvm/test/CodeGen/X86/vector-partial-reduce-fmla-bf16.ll
    A llvm/test/CodeGen/X86/vector-partial-reduce-fmla-f16.ll
    A llvm/test/CodeGen/X86/vector-partial-reduce-smla.ll
    A llvm/test/CodeGen/X86/vector-partial-reduce-sumla.ll
    A llvm/test/CodeGen/X86/vector-partial-reduce-umla.ll
    A llvm/test/Transforms/LoopVectorize/X86/CostModel/partial-reduce-dot-product.ll

  Log Message:
  -----------
  [X86][CodeGen] Support partial-reduce dot products (#205373)

Part of #187779 

Lower the llvm.vector.partial.reduce dot product shapes onto the X86 dot
product instructions:

    i8 x i8 -> i32:     vpdpbusd (AVX512-VNNI, AVX-VNNI)
                        vpdpbssd / vpdpbuud (AVX-VNNI-INT8, AVX10.2)
    i16 x i16 -> i32:   vpdpwssd (AVX512-VNNI, AVX-VNNI)
                        vpdpwsud / vpdpwuud (AVX-VNNI-INT16, AVX10.2)
    bf16 x bf16 -> f32: vdpbf16ps (AVX512-BF16)
    f16 x f16 -> f32:   vdpphps (AVX10.2)

The 128/256-bit shapes, and the 512-bit shapes on targets using 512-bit
registers, are registered as Custom. 512-bit shapes for which only the
VEX
encoding exists are split into two 256-bit dot products, and the
128/256-bit
shapes are widened to 512 bits on targets without VLX.

Teach X86TTIImpl::getPartialReductionCost about these shapes, costing
one
dot product per legal accumulator vector and otherwise falling back to
the
generic expansion cost, so that LoopVectorize can form partial
reductions
for them.

The DAG combiner forms partial reductions with narrower-than-legal types
when the widened shape is supported, so also widen the integer
PARTIAL_REDUCE_*MLA results the same way as PARTIAL_REDUCE_FMLA (expand,
then widen).



To unsubscribe from these emails, change your notification settings at https://github.com/llvm/llvm-project/settings/notifications