[llvm-branch-commits] [llvm] [X86] Add missing v8bf16/v16bf16 VPBROADCASTW patterns without AVX512BW+VLX (PR #240805)

Emilio Cota via llvm-branch-commits llvm-branch-commits at lists.llvm.org
Tue Nov 10 04:59:00 PST 2026


https://github.com/cota updated https://github.com/llvm/llvm-project/pull/240805

>From 66c747435b3763455317f461280f167b4c6d66f7 Mon Sep 17 00:00:00 2001
From: Emilio Cota <ecg at google.com>
Date: Tue, 10 Nov 2026 12:52:54 +0000
Subject: [PATCH] [X86] Fix bf16 VBROADCAST lowering and selection without
 AVX512BW+VLX

`bf16` vector shuffles are lowered as `vXi16` shuffles, and
`combineTargetShuffle` turns `VBROADCAST(bitcast(x))` into
`bitcast(VBROADCAST(x))` whenever `x` has a legal type. If `x` is an
un-bitcast `v8bf16` value (e.g. a register live across basic blocks, a
volatile load, or an AVX-NE-CONVERT conversion result), this creates a
`v8bf16` or `v16bf16` `X86ISD::VBROADCAST` (sources wider than 128 bits are
extracted down to `v8bf16`). If the source is a load whose other uses are
simplified away later, it becomes an `X86ISD::VBROADCAST_LOAD` instead. Both
only had isel patterns under `[HasBWI, HasVLX]`:

    LLVM ERROR: Cannot select: v8bf16 = X86ISD::VBROADCAST ...

This has been latent with AVX-NE-CONVERT (LLVM 18 already fails with
`-mcpu=sierraforest`). 93d242ed5416 (#230953) made `bf16` vectors legal with
AVX512BW, which exposed it without AVX512VL, and 3fdac7de5496 (#221225) made
`v8bf16` legal with AVX, so it now fails on every AVX2 target without
AVX512BW+VLX (e.g. `-mcpu=znver2`/`znver3`).

In addition, because `v16bf16` and `v32bf16` can be illegal when `v8bf16` is
legal, the `broadcast(bitcast(src))` and `vbroadcast(bitcast(vbroadcast(src)))`
folds in `combineTargetShuffle` can create illegal `bf16` broadcasts after
type legalization. These trip "Unexpected illegal type!" with assertions and
can fail to select without them.

Add VEX `VPBROADCASTW{,Y}{rr,rm}` patterns for `v8bf16`/`v16bf16` to the
`[HasAVX2, NoVLX_Or_NoBWI]` block, mirroring the existing `f16` patterns and
the `[HasBWI, HasVLX]` `bf16` patterns in `X86InstrAVX512.td`. Scalar `bf16`
broadcast patterns are omitted because scalar `bf16` is not a legal type.
Only apply the two folds if the new broadcast type is legal.

The tests were precommitted in #240804 with only the default target and
`-mattr=+avx512bw,+avx512vl`, because the other configurations crashed. This
adds RUN lines for `+avx2`, `+avx512f`, `+avx512vl`, `+avxneconvert` and
`+avx512bw`.

Assisted-by: Antigravity
---
 llvm/lib/Target/X86/X86ISelLowering.cpp   |   9 +-
 llvm/lib/Target/X86/X86InstrSSE.td        |  11 ++
 llvm/test/CodeGen/X86/bfloat-broadcast.ll | 125 +++++++++++++++-------
 3 files changed, 104 insertions(+), 41 deletions(-)

diff --git a/llvm/lib/Target/X86/X86ISelLowering.cpp b/llvm/lib/Target/X86/X86ISelLowering.cpp
index 4d081b7ecbf43d..1e0aebae36dbb6 100644
--- a/llvm/lib/Target/X86/X86ISelLowering.cpp
+++ b/llvm/lib/Target/X86/X86ISelLowering.cpp
@@ -43643,7 +43643,10 @@ static SDValue combineTargetShuffle(SDValue N, const SDLoc &DL,
             BCVT.getScalarType().getTypeForEVT(*DAG.getContext()))) {
       EVT NewVT = EVT::getVectorVT(*DAG.getContext(), BCVT.getScalarType(),
                                    VT.getVectorNumElements());
-      return DAG.getBitcast(VT, DAG.getNode(X86ISD::VBROADCAST, DL, NewVT, BC));
+      // NewVT can be illegal even if BCVT is legal (e.g. v16bf16 vs. v8bf16).
+      if (TLI.isTypeLegal(NewVT))
+        return DAG.getBitcast(VT,
+                              DAG.getNode(X86ISD::VBROADCAST, DL, NewVT, BC));
     }
 
     // vbroadcast(bitcast(vbroadcast(src))) -> bitcast(vbroadcast(src))
@@ -43658,7 +43661,9 @@ static SDValue combineTargetShuffle(SDValue N, const SDLoc &DL,
       MVT NewVT =
           MVT::getVectorVT(BCVT.getSimpleVT().getScalarType(),
                            VT.getSizeInBits() / BCVT.getScalarSizeInBits());
-      return DAG.getBitcast(VT, DAG.getNode(X86ISD::VBROADCAST, DL, NewVT, BC));
+      if (TLI.isTypeLegal(NewVT))
+        return DAG.getBitcast(VT,
+                              DAG.getNode(X86ISD::VBROADCAST, DL, NewVT, BC));
     }
 
     // Reduce broadcast source vector to lowest 128-bits.
diff --git a/llvm/lib/Target/X86/X86InstrSSE.td b/llvm/lib/Target/X86/X86InstrSSE.td
index 95426695f7b22a..00b393376a40d9 100644
--- a/llvm/lib/Target/X86/X86InstrSSE.td
+++ b/llvm/lib/Target/X86/X86InstrSSE.td
@@ -7852,6 +7852,17 @@ let Predicates = [HasAVX2, NoVLX_Or_NoBWI] in {
             (VPBROADCASTWrr (COPY_TO_REGCLASS FR16:$src, VR128))>;
   def : Pat<(v16f16 (X86VBroadcast (f16 FR16:$src))),
             (VPBROADCASTWYrr (COPY_TO_REGCLASS FR16:$src, VR128))>;
+
+  // No scalar bf16 broadcast patterns: scalar bf16 is not a legal type.
+  def : Pat<(v8bf16 (X86VBroadcastld16 addr:$src)),
+            (VPBROADCASTWrm addr:$src)>;
+  def : Pat<(v16bf16 (X86VBroadcastld16 addr:$src)),
+            (VPBROADCASTWYrm addr:$src)>;
+
+  def : Pat<(v8bf16 (X86VBroadcast (v8bf16 VR128:$src))),
+            (VPBROADCASTWrr VR128:$src)>;
+  def : Pat<(v16bf16 (X86VBroadcast (v8bf16 VR128:$src))),
+            (VPBROADCASTWYrr VR128:$src)>;
 }
 let Predicates = [HasAVX2, NoVLX] in {
   def : Pat<(v4i32 (X86VBroadcast GR32:$src)),
diff --git a/llvm/test/CodeGen/X86/bfloat-broadcast.ll b/llvm/test/CodeGen/X86/bfloat-broadcast.ll
index 9aa00cabc85c5d..12ad6bc1890358 100644
--- a/llvm/test/CodeGen/X86/bfloat-broadcast.ll
+++ b/llvm/test/CodeGen/X86/bfloat-broadcast.ll
@@ -1,12 +1,18 @@
 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
 ; RUN: llc < %s -mtriple=x86_64-- | FileCheck %s --check-prefixes=SSE
-; RUN: llc < %s -mtriple=x86_64-- -mattr=+avx512bw,+avx512vl | FileCheck %s --check-prefixes=AVX
+; RUN: llc < %s -mtriple=x86_64-- -mattr=+avx2 | FileCheck %s --check-prefixes=AVX,AVX2
+; RUN: llc < %s -mtriple=x86_64-- -mattr=+avx512f | FileCheck %s --check-prefixes=AVX,AVX2
+; RUN: llc < %s -mtriple=x86_64-- -mattr=+avx512vl | FileCheck %s --check-prefixes=AVX,AVX2
+; RUN: llc < %s -mtriple=x86_64-- -mattr=+avxneconvert | FileCheck %s --check-prefixes=AVX,V16BF16
+; RUN: llc < %s -mtriple=x86_64-- -mattr=+avx512bw | FileCheck %s --check-prefixes=AVX,V16BF16
+; RUN: llc < %s -mtriple=x86_64-- -mattr=+avx512bw,+avx512vl | FileCheck %s --check-prefixes=AVX,V16BF16
 
 ;; bf16 splats are lowered as vXi16 shuffles, but combineTargetShuffle turns
 ;; VBROADCAST(bitcast(x)) into bitcast(VBROADCAST(x)) when x is a legal bf16
 ;; vector, e.g. a value that is live across basic blocks or a volatile load.
-;; FIXME: AVX2 targets without AVX512BW+VLX crash on these, so they are not
-;; tested yet.
+;; On AVX2 targets without AVX512BW+VLX, the resulting bf16 VBROADCAST used to
+;; fail to select. v16bf16 is only legal with AVX-NE-CONVERT or AVX512BW
+;; (V16BF16); otherwise it is split into v8bf16 halves (AVX2).
 
 define void @splat_v8bf16_cross_block(<8 x bfloat> %v, i1 %c, ptr %p) nounwind {
 ; SSE-LABEL: splat_v8bf16_cross_block:
@@ -75,16 +81,27 @@ define void @splat_v16bf16_from_v8bf16_cross_block(<8 x bfloat> %v, i1 %c, ptr %
 ; SSE-NEXT:  .LBB1_2: # %exit
 ; SSE-NEXT:    retq
 ;
-; AVX-LABEL: splat_v16bf16_from_v8bf16_cross_block:
-; AVX:       # %bb.0: # %entry
-; AVX-NEXT:    testb $1, %dil
-; AVX-NEXT:    je .LBB1_2
-; AVX-NEXT:  # %bb.1: # %splat
-; AVX-NEXT:    vpbroadcastw %xmm0, %ymm0
-; AVX-NEXT:    vmovdqa %ymm0, (%rsi)
-; AVX-NEXT:  .LBB1_2: # %exit
-; AVX-NEXT:    vzeroupper
-; AVX-NEXT:    retq
+; AVX2-LABEL: splat_v16bf16_from_v8bf16_cross_block:
+; AVX2:       # %bb.0: # %entry
+; AVX2-NEXT:    testb $1, %dil
+; AVX2-NEXT:    je .LBB1_2
+; AVX2-NEXT:  # %bb.1: # %splat
+; AVX2-NEXT:    vpbroadcastw %xmm0, %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, 16(%rsi)
+; AVX2-NEXT:    vmovdqa %xmm0, (%rsi)
+; AVX2-NEXT:  .LBB1_2: # %exit
+; AVX2-NEXT:    retq
+;
+; V16BF16-LABEL: splat_v16bf16_from_v8bf16_cross_block:
+; V16BF16:       # %bb.0: # %entry
+; V16BF16-NEXT:    testb $1, %dil
+; V16BF16-NEXT:    je .LBB1_2
+; V16BF16-NEXT:  # %bb.1: # %splat
+; V16BF16-NEXT:    vpbroadcastw %xmm0, %ymm0
+; V16BF16-NEXT:    vmovdqa %ymm0, (%rsi)
+; V16BF16-NEXT:  .LBB1_2: # %exit
+; V16BF16-NEXT:    vzeroupper
+; V16BF16-NEXT:    retq
 entry:
   br i1 %c, label %splat, label %exit
 splat:
@@ -123,16 +140,28 @@ define void @splat_v16bf16_cross_block(<16 x bfloat> %v, i1 %c, ptr %p) nounwind
 ; SSE-NEXT:  .LBB2_2: # %exit
 ; SSE-NEXT:    retq
 ;
-; AVX-LABEL: splat_v16bf16_cross_block:
-; AVX:       # %bb.0: # %entry
-; AVX-NEXT:    testb $1, %dil
-; AVX-NEXT:    je .LBB2_2
-; AVX-NEXT:  # %bb.1: # %splat
-; AVX-NEXT:    vpbroadcastw %xmm0, %ymm0
-; AVX-NEXT:    vmovdqa %ymm0, (%rsi)
-; AVX-NEXT:  .LBB2_2: # %exit
-; AVX-NEXT:    vzeroupper
-; AVX-NEXT:    retq
+; AVX2-LABEL: splat_v16bf16_cross_block:
+; AVX2:       # %bb.0: # %entry
+; AVX2-NEXT:    testb $1, %dil
+; AVX2-NEXT:    je .LBB2_2
+; AVX2-NEXT:  # %bb.1: # %splat
+; AVX2-NEXT:    vpbroadcastw %xmm0, %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, 16(%rsi)
+; AVX2-NEXT:    vmovdqa %xmm0, (%rsi)
+; AVX2-NEXT:  .LBB2_2: # %exit
+; AVX2-NEXT:    vzeroupper
+; AVX2-NEXT:    retq
+;
+; V16BF16-LABEL: splat_v16bf16_cross_block:
+; V16BF16:       # %bb.0: # %entry
+; V16BF16-NEXT:    testb $1, %dil
+; V16BF16-NEXT:    je .LBB2_2
+; V16BF16-NEXT:  # %bb.1: # %splat
+; V16BF16-NEXT:    vpbroadcastw %xmm0, %ymm0
+; V16BF16-NEXT:    vmovdqa %ymm0, (%rsi)
+; V16BF16-NEXT:  .LBB2_2: # %exit
+; V16BF16-NEXT:    vzeroupper
+; V16BF16-NEXT:    retq
 entry:
   br i1 %c, label %splat, label %exit
 splat:
@@ -254,14 +283,22 @@ define void @splat_v16bf16_broadcast_load(ptr %p, <8 x bfloat> %other, ptr %dst1
 ; SSE-NEXT:    movw %ax, 2(%rdx)
 ; SSE-NEXT:    retq
 ;
-; AVX-LABEL: splat_v16bf16_broadcast_load:
-; AVX:       # %bb.0:
-; AVX-NEXT:    vpbroadcastw (%rdi), %ymm1
-; AVX-NEXT:    vmovdqa %ymm1, (%rsi)
-; AVX-NEXT:    vmovddup {{.*#+}} xmm0 = xmm0[0,0]
-; AVX-NEXT:    vmovaps %xmm0, (%rdx)
-; AVX-NEXT:    vzeroupper
-; AVX-NEXT:    retq
+; AVX2-LABEL: splat_v16bf16_broadcast_load:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    vpbroadcastw (%rdi), %xmm1
+; AVX2-NEXT:    vmovdqa %xmm1, 16(%rsi)
+; AVX2-NEXT:    vmovddup {{.*#+}} xmm0 = xmm0[0,0]
+; AVX2-NEXT:    vmovaps %xmm0, (%rdx)
+; AVX2-NEXT:    retq
+;
+; V16BF16-LABEL: splat_v16bf16_broadcast_load:
+; V16BF16:       # %bb.0:
+; V16BF16-NEXT:    vpbroadcastw (%rdi), %ymm1
+; V16BF16-NEXT:    vmovdqa %ymm1, (%rsi)
+; V16BF16-NEXT:    vmovddup {{.*#+}} xmm0 = xmm0[0,0]
+; V16BF16-NEXT:    vmovaps %xmm0, (%rdx)
+; V16BF16-NEXT:    vzeroupper
+; V16BF16-NEXT:    retq
   %ld = load <8 x bfloat>, ptr %p
   %shuf = shufflevector <8 x bfloat> %ld, <8 x bfloat> %other, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 8, i32 9, i32 10, i32 11>
   %splat0 = shufflevector <8 x bfloat> %shuf, <8 x bfloat> poison, <8 x i32> zeroinitializer
@@ -338,14 +375,24 @@ define void @rebroadcast_v8i32_from_v8bf16(ptr %src, ptr %dst1, ptr %dst2) nounw
 ; SSE-NEXT:    movdqa %xmm0, (%rdx)
 ; SSE-NEXT:    retq
 ;
-; AVX-LABEL: rebroadcast_v8i32_from_v8bf16:
-; AVX:       # %bb.0:
-; AVX-NEXT:    vmovdqa (%rdi), %xmm0
-; AVX-NEXT:    vpbroadcastw %xmm0, %ymm0
-; AVX-NEXT:    vmovdqa %xmm0, (%rsi)
-; AVX-NEXT:    vmovdqa %ymm0, (%rdx)
-; AVX-NEXT:    vzeroupper
-; AVX-NEXT:    retq
+; AVX2-LABEL: rebroadcast_v8i32_from_v8bf16:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    vmovdqa (%rdi), %xmm0
+; AVX2-NEXT:    vpbroadcastw %xmm0, %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, (%rsi)
+; AVX2-NEXT:    vpbroadcastd %xmm0, %ymm0
+; AVX2-NEXT:    vmovdqa %ymm0, (%rdx)
+; AVX2-NEXT:    vzeroupper
+; AVX2-NEXT:    retq
+;
+; V16BF16-LABEL: rebroadcast_v8i32_from_v8bf16:
+; V16BF16:       # %bb.0:
+; V16BF16-NEXT:    vmovdqa (%rdi), %xmm0
+; V16BF16-NEXT:    vpbroadcastw %xmm0, %ymm0
+; V16BF16-NEXT:    vmovdqa %xmm0, (%rsi)
+; V16BF16-NEXT:    vmovdqa %ymm0, (%rdx)
+; V16BF16-NEXT:    vzeroupper
+; V16BF16-NEXT:    retq
   %v = load volatile <8 x bfloat>, ptr %src
   %s8 = shufflevector <8 x bfloat> %v, <8 x bfloat> poison, <8 x i32> zeroinitializer
   store <8 x bfloat> %s8, ptr %dst1



More information about the llvm-branch-commits mailing list