[llvm-branch-commits] [llvm] [X86] Add missing v8bf16/v16bf16 VPBROADCASTW patterns without AVX512BW+VLX (PR #240805)
Emilio Cota via llvm-branch-commits
llvm-branch-commits at lists.llvm.org
Tue Nov 10 04:59:00 PST 2026
https://github.com/cota updated https://github.com/llvm/llvm-project/pull/240805
>From 66c747435b3763455317f461280f167b4c6d66f7 Mon Sep 17 00:00:00 2001
From: Emilio Cota <ecg at google.com>
Date: Tue, 10 Nov 2026 12:52:54 +0000
Subject: [PATCH] [X86] Fix bf16 VBROADCAST lowering and selection without
AVX512BW+VLX
`bf16` vector shuffles are lowered as `vXi16` shuffles, and
`combineTargetShuffle` turns `VBROADCAST(bitcast(x))` into
`bitcast(VBROADCAST(x))` whenever `x` has a legal type. If `x` is an
un-bitcast `v8bf16` value (e.g. a register live across basic blocks, a
volatile load, or an AVX-NE-CONVERT conversion result), this creates a
`v8bf16` or `v16bf16` `X86ISD::VBROADCAST` (sources wider than 128 bits are
extracted down to `v8bf16`). If the source is a load whose other uses are
simplified away later, it becomes an `X86ISD::VBROADCAST_LOAD` instead. Both
only had isel patterns under `[HasBWI, HasVLX]`:
LLVM ERROR: Cannot select: v8bf16 = X86ISD::VBROADCAST ...
This has been latent with AVX-NE-CONVERT (LLVM 18 already fails with
`-mcpu=sierraforest`). 93d242ed5416 (#230953) made `bf16` vectors legal with
AVX512BW, which exposed it without AVX512VL, and 3fdac7de5496 (#221225) made
`v8bf16` legal with AVX, so it now fails on every AVX2 target without
AVX512BW+VLX (e.g. `-mcpu=znver2`/`znver3`).
In addition, because `v16bf16` and `v32bf16` can be illegal when `v8bf16` is
legal, the `broadcast(bitcast(src))` and `vbroadcast(bitcast(vbroadcast(src)))`
folds in `combineTargetShuffle` can create illegal `bf16` broadcasts after
type legalization. These trip "Unexpected illegal type!" with assertions and
can fail to select without them.
Add VEX `VPBROADCASTW{,Y}{rr,rm}` patterns for `v8bf16`/`v16bf16` to the
`[HasAVX2, NoVLX_Or_NoBWI]` block, mirroring the existing `f16` patterns and
the `[HasBWI, HasVLX]` `bf16` patterns in `X86InstrAVX512.td`. Scalar `bf16`
broadcast patterns are omitted because scalar `bf16` is not a legal type.
Only apply the two folds if the new broadcast type is legal.
The tests were precommitted in #240804 with only the default target and
`-mattr=+avx512bw,+avx512vl`, because the other configurations crashed. This
adds RUN lines for `+avx2`, `+avx512f`, `+avx512vl`, `+avxneconvert` and
`+avx512bw`.
Assisted-by: Antigravity
---
llvm/lib/Target/X86/X86ISelLowering.cpp | 9 +-
llvm/lib/Target/X86/X86InstrSSE.td | 11 ++
llvm/test/CodeGen/X86/bfloat-broadcast.ll | 125 +++++++++++++++-------
3 files changed, 104 insertions(+), 41 deletions(-)
diff --git a/llvm/lib/Target/X86/X86ISelLowering.cpp b/llvm/lib/Target/X86/X86ISelLowering.cpp
index 4d081b7ecbf43d..1e0aebae36dbb6 100644
--- a/llvm/lib/Target/X86/X86ISelLowering.cpp
+++ b/llvm/lib/Target/X86/X86ISelLowering.cpp
@@ -43643,7 +43643,10 @@ static SDValue combineTargetShuffle(SDValue N, const SDLoc &DL,
BCVT.getScalarType().getTypeForEVT(*DAG.getContext()))) {
EVT NewVT = EVT::getVectorVT(*DAG.getContext(), BCVT.getScalarType(),
VT.getVectorNumElements());
- return DAG.getBitcast(VT, DAG.getNode(X86ISD::VBROADCAST, DL, NewVT, BC));
+ // NewVT can be illegal even if BCVT is legal (e.g. v16bf16 vs. v8bf16).
+ if (TLI.isTypeLegal(NewVT))
+ return DAG.getBitcast(VT,
+ DAG.getNode(X86ISD::VBROADCAST, DL, NewVT, BC));
}
// vbroadcast(bitcast(vbroadcast(src))) -> bitcast(vbroadcast(src))
@@ -43658,7 +43661,9 @@ static SDValue combineTargetShuffle(SDValue N, const SDLoc &DL,
MVT NewVT =
MVT::getVectorVT(BCVT.getSimpleVT().getScalarType(),
VT.getSizeInBits() / BCVT.getScalarSizeInBits());
- return DAG.getBitcast(VT, DAG.getNode(X86ISD::VBROADCAST, DL, NewVT, BC));
+ if (TLI.isTypeLegal(NewVT))
+ return DAG.getBitcast(VT,
+ DAG.getNode(X86ISD::VBROADCAST, DL, NewVT, BC));
}
// Reduce broadcast source vector to lowest 128-bits.
diff --git a/llvm/lib/Target/X86/X86InstrSSE.td b/llvm/lib/Target/X86/X86InstrSSE.td
index 95426695f7b22a..00b393376a40d9 100644
--- a/llvm/lib/Target/X86/X86InstrSSE.td
+++ b/llvm/lib/Target/X86/X86InstrSSE.td
@@ -7852,6 +7852,17 @@ let Predicates = [HasAVX2, NoVLX_Or_NoBWI] in {
(VPBROADCASTWrr (COPY_TO_REGCLASS FR16:$src, VR128))>;
def : Pat<(v16f16 (X86VBroadcast (f16 FR16:$src))),
(VPBROADCASTWYrr (COPY_TO_REGCLASS FR16:$src, VR128))>;
+
+ // No scalar bf16 broadcast patterns: scalar bf16 is not a legal type.
+ def : Pat<(v8bf16 (X86VBroadcastld16 addr:$src)),
+ (VPBROADCASTWrm addr:$src)>;
+ def : Pat<(v16bf16 (X86VBroadcastld16 addr:$src)),
+ (VPBROADCASTWYrm addr:$src)>;
+
+ def : Pat<(v8bf16 (X86VBroadcast (v8bf16 VR128:$src))),
+ (VPBROADCASTWrr VR128:$src)>;
+ def : Pat<(v16bf16 (X86VBroadcast (v8bf16 VR128:$src))),
+ (VPBROADCASTWYrr VR128:$src)>;
}
let Predicates = [HasAVX2, NoVLX] in {
def : Pat<(v4i32 (X86VBroadcast GR32:$src)),
diff --git a/llvm/test/CodeGen/X86/bfloat-broadcast.ll b/llvm/test/CodeGen/X86/bfloat-broadcast.ll
index 9aa00cabc85c5d..12ad6bc1890358 100644
--- a/llvm/test/CodeGen/X86/bfloat-broadcast.ll
+++ b/llvm/test/CodeGen/X86/bfloat-broadcast.ll
@@ -1,12 +1,18 @@
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
; RUN: llc < %s -mtriple=x86_64-- | FileCheck %s --check-prefixes=SSE
-; RUN: llc < %s -mtriple=x86_64-- -mattr=+avx512bw,+avx512vl | FileCheck %s --check-prefixes=AVX
+; RUN: llc < %s -mtriple=x86_64-- -mattr=+avx2 | FileCheck %s --check-prefixes=AVX,AVX2
+; RUN: llc < %s -mtriple=x86_64-- -mattr=+avx512f | FileCheck %s --check-prefixes=AVX,AVX2
+; RUN: llc < %s -mtriple=x86_64-- -mattr=+avx512vl | FileCheck %s --check-prefixes=AVX,AVX2
+; RUN: llc < %s -mtriple=x86_64-- -mattr=+avxneconvert | FileCheck %s --check-prefixes=AVX,V16BF16
+; RUN: llc < %s -mtriple=x86_64-- -mattr=+avx512bw | FileCheck %s --check-prefixes=AVX,V16BF16
+; RUN: llc < %s -mtriple=x86_64-- -mattr=+avx512bw,+avx512vl | FileCheck %s --check-prefixes=AVX,V16BF16
;; bf16 splats are lowered as vXi16 shuffles, but combineTargetShuffle turns
;; VBROADCAST(bitcast(x)) into bitcast(VBROADCAST(x)) when x is a legal bf16
;; vector, e.g. a value that is live across basic blocks or a volatile load.
-;; FIXME: AVX2 targets without AVX512BW+VLX crash on these, so they are not
-;; tested yet.
+;; On AVX2 targets without AVX512BW+VLX, the resulting bf16 VBROADCAST used to
+;; fail to select. v16bf16 is only legal with AVX-NE-CONVERT or AVX512BW
+;; (V16BF16); otherwise it is split into v8bf16 halves (AVX2).
define void @splat_v8bf16_cross_block(<8 x bfloat> %v, i1 %c, ptr %p) nounwind {
; SSE-LABEL: splat_v8bf16_cross_block:
@@ -75,16 +81,27 @@ define void @splat_v16bf16_from_v8bf16_cross_block(<8 x bfloat> %v, i1 %c, ptr %
; SSE-NEXT: .LBB1_2: # %exit
; SSE-NEXT: retq
;
-; AVX-LABEL: splat_v16bf16_from_v8bf16_cross_block:
-; AVX: # %bb.0: # %entry
-; AVX-NEXT: testb $1, %dil
-; AVX-NEXT: je .LBB1_2
-; AVX-NEXT: # %bb.1: # %splat
-; AVX-NEXT: vpbroadcastw %xmm0, %ymm0
-; AVX-NEXT: vmovdqa %ymm0, (%rsi)
-; AVX-NEXT: .LBB1_2: # %exit
-; AVX-NEXT: vzeroupper
-; AVX-NEXT: retq
+; AVX2-LABEL: splat_v16bf16_from_v8bf16_cross_block:
+; AVX2: # %bb.0: # %entry
+; AVX2-NEXT: testb $1, %dil
+; AVX2-NEXT: je .LBB1_2
+; AVX2-NEXT: # %bb.1: # %splat
+; AVX2-NEXT: vpbroadcastw %xmm0, %xmm0
+; AVX2-NEXT: vmovdqa %xmm0, 16(%rsi)
+; AVX2-NEXT: vmovdqa %xmm0, (%rsi)
+; AVX2-NEXT: .LBB1_2: # %exit
+; AVX2-NEXT: retq
+;
+; V16BF16-LABEL: splat_v16bf16_from_v8bf16_cross_block:
+; V16BF16: # %bb.0: # %entry
+; V16BF16-NEXT: testb $1, %dil
+; V16BF16-NEXT: je .LBB1_2
+; V16BF16-NEXT: # %bb.1: # %splat
+; V16BF16-NEXT: vpbroadcastw %xmm0, %ymm0
+; V16BF16-NEXT: vmovdqa %ymm0, (%rsi)
+; V16BF16-NEXT: .LBB1_2: # %exit
+; V16BF16-NEXT: vzeroupper
+; V16BF16-NEXT: retq
entry:
br i1 %c, label %splat, label %exit
splat:
@@ -123,16 +140,28 @@ define void @splat_v16bf16_cross_block(<16 x bfloat> %v, i1 %c, ptr %p) nounwind
; SSE-NEXT: .LBB2_2: # %exit
; SSE-NEXT: retq
;
-; AVX-LABEL: splat_v16bf16_cross_block:
-; AVX: # %bb.0: # %entry
-; AVX-NEXT: testb $1, %dil
-; AVX-NEXT: je .LBB2_2
-; AVX-NEXT: # %bb.1: # %splat
-; AVX-NEXT: vpbroadcastw %xmm0, %ymm0
-; AVX-NEXT: vmovdqa %ymm0, (%rsi)
-; AVX-NEXT: .LBB2_2: # %exit
-; AVX-NEXT: vzeroupper
-; AVX-NEXT: retq
+; AVX2-LABEL: splat_v16bf16_cross_block:
+; AVX2: # %bb.0: # %entry
+; AVX2-NEXT: testb $1, %dil
+; AVX2-NEXT: je .LBB2_2
+; AVX2-NEXT: # %bb.1: # %splat
+; AVX2-NEXT: vpbroadcastw %xmm0, %xmm0
+; AVX2-NEXT: vmovdqa %xmm0, 16(%rsi)
+; AVX2-NEXT: vmovdqa %xmm0, (%rsi)
+; AVX2-NEXT: .LBB2_2: # %exit
+; AVX2-NEXT: vzeroupper
+; AVX2-NEXT: retq
+;
+; V16BF16-LABEL: splat_v16bf16_cross_block:
+; V16BF16: # %bb.0: # %entry
+; V16BF16-NEXT: testb $1, %dil
+; V16BF16-NEXT: je .LBB2_2
+; V16BF16-NEXT: # %bb.1: # %splat
+; V16BF16-NEXT: vpbroadcastw %xmm0, %ymm0
+; V16BF16-NEXT: vmovdqa %ymm0, (%rsi)
+; V16BF16-NEXT: .LBB2_2: # %exit
+; V16BF16-NEXT: vzeroupper
+; V16BF16-NEXT: retq
entry:
br i1 %c, label %splat, label %exit
splat:
@@ -254,14 +283,22 @@ define void @splat_v16bf16_broadcast_load(ptr %p, <8 x bfloat> %other, ptr %dst1
; SSE-NEXT: movw %ax, 2(%rdx)
; SSE-NEXT: retq
;
-; AVX-LABEL: splat_v16bf16_broadcast_load:
-; AVX: # %bb.0:
-; AVX-NEXT: vpbroadcastw (%rdi), %ymm1
-; AVX-NEXT: vmovdqa %ymm1, (%rsi)
-; AVX-NEXT: vmovddup {{.*#+}} xmm0 = xmm0[0,0]
-; AVX-NEXT: vmovaps %xmm0, (%rdx)
-; AVX-NEXT: vzeroupper
-; AVX-NEXT: retq
+; AVX2-LABEL: splat_v16bf16_broadcast_load:
+; AVX2: # %bb.0:
+; AVX2-NEXT: vpbroadcastw (%rdi), %xmm1
+; AVX2-NEXT: vmovdqa %xmm1, 16(%rsi)
+; AVX2-NEXT: vmovddup {{.*#+}} xmm0 = xmm0[0,0]
+; AVX2-NEXT: vmovaps %xmm0, (%rdx)
+; AVX2-NEXT: retq
+;
+; V16BF16-LABEL: splat_v16bf16_broadcast_load:
+; V16BF16: # %bb.0:
+; V16BF16-NEXT: vpbroadcastw (%rdi), %ymm1
+; V16BF16-NEXT: vmovdqa %ymm1, (%rsi)
+; V16BF16-NEXT: vmovddup {{.*#+}} xmm0 = xmm0[0,0]
+; V16BF16-NEXT: vmovaps %xmm0, (%rdx)
+; V16BF16-NEXT: vzeroupper
+; V16BF16-NEXT: retq
%ld = load <8 x bfloat>, ptr %p
%shuf = shufflevector <8 x bfloat> %ld, <8 x bfloat> %other, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 8, i32 9, i32 10, i32 11>
%splat0 = shufflevector <8 x bfloat> %shuf, <8 x bfloat> poison, <8 x i32> zeroinitializer
@@ -338,14 +375,24 @@ define void @rebroadcast_v8i32_from_v8bf16(ptr %src, ptr %dst1, ptr %dst2) nounw
; SSE-NEXT: movdqa %xmm0, (%rdx)
; SSE-NEXT: retq
;
-; AVX-LABEL: rebroadcast_v8i32_from_v8bf16:
-; AVX: # %bb.0:
-; AVX-NEXT: vmovdqa (%rdi), %xmm0
-; AVX-NEXT: vpbroadcastw %xmm0, %ymm0
-; AVX-NEXT: vmovdqa %xmm0, (%rsi)
-; AVX-NEXT: vmovdqa %ymm0, (%rdx)
-; AVX-NEXT: vzeroupper
-; AVX-NEXT: retq
+; AVX2-LABEL: rebroadcast_v8i32_from_v8bf16:
+; AVX2: # %bb.0:
+; AVX2-NEXT: vmovdqa (%rdi), %xmm0
+; AVX2-NEXT: vpbroadcastw %xmm0, %xmm0
+; AVX2-NEXT: vmovdqa %xmm0, (%rsi)
+; AVX2-NEXT: vpbroadcastd %xmm0, %ymm0
+; AVX2-NEXT: vmovdqa %ymm0, (%rdx)
+; AVX2-NEXT: vzeroupper
+; AVX2-NEXT: retq
+;
+; V16BF16-LABEL: rebroadcast_v8i32_from_v8bf16:
+; V16BF16: # %bb.0:
+; V16BF16-NEXT: vmovdqa (%rdi), %xmm0
+; V16BF16-NEXT: vpbroadcastw %xmm0, %ymm0
+; V16BF16-NEXT: vmovdqa %xmm0, (%rsi)
+; V16BF16-NEXT: vmovdqa %ymm0, (%rdx)
+; V16BF16-NEXT: vzeroupper
+; V16BF16-NEXT: retq
%v = load volatile <8 x bfloat>, ptr %src
%s8 = shufflevector <8 x bfloat> %v, <8 x bfloat> poison, <8 x i32> zeroinitializer
store <8 x bfloat> %s8, ptr %dst1
More information about the llvm-branch-commits
mailing list