[llvm] [X86] Lower zext-byte add reductions to vpsadbw (PR #201076)
via llvm-commits
llvm-commits at lists.llvm.org
Tue Jun 2 02:27:23 PDT 2026
https://github.com/mbhade-amd created https://github.com/llvm/llvm-project/pull/201076
Loops of the form `for (i) sum += bytes[i];` (`uint8_t` input, `i32`/`i64`
accumulator) lower to `vpmovzxbd` + `vpaddd` today, although `PSADBW(x, 0)`
computes the same sum in one instruction per 128/256/512-bit lane.
Teach `X86PartialReduction` to rewrite the `zext <N x i8> to <N x i32|i64>`
leaves of an add reduction (N >= 16) into `PSADBW(x, 0)`, split across
SSE2/AVX2/AVX-512BW lanes per the subtarget. `i64` accumulators consume
`PSADBW`'s natural `<N/8 x i64>` output directly.
Tests: `x86-partial-reduction-byte-sum*.ll` (matcher in isolation),
`byte-sum-{positive,negative}.ll` (full CodeGen on +sse2/+avx2/+avx512bw).
### Scope
Backend-late matcher; doesn't drive LV's VF. AVX2 default VF for an `i32`
reduction is 8, so `-vectorizer-maximize-bandwidth` is needed to reach the
N >= 16 floor. AVX-512BW (`-march=znver5`) default VF is already 16 and
the matcher fires at plain `-O3`. A cost-model fix is a sensible follow-up,
out of scope here.
>From 2b170426fa13172733ab325ed70780713fa00608 Mon Sep 17 00:00:00 2001
From: mbhade <mbhade at amd.com>
Date: Tue, 2 Jun 2026 13:20:37 +0530
Subject: [PATCH] [X86] Lower zext-byte add reductions to vpsadbw
Loops of the form `for (i) sum += bytes[i];` (uint8_t input, i32 or
i64 accumulator) currently lower to vpmovzxbd + vpaddd even though
PSADBW(x, 0) computes the same sum natively in one instruction per
128/256/512-bit lane.
Teach X86PartialReduction's matcher to rewrite the
`zext <N x i8> to <N x i32|i64>` leaves of an add reduction
(N >= 16) into PSADBW(x, 0). The rewrite splits the source into as
many SSE2/AVX2/AVX-512BW lanes as the subtarget and VF require;
i64 accumulators consume PSADBW's natural <N/8 x i64> output
directly without an intermediate i32 bitcast.
x86-partial-reduction-byte-sum*.ll isolate the matcher with
`opt -passes=x86-partial-reduction` (positive / negative / wide-i64
/ debug-location preservation). byte-sum-{positive,negative}.ll
exercise the full CodeGen pipeline with `llc` on +sse2, +avx2 and
+avx512bw.
---
llvm/lib/Target/X86/X86PartialReduction.cpp | 151 +++++-
llvm/test/CodeGen/X86/byte-sum-negative.ll | 206 ++++++++
llvm/test/CodeGen/X86/byte-sum-positive.ll | 478 ++++++++++++++++++
...x86-partial-reduction-byte-sum-debugloc.ll | 51 ++
...x86-partial-reduction-byte-sum-negative.ll | 49 ++
...x86-partial-reduction-byte-sum-wide-i64.ll | 81 +++
.../X86/x86-partial-reduction-byte-sum.ll | 34 ++
7 files changed, 1049 insertions(+), 1 deletion(-)
create mode 100644 llvm/test/CodeGen/X86/byte-sum-negative.ll
create mode 100644 llvm/test/CodeGen/X86/byte-sum-positive.ll
create mode 100644 llvm/test/CodeGen/X86/x86-partial-reduction-byte-sum-debugloc.ll
create mode 100644 llvm/test/CodeGen/X86/x86-partial-reduction-byte-sum-negative.ll
create mode 100644 llvm/test/CodeGen/X86/x86-partial-reduction-byte-sum-wide-i64.ll
create mode 100644 llvm/test/CodeGen/X86/x86-partial-reduction-byte-sum.ll
diff --git a/llvm/lib/Target/X86/X86PartialReduction.cpp b/llvm/lib/Target/X86/X86PartialReduction.cpp
index 898c83cf9b468..75dcd7cf9aaed 100644
--- a/llvm/lib/Target/X86/X86PartialReduction.cpp
+++ b/llvm/lib/Target/X86/X86PartialReduction.cpp
@@ -18,6 +18,7 @@
#include "llvm/CodeGen/TargetPassConfig.h"
#include "llvm/IR/Analysis.h"
#include "llvm/IR/Constants.h"
+#include "llvm/IR/DerivedTypes.h"
#include "llvm/IR/IRBuilder.h"
#include "llvm/IR/Instructions.h"
#include "llvm/IR/IntrinsicsX86.h"
@@ -44,6 +45,7 @@ class X86PartialReduction {
private:
bool tryMAddReplacement(Instruction *Op, bool ReduceInOneBB);
bool trySADReplacement(Instruction *Op);
+ bool tryByteSumReplacement(Instruction *Op);
};
class X86PartialReductionLegacy : public FunctionPass {
@@ -353,6 +355,141 @@ bool X86PartialReduction::trySADReplacement(Instruction *Op) {
return true;
}
+bool X86PartialReduction::tryByteSumReplacement(Instruction *Op) {
+ // Gate on SSE2.
+ if (!ST->hasSSE2())
+ return false;
+
+ // Element type must be i32 or i64.
+ // - i32 is the leaf of the canonical i32 add-reduction tree LV emits
+ // when the source code uses a uint32_t accumulator.
+ // - i64 is what LV emits directly (skipping the i32 intermediate) when
+ // the source code uses a uint64_t accumulator: see for example the
+ // `uint64_t sum += a[i:i8]` shape, where LV produces a wide
+ // `zext <N x i8> to <N x i64>` feeding an i64 add-reduce. psadbw's
+ // natural output is already i64-per-8-byte-lane, so handling this
+ // case requires only skipping the i32 bitcast below.
+ auto *OpTy = dyn_cast<FixedVectorType>(Op->getType());
+ if (!OpTy)
+ return false;
+ unsigned ElemBits = OpTy->getElementType()->getScalarSizeInBits();
+ if (ElemBits != 32 && ElemBits != 64)
+ return false;
+
+ // Must be a zext from i8.
+ auto *ZExt = dyn_cast<ZExtInst>(Op);
+ if (!ZExt)
+ return false;
+
+ auto *SrcTy = dyn_cast<FixedVectorType>(ZExt->getOperand(0)->getType());
+ if (!SrcTy || !SrcTy->getElementType()->isIntegerTy(8))
+ return false;
+
+ unsigned NumElts = OpTy->getNumElements();
+
+ // Only match shapes that are at least one native psadbw width wide.
+ // Below that (e.g. <8 x i8>) SelectionDAG's own SAD matcher already
+ // produces optimal code and padding up to 16 elements here can actually
+ // regress small cases by forcing wider horizontal reduction sequences.
+ if (NumElts < 16)
+ return false;
+
+ // Select the widest psadbw intrinsic the subtarget supports that can cover
+ // the input in whole 16/32/64-byte slices. Same dispatch shape as the
+ // sibling trySADReplacement matcher.
+ unsigned IntrinsicNumElts;
+ Intrinsic::ID IID;
+ if (ST->hasBWI() && NumElts >= 64) {
+ IID = Intrinsic::x86_avx512_psad_bw_512;
+ IntrinsicNumElts = 64;
+ } else if (ST->hasAVX2() && NumElts >= 32) {
+ IID = Intrinsic::x86_avx2_psad_bw;
+ IntrinsicNumElts = 32;
+ } else {
+ IID = Intrinsic::x86_sse2_psad_bw;
+ IntrinsicNumElts = 16;
+ }
+
+ // Reject non-divisible or non-power-of-two split counts up front so the
+ // function is side-effect-free on every bail path (in particular we do
+ // not insert an intrinsic declaration into the module if we are going to
+ // return false anyway).
+ if (NumElts % IntrinsicNumElts != 0 ||
+ !isPowerOf2_32(NumElts / IntrinsicNumElts))
+ return false;
+ unsigned NumSplits = NumElts / IntrinsicNumElts;
+
+ // Build the replacement. Carry over the original instruction's DebugLoc
+ // so the new IR keeps a useful source location under -g.
+ IRBuilder<> Builder(Op);
+ Builder.SetCurrentDebugLocation(Op->getDebugLoc());
+
+ Function *PSADBWFn = Intrinsic::getOrInsertDeclaration(Op->getModule(), IID);
+
+ // psadbw computes |a - b| per byte and then horizontally sums 8 bytes per
+ // lane. Using a zero second operand turns it into the unsigned byte sum
+ // of the first operand.
+ auto *I8VecTy = FixedVectorType::get(Builder.getInt8Ty(), IntrinsicNumElts);
+ Value *Zeroes = Constant::getNullValue(I8VecTy);
+
+ // psadbw natively returns <IntrinsicNumElts/8 x i64>. For an i32
+ // accumulator we bitcast each i64 lane to two i32 lanes so the
+ // downstream reduction tree sees the expected element width. Per-lane
+ // sums are at most 8*255 = 2040 (well within i32), so the upper i32 of
+ // every i64 is zero and the reduction sums the correct totals. For an
+ // i64 accumulator the native <.../8 x i64> shape is already correct.
+ FixedVectorType *I32PerSplitTy =
+ ElemBits == 32
+ ? FixedVectorType::get(Builder.getInt32Ty(), IntrinsicNumElts / 4)
+ : nullptr;
+
+ // Carve the input into `NumSplits` lanes of `IntrinsicNumElts` bytes,
+ // compute each lane's byte sum with psadbw(lane, 0), and bitcast back to
+ // the per-split result type chosen above.
+ Value *Src = ZExt->getOperand(0);
+ SmallVector<Value *, 4> Ops(NumSplits);
+ for (unsigned i = 0; i != NumSplits; ++i) {
+ SmallVector<int, 64> ExtractMask(IntrinsicNumElts);
+ std::iota(ExtractMask.begin(), ExtractMask.end(), i * IntrinsicNumElts);
+ Value *ExtractSrc = Builder.CreateShuffleVector(Src, Src, ExtractMask);
+ Ops[i] = Builder.CreateCall(PSADBWFn, {ExtractSrc, Zeroes});
+ if (I32PerSplitTy)
+ Ops[i] = Builder.CreateBitCast(Ops[i], I32PerSplitTy);
+ }
+
+ // Concat the per-split results back into a single wide vector using a
+ // pairwise log2(NumSplits) shuffle tree, mirroring trySADReplacement.
+ unsigned Stages = Log2_32(NumSplits);
+ for (unsigned S = Stages; S > 0; --S) {
+ unsigned NumConcatElts =
+ cast<FixedVectorType>(Ops[0]->getType())->getNumElements() * 2;
+ for (unsigned i = 0; i != 1U << (S - 1); ++i) {
+ SmallVector<int, 64> ConcatMask(NumConcatElts);
+ std::iota(ConcatMask.begin(), ConcatMask.end(), 0);
+ Ops[i] =
+ Builder.CreateShuffleVector(Ops[i * 2], Ops[i * 2 + 1], ConcatMask);
+ }
+ }
+
+ // After the psadbw collapse, the result has fewer lanes than the original
+ // value `Op` had. Pad with zeros so the type matches what downstream uses
+ // of `Op` expect (the existing reduction tree shrinks them again). The
+ // early-out above guarantees NumElts >= 16, so the result always strictly
+ // exceeds the per-split width and needs the zero pad below.
+ SmallVector<int, 32> ConcatMask(NumElts);
+ unsigned SubElts = cast<FixedVectorType>(Ops[0]->getType())->getNumElements();
+ for (unsigned i = 0; i != SubElts; ++i)
+ ConcatMask[i] = i;
+ for (unsigned i = SubElts; i != NumElts; ++i)
+ ConcatMask[i] = (i % SubElts) + SubElts;
+ Value *Zero = Constant::getNullValue(Ops[0]->getType());
+ Ops[0] = Builder.CreateShuffleVector(Ops[0], Zero, ConcatMask);
+
+ Op->replaceAllUsesWith(Ops[0]);
+ Op->eraseFromParent();
+ return true;
+}
+
// Walk backwards from the ExtractElementInst and determine if it is the end of
// a horizontal reduction. Return the input to the reduction if we find one.
static Value *matchAddReduction(const ExtractElementInst &EE,
@@ -530,8 +667,20 @@ bool X86PartialReduction::run(Function &F) {
// Don't do SAD matching on the root node. SelectionDAG already
// has support for that and currently generates better code.
- if (I != Root && trySADReplacement(I))
+ if (I != Root && trySADReplacement(I)) {
MadeChange = true;
+ continue;
+ }
+
+ // Byte sum via psadbw(x, 0). Same rationale as trySADReplacement:
+ // don't match on the root node because SelectionDAG already handles
+ // small single-vector patterns and generally emits better code for
+ // them. We only help on wider intermediate shapes that reach us
+ // from loop vectorization.
+ if (I != Root && tryByteSumReplacement(I)) {
+ MadeChange = true;
+ continue;
+ }
}
}
}
diff --git a/llvm/test/CodeGen/X86/byte-sum-negative.ll b/llvm/test/CodeGen/X86/byte-sum-negative.ll
new file mode 100644
index 0000000000000..41e5ed69f5aac
--- /dev/null
+++ b/llvm/test/CodeGen/X86/byte-sum-negative.ll
@@ -0,0 +1,206 @@
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx2 | FileCheck %s --check-prefix=NO-PSADBW
+
+; Negative CodeGen tests: must NOT lower to vpsadbw via tryByteSumReplacement.
+
+ at a = dso_local global [1024 x i8] zeroinitializer, align 16
+
+define dso_local i32 @byte_sum_v16_i32_sext() nounwind {
+; NO-PSADBW-LABEL: byte_sum_v16_i32_sext:
+entry:
+ br label %vector.body
+
+vector.body:
+ %index = phi i64 [ 0, %entry ], [ %index.next, %vector.body ]
+ %vec.phi = phi <16 x i32> [ zeroinitializer, %entry ], [ %add, %vector.body ]
+ %p = getelementptr inbounds [1024 x i8], ptr @a, i64 0, i64 %index
+ %wide.load = load <16 x i8>, ptr %p, align 16
+ %z = sext <16 x i8> %wide.load to <16 x i32>
+ %add = add nsw <16 x i32> %z, %vec.phi
+ %index.next = add i64 %index, 16
+ %cmp = icmp eq i64 %index.next, 1024
+ br i1 %cmp, label %middle.block, label %vector.body
+
+middle.block:
+ %s1 = shufflevector <16 x i32> %add, <16 x i32> undef, <16 x i32> <i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r1 = add <16 x i32> %add, %s1
+ %s2 = shufflevector <16 x i32> %r1, <16 x i32> undef, <16 x i32> <i32 4, i32 5, i32 6, i32 7, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r2 = add <16 x i32> %r1, %s2
+ %s3 = shufflevector <16 x i32> %r2, <16 x i32> undef, <16 x i32> <i32 2, i32 3, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r3 = add <16 x i32> %r2, %s3
+ %s4 = shufflevector <16 x i32> %r3, <16 x i32> undef, <16 x i32> <i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r4 = add <16 x i32> %r3, %s4
+ %ext = extractelement <16 x i32> %r4, i32 0
+ ret i32 %ext
+; NO-PSADBW-NOT: psadbw
+}
+
+;----------------------------------------------------------------------------
+; NEGATIVE: i16 source instead of i8.
+;
+; psadbw operates on bytes; widening from i16 is the wrong shape so the pass
+; must bail. Nothing here looks like a SAD/MAdd either, so SDAG also will
+; not invent a psadbw.
+;----------------------------------------------------------------------------
+define dso_local i32 @word_sum_v16_i32(ptr nocapture readonly %src) nounwind {
+; NO-PSADBW-LABEL: word_sum_v16_i32:
+entry:
+ br label %vector.body
+
+vector.body:
+ %index = phi i64 [ 0, %entry ], [ %index.next, %vector.body ]
+ %vec.phi = phi <16 x i32> [ zeroinitializer, %entry ], [ %add, %vector.body ]
+ %p = getelementptr inbounds i16, ptr %src, i64 %index
+ %wide.load = load <16 x i16>, ptr %p, align 2
+ %z = zext <16 x i16> %wide.load to <16 x i32>
+ %add = add nsw <16 x i32> %z, %vec.phi
+ %index.next = add i64 %index, 16
+ %cmp = icmp eq i64 %index.next, 1024
+ br i1 %cmp, label %middle.block, label %vector.body
+
+middle.block:
+ %s1 = shufflevector <16 x i32> %add, <16 x i32> undef, <16 x i32> <i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r1 = add <16 x i32> %add, %s1
+ %s2 = shufflevector <16 x i32> %r1, <16 x i32> undef, <16 x i32> <i32 4, i32 5, i32 6, i32 7, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r2 = add <16 x i32> %r1, %s2
+ %s3 = shufflevector <16 x i32> %r2, <16 x i32> undef, <16 x i32> <i32 2, i32 3, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r3 = add <16 x i32> %r2, %s3
+ %s4 = shufflevector <16 x i32> %r3, <16 x i32> undef, <16 x i32> <i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r4 = add <16 x i32> %r3, %s4
+ %ext = extractelement <16 x i32> %r4, i32 0
+ ret i32 %ext
+; NO-PSADBW-NOT: psadbw
+}
+
+;----------------------------------------------------------------------------
+; NEGATIVE: i16 accumulator (element type < 32 bits).
+;
+; psadbw produces i64 partial sums; the rewrite needs at least i32 lanes to
+; hold them. An i16 reduction must not be rewritten.
+;----------------------------------------------------------------------------
+define dso_local i16 @byte_sum_v16_i16() nounwind {
+; NO-PSADBW-LABEL: byte_sum_v16_i16:
+entry:
+ br label %vector.body
+
+vector.body:
+ %index = phi i64 [ 0, %entry ], [ %index.next, %vector.body ]
+ %vec.phi = phi <16 x i16> [ zeroinitializer, %entry ], [ %add, %vector.body ]
+ %p = getelementptr inbounds [1024 x i8], ptr @a, i64 0, i64 %index
+ %wide.load = load <16 x i8>, ptr %p, align 16
+ %z = zext <16 x i8> %wide.load to <16 x i16>
+ %add = add nsw <16 x i16> %z, %vec.phi
+ %index.next = add i64 %index, 16
+ %cmp = icmp eq i64 %index.next, 1024
+ br i1 %cmp, label %middle.block, label %vector.body
+
+middle.block:
+ %s1 = shufflevector <16 x i16> %add, <16 x i16> undef, <16 x i32> <i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r1 = add <16 x i16> %add, %s1
+ %s2 = shufflevector <16 x i16> %r1, <16 x i16> undef, <16 x i32> <i32 4, i32 5, i32 6, i32 7, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r2 = add <16 x i16> %r1, %s2
+ %s3 = shufflevector <16 x i16> %r2, <16 x i16> undef, <16 x i32> <i32 2, i32 3, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r3 = add <16 x i16> %r2, %s3
+ %s4 = shufflevector <16 x i16> %r3, <16 x i16> undef, <16 x i32> <i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r4 = add <16 x i16> %r3, %s4
+ %ext = extractelement <16 x i16> %r4, i32 0
+ ret i16 %ext
+; NO-PSADBW-NOT: psadbw
+}
+
+;----------------------------------------------------------------------------
+; NEGATIVE: leaf is not a zext.
+;
+; The leaf operand of the reduction is an already-i32 value (no zext, no
+; underlying i8 source). tryByteSumReplacement must bail because there is
+; no way to know the values fit in a byte.
+;----------------------------------------------------------------------------
+define dso_local i32 @dword_sum_v16_i32(ptr nocapture readonly %src) nounwind {
+; NO-PSADBW-LABEL: dword_sum_v16_i32:
+entry:
+ br label %vector.body
+
+vector.body:
+ %index = phi i64 [ 0, %entry ], [ %index.next, %vector.body ]
+ %vec.phi = phi <16 x i32> [ zeroinitializer, %entry ], [ %add, %vector.body ]
+ %p = getelementptr inbounds i32, ptr %src, i64 %index
+ %wide.load = load <16 x i32>, ptr %p, align 4
+ %add = add nsw <16 x i32> %wide.load, %vec.phi
+ %index.next = add i64 %index, 16
+ %cmp = icmp eq i64 %index.next, 1024
+ br i1 %cmp, label %middle.block, label %vector.body
+
+middle.block:
+ %s1 = shufflevector <16 x i32> %add, <16 x i32> undef, <16 x i32> <i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r1 = add <16 x i32> %add, %s1
+ %s2 = shufflevector <16 x i32> %r1, <16 x i32> undef, <16 x i32> <i32 4, i32 5, i32 6, i32 7, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r2 = add <16 x i32> %r1, %s2
+ %s3 = shufflevector <16 x i32> %r2, <16 x i32> undef, <16 x i32> <i32 2, i32 3, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r3 = add <16 x i32> %r2, %s3
+ %s4 = shufflevector <16 x i32> %r3, <16 x i32> undef, <16 x i32> <i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r4 = add <16 x i32> %r3, %s4
+ %ext = extractelement <16 x i32> %r4, i32 0
+ ret i32 %ext
+; NO-PSADBW-NOT: psadbw
+}
+
+;----------------------------------------------------------------------------
+; NEGATIVE: non-power-of-two lane count (24).
+;----------------------------------------------------------------------------
+define dso_local i32 @byte_sum_v24_i32() nounwind {
+; NO-PSADBW-LABEL: byte_sum_v24_i32:
+entry:
+ br label %vector.body
+
+vector.body:
+ %index = phi i64 [ 0, %entry ], [ %index.next, %vector.body ]
+ %vec.phi = phi <24 x i32> [ zeroinitializer, %entry ], [ %add, %vector.body ]
+ %p = getelementptr inbounds [1024 x i8], ptr @a, i64 0, i64 %index
+ %wide.load = load <24 x i8>, ptr %p, align 8
+ %z = zext <24 x i8> %wide.load to <24 x i32>
+ %add = add nsw <24 x i32> %z, %vec.phi
+ %index.next = add i64 %index, 24
+ %cmp = icmp eq i64 %index.next, 1024
+ br i1 %cmp, label %middle.block, label %vector.body
+
+middle.block:
+ %ext = extractelement <24 x i32> %add, i32 0
+ ret i32 %ext
+; NO-PSADBW-NOT: psadbw
+}
+
+;----------------------------------------------------------------------------
+; NEGATIVE: VF = 8 (fewer than 16 lanes).
+;
+; tryByteSumReplacement only fires for NumElts >= 16. Padding a v8 byte
+; reduction up to 16 lanes would force an extra shuffle and a wider
+; horizontal reduction, regressing the small case that SelectionDAG handles
+; well by itself. No psadbw is expected for this shape from any of the
+; rewriters along the way.
+;----------------------------------------------------------------------------
+define dso_local i32 @byte_sum_v8_i32() nounwind {
+; NO-PSADBW-LABEL: byte_sum_v8_i32:
+entry:
+ br label %vector.body
+
+vector.body:
+ %index = phi i64 [ 0, %entry ], [ %index.next, %vector.body ]
+ %vec.phi = phi <8 x i32> [ zeroinitializer, %entry ], [ %add, %vector.body ]
+ %p = getelementptr inbounds [1024 x i8], ptr @a, i64 0, i64 %index
+ %wide.load = load <8 x i8>, ptr %p, align 8
+ %z = zext <8 x i8> %wide.load to <8 x i32>
+ %add = add nsw <8 x i32> %z, %vec.phi
+ %index.next = add i64 %index, 8
+ %cmp = icmp eq i64 %index.next, 1024
+ br i1 %cmp, label %middle.block, label %vector.body
+
+middle.block:
+ %s1 = shufflevector <8 x i32> %add, <8 x i32> undef, <8 x i32> <i32 4, i32 5, i32 6, i32 7, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r1 = add <8 x i32> %add, %s1
+ %s2 = shufflevector <8 x i32> %r1, <8 x i32> undef, <8 x i32> <i32 2, i32 3, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r2 = add <8 x i32> %r1, %s2
+ %s3 = shufflevector <8 x i32> %r2, <8 x i32> undef, <8 x i32> <i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r3 = add <8 x i32> %r2, %s3
+ %ext = extractelement <8 x i32> %r3, i32 0
+ ret i32 %ext
+; NO-PSADBW-NOT: psadbw
+}
diff --git a/llvm/test/CodeGen/X86/byte-sum-positive.ll b/llvm/test/CodeGen/X86/byte-sum-positive.ll
new file mode 100644
index 0000000000000..cd118e2069bde
--- /dev/null
+++ b/llvm/test/CodeGen/X86/byte-sum-positive.ll
@@ -0,0 +1,478 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+sse2 | FileCheck %s --check-prefixes=SSE2
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx2 | FileCheck %s --check-prefixes=AVX2
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx512bw | FileCheck %s --check-prefixes=AVX512BW
+
+; Positive CodeGen tests for X86PartialReduction::tryByteSumReplacement.
+
+ at a = dso_local global [1024 x i8] zeroinitializer, align 16
+
+define dso_local i32 @byte_sum_v16_i32() nounwind {
+; SSE2-LABEL: byte_sum_v16_i32:
+; SSE2: # %bb.0: # %entry
+; SSE2-NEXT: pxor %xmm0, %xmm0
+; SSE2-NEXT: movq $-1024, %rax # imm = 0xFC00
+; SSE2-NEXT: pxor %xmm1, %xmm1
+; SSE2-NEXT: .p2align 4
+; SSE2-NEXT: .LBB0_1: # %vector.body
+; SSE2-NEXT: # =>This Inner Loop Header: Depth=1
+; SSE2-NEXT: movdqa a+1024(%rax), %xmm2
+; SSE2-NEXT: psadbw %xmm0, %xmm2
+; SSE2-NEXT: paddd %xmm2, %xmm1
+; SSE2-NEXT: addq $16, %rax
+; SSE2-NEXT: jne .LBB0_1
+; SSE2-NEXT: # %bb.2: # %middle.block
+; SSE2-NEXT: paddd %xmm0, %xmm1
+; SSE2-NEXT: paddd %xmm0, %xmm0
+; SSE2-NEXT: paddd %xmm1, %xmm0
+; SSE2-NEXT: pshufd {{.*#+}} xmm1 = xmm0[2,3,2,3]
+; SSE2-NEXT: paddd %xmm0, %xmm1
+; SSE2-NEXT: pshufd {{.*#+}} xmm0 = xmm1[1,1,1,1]
+; SSE2-NEXT: paddd %xmm1, %xmm0
+; SSE2-NEXT: movd %xmm0, %eax
+; SSE2-NEXT: retq
+;
+; AVX2-LABEL: byte_sum_v16_i32:
+; AVX2: # %bb.0: # %entry
+; AVX2-NEXT: vpxor %xmm0, %xmm0, %xmm0
+; AVX2-NEXT: movq $-1024, %rax # imm = 0xFC00
+; AVX2-NEXT: vpxor %xmm1, %xmm1, %xmm1
+; AVX2-NEXT: vpxor %xmm2, %xmm2, %xmm2
+; AVX2-NEXT: .p2align 4
+; AVX2-NEXT: .LBB0_1: # %vector.body
+; AVX2-NEXT: # =>This Inner Loop Header: Depth=1
+; AVX2-NEXT: vpsadbw a+1024(%rax), %xmm1, %xmm3
+; AVX2-NEXT: vpaddd %ymm2, %ymm3, %ymm2
+; AVX2-NEXT: addq $16, %rax
+; AVX2-NEXT: jne .LBB0_1
+; AVX2-NEXT: # %bb.2: # %middle.block
+; AVX2-NEXT: vpaddd %ymm0, %ymm2, %ymm0
+; AVX2-NEXT: vextracti128 $1, %ymm0, %xmm1
+; AVX2-NEXT: vpaddd %xmm1, %xmm0, %xmm0
+; AVX2-NEXT: vpshufd {{.*#+}} xmm1 = xmm0[2,3,2,3]
+; AVX2-NEXT: vpaddd %xmm1, %xmm0, %xmm0
+; AVX2-NEXT: vpshufd {{.*#+}} xmm1 = xmm0[1,1,1,1]
+; AVX2-NEXT: vpaddd %xmm1, %xmm0, %xmm0
+; AVX2-NEXT: vmovd %xmm0, %eax
+; AVX2-NEXT: vzeroupper
+; AVX2-NEXT: retq
+;
+; AVX512BW-LABEL: byte_sum_v16_i32:
+; AVX512BW: # %bb.0: # %entry
+; AVX512BW-NEXT: vpxor %xmm0, %xmm0, %xmm0
+; AVX512BW-NEXT: movq $-1024, %rax # imm = 0xFC00
+; AVX512BW-NEXT: vpxor %xmm1, %xmm1, %xmm1
+; AVX512BW-NEXT: .p2align 4
+; AVX512BW-NEXT: .LBB0_1: # %vector.body
+; AVX512BW-NEXT: # =>This Inner Loop Header: Depth=1
+; AVX512BW-NEXT: vpsadbw a+1024(%rax), %xmm1, %xmm2
+; AVX512BW-NEXT: vpaddd %zmm0, %zmm2, %zmm0
+; AVX512BW-NEXT: addq $16, %rax
+; AVX512BW-NEXT: jne .LBB0_1
+; AVX512BW-NEXT: # %bb.2: # %middle.block
+; AVX512BW-NEXT: vextracti64x4 $1, %zmm0, %ymm1
+; AVX512BW-NEXT: vpaddd %zmm1, %zmm0, %zmm0
+; AVX512BW-NEXT: vextracti128 $1, %ymm0, %xmm1
+; AVX512BW-NEXT: vpaddd %xmm1, %xmm0, %xmm0
+; AVX512BW-NEXT: vpshufd {{.*#+}} xmm1 = xmm0[2,3,2,3]
+; AVX512BW-NEXT: vpaddd %xmm1, %xmm0, %xmm0
+; AVX512BW-NEXT: vpshufd {{.*#+}} xmm1 = xmm0[1,1,1,1]
+; AVX512BW-NEXT: vpaddd %xmm1, %xmm0, %xmm0
+; AVX512BW-NEXT: vmovd %xmm0, %eax
+; AVX512BW-NEXT: vzeroupper
+; AVX512BW-NEXT: retq
+entry:
+ br label %vector.body
+
+vector.body:
+ %index = phi i64 [ 0, %entry ], [ %index.next, %vector.body ]
+ %vec.phi = phi <16 x i32> [ zeroinitializer, %entry ], [ %add, %vector.body ]
+ %p = getelementptr inbounds [1024 x i8], ptr @a, i64 0, i64 %index
+ %wide.load = load <16 x i8>, ptr %p, align 16
+ %z = zext <16 x i8> %wide.load to <16 x i32>
+ %add = add nsw <16 x i32> %z, %vec.phi
+ %index.next = add i64 %index, 16
+ %cmp = icmp eq i64 %index.next, 1024
+ br i1 %cmp, label %middle.block, label %vector.body
+
+middle.block:
+ %s1 = shufflevector <16 x i32> %add, <16 x i32> undef, <16 x i32> <i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r1 = add <16 x i32> %add, %s1
+ %s2 = shufflevector <16 x i32> %r1, <16 x i32> undef, <16 x i32> <i32 4, i32 5, i32 6, i32 7, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r2 = add <16 x i32> %r1, %s2
+ %s3 = shufflevector <16 x i32> %r2, <16 x i32> undef, <16 x i32> <i32 2, i32 3, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r3 = add <16 x i32> %r2, %s3
+ %s4 = shufflevector <16 x i32> %r3, <16 x i32> undef, <16 x i32> <i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r4 = add <16 x i32> %r3, %s4
+ %ext = extractelement <16 x i32> %r4, i32 0
+ ret i32 %ext
+}
+
+;----------------------------------------------------------------------------
+; POSITIVE: i32 accumulator, VF = 32.
+; SSE2 splits into two 128-bit psadbw per iteration; AVX2 and AVX-512BW
+; emit a single 256-bit vpsadbw (AVX-512BW takes the 256-bit path because
+; NumElts < 64).
+;----------------------------------------------------------------------------
+define dso_local i32 @byte_sum_v32_i32() nounwind {
+; SSE2-LABEL: byte_sum_v32_i32:
+; SSE2: # %bb.0: # %entry
+; SSE2-NEXT: pxor %xmm0, %xmm0
+; SSE2-NEXT: movq $-1024, %rax # imm = 0xFC00
+; SSE2-NEXT: pxor %xmm1, %xmm1
+; SSE2-NEXT: pxor %xmm2, %xmm2
+; SSE2-NEXT: .p2align 4
+; SSE2-NEXT: .LBB1_1: # %vector.body
+; SSE2-NEXT: # =>This Inner Loop Header: Depth=1
+; SSE2-NEXT: movdqa a+1024(%rax), %xmm3
+; SSE2-NEXT: psadbw %xmm0, %xmm3
+; SSE2-NEXT: paddd %xmm3, %xmm1
+; SSE2-NEXT: movdqa a+1040(%rax), %xmm3
+; SSE2-NEXT: psadbw %xmm0, %xmm3
+; SSE2-NEXT: paddd %xmm3, %xmm2
+; SSE2-NEXT: addq $32, %rax
+; SSE2-NEXT: jne .LBB1_1
+; SSE2-NEXT: # %bb.2: # %middle.block
+; SSE2-NEXT: paddd %xmm0, %xmm2
+; SSE2-NEXT: paddd %xmm0, %xmm1
+; SSE2-NEXT: paddd %xmm0, %xmm0
+; SSE2-NEXT: paddd %xmm0, %xmm1
+; SSE2-NEXT: paddd %xmm2, %xmm0
+; SSE2-NEXT: paddd %xmm1, %xmm0
+; SSE2-NEXT: pshufd {{.*#+}} xmm1 = xmm0[2,3,2,3]
+; SSE2-NEXT: paddd %xmm0, %xmm1
+; SSE2-NEXT: pshufd {{.*#+}} xmm0 = xmm1[1,1,1,1]
+; SSE2-NEXT: paddd %xmm1, %xmm0
+; SSE2-NEXT: movd %xmm0, %eax
+; SSE2-NEXT: retq
+;
+; AVX2-LABEL: byte_sum_v32_i32:
+; AVX2: # %bb.0: # %entry
+; AVX2-NEXT: vpxor %xmm0, %xmm0, %xmm0
+; AVX2-NEXT: movq $-1024, %rax # imm = 0xFC00
+; AVX2-NEXT: vpxor %xmm1, %xmm1, %xmm1
+; AVX2-NEXT: .p2align 4
+; AVX2-NEXT: .LBB1_1: # %vector.body
+; AVX2-NEXT: # =>This Inner Loop Header: Depth=1
+; AVX2-NEXT: vpsadbw a+1024(%rax), %ymm0, %ymm2
+; AVX2-NEXT: vpaddd %ymm1, %ymm2, %ymm1
+; AVX2-NEXT: addq $32, %rax
+; AVX2-NEXT: jne .LBB1_1
+; AVX2-NEXT: # %bb.2: # %middle.block
+; AVX2-NEXT: vpaddd %ymm0, %ymm1, %ymm1
+; AVX2-NEXT: vpaddd %ymm0, %ymm0, %ymm0
+; AVX2-NEXT: vpaddd %ymm0, %ymm1, %ymm0
+; AVX2-NEXT: vextracti128 $1, %ymm0, %xmm1
+; AVX2-NEXT: vpaddd %xmm1, %xmm0, %xmm0
+; AVX2-NEXT: vpshufd {{.*#+}} xmm1 = xmm0[2,3,2,3]
+; AVX2-NEXT: vpaddd %xmm1, %xmm0, %xmm0
+; AVX2-NEXT: vpshufd {{.*#+}} xmm1 = xmm0[1,1,1,1]
+; AVX2-NEXT: vpaddd %xmm1, %xmm0, %xmm0
+; AVX2-NEXT: vmovd %xmm0, %eax
+; AVX2-NEXT: vzeroupper
+; AVX2-NEXT: retq
+;
+; AVX512BW-LABEL: byte_sum_v32_i32:
+; AVX512BW: # %bb.0: # %entry
+; AVX512BW-NEXT: vpxor %xmm0, %xmm0, %xmm0
+; AVX512BW-NEXT: movq $-1024, %rax # imm = 0xFC00
+; AVX512BW-NEXT: vpxor %xmm1, %xmm1, %xmm1
+; AVX512BW-NEXT: vpxor %xmm2, %xmm2, %xmm2
+; AVX512BW-NEXT: .p2align 4
+; AVX512BW-NEXT: .LBB1_1: # %vector.body
+; AVX512BW-NEXT: # =>This Inner Loop Header: Depth=1
+; AVX512BW-NEXT: vpsadbw a+1024(%rax), %ymm1, %ymm3
+; AVX512BW-NEXT: vpaddd %zmm2, %zmm3, %zmm2
+; AVX512BW-NEXT: addq $32, %rax
+; AVX512BW-NEXT: jne .LBB1_1
+; AVX512BW-NEXT: # %bb.2: # %middle.block
+; AVX512BW-NEXT: vpaddd %zmm0, %zmm2, %zmm0
+; AVX512BW-NEXT: vextracti64x4 $1, %zmm0, %ymm1
+; AVX512BW-NEXT: vpaddd %zmm1, %zmm0, %zmm0
+; AVX512BW-NEXT: vextracti128 $1, %ymm0, %xmm1
+; AVX512BW-NEXT: vpaddd %xmm1, %xmm0, %xmm0
+; AVX512BW-NEXT: vpshufd {{.*#+}} xmm1 = xmm0[2,3,2,3]
+; AVX512BW-NEXT: vpaddd %xmm1, %xmm0, %xmm0
+; AVX512BW-NEXT: vpshufd {{.*#+}} xmm1 = xmm0[1,1,1,1]
+; AVX512BW-NEXT: vpaddd %xmm1, %xmm0, %xmm0
+; AVX512BW-NEXT: vmovd %xmm0, %eax
+; AVX512BW-NEXT: vzeroupper
+; AVX512BW-NEXT: retq
+entry:
+ br label %vector.body
+
+vector.body:
+ %index = phi i64 [ 0, %entry ], [ %index.next, %vector.body ]
+ %vec.phi = phi <32 x i32> [ zeroinitializer, %entry ], [ %add, %vector.body ]
+ %p = getelementptr inbounds [1024 x i8], ptr @a, i64 0, i64 %index
+ %wide.load = load <32 x i8>, ptr %p, align 16
+ %z = zext <32 x i8> %wide.load to <32 x i32>
+ %add = add nsw <32 x i32> %z, %vec.phi
+ %index.next = add i64 %index, 32
+ %cmp = icmp eq i64 %index.next, 1024
+ br i1 %cmp, label %middle.block, label %vector.body
+
+middle.block:
+ %s1 = shufflevector <32 x i32> %add, <32 x i32> undef, <32 x i32> <i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30, i32 31, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r1 = add <32 x i32> %add, %s1
+ %s2 = shufflevector <32 x i32> %r1, <32 x i32> undef, <32 x i32> <i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r2 = add <32 x i32> %r1, %s2
+ %s3 = shufflevector <32 x i32> %r2, <32 x i32> undef, <32 x i32> <i32 4, i32 5, i32 6, i32 7, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r3 = add <32 x i32> %r2, %s3
+ %s4 = shufflevector <32 x i32> %r3, <32 x i32> undef, <32 x i32> <i32 2, i32 3, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r4 = add <32 x i32> %r3, %s4
+ %s5 = shufflevector <32 x i32> %r4, <32 x i32> undef, <32 x i32> <i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r5 = add <32 x i32> %r4, %s5
+ %ext = extractelement <32 x i32> %r5, i32 0
+ ret i32 %ext
+}
+
+;----------------------------------------------------------------------------
+; POSITIVE: i32 accumulator, VF = 64 (AVX-512BW one-shot psadbw).
+;----------------------------------------------------------------------------
+define dso_local i32 @byte_sum_v64_i32() nounwind {
+; SSE2-LABEL: byte_sum_v64_i32:
+; SSE2: # %bb.0: # %entry
+; SSE2-NEXT: pxor %xmm4, %xmm4
+; SSE2-NEXT: movq $-1024, %rax # imm = 0xFC00
+; SSE2-NEXT: pxor %xmm0, %xmm0
+; SSE2-NEXT: pxor %xmm3, %xmm3
+; SSE2-NEXT: pxor %xmm2, %xmm2
+; SSE2-NEXT: pxor %xmm1, %xmm1
+; SSE2-NEXT: .p2align 4
+; SSE2-NEXT: .LBB2_1: # %vector.body
+; SSE2-NEXT: # =>This Inner Loop Header: Depth=1
+; SSE2-NEXT: movdqa a+1024(%rax), %xmm5
+; SSE2-NEXT: psadbw %xmm4, %xmm5
+; SSE2-NEXT: paddd %xmm5, %xmm0
+; SSE2-NEXT: movdqa a+1040(%rax), %xmm5
+; SSE2-NEXT: psadbw %xmm4, %xmm5
+; SSE2-NEXT: paddd %xmm5, %xmm3
+; SSE2-NEXT: movdqa a+1056(%rax), %xmm5
+; SSE2-NEXT: psadbw %xmm4, %xmm5
+; SSE2-NEXT: paddd %xmm5, %xmm2
+; SSE2-NEXT: movdqa a+1072(%rax), %xmm5
+; SSE2-NEXT: psadbw %xmm4, %xmm5
+; SSE2-NEXT: paddd %xmm5, %xmm1
+; SSE2-NEXT: addq $64, %rax
+; SSE2-NEXT: jne .LBB2_1
+; SSE2-NEXT: # %bb.2: # %middle.block
+; SSE2-NEXT: paddd %xmm4, %xmm2
+; SSE2-NEXT: pxor %xmm5, %xmm5
+; SSE2-NEXT: paddd %xmm5, %xmm5
+; SSE2-NEXT: paddd %xmm4, %xmm0
+; SSE2-NEXT: paddd %xmm4, %xmm1
+; SSE2-NEXT: paddd %xmm4, %xmm3
+; SSE2-NEXT: paddd %xmm5, %xmm3
+; SSE2-NEXT: paddd %xmm5, %xmm1
+; SSE2-NEXT: paddd %xmm3, %xmm1
+; SSE2-NEXT: paddd %xmm5, %xmm0
+; SSE2-NEXT: paddd %xmm2, %xmm5
+; SSE2-NEXT: paddd %xmm0, %xmm5
+; SSE2-NEXT: paddd %xmm1, %xmm5
+; SSE2-NEXT: pshufd {{.*#+}} xmm0 = xmm5[2,3,2,3]
+; SSE2-NEXT: paddd %xmm5, %xmm0
+; SSE2-NEXT: pshufd {{.*#+}} xmm1 = xmm0[1,1,1,1]
+; SSE2-NEXT: paddd %xmm0, %xmm1
+; SSE2-NEXT: movd %xmm1, %eax
+; SSE2-NEXT: retq
+;
+; AVX2-LABEL: byte_sum_v64_i32:
+; AVX2: # %bb.0: # %entry
+; AVX2-NEXT: vpxor %xmm0, %xmm0, %xmm0
+; AVX2-NEXT: movq $-1024, %rax # imm = 0xFC00
+; AVX2-NEXT: vpxor %xmm1, %xmm1, %xmm1
+; AVX2-NEXT: vpxor %xmm2, %xmm2, %xmm2
+; AVX2-NEXT: .p2align 4
+; AVX2-NEXT: .LBB2_1: # %vector.body
+; AVX2-NEXT: # =>This Inner Loop Header: Depth=1
+; AVX2-NEXT: vpsadbw a+1024(%rax), %ymm0, %ymm3
+; AVX2-NEXT: vpaddd %ymm1, %ymm3, %ymm1
+; AVX2-NEXT: vpsadbw a+1056(%rax), %ymm0, %ymm3
+; AVX2-NEXT: vpaddd %ymm2, %ymm3, %ymm2
+; AVX2-NEXT: addq $64, %rax
+; AVX2-NEXT: jne .LBB2_1
+; AVX2-NEXT: # %bb.2: # %middle.block
+; AVX2-NEXT: vpaddd %ymm0, %ymm2, %ymm2
+; AVX2-NEXT: vpaddd %ymm0, %ymm0, %ymm3
+; AVX2-NEXT: vpaddd %ymm0, %ymm1, %ymm0
+; AVX2-NEXT: vpaddd %ymm3, %ymm0, %ymm0
+; AVX2-NEXT: vpaddd %ymm3, %ymm2, %ymm1
+; AVX2-NEXT: vpaddd %ymm1, %ymm0, %ymm0
+; AVX2-NEXT: vextracti128 $1, %ymm0, %xmm1
+; AVX2-NEXT: vpaddd %xmm1, %xmm0, %xmm0
+; AVX2-NEXT: vpshufd {{.*#+}} xmm1 = xmm0[2,3,2,3]
+; AVX2-NEXT: vpaddd %xmm1, %xmm0, %xmm0
+; AVX2-NEXT: vpshufd {{.*#+}} xmm1 = xmm0[1,1,1,1]
+; AVX2-NEXT: vpaddd %xmm1, %xmm0, %xmm0
+; AVX2-NEXT: vmovd %xmm0, %eax
+; AVX2-NEXT: vzeroupper
+; AVX2-NEXT: retq
+;
+; AVX512BW-LABEL: byte_sum_v64_i32:
+; AVX512BW: # %bb.0: # %entry
+; AVX512BW-NEXT: vpxor %xmm0, %xmm0, %xmm0
+; AVX512BW-NEXT: movq $-1024, %rax # imm = 0xFC00
+; AVX512BW-NEXT: vpxor %xmm1, %xmm1, %xmm1
+; AVX512BW-NEXT: .p2align 4
+; AVX512BW-NEXT: .LBB2_1: # %vector.body
+; AVX512BW-NEXT: # =>This Inner Loop Header: Depth=1
+; AVX512BW-NEXT: vpsadbw a+1024(%rax), %zmm0, %zmm2
+; AVX512BW-NEXT: vpaddd %zmm1, %zmm2, %zmm1
+; AVX512BW-NEXT: addq $64, %rax
+; AVX512BW-NEXT: jne .LBB2_1
+; AVX512BW-NEXT: # %bb.2: # %middle.block
+; AVX512BW-NEXT: vpaddd %zmm0, %zmm1, %zmm1
+; AVX512BW-NEXT: vpaddd %zmm0, %zmm0, %zmm0
+; AVX512BW-NEXT: vpaddd %zmm0, %zmm1, %zmm0
+; AVX512BW-NEXT: vextracti64x4 $1, %zmm0, %ymm1
+; AVX512BW-NEXT: vpaddd %zmm1, %zmm0, %zmm0
+; AVX512BW-NEXT: vextracti128 $1, %ymm0, %xmm1
+; AVX512BW-NEXT: vpaddd %xmm1, %xmm0, %xmm0
+; AVX512BW-NEXT: vpshufd {{.*#+}} xmm1 = xmm0[2,3,2,3]
+; AVX512BW-NEXT: vpaddd %xmm1, %xmm0, %xmm0
+; AVX512BW-NEXT: vpshufd {{.*#+}} xmm1 = xmm0[1,1,1,1]
+; AVX512BW-NEXT: vpaddd %xmm1, %xmm0, %xmm0
+; AVX512BW-NEXT: vmovd %xmm0, %eax
+; AVX512BW-NEXT: vzeroupper
+; AVX512BW-NEXT: retq
+entry:
+ br label %vector.body
+
+vector.body:
+ %index = phi i64 [ 0, %entry ], [ %index.next, %vector.body ]
+ %vec.phi = phi <64 x i32> [ zeroinitializer, %entry ], [ %add, %vector.body ]
+ %p = getelementptr inbounds [1024 x i8], ptr @a, i64 0, i64 %index
+ %wide.load = load <64 x i8>, ptr %p, align 16
+ %z = zext <64 x i8> %wide.load to <64 x i32>
+ %add = add nsw <64 x i32> %z, %vec.phi
+ %index.next = add i64 %index, 64
+ %cmp = icmp eq i64 %index.next, 1024
+ br i1 %cmp, label %middle.block, label %vector.body
+
+middle.block:
+ %s0 = shufflevector <64 x i32> %add, <64 x i32> poison, <64 x i32> <i32 32, i32 33, i32 34, i32 35, i32 36, i32 37, i32 38, i32 39, i32 40, i32 41, i32 42, i32 43, i32 44, i32 45, i32 46, i32 47, i32 48, i32 49, i32 50, i32 51, i32 52, i32 53, i32 54, i32 55, i32 56, i32 57, i32 58, i32 59, i32 60, i32 61, i32 62, i32 63, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison>
+ %r0 = add <64 x i32> %add, %s0
+ %s1 = shufflevector <64 x i32> %r0, <64 x i32> poison, <64 x i32> <i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30, i32 31, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison>
+ %r1 = add <64 x i32> %r0, %s1
+ %s2 = shufflevector <64 x i32> %r1, <64 x i32> poison, <64 x i32> <i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison>
+ %r2 = add <64 x i32> %r1, %s2
+ %s3 = shufflevector <64 x i32> %r2, <64 x i32> poison, <64 x i32> <i32 4, i32 5, i32 6, i32 7, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison>
+ %r3 = add <64 x i32> %r2, %s3
+ %s4 = shufflevector <64 x i32> %r3, <64 x i32> poison, <64 x i32> <i32 2, i32 3, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison>
+ %r4 = add <64 x i32> %r3, %s4
+ %s5 = shufflevector <64 x i32> %r4, <64 x i32> poison, <64 x i32> <i32 1, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison>
+ %r5 = add <64 x i32> %r4, %s5
+ %ext = extractelement <64 x i32> %r5, i32 0
+ ret i32 %ext
+}
+
+;----------------------------------------------------------------------------
+; POSITIVE: i64 accumulator, VF = 16.
+;
+; LV emits a `zext <16 x i8> to <16 x i64>` directly (no i32 intermediate)
+; when the source uses a uint64_t accumulator. tryByteSumReplacement must
+; handle this shape too: psadbw natively returns <2 x i64> per 16-byte lane,
+; so no i64<->i32 bitcast is needed.
+;----------------------------------------------------------------------------
+define dso_local i64 @byte_sum_v16_i64() nounwind {
+; SSE2-LABEL: byte_sum_v16_i64:
+; SSE2: # %bb.0: # %entry
+; SSE2-NEXT: pxor %xmm0, %xmm0
+; SSE2-NEXT: movq $-1024, %rax # imm = 0xFC00
+; SSE2-NEXT: pxor %xmm1, %xmm1
+; SSE2-NEXT: .p2align 4
+; SSE2-NEXT: .LBB3_1: # %vector.body
+; SSE2-NEXT: # =>This Inner Loop Header: Depth=1
+; SSE2-NEXT: movdqa a+1024(%rax), %xmm2
+; SSE2-NEXT: psadbw %xmm0, %xmm2
+; SSE2-NEXT: paddq %xmm2, %xmm1
+; SSE2-NEXT: addq $16, %rax
+; SSE2-NEXT: jne .LBB3_1
+; SSE2-NEXT: # %bb.2: # %middle.block
+; SSE2-NEXT: paddq %xmm0, %xmm1
+; SSE2-NEXT: paddq %xmm0, %xmm0
+; SSE2-NEXT: movdqa %xmm0, %xmm2
+; SSE2-NEXT: paddq %xmm0, %xmm2
+; SSE2-NEXT: paddq %xmm0, %xmm2
+; SSE2-NEXT: paddq %xmm1, %xmm2
+; SSE2-NEXT: pshufd {{.*#+}} xmm0 = xmm2[2,3,2,3]
+; SSE2-NEXT: paddq %xmm2, %xmm0
+; SSE2-NEXT: movq %xmm0, %rax
+; SSE2-NEXT: retq
+;
+; AVX2-LABEL: byte_sum_v16_i64:
+; AVX2: # %bb.0: # %entry
+; AVX2-NEXT: vpxor %xmm0, %xmm0, %xmm0
+; AVX2-NEXT: movq $-1024, %rax # imm = 0xFC00
+; AVX2-NEXT: vpxor %xmm1, %xmm1, %xmm1
+; AVX2-NEXT: vpxor %xmm2, %xmm2, %xmm2
+; AVX2-NEXT: .p2align 4
+; AVX2-NEXT: .LBB3_1: # %vector.body
+; AVX2-NEXT: # =>This Inner Loop Header: Depth=1
+; AVX2-NEXT: vpsadbw a+1024(%rax), %xmm1, %xmm3
+; AVX2-NEXT: vpaddq %ymm2, %ymm3, %ymm2
+; AVX2-NEXT: addq $16, %rax
+; AVX2-NEXT: jne .LBB3_1
+; AVX2-NEXT: # %bb.2: # %middle.block
+; AVX2-NEXT: vpaddq %ymm0, %ymm2, %ymm1
+; AVX2-NEXT: vpaddq %ymm0, %ymm0, %ymm0
+; AVX2-NEXT: vpaddq %ymm0, %ymm1, %ymm0
+; AVX2-NEXT: vextracti128 $1, %ymm0, %xmm1
+; AVX2-NEXT: vpaddq %xmm1, %xmm0, %xmm0
+; AVX2-NEXT: vpshufd {{.*#+}} xmm1 = xmm0[2,3,2,3]
+; AVX2-NEXT: vpaddq %xmm1, %xmm0, %xmm0
+; AVX2-NEXT: vmovq %xmm0, %rax
+; AVX2-NEXT: vzeroupper
+; AVX2-NEXT: retq
+;
+; AVX512BW-LABEL: byte_sum_v16_i64:
+; AVX512BW: # %bb.0: # %entry
+; AVX512BW-NEXT: vpxor %xmm0, %xmm0, %xmm0
+; AVX512BW-NEXT: movq $-1024, %rax # imm = 0xFC00
+; AVX512BW-NEXT: vpxor %xmm1, %xmm1, %xmm1
+; AVX512BW-NEXT: vpxor %xmm2, %xmm2, %xmm2
+; AVX512BW-NEXT: .p2align 4
+; AVX512BW-NEXT: .LBB3_1: # %vector.body
+; AVX512BW-NEXT: # =>This Inner Loop Header: Depth=1
+; AVX512BW-NEXT: vpsadbw a+1024(%rax), %xmm1, %xmm3
+; AVX512BW-NEXT: vpaddq %zmm2, %zmm3, %zmm2
+; AVX512BW-NEXT: addq $16, %rax
+; AVX512BW-NEXT: jne .LBB3_1
+; AVX512BW-NEXT: # %bb.2: # %middle.block
+; AVX512BW-NEXT: vpaddq %zmm0, %zmm2, %zmm0
+; AVX512BW-NEXT: vextracti64x4 $1, %zmm0, %ymm1
+; AVX512BW-NEXT: vpaddq %zmm1, %zmm0, %zmm0
+; AVX512BW-NEXT: vextracti128 $1, %ymm0, %xmm1
+; AVX512BW-NEXT: vpaddq %xmm1, %xmm0, %xmm0
+; AVX512BW-NEXT: vpshufd {{.*#+}} xmm1 = xmm0[2,3,2,3]
+; AVX512BW-NEXT: vpaddq %xmm1, %xmm0, %xmm0
+; AVX512BW-NEXT: vmovq %xmm0, %rax
+; AVX512BW-NEXT: vzeroupper
+; AVX512BW-NEXT: retq
+entry:
+ br label %vector.body
+
+vector.body:
+ %index = phi i64 [ 0, %entry ], [ %index.next, %vector.body ]
+ %vec.phi = phi <16 x i64> [ zeroinitializer, %entry ], [ %add, %vector.body ]
+ %p = getelementptr inbounds [1024 x i8], ptr @a, i64 0, i64 %index
+ %wide.load = load <16 x i8>, ptr %p, align 16
+ %z = zext <16 x i8> %wide.load to <16 x i64>
+ %add = add nsw <16 x i64> %z, %vec.phi
+ %index.next = add i64 %index, 16
+ %cmp = icmp eq i64 %index.next, 1024
+ br i1 %cmp, label %middle.block, label %vector.body
+
+middle.block:
+ %s1 = shufflevector <16 x i64> %add, <16 x i64> undef, <16 x i32> <i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r1 = add <16 x i64> %add, %s1
+ %s2 = shufflevector <16 x i64> %r1, <16 x i64> undef, <16 x i32> <i32 4, i32 5, i32 6, i32 7, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r2 = add <16 x i64> %r1, %s2
+ %s3 = shufflevector <16 x i64> %r2, <16 x i64> undef, <16 x i32> <i32 2, i32 3, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r3 = add <16 x i64> %r2, %s3
+ %s4 = shufflevector <16 x i64> %r3, <16 x i64> undef, <16 x i32> <i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+ %r4 = add <16 x i64> %r3, %s4
+ %ext = extractelement <16 x i64> %r4, i32 0
+ ret i64 %ext
+}
diff --git a/llvm/test/CodeGen/X86/x86-partial-reduction-byte-sum-debugloc.ll b/llvm/test/CodeGen/X86/x86-partial-reduction-byte-sum-debugloc.ll
new file mode 100644
index 0000000000000..8261e2c25052b
--- /dev/null
+++ b/llvm/test/CodeGen/X86/x86-partial-reduction-byte-sum-debugloc.ll
@@ -0,0 +1,51 @@
+; RUN: opt < %s -passes=x86-partial-reduction -mtriple=x86_64-unknown-unknown -mattr=+sse2 -S | FileCheck %s
+
+; Verify that X86PartialReduction::tryByteSumReplacement carries the
+; original add instruction's !dbg location onto the freshly emitted
+; psadbw / shuffle sequence so source locations survive into the
+; optimized IR under -g.
+
+ at a = dso_local global [1024 x i8] zeroinitializer, align 16
+
+; CHECK-LABEL: @byte_sum_v16_i32
+; CHECK: call <2 x i64> @llvm.x86.sse2.psad.bw({{.*}}), !dbg ![[#LOC:]]
+; CHECK: ![[#LOC]] = !DILocation(line: 42,
+define dso_local i32 @byte_sum_v16_i32() nounwind !dbg !6 {
+entry:
+ br label %vector.body
+
+vector.body:
+ %index = phi i64 [ 0, %entry ], [ %index.next, %vector.body ]
+ %vec.phi = phi <16 x i32> [ zeroinitializer, %entry ], [ %add, %vector.body ]
+ %p = getelementptr inbounds [1024 x i8], ptr @a, i64 0, i64 %index
+ %wide.load = load <16 x i8>, ptr %p, align 16
+ %z = zext <16 x i8> %wide.load to <16 x i32>, !dbg !8
+ %add = add nsw <16 x i32> %z, %vec.phi
+ %index.next = add i64 %index, 16
+ %cmp = icmp eq i64 %index.next, 1024
+ br i1 %cmp, label %middle.block, label %vector.body
+
+middle.block:
+ %s1 = shufflevector <16 x i32> %add, <16 x i32> poison, <16 x i32> <i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison>
+ %r1 = add <16 x i32> %add, %s1
+ %s2 = shufflevector <16 x i32> %r1, <16 x i32> poison, <16 x i32> <i32 4, i32 5, i32 6, i32 7, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison>
+ %r2 = add <16 x i32> %r1, %s2
+ %s3 = shufflevector <16 x i32> %r2, <16 x i32> poison, <16 x i32> <i32 2, i32 3, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison>
+ %r3 = add <16 x i32> %r2, %s3
+ %s4 = shufflevector <16 x i32> %r3, <16 x i32> poison, <16 x i32> <i32 1, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison>
+ %r4 = add <16 x i32> %r3, %s4
+ %ext = extractelement <16 x i32> %r4, i32 0
+ ret i32 %ext
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3, !4, !5}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug)
+!1 = !DIFile(filename: "byte-sum-debugloc.c", directory: "/tmp")
+!3 = !{i32 7, !"Dwarf Version", i32 4}
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = !{i32 1, !"wchar_size", i32 4}
+!6 = distinct !DISubprogram(name: "byte_sum_v16_i32", scope: !1, file: !1, line: 1, type: !7, scopeLine: 1, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !0)
+!7 = !DISubroutineType(types: !{})
+!8 = !DILocation(line: 42, column: 1, scope: !6)
diff --git a/llvm/test/CodeGen/X86/x86-partial-reduction-byte-sum-negative.ll b/llvm/test/CodeGen/X86/x86-partial-reduction-byte-sum-negative.ll
new file mode 100644
index 0000000000000..35b751002f265
--- /dev/null
+++ b/llvm/test/CodeGen/X86/x86-partial-reduction-byte-sum-negative.ll
@@ -0,0 +1,49 @@
+; RUN: opt < %s -passes=x86-partial-reduction -mtriple=x86_64-unknown-unknown -mattr=+avx2 -S | FileCheck %s
+
+; Shapes that tryByteSumReplacement must not rewrite.
+
+ at a = dso_local global [1024 x i8] zeroinitializer, align 16
+
+; CHECK-LABEL: @byte_sum_v8_i32
+; CHECK-NOT: psad.bw
+define dso_local i32 @byte_sum_v8_i32() nounwind {
+entry:
+ br label %vector.body
+
+vector.body:
+ %index = phi i64 [ 0, %entry ], [ %index.next, %vector.body ]
+ %vec.phi = phi <8 x i32> [ zeroinitializer, %entry ], [ %add, %vector.body ]
+ %p = getelementptr inbounds [1024 x i8], ptr @a, i64 0, i64 %index
+ %wide.load = load <8 x i8>, ptr %p, align 8
+ %z = zext <8 x i8> %wide.load to <8 x i32>
+ %add = add nsw <8 x i32> %z, %vec.phi
+ %index.next = add i64 %index, 8
+ %cmp = icmp eq i64 %index.next, 1024
+ br i1 %cmp, label %middle.block, label %vector.body
+
+middle.block:
+ %ext = extractelement <8 x i32> %add, i32 0
+ ret i32 %ext
+}
+
+; CHECK-LABEL: @byte_sum_v24_i32
+; CHECK-NOT: psad.bw
+define dso_local i32 @byte_sum_v24_i32() nounwind {
+entry:
+ br label %vector.body
+
+vector.body:
+ %index = phi i64 [ 0, %entry ], [ %index.next, %vector.body ]
+ %vec.phi = phi <24 x i32> [ zeroinitializer, %entry ], [ %add, %vector.body ]
+ %p = getelementptr inbounds [1024 x i8], ptr @a, i64 0, i64 %index
+ %wide.load = load <24 x i8>, ptr %p, align 8
+ %z = zext <24 x i8> %wide.load to <24 x i32>
+ %add = add nsw <24 x i32> %z, %vec.phi
+ %index.next = add i64 %index, 24
+ %cmp = icmp eq i64 %index.next, 1024
+ br i1 %cmp, label %middle.block, label %vector.body
+
+middle.block:
+ %ext = extractelement <24 x i32> %add, i32 0
+ ret i32 %ext
+}
diff --git a/llvm/test/CodeGen/X86/x86-partial-reduction-byte-sum-wide-i64.ll b/llvm/test/CodeGen/X86/x86-partial-reduction-byte-sum-wide-i64.ll
new file mode 100644
index 0000000000000..78d87fda0d5f9
--- /dev/null
+++ b/llvm/test/CodeGen/X86/x86-partial-reduction-byte-sum-wide-i64.ll
@@ -0,0 +1,81 @@
+; RUN: opt < %s -passes=x86-partial-reduction -mtriple=x86_64-unknown-unknown -mattr=+avx2 -S | FileCheck %s --check-prefix=AVX2
+; RUN: opt < %s -passes=x86-partial-reduction -mtriple=x86_64-unknown-unknown -mattr=+avx512bw -S | FileCheck %s --check-prefix=AVX512
+
+; Wider-VF i64-accumulator positive shapes for tryByteSumReplacement.
+; These exercise the PerSplitTy = <IntrinsicNumElts/8 x i64> branch where the
+; matcher must NOT bitcast the psadbw result back to i32.
+;
+; VF=32 i64 uses the AVX2 256-bit lane (one avx2.psad.bw call). On
+; +avx512bw the dispatch falls into the same AVX2 path because NumElts<64.
+;
+; VF=64 i64 uses the AVX-512BW 512-bit lane (one avx512.psad.bw.512 call).
+
+ at a = dso_local global [1024 x i8] zeroinitializer, align 16
+
+; AVX2-LABEL: @byte_sum_v32_i64(
+; AVX2: call <4 x i64> @llvm.x86.avx2.psad.bw(
+; AVX512-LABEL: @byte_sum_v32_i64(
+; AVX512: call <4 x i64> @llvm.x86.avx2.psad.bw(
+define dso_local i64 @byte_sum_v32_i64() nounwind {
+entry:
+ br label %vector.body
+
+vector.body:
+ %index = phi i64 [ 0, %entry ], [ %index.next, %vector.body ]
+ %vec.phi = phi <32 x i64> [ zeroinitializer, %entry ], [ %add, %vector.body ]
+ %p = getelementptr inbounds [1024 x i8], ptr @a, i64 0, i64 %index
+ %wide.load = load <32 x i8>, ptr %p, align 16
+ %z = zext <32 x i8> %wide.load to <32 x i64>
+ %add = add nsw <32 x i64> %z, %vec.phi
+ %index.next = add i64 %index, 32
+ %cmp = icmp eq i64 %index.next, 1024
+ br i1 %cmp, label %middle.block, label %vector.body
+
+middle.block:
+ %s0 = shufflevector <32 x i64> %add, <32 x i64> poison, <32 x i32> <i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30, i32 31, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison>
+ %r0 = add <32 x i64> %add, %s0
+ %s1 = shufflevector <32 x i64> %r0, <32 x i64> poison, <32 x i32> <i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison>
+ %r1 = add <32 x i64> %r0, %s1
+ %s2 = shufflevector <32 x i64> %r1, <32 x i64> poison, <32 x i32> <i32 4, i32 5, i32 6, i32 7, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison>
+ %r2 = add <32 x i64> %r1, %s2
+ %s3 = shufflevector <32 x i64> %r2, <32 x i64> poison, <32 x i32> <i32 2, i32 3, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison>
+ %r3 = add <32 x i64> %r2, %s3
+ %s4 = shufflevector <32 x i64> %r3, <32 x i64> poison, <32 x i32> <i32 1, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison>
+ %r4 = add <32 x i64> %r3, %s4
+ %ext = extractelement <32 x i64> %r4, i32 0
+ ret i64 %ext
+}
+
+; AVX512-LABEL: @byte_sum_v64_i64(
+; AVX512: call <8 x i64> @llvm.x86.avx512.psad.bw.512(
+define dso_local i64 @byte_sum_v64_i64() nounwind {
+entry:
+ br label %vector.body
+
+vector.body:
+ %index = phi i64 [ 0, %entry ], [ %index.next, %vector.body ]
+ %vec.phi = phi <64 x i64> [ zeroinitializer, %entry ], [ %add, %vector.body ]
+ %p = getelementptr inbounds [1024 x i8], ptr @a, i64 0, i64 %index
+ %wide.load = load <64 x i8>, ptr %p, align 16
+ %z = zext <64 x i8> %wide.load to <64 x i64>
+ %add = add nsw <64 x i64> %z, %vec.phi
+ %index.next = add i64 %index, 64
+ %cmp = icmp eq i64 %index.next, 1024
+ br i1 %cmp, label %middle.block, label %vector.body
+
+middle.block:
+ %s0 = shufflevector <64 x i64> %add, <64 x i64> poison, <64 x i32> <i32 32, i32 33, i32 34, i32 35, i32 36, i32 37, i32 38, i32 39, i32 40, i32 41, i32 42, i32 43, i32 44, i32 45, i32 46, i32 47, i32 48, i32 49, i32 50, i32 51, i32 52, i32 53, i32 54, i32 55, i32 56, i32 57, i32 58, i32 59, i32 60, i32 61, i32 62, i32 63, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison>
+ %r0 = add <64 x i64> %add, %s0
+ %s1 = shufflevector <64 x i64> %r0, <64 x i64> poison, <64 x i32> <i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30, i32 31, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison>
+ %r1 = add <64 x i64> %r0, %s1
+ %s2 = shufflevector <64 x i64> %r1, <64 x i64> poison, <64 x i32> <i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison>
+ %r2 = add <64 x i64> %r1, %s2
+ %s3 = shufflevector <64 x i64> %r2, <64 x i64> poison, <64 x i32> <i32 4, i32 5, i32 6, i32 7, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison>
+ %r3 = add <64 x i64> %r2, %s3
+ %s4 = shufflevector <64 x i64> %r3, <64 x i64> poison, <64 x i32> <i32 2, i32 3, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison>
+ %r4 = add <64 x i64> %r3, %s4
+ %s5 = shufflevector <64 x i64> %r4, <64 x i64> poison, <64 x i32> <i32 1, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison>
+ %r5 = add <64 x i64> %r4, %s5
+ %ext = extractelement <64 x i64> %r5, i32 0
+ ret i64 %ext
+}
diff --git a/llvm/test/CodeGen/X86/x86-partial-reduction-byte-sum.ll b/llvm/test/CodeGen/X86/x86-partial-reduction-byte-sum.ll
new file mode 100644
index 0000000000000..d0d6cb66a477a
--- /dev/null
+++ b/llvm/test/CodeGen/X86/x86-partial-reduction-byte-sum.ll
@@ -0,0 +1,34 @@
+; RUN: opt < %s -passes=x86-partial-reduction -mtriple=x86_64-unknown-unknown -mattr=+sse2 -S | FileCheck %s
+
+; Isolate X86PartialReduction::tryByteSumReplacement on a positive shape.
+
+ at a = dso_local global [1024 x i8] zeroinitializer, align 16
+
+; CHECK: call <2 x i64> @llvm.x86.sse2.psad.bw(
+define dso_local i32 @byte_sum_v16_i32() nounwind {
+entry:
+ br label %vector.body
+
+vector.body:
+ %index = phi i64 [ 0, %entry ], [ %index.next, %vector.body ]
+ %vec.phi = phi <16 x i32> [ zeroinitializer, %entry ], [ %add, %vector.body ]
+ %p = getelementptr inbounds [1024 x i8], ptr @a, i64 0, i64 %index
+ %wide.load = load <16 x i8>, ptr %p, align 16
+ %z = zext <16 x i8> %wide.load to <16 x i32>
+ %add = add nsw <16 x i32> %z, %vec.phi
+ %index.next = add i64 %index, 16
+ %cmp = icmp eq i64 %index.next, 1024
+ br i1 %cmp, label %middle.block, label %vector.body
+
+middle.block:
+ %s1 = shufflevector <16 x i32> %add, <16 x i32> poison, <16 x i32> <i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison>
+ %r1 = add <16 x i32> %add, %s1
+ %s2 = shufflevector <16 x i32> %r1, <16 x i32> poison, <16 x i32> <i32 4, i32 5, i32 6, i32 7, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison>
+ %r2 = add <16 x i32> %r1, %s2
+ %s3 = shufflevector <16 x i32> %r2, <16 x i32> poison, <16 x i32> <i32 2, i32 3, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison>
+ %r3 = add <16 x i32> %r2, %s3
+ %s4 = shufflevector <16 x i32> %r3, <16 x i32> poison, <16 x i32> <i32 1, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison>
+ %r4 = add <16 x i32> %r3, %s4
+ %ext = extractelement <16 x i32> %r4, i32 0
+ ret i32 %ext
+}
More information about the llvm-commits
mailing list