[llvm] [X86][CostModel] Add per-shape gather/scatter cost tables for AMD znver4+ (PR #199488)
Sumukh J Bharadwaj via llvm-commits
llvm-commits at lists.llvm.org
Thu Jul 30 00:01:33 PDT 2026
https://github.com/amd-subharad updated https://github.com/llvm/llvm-project/pull/199488
>From c1de785264f8c505e5d6443df2fc978cc05aff69 Mon Sep 17 00:00:00 2001
From: Sumukh Bharadwaj <Sumukh.Bharadwaj at amd.com>
Date: Wed, 27 May 2026 15:45:46 +0530
Subject: [PATCH 1/6] [X86][CostModel] Add per-shape gather/scatter cost tables
for AMD znver4+
The X86 cost model currently returns a single flat overhead from
getGatherOverhead / getScatterOverhead, applied to every shape of
masked gather or scatter on every X86 subtarget that reaches the
gather/scatter path. On modern AMD parts the actual cost of these
instructions varies substantially with the vector width and element
size, and the single flat number forces the LoopVectorizer to either
under- or over-estimate the profitability of vectorising loops that
need indirect memory access.
This change adds a subtarget tuning bit, TuningPreferAMDZenGSCost,
attached to ZN4Tuning so znver4 and znver5 pick it up automatically.
Pre-AVX-512 Zen parts (znver1..3) take the scalarise path for masked
gather and never reach the new code, so the bit is intentionally NOT
placed in ZNTuning; flagging older Zen parts with the feature would
be misleading.
When the tuning bit is set, getGatherOverhead / getScatterOverhead
look the source vector type up in per-shape cost tables before
falling back to the existing generic flat overhead. The tables only
contain the shapes that are actually reached at the lookup site:
VF=2 is force-scalarised on AVX-512 (forceScalarizeMaskedGather), and
v16f64 is split via type legalisation in getGSVectorCost (1024-bit
data exceeds a single zmm), so neither row would ever be queried and
both are omitted. The live tables cover gather and scatter for
VF=4..16 over i32 / f32 / f64, plus VF=4 and VF=8 for i64.
Methodology
The numbers are empirical break-even costs measured on znver4 and
znver5 hardware. The methodology, summarised:
1. Take a controlled gather/scatter micro-benchmark with one
indirect memory access per inner-loop iteration.
2. Sweep the gather/scatter overhead via a local cl::opt patch on
X86TTI (the upstream tree has no such knob today; this is a
standalone local instrument that returns the forced value from
getGatherOverhead / getScatterOverhead). A reproducible version
of the patch lives on the author's zen-gs-i64-sweep branch.
3. For each (element type, VF) compile the micro-benchmark at a
range of forced overheads and identify the "flip" cost above
which the LoopVectorizer stops emitting the gather / scatter
instruction (it switches to an extract-load-insert lowering or
to a pure scalar loop). The tabulated cost is the highest value
at which gather/scatter emission was the right call: the
vectoriser still selects it AND the resulting binary is at least
as fast as the post-flip alternatives on the test hardware.
4. The sweep is run independently for each (element type, VF) on
Genoa, Milan and Turin and re-validated on Zen 5.
Notes on individual entries
* i64 entries are higher than their f64 counterparts at the same
VF. The scalar alternative for i64 runs on the integer pipeline
(cheaper than f64 on the FP pipeline), so gather has to be
cheaper to win. At the f64-style break-even, i64 gather was
1.7-3.5x slower than the scalarised lowering across stride
patterns on Zen 5, so the i64 break-even sits at the minimum
cost that suppresses vpgatherqq / vpscatterqq emission for the
measured patterns (which include the libquantum-style indirect
scatter cited in PR #198850).
* f32 rows for both tables mirror i32 rows: the original sweep
only characterised i32 and f64 lanes, and the f32 rows were
derived by symmetry because vpgatherdd / vpscatterdd and the
corresponding ps variants share the same physical lane width on
Zen. The runtime equivalence of vpscatterdd and vscatterdps was
verified directly (within 3% across VF and stride patterns).
Tests
The cost-model test
llvm/test/Analysis/CostModel/X86/masked-gather-scatter-amd-zen.ll
covers every live shape on znver4 / znver5 and pins the unchanged
behaviour for znver3 (scalarise path) and skx (generic flat
overhead). It also covers the 32-bit-reducible GEP form for v16f32
and v16i32 scatter, which is the only path that actually queries the
v16 row of the scatter table (the <16 x ptr> form recurses to v8
through type legalisation).
A second test
llvm/test/Transforms/LoopVectorize/X86/amd-zen-gather-scatter-decisions.ll
pins the resulting vectoriser decisions end-to-end: gather IS emitted
for an f64 indirect-load reduction on znver5; gather is NOT emitted
for an i64 indirect-load reduction (where the cost table is meant to
suppress it); and a unit-stride load must not become a gather
regardless of cost-table values (regression guard for issue #91370).
---
llvm/docs/ReleaseNotes.md | 5 +
llvm/lib/Target/X86/X86.td | 14 +-
.../lib/Target/X86/X86TargetTransformInfo.cpp | 84 ++-
llvm/lib/Target/X86/X86TargetTransformInfo.h | 4 +-
.../X86/masked-gather-scatter-amd-zen.ll | 612 ++++++++++++++++++
.../X86/amd-zen-gather-scatter-decisions.ll | 97 +++
6 files changed, 809 insertions(+), 7 deletions(-)
create mode 100644 llvm/test/Analysis/CostModel/X86/masked-gather-scatter-amd-zen.ll
create mode 100644 llvm/test/Transforms/LoopVectorize/X86/amd-zen-gather-scatter-decisions.ll
diff --git a/llvm/docs/ReleaseNotes.md b/llvm/docs/ReleaseNotes.md
index 013b5d0a52067..d1b1bef07ebd3 100644
--- a/llvm/docs/ReleaseNotes.md
+++ b/llvm/docs/ReleaseNotes.md
@@ -240,6 +240,11 @@ Makes programs 10x faster by doing Special New Thing.
* `.att_syntax` directive is now emitted for assembly files when AT&T syntax is
in use. This matches the behaviour of Intel syntax and aids with
compatibility when changing the default Clang syntax to the Intel syntax.
+* Masked gather and scatter cost overheads are now per-shape on AMD znver4
+ and znver5 targets via a new `TuningPreferAMDZenGSCost` subtarget
+ feature, replacing the single flat overhead inherited from the generic
+ AVX-512 path. The per-shape costs use empirical break-even values
+ measured on Zen 4 / Zen 5 hardware.
### Changes to the OCaml bindings
diff --git a/llvm/lib/Target/X86/X86.td b/llvm/lib/Target/X86/X86.td
index 50fb7204ebfa1..28bbd639649bb 100644
--- a/llvm/lib/Target/X86/X86.td
+++ b/llvm/lib/Target/X86/X86.td
@@ -721,6 +721,17 @@ def TuningFastGather
: SubtargetFeature<"fast-gather", "HasFastGather", "true",
"Indicates if gather is reasonably fast (this is true for Skylake client and all AVX-512 CPUs)">;
+// Use AMD Zen-tuned cost tables for masked gather/scatter intrinsics in the
+// X86 TargetTransformInfo cost model. Refines the flat overhead used by other
+// AVX-512 targets with per-element-type/per-VL costs measured on znver4 and
+// znver5. Inherited automatically by every znver4+ CPU via ZN4Tuning; not
+// applied to pre-AVX-512 Zen parts (znver1..3), which take the scalarise
+// path for masked gather anyway.
+def TuningPreferAMDZenGSCost
+ : SubtargetFeature<"prefer-amd-zen-gs-cost",
+ "HasPreferAMDZenGSCost", "true",
+ "Use AMD Zen-tuned gather/scatter cost tables in the cost model">;
+
// Generate vpdpwssd instead of vpmaddwd+vpaddd sequence.
def TuningFastDPWSSD
: SubtargetFeature<
@@ -1631,7 +1642,8 @@ def ProcessorFeatures {
list<SubtargetFeature> ZN3Features =
!listconcat(ZN2Features, ZN3AdditionalFeatures);
- list<SubtargetFeature> ZN4AdditionalTuning = [TuningFastDPWSSD];
+ list<SubtargetFeature> ZN4AdditionalTuning = [TuningFastDPWSSD,
+ TuningPreferAMDZenGSCost];
list<SubtargetFeature> ZN4Tuning =
!listconcat(ZN3Tuning, ZN4AdditionalTuning);
list<SubtargetFeature> ZN4AdditionalFeatures = [FeatureAVX512,
diff --git a/llvm/lib/Target/X86/X86TargetTransformInfo.cpp b/llvm/lib/Target/X86/X86TargetTransformInfo.cpp
index 62005a729b6fd..455869b233eec 100644
--- a/llvm/lib/Target/X86/X86TargetTransformInfo.cpp
+++ b/llvm/lib/Target/X86/X86TargetTransformInfo.cpp
@@ -6362,20 +6362,95 @@ InstructionCost X86TTIImpl::getCFInstrCost(unsigned Opcode,
return TTI::TCC_Free;
}
-int X86TTIImpl::getGatherOverhead() const {
+int X86TTIImpl::getGatherOverhead(Type *SrcVTy) const {
// Some CPUs have more overhead for gather. The specified overhead is relative
// to the Load operation. "2" is the number provided by Intel architects. This
// parameter is used for cost estimation of Gather Op and comparison with
// other alternatives.
// TODO: Remove the explicit hasAVX512()?, That would mean we would only
// enable gather with a -march.
+
+ // AMD znver4+ targets enable per-shape costs measured on the hardware via
+ // TuningPreferAMDZenGSCost (set in ZN4Tuning). Pre-AVX-512 Zen parts
+ // (znver1..3) take the scalarise path for masked gather and never reach
+ // this code, so the table only needs to cover AVX-512 widths.
+ if (ST->hasPreferAMDZenGSCost() && SrcVTy) {
+ // Per-shape gather costs for AMD znver4+ targets.
+ //
+ // The numbers are the empirical "break-even" (lower-bound) costs
+ // measured by sweeping a forced gather cost while compiling a
+ // controlled gather micro-benchmark and observing the point at which
+ // the LoopVectorizer still chose the gather lowering over the scalar
+ // fallback. The sweep was run independently for every (data type,
+ // VF) combination on Genoa / Milan / Turin and re-validated on Zen 5;
+ // the value tabulated below is the cost at which gather emission
+ // was the right call for that shape.
+ //
+ // i64 entries were measured separately and intentionally exceed the
+ // f64 entries for the same VF: the scalar alternative for i64 runs
+ // on the integer pipeline (faster than f64 on the FP pipeline), so
+ // gather has to be cheaper to win. At the f64-style break-even, i64
+ // gather was 1.7-3.5x slower than the scalarised lowering across
+ // stride patterns, so the i64 break-even sits at the minimum cost
+ // that suppresses vpgatherqq emission on this microbench.
+ // VF=2 is force-scalarised on AVX-512 (forceScalarizeMaskedGather)
+ // and v16f64 is split via type legalisation in getGSVectorCost
+ // (1024-bit data exceeds zmm), so neither shape ever reaches this
+ // lookup -- corresponding rows are omitted to make the live set
+ // explicit. Adding rows for them would have no observable effect.
+ static const CostTblEntry ZenGatherCostTable[] = {
+ {ISD::LOAD, MVT::v4i32, 7}, {ISD::LOAD, MVT::v8i32, 17},
+ {ISD::LOAD, MVT::v16i32, 14},
+ {ISD::LOAD, MVT::v4f32, 7}, {ISD::LOAD, MVT::v8f32, 17},
+ {ISD::LOAD, MVT::v16f32, 14},
+ {ISD::LOAD, MVT::v4f64, 7}, {ISD::LOAD, MVT::v8f64, 17},
+ {ISD::LOAD, MVT::v4i64, 10}, {ISD::LOAD, MVT::v8i64, 22},
+ };
+ EVT VT = TLI->getValueType(DL, SrcVTy);
+ if (VT.isSimple())
+ if (const auto *E = CostTableLookup(ZenGatherCostTable, ISD::LOAD,
+ VT.getSimpleVT()))
+ return E->Cost;
+ }
+
if (ST->hasAVX512() || (ST->hasAVX2() && ST->hasFastGather()))
return 2;
return 1024;
}
-int X86TTIImpl::getScatterOverhead() const {
+int X86TTIImpl::getScatterOverhead(Type *SrcVTy) const {
+ // AMD znver4+ targets use per-shape scatter costs measured on the hardware
+ // via TuningPreferAMDZenGSCost (set in ZN4Tuning). Fall through to the
+ // generic flat overhead for shapes we have not characterised.
+ if (ST->hasPreferAMDZenGSCost() && ST->hasAVX512() && SrcVTy) {
+ // Per-shape scatter costs for AMD znver4+ targets, measured with the
+ // same break-even methodology as the gather table above. The
+ // original sweep characterised i32 and f64 lanes; the f32 rows
+ // mirror i32 because vpscatterdd and vscatterdps are the same
+ // physical 16-lane 32-bit scatter on Zen and were measured as
+ // runtime-equivalent (within 3% across VF and stride patterns).
+ // i64 entries match the i64 gather rationale: scatter is harmful
+ // for all stride patterns tested (1.2-1.4x slower than scalarised
+ // on Zen 5), so the entry is set to the minimum cost that
+ // suppresses vpscatterqq emission.
+ // As with the gather table: VF=2 is force-scalarised and v16f64 is
+ // split via type legalisation, so those rows are omitted.
+ static const CostTblEntry ZenScatterCostTable[] = {
+ {ISD::STORE, MVT::v4i32, 12}, {ISD::STORE, MVT::v8i32, 14},
+ {ISD::STORE, MVT::v16i32, 6},
+ {ISD::STORE, MVT::v4f32, 12}, {ISD::STORE, MVT::v8f32, 14},
+ {ISD::STORE, MVT::v16f32, 6},
+ {ISD::STORE, MVT::v4f64, 5}, {ISD::STORE, MVT::v8f64, 15},
+ {ISD::STORE, MVT::v4i64, 10}, {ISD::STORE, MVT::v8i64, 22},
+ };
+ EVT VT = TLI->getValueType(DL, SrcVTy);
+ if (VT.isSimple())
+ if (const auto *E = CostTableLookup(ZenScatterCostTable, ISD::STORE,
+ VT.getSimpleVT()))
+ return E->Cost;
+ }
+
if (ST->hasAVX512())
return 2;
@@ -6447,8 +6522,9 @@ InstructionCost X86TTIImpl::getGSVectorCost(unsigned Opcode,
// The gather / scatter cost is given by Intel architects. It is a rough
// number since we are looking at one instruction in a time.
- const int GSOverhead = (Opcode == Instruction::Load) ? getGatherOverhead()
- : getScatterOverhead();
+ const int GSOverhead = (Opcode == Instruction::Load)
+ ? getGatherOverhead(SrcVTy)
+ : getScatterOverhead(SrcVTy);
return GSOverhead + VF * getMemoryOpCost(Opcode, SrcVTy->getScalarType(),
Alignment, AddressSpace, CostKind);
}
diff --git a/llvm/lib/Target/X86/X86TargetTransformInfo.h b/llvm/lib/Target/X86/X86TargetTransformInfo.h
index 4120421622b21..f44caa769be64 100644
--- a/llvm/lib/Target/X86/X86TargetTransformInfo.h
+++ b/llvm/lib/Target/X86/X86TargetTransformInfo.h
@@ -348,8 +348,8 @@ class X86TTIImpl final : public BasicTTIImplBase<X86TTIImpl> {
Type *DataTy, const Value *Ptr,
Align Alignment, unsigned AddressSpace) const;
- int getGatherOverhead() const;
- int getScatterOverhead() const;
+ int getGatherOverhead(Type *SrcVTy) const;
+ int getScatterOverhead(Type *SrcVTy) const;
/// @}
};
diff --git a/llvm/test/Analysis/CostModel/X86/masked-gather-scatter-amd-zen.ll b/llvm/test/Analysis/CostModel/X86/masked-gather-scatter-amd-zen.ll
new file mode 100644
index 0000000000000..f2bce4b460f77
--- /dev/null
+++ b/llvm/test/Analysis/CostModel/X86/masked-gather-scatter-amd-zen.ll
@@ -0,0 +1,612 @@
+; NOTE: Assertions have been autogenerated by utils/update_analyze_test_checks.py
+; Cost-model coverage for AMD Zen-tuned masked gather/scatter overheads.
+;
+; ZNVER4 / ZNVER5 enable the per-shape Zen cost tables via
+; TuningPreferAMDZenGSCost (set in ZN4Tuning and inherited by ZN5Tuning) and
+; have AVX-512, so the new tables are consulted in getGSVectorCost.
+; ZNVER3 does NOT carry TuningPreferAMDZenGSCost and lacks both AVX-512 and
+; TuningFastGather, so isLegalMaskedGather() returns false and the cost model
+; walks the scalarise path (getGSScalarCost). The ZNVER3 numbers below are the
+; unchanged scalar fallback cost, included here only to lock in that this
+; change does not regress pre-AVX-512 Zen targets.
+; SKX is a non-Zen AVX-512 baseline showing the generic flat overhead of 2.
+;
+; RUN: opt < %s -S -mtriple=x86_64-unknown-linux-gnu -passes="print<cost-model>" 2>&1 -disable-output -cost-kind=throughput -mcpu=znver4 | FileCheck %s --check-prefix=ZNVER4
+; RUN: opt < %s -S -mtriple=x86_64-unknown-linux-gnu -passes="print<cost-model>" 2>&1 -disable-output -cost-kind=throughput -mcpu=znver5 | FileCheck %s --check-prefix=ZNVER5
+; RUN: opt < %s -S -mtriple=x86_64-unknown-linux-gnu -passes="print<cost-model>" 2>&1 -disable-output -cost-kind=throughput -mcpu=znver3 | FileCheck %s --check-prefix=ZNVER3
+; RUN: opt < %s -S -mtriple=x86_64-unknown-linux-gnu -passes="print<cost-model>" 2>&1 -disable-output -cost-kind=throughput -mcpu=skx | FileCheck %s --check-prefix=SKX
+
+;------------------------------------------------------------------------------
+; Masked gather - i32 element type
+;------------------------------------------------------------------------------
+
+define <2 x i32> @gather_v2i32(<2 x ptr> %ptrs, <2 x i1> %mask) {
+; ZNVER4-LABEL: 'gather_v2i32'
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i32> @llvm.masked.gather.v2i32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x i32> undef)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i32> %v
+;
+; ZNVER5-LABEL: 'gather_v2i32'
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i32> @llvm.masked.gather.v2i32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x i32> undef)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i32> %v
+;
+; ZNVER3-LABEL: 'gather_v2i32'
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x i32> @llvm.masked.gather.v2i32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x i32> undef)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i32> %v
+;
+; SKX-LABEL: 'gather_v2i32'
+; SKX-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i32> @llvm.masked.gather.v2i32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x i32> undef)
+; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i32> %v
+;
+ %v = call <2 x i32> @llvm.masked.gather.v2i32.v2p0(<2 x ptr> %ptrs, i32 4, <2 x i1> %mask, <2 x i32> undef)
+ ret <2 x i32> %v
+}
+
+define <4 x i32> @gather_v4i32(<4 x ptr> %ptrs, <4 x i1> %mask) {
+; ZNVER4-LABEL: 'gather_v4i32'
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 11 for instruction: %v = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x i32> undef)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i32> %v
+;
+; ZNVER5-LABEL: 'gather_v4i32'
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 11 for instruction: %v = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x i32> undef)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i32> %v
+;
+; ZNVER3-LABEL: 'gather_v4i32'
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 14 for instruction: %v = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x i32> undef)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i32> %v
+;
+; SKX-LABEL: 'gather_v4i32'
+; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x i32> undef)
+; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i32> %v
+;
+ %v = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> %ptrs, i32 4, <4 x i1> %mask, <4 x i32> undef)
+ ret <4 x i32> %v
+}
+
+define <8 x i32> @gather_v8i32(<8 x ptr> %ptrs, <8 x i1> %mask) {
+; ZNVER4-LABEL: 'gather_v8i32'
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x i32> undef)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i32> %v
+;
+; ZNVER5-LABEL: 'gather_v8i32'
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x i32> undef)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i32> %v
+;
+; ZNVER3-LABEL: 'gather_v8i32'
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 28 for instruction: %v = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x i32> undef)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i32> %v
+;
+; SKX-LABEL: 'gather_v8i32'
+; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x i32> undef)
+; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i32> %v
+;
+ %v = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> %ptrs, i32 4, <8 x i1> %mask, <8 x i32> undef)
+ ret <8 x i32> %v
+}
+
+define <16 x i32> @gather_v16i32(<16 x ptr> %ptrs, <16 x i1> %mask) {
+; ZNVER4-LABEL: 'gather_v16i32'
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 50 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> undef)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i32> %v
+;
+; ZNVER5-LABEL: 'gather_v16i32'
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 50 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> undef)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i32> %v
+;
+; ZNVER3-LABEL: 'gather_v16i32'
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 55 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> undef)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i32> %v
+;
+; SKX-LABEL: 'gather_v16i32'
+; SKX-NEXT: Cost Model: Found an estimated cost of 20 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> undef)
+; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i32> %v
+;
+ %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> %ptrs, i32 4, <16 x i1> %mask, <16 x i32> undef)
+ ret <16 x i32> %v
+}
+
+;------------------------------------------------------------------------------
+; Masked gather - i64 element type
+;------------------------------------------------------------------------------
+
+define <2 x i64> @gather_v2i64(<2 x ptr> %ptrs, <2 x i1> %mask) {
+; ZNVER4-LABEL: 'gather_v2i64'
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i64> @llvm.masked.gather.v2i64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x i64> undef)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i64> %v
+;
+; ZNVER5-LABEL: 'gather_v2i64'
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i64> @llvm.masked.gather.v2i64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x i64> undef)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i64> %v
+;
+; ZNVER3-LABEL: 'gather_v2i64'
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x i64> @llvm.masked.gather.v2i64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x i64> undef)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i64> %v
+;
+; SKX-LABEL: 'gather_v2i64'
+; SKX-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i64> @llvm.masked.gather.v2i64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x i64> undef)
+; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i64> %v
+;
+ %v = call <2 x i64> @llvm.masked.gather.v2i64.v2p0(<2 x ptr> %ptrs, i32 8, <2 x i1> %mask, <2 x i64> undef)
+ ret <2 x i64> %v
+}
+
+define <4 x i64> @gather_v4i64(<4 x ptr> %ptrs, <4 x i1> %mask) {
+; ZNVER4-LABEL: 'gather_v4i64'
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 14 for instruction: %v = call <4 x i64> @llvm.masked.gather.v4i64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x i64> undef)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i64> %v
+;
+; ZNVER5-LABEL: 'gather_v4i64'
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 14 for instruction: %v = call <4 x i64> @llvm.masked.gather.v4i64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x i64> undef)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i64> %v
+;
+; ZNVER3-LABEL: 'gather_v4i64'
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 15 for instruction: %v = call <4 x i64> @llvm.masked.gather.v4i64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x i64> undef)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i64> %v
+;
+; SKX-LABEL: 'gather_v4i64'
+; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x i64> @llvm.masked.gather.v4i64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x i64> undef)
+; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i64> %v
+;
+ %v = call <4 x i64> @llvm.masked.gather.v4i64.v4p0(<4 x ptr> %ptrs, i32 8, <4 x i1> %mask, <4 x i64> undef)
+ ret <4 x i64> %v
+}
+
+define <8 x i64> @gather_v8i64(<8 x ptr> %ptrs, <8 x i1> %mask) {
+; ZNVER4-LABEL: 'gather_v8i64'
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 30 for instruction: %v = call <8 x i64> @llvm.masked.gather.v8i64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x i64> undef)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i64> %v
+;
+; ZNVER5-LABEL: 'gather_v8i64'
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 30 for instruction: %v = call <8 x i64> @llvm.masked.gather.v8i64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x i64> undef)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i64> %v
+;
+; ZNVER3-LABEL: 'gather_v8i64'
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 29 for instruction: %v = call <8 x i64> @llvm.masked.gather.v8i64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x i64> undef)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i64> %v
+;
+; SKX-LABEL: 'gather_v8i64'
+; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x i64> @llvm.masked.gather.v8i64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x i64> undef)
+; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i64> %v
+;
+ %v = call <8 x i64> @llvm.masked.gather.v8i64.v8p0(<8 x ptr> %ptrs, i32 8, <8 x i1> %mask, <8 x i64> undef)
+ ret <8 x i64> %v
+}
+
+;------------------------------------------------------------------------------
+; Masked gather - f32 element type
+;------------------------------------------------------------------------------
+
+define <2 x float> @gather_v2f32(<2 x ptr> %ptrs, <2 x i1> %mask) {
+; ZNVER4-LABEL: 'gather_v2f32'
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x float> @llvm.masked.gather.v2f32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x float> undef)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x float> %v
+;
+; ZNVER5-LABEL: 'gather_v2f32'
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x float> @llvm.masked.gather.v2f32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x float> undef)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x float> %v
+;
+; ZNVER3-LABEL: 'gather_v2f32'
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <2 x float> @llvm.masked.gather.v2f32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x float> undef)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x float> %v
+;
+; SKX-LABEL: 'gather_v2f32'
+; SKX-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x float> @llvm.masked.gather.v2f32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x float> undef)
+; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x float> %v
+;
+ %v = call <2 x float> @llvm.masked.gather.v2f32.v2p0(<2 x ptr> %ptrs, i32 4, <2 x i1> %mask, <2 x float> undef)
+ ret <2 x float> %v
+}
+
+define <4 x float> @gather_v4f32(<4 x ptr> %ptrs, <4 x i1> %mask) {
+; ZNVER4-LABEL: 'gather_v4f32'
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 11 for instruction: %v = call <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x float> undef)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x float> %v
+;
+; ZNVER5-LABEL: 'gather_v4f32'
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 11 for instruction: %v = call <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x float> undef)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x float> %v
+;
+; ZNVER3-LABEL: 'gather_v4f32'
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 13 for instruction: %v = call <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x float> undef)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x float> %v
+;
+; SKX-LABEL: 'gather_v4f32'
+; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x float> undef)
+; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x float> %v
+;
+ %v = call <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr> %ptrs, i32 4, <4 x i1> %mask, <4 x float> undef)
+ ret <4 x float> %v
+}
+
+define <8 x float> @gather_v8f32(<8 x ptr> %ptrs, <8 x i1> %mask) {
+; ZNVER4-LABEL: 'gather_v8f32'
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x float> undef)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x float> %v
+;
+; ZNVER5-LABEL: 'gather_v8f32'
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x float> undef)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x float> %v
+;
+; ZNVER3-LABEL: 'gather_v8f32'
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 26 for instruction: %v = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x float> undef)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x float> %v
+;
+; SKX-LABEL: 'gather_v8f32'
+; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x float> undef)
+; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x float> %v
+;
+ %v = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> %ptrs, i32 4, <8 x i1> %mask, <8 x float> undef)
+ ret <8 x float> %v
+}
+
+define <16 x float> @gather_v16f32(<16 x ptr> %ptrs, <16 x i1> %mask) {
+; ZNVER4-LABEL: 'gather_v16f32'
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 50 for instruction: %v = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x float> undef)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x float> %v
+;
+; ZNVER5-LABEL: 'gather_v16f32'
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 50 for instruction: %v = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x float> undef)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x float> %v
+;
+; ZNVER3-LABEL: 'gather_v16f32'
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 51 for instruction: %v = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x float> undef)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x float> %v
+;
+; SKX-LABEL: 'gather_v16f32'
+; SKX-NEXT: Cost Model: Found an estimated cost of 20 for instruction: %v = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x float> undef)
+; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x float> %v
+;
+ %v = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> %ptrs, i32 4, <16 x i1> %mask, <16 x float> undef)
+ ret <16 x float> %v
+}
+
+;------------------------------------------------------------------------------
+; Masked gather - f64 element type
+;------------------------------------------------------------------------------
+
+define <2 x double> @gather_v2f64(<2 x ptr> %ptrs, <2 x i1> %mask) {
+; ZNVER4-LABEL: 'gather_v2f64'
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x double> @llvm.masked.gather.v2f64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x double> undef)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x double> %v
+;
+; ZNVER5-LABEL: 'gather_v2f64'
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x double> @llvm.masked.gather.v2f64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x double> undef)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x double> %v
+;
+; ZNVER3-LABEL: 'gather_v2f64'
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <2 x double> @llvm.masked.gather.v2f64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x double> undef)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x double> %v
+;
+; SKX-LABEL: 'gather_v2f64'
+; SKX-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x double> @llvm.masked.gather.v2f64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x double> undef)
+; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x double> %v
+;
+ %v = call <2 x double> @llvm.masked.gather.v2f64.v2p0(<2 x ptr> %ptrs, i32 8, <2 x i1> %mask, <2 x double> undef)
+ ret <2 x double> %v
+}
+
+define <4 x double> @gather_v4f64(<4 x ptr> %ptrs, <4 x i1> %mask) {
+; ZNVER4-LABEL: 'gather_v4f64'
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 11 for instruction: %v = call <4 x double> @llvm.masked.gather.v4f64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x double> undef)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x double> %v
+;
+; ZNVER5-LABEL: 'gather_v4f64'
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 11 for instruction: %v = call <4 x double> @llvm.masked.gather.v4f64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x double> undef)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x double> %v
+;
+; ZNVER3-LABEL: 'gather_v4f64'
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 13 for instruction: %v = call <4 x double> @llvm.masked.gather.v4f64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x double> undef)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x double> %v
+;
+; SKX-LABEL: 'gather_v4f64'
+; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x double> @llvm.masked.gather.v4f64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x double> undef)
+; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x double> %v
+;
+ %v = call <4 x double> @llvm.masked.gather.v4f64.v4p0(<4 x ptr> %ptrs, i32 8, <4 x i1> %mask, <4 x double> undef)
+ ret <4 x double> %v
+}
+
+define <8 x double> @gather_v8f64(<8 x ptr> %ptrs, <8 x i1> %mask) {
+; ZNVER4-LABEL: 'gather_v8f64'
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x double> @llvm.masked.gather.v8f64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x double> undef)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x double> %v
+;
+; ZNVER5-LABEL: 'gather_v8f64'
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x double> @llvm.masked.gather.v8f64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x double> undef)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x double> %v
+;
+; ZNVER3-LABEL: 'gather_v8f64'
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x double> @llvm.masked.gather.v8f64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x double> undef)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x double> %v
+;
+; SKX-LABEL: 'gather_v8f64'
+; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x double> @llvm.masked.gather.v8f64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x double> undef)
+; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x double> %v
+;
+ %v = call <8 x double> @llvm.masked.gather.v8f64.v8p0(<8 x ptr> %ptrs, i32 8, <8 x i1> %mask, <8 x double> undef)
+ ret <8 x double> %v
+}
+
+;------------------------------------------------------------------------------
+; Masked scatter - i32 element type
+;------------------------------------------------------------------------------
+
+define void @scatter_v4i32(<4 x i32> %src, <4 x ptr> %ptrs, <4 x i1> %mask) {
+; ZNVER4-LABEL: 'scatter_v4i32'
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 16 for instruction: call void @llvm.masked.scatter.v4i32.v4p0(<4 x i32> %src, <4 x ptr> align 4 %ptrs, <4 x i1> %mask)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; ZNVER5-LABEL: 'scatter_v4i32'
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 16 for instruction: call void @llvm.masked.scatter.v4i32.v4p0(<4 x i32> %src, <4 x ptr> align 4 %ptrs, <4 x i1> %mask)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; ZNVER3-LABEL: 'scatter_v4i32'
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 14 for instruction: call void @llvm.masked.scatter.v4i32.v4p0(<4 x i32> %src, <4 x ptr> align 4 %ptrs, <4 x i1> %mask)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; SKX-LABEL: 'scatter_v4i32'
+; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: call void @llvm.masked.scatter.v4i32.v4p0(<4 x i32> %src, <4 x ptr> align 4 %ptrs, <4 x i1> %mask)
+; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+ call void @llvm.masked.scatter.v4i32.v4p0(<4 x i32> %src, <4 x ptr> %ptrs, i32 4, <4 x i1> %mask)
+ ret void
+}
+
+define void @scatter_v8i32(<8 x i32> %src, <8 x ptr> %ptrs, <8 x i1> %mask) {
+; ZNVER4-LABEL: 'scatter_v8i32'
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 22 for instruction: call void @llvm.masked.scatter.v8i32.v8p0(<8 x i32> %src, <8 x ptr> align 4 %ptrs, <8 x i1> %mask)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; ZNVER5-LABEL: 'scatter_v8i32'
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 22 for instruction: call void @llvm.masked.scatter.v8i32.v8p0(<8 x i32> %src, <8 x ptr> align 4 %ptrs, <8 x i1> %mask)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; ZNVER3-LABEL: 'scatter_v8i32'
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 28 for instruction: call void @llvm.masked.scatter.v8i32.v8p0(<8 x i32> %src, <8 x ptr> align 4 %ptrs, <8 x i1> %mask)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; SKX-LABEL: 'scatter_v8i32'
+; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: call void @llvm.masked.scatter.v8i32.v8p0(<8 x i32> %src, <8 x ptr> align 4 %ptrs, <8 x i1> %mask)
+; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+ call void @llvm.masked.scatter.v8i32.v8p0(<8 x i32> %src, <8 x ptr> %ptrs, i32 4, <8 x i1> %mask)
+ ret void
+}
+
+define void @scatter_v16i32(<16 x i32> %src, <16 x ptr> %ptrs, <16 x i1> %mask) {
+; ZNVER4-LABEL: 'scatter_v16i32'
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 44 for instruction: call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32> %src, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; ZNVER5-LABEL: 'scatter_v16i32'
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 44 for instruction: call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32> %src, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; ZNVER3-LABEL: 'scatter_v16i32'
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 55 for instruction: call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32> %src, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; SKX-LABEL: 'scatter_v16i32'
+; SKX-NEXT: Cost Model: Found an estimated cost of 20 for instruction: call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32> %src, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
+; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+ call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32> %src, <16 x ptr> %ptrs, i32 4, <16 x i1> %mask)
+ ret void
+}
+
+;------------------------------------------------------------------------------
+; Masked scatter - i64 element type
+;------------------------------------------------------------------------------
+
+define void @scatter_v4i64(<4 x i64> %src, <4 x ptr> %ptrs, <4 x i1> %mask) {
+; ZNVER4-LABEL: 'scatter_v4i64'
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 14 for instruction: call void @llvm.masked.scatter.v4i64.v4p0(<4 x i64> %src, <4 x ptr> align 8 %ptrs, <4 x i1> %mask)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; ZNVER5-LABEL: 'scatter_v4i64'
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 14 for instruction: call void @llvm.masked.scatter.v4i64.v4p0(<4 x i64> %src, <4 x ptr> align 8 %ptrs, <4 x i1> %mask)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; ZNVER3-LABEL: 'scatter_v4i64'
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 15 for instruction: call void @llvm.masked.scatter.v4i64.v4p0(<4 x i64> %src, <4 x ptr> align 8 %ptrs, <4 x i1> %mask)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; SKX-LABEL: 'scatter_v4i64'
+; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: call void @llvm.masked.scatter.v4i64.v4p0(<4 x i64> %src, <4 x ptr> align 8 %ptrs, <4 x i1> %mask)
+; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+ call void @llvm.masked.scatter.v4i64.v4p0(<4 x i64> %src, <4 x ptr> %ptrs, i32 8, <4 x i1> %mask)
+ ret void
+}
+
+define void @scatter_v8i64(<8 x i64> %src, <8 x ptr> %ptrs, <8 x i1> %mask) {
+; ZNVER4-LABEL: 'scatter_v8i64'
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 30 for instruction: call void @llvm.masked.scatter.v8i64.v8p0(<8 x i64> %src, <8 x ptr> align 8 %ptrs, <8 x i1> %mask)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; ZNVER5-LABEL: 'scatter_v8i64'
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 30 for instruction: call void @llvm.masked.scatter.v8i64.v8p0(<8 x i64> %src, <8 x ptr> align 8 %ptrs, <8 x i1> %mask)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; ZNVER3-LABEL: 'scatter_v8i64'
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 29 for instruction: call void @llvm.masked.scatter.v8i64.v8p0(<8 x i64> %src, <8 x ptr> align 8 %ptrs, <8 x i1> %mask)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; SKX-LABEL: 'scatter_v8i64'
+; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: call void @llvm.masked.scatter.v8i64.v8p0(<8 x i64> %src, <8 x ptr> align 8 %ptrs, <8 x i1> %mask)
+; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+ call void @llvm.masked.scatter.v8i64.v8p0(<8 x i64> %src, <8 x ptr> %ptrs, i32 8, <8 x i1> %mask)
+ ret void
+}
+
+;------------------------------------------------------------------------------
+; Masked scatter - f32 element type
+;------------------------------------------------------------------------------
+
+define void @scatter_v4f32(<4 x float> %src, <4 x ptr> %ptrs, <4 x i1> %mask) {
+; ZNVER4-LABEL: 'scatter_v4f32'
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 16 for instruction: call void @llvm.masked.scatter.v4f32.v4p0(<4 x float> %src, <4 x ptr> align 4 %ptrs, <4 x i1> %mask)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; ZNVER5-LABEL: 'scatter_v4f32'
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 16 for instruction: call void @llvm.masked.scatter.v4f32.v4p0(<4 x float> %src, <4 x ptr> align 4 %ptrs, <4 x i1> %mask)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; ZNVER3-LABEL: 'scatter_v4f32'
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 13 for instruction: call void @llvm.masked.scatter.v4f32.v4p0(<4 x float> %src, <4 x ptr> align 4 %ptrs, <4 x i1> %mask)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; SKX-LABEL: 'scatter_v4f32'
+; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: call void @llvm.masked.scatter.v4f32.v4p0(<4 x float> %src, <4 x ptr> align 4 %ptrs, <4 x i1> %mask)
+; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+ call void @llvm.masked.scatter.v4f32.v4p0(<4 x float> %src, <4 x ptr> %ptrs, i32 4, <4 x i1> %mask)
+ ret void
+}
+
+define void @scatter_v8f32(<8 x float> %src, <8 x ptr> %ptrs, <8 x i1> %mask) {
+; ZNVER4-LABEL: 'scatter_v8f32'
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 22 for instruction: call void @llvm.masked.scatter.v8f32.v8p0(<8 x float> %src, <8 x ptr> align 4 %ptrs, <8 x i1> %mask)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; ZNVER5-LABEL: 'scatter_v8f32'
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 22 for instruction: call void @llvm.masked.scatter.v8f32.v8p0(<8 x float> %src, <8 x ptr> align 4 %ptrs, <8 x i1> %mask)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; ZNVER3-LABEL: 'scatter_v8f32'
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 26 for instruction: call void @llvm.masked.scatter.v8f32.v8p0(<8 x float> %src, <8 x ptr> align 4 %ptrs, <8 x i1> %mask)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; SKX-LABEL: 'scatter_v8f32'
+; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: call void @llvm.masked.scatter.v8f32.v8p0(<8 x float> %src, <8 x ptr> align 4 %ptrs, <8 x i1> %mask)
+; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+ call void @llvm.masked.scatter.v8f32.v8p0(<8 x float> %src, <8 x ptr> %ptrs, i32 4, <8 x i1> %mask)
+ ret void
+}
+
+define void @scatter_v16f32(<16 x float> %src, <16 x ptr> %ptrs, <16 x i1> %mask) {
+; ZNVER4-LABEL: 'scatter_v16f32'
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 44 for instruction: call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> %src, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; ZNVER5-LABEL: 'scatter_v16f32'
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 44 for instruction: call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> %src, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; ZNVER3-LABEL: 'scatter_v16f32'
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 51 for instruction: call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> %src, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; SKX-LABEL: 'scatter_v16f32'
+; SKX-NEXT: Cost Model: Found an estimated cost of 20 for instruction: call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> %src, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
+; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+ call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> %src, <16 x ptr> %ptrs, i32 4, <16 x i1> %mask)
+ ret void
+}
+
+;------------------------------------------------------------------------------
+; Masked scatter - f64 element type
+;------------------------------------------------------------------------------
+
+define void @scatter_v4f64(<4 x double> %src, <4 x ptr> %ptrs, <4 x i1> %mask) {
+; ZNVER4-LABEL: 'scatter_v4f64'
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 9 for instruction: call void @llvm.masked.scatter.v4f64.v4p0(<4 x double> %src, <4 x ptr> align 8 %ptrs, <4 x i1> %mask)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; ZNVER5-LABEL: 'scatter_v4f64'
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 9 for instruction: call void @llvm.masked.scatter.v4f64.v4p0(<4 x double> %src, <4 x ptr> align 8 %ptrs, <4 x i1> %mask)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; ZNVER3-LABEL: 'scatter_v4f64'
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 13 for instruction: call void @llvm.masked.scatter.v4f64.v4p0(<4 x double> %src, <4 x ptr> align 8 %ptrs, <4 x i1> %mask)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; SKX-LABEL: 'scatter_v4f64'
+; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: call void @llvm.masked.scatter.v4f64.v4p0(<4 x double> %src, <4 x ptr> align 8 %ptrs, <4 x i1> %mask)
+; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+ call void @llvm.masked.scatter.v4f64.v4p0(<4 x double> %src, <4 x ptr> %ptrs, i32 8, <4 x i1> %mask)
+ ret void
+}
+
+define void @scatter_v8f64(<8 x double> %src, <8 x ptr> %ptrs, <8 x i1> %mask) {
+; ZNVER4-LABEL: 'scatter_v8f64'
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 23 for instruction: call void @llvm.masked.scatter.v8f64.v8p0(<8 x double> %src, <8 x ptr> align 8 %ptrs, <8 x i1> %mask)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; ZNVER5-LABEL: 'scatter_v8f64'
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 23 for instruction: call void @llvm.masked.scatter.v8f64.v8p0(<8 x double> %src, <8 x ptr> align 8 %ptrs, <8 x i1> %mask)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; ZNVER3-LABEL: 'scatter_v8f64'
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 25 for instruction: call void @llvm.masked.scatter.v8f64.v8p0(<8 x double> %src, <8 x ptr> align 8 %ptrs, <8 x i1> %mask)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; SKX-LABEL: 'scatter_v8f64'
+; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: call void @llvm.masked.scatter.v8f64.v8p0(<8 x double> %src, <8 x ptr> align 8 %ptrs, <8 x i1> %mask)
+; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+ call void @llvm.masked.scatter.v8f64.v8p0(<8 x double> %src, <8 x ptr> %ptrs, i32 8, <8 x i1> %mask)
+ ret void
+}
+
+; Coverage for the v16 entries that are only reachable when the index can be
+; reduced to i32 (no SplitFactor in getGSVectorCost). Without this stanza, the
+; <16 x ptr>-form scatter splits into two <8 x ...> scatters and the v16 row of
+; the cost table is never consulted, masking any divergence between i32 and f32
+; scatter costs at VF=16 (vpscatterdd vs vscatterdps -- same physical store on
+; Zen, same expected cost).
+
+define void @scatter_v16f32_gep(ptr %base, <16 x i32> %idx, <16 x float> %val, <16 x i1> %mask) {
+; ZNVER4-LABEL: 'scatter_v16f32_gep'
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr float, ptr %base, <16 x i32> %idx
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 22 for instruction: call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> %val, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; ZNVER5-LABEL: 'scatter_v16f32_gep'
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr float, ptr %base, <16 x i32> %idx
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 22 for instruction: call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> %val, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; ZNVER3-LABEL: 'scatter_v16f32_gep'
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr float, ptr %base, <16 x i32> %idx
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 51 for instruction: call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> %val, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; SKX-LABEL: 'scatter_v16f32_gep'
+; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr float, ptr %base, <16 x i32> %idx
+; SKX-NEXT: Cost Model: Found an estimated cost of 18 for instruction: call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> %val, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
+; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+ %ptrs = getelementptr float, ptr %base, <16 x i32> %idx
+ call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> %val, <16 x ptr> %ptrs, i32 4, <16 x i1> %mask)
+ ret void
+}
+
+define void @scatter_v16i32_gep(ptr %base, <16 x i32> %idx, <16 x i32> %val, <16 x i1> %mask) {
+; ZNVER4-LABEL: 'scatter_v16i32_gep'
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 22 for instruction: call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32> %val, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; ZNVER5-LABEL: 'scatter_v16i32_gep'
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 22 for instruction: call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32> %val, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; ZNVER3-LABEL: 'scatter_v16i32_gep'
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 55 for instruction: call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32> %val, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; SKX-LABEL: 'scatter_v16i32_gep'
+; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
+; SKX-NEXT: Cost Model: Found an estimated cost of 18 for instruction: call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32> %val, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
+; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+ %ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
+ call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32> %val, <16 x ptr> %ptrs, i32 4, <16 x i1> %mask)
+ ret void
+}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/amd-zen-gather-scatter-decisions.ll b/llvm/test/Transforms/LoopVectorize/X86/amd-zen-gather-scatter-decisions.ll
new file mode 100644
index 0000000000000..0b187f724781d
--- /dev/null
+++ b/llvm/test/Transforms/LoopVectorize/X86/amd-zen-gather-scatter-decisions.ll
@@ -0,0 +1,97 @@
+; End-to-end loop-vectorize decisions driven by the AMD Zen per-shape
+; gather/scatter cost tables (TuningPreferAMDZenGSCost, set on znver4+).
+;
+; The companion cost-model test
+; llvm/test/Analysis/CostModel/X86/masked-gather-scatter-amd-zen.ll
+; pins individual cost numbers; this test pins the resulting vectorizer
+; decisions so future cost-model refactors that accidentally re-enable
+; harmful gathers (or suppress profitable ones) are caught here.
+;
+; The three cases below correspond to:
+; 1. f64 indirect-load reduction -- gather IS chosen on znver5
+; (the lbm-style win the cost table exists to enable).
+; 2. i64 indirect-load reduction -- gather is NOT chosen on znver5;
+; the i64 entry was measured separately and deliberately set above
+; the break-even so vpgatherqq is suppressed for harmful patterns
+; (cf. PR #198850 / libquantum regression).
+; 3. Unit-stride load -- the vectorizer must emit a plain wide load
+; (not @llvm.masked.gather) regardless of cost-table values.
+; Regression guard for issue #91370.
+;
+; RUN: opt < %s -S -passes=loop-vectorize -mtriple=x86_64-unknown-linux-gnu -mcpu=znver5 | FileCheck %s
+
+target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+
+; --- Case 1: f64 indirect-load gather IS chosen on znver5 ----------------
+; CHECK-LABEL: define double @f64_indirect_gather_chosen
+; CHECK: call <{{[0-9]+}} x double> @llvm.masked.gather.v{{[0-9]+}}f64
+define double @f64_indirect_gather_chosen(ptr noundef readonly %data, ptr noundef readonly %idx, i32 noundef %n) {
+entry:
+ %cmp = icmp ugt i32 %n, 0
+ br i1 %cmp, label %loop, label %exit
+
+loop:
+ %i = phi i32 [ 0, %entry ], [ %inc, %loop ]
+ %acc = phi double [ 0.0, %entry ], [ %acc.next, %loop ]
+ %idx.gep = getelementptr inbounds i32, ptr %idx, i32 %i
+ %idx.val = load i32, ptr %idx.gep, align 4
+ %idx.sext = sext i32 %idx.val to i64
+ %data.gep = getelementptr inbounds double, ptr %data, i64 %idx.sext
+ %data.val = load double, ptr %data.gep, align 8
+ %acc.next = fadd fast double %acc, %data.val
+ %inc = add nuw nsw i32 %i, 1
+ %done = icmp eq i32 %inc, %n
+ br i1 %done, label %exit, label %loop
+
+exit:
+ %ret = phi double [ 0.0, %entry ], [ %acc.next, %loop ]
+ ret double %ret
+}
+
+; --- Case 2: i64 indirect-load gather is NOT chosen on znver5 ------------
+; CHECK-LABEL: define i64 @i64_indirect_gather_avoided
+; CHECK-NOT: call <{{[0-9]+}} x i64> @llvm.masked.gather.v{{[0-9]+}}i64
+define i64 @i64_indirect_gather_avoided(ptr noundef readonly %data, ptr noundef readonly %idx, i32 noundef %n) {
+entry:
+ %cmp = icmp ugt i32 %n, 0
+ br i1 %cmp, label %loop, label %exit
+
+loop:
+ %i = phi i32 [ 0, %entry ], [ %inc, %loop ]
+ %acc = phi i64 [ 0, %entry ], [ %acc.next, %loop ]
+ %idx.gep = getelementptr inbounds i64, ptr %idx, i32 %i
+ %idx.val = load i64, ptr %idx.gep, align 8
+ %data.gep = getelementptr inbounds i64, ptr %data, i64 %idx.val
+ %data.val = load i64, ptr %data.gep, align 8
+ %acc.next = add i64 %acc, %data.val
+ %inc = add nuw nsw i32 %i, 1
+ %done = icmp eq i32 %inc, %n
+ br i1 %done, label %exit, label %loop
+
+exit:
+ %ret = phi i64 [ 0, %entry ], [ %acc.next, %loop ]
+ ret i64 %ret
+}
+
+; --- Case 3: unit-stride load must NOT become a gather (#91370 guard) -----
+; CHECK-LABEL: define void @unit_stride_no_gather
+; CHECK-NOT: call <{{[0-9]+}} x double> @llvm.masked.gather
+define void @unit_stride_no_gather(ptr noundef writeonly %out, ptr noundef readonly %in, i32 noundef %n) {
+entry:
+ %cmp = icmp ugt i32 %n, 0
+ br i1 %cmp, label %loop, label %exit
+
+loop:
+ %i = phi i32 [ 0, %entry ], [ %inc, %loop ]
+ %in.gep = getelementptr inbounds double, ptr %in, i32 %i
+ %in.val = load double, ptr %in.gep, align 8
+ %mul = fmul fast double %in.val, 2.000000e+00
+ %out.gep = getelementptr inbounds double, ptr %out, i32 %i
+ store double %mul, ptr %out.gep, align 8
+ %inc = add nuw nsw i32 %i, 1
+ %done = icmp eq i32 %inc, %n
+ br i1 %done, label %exit, label %loop
+
+exit:
+ ret void
+}
>From 3ba555a3a4a75ffa4232339ae566c863066590ff Mon Sep 17 00:00:00 2001
From: Sumukh Bharadwaj <Sumukh.Bharadwaj at amd.com>
Date: Thu, 28 May 2026 09:39:47 +0530
Subject: [PATCH 2/6] [X86][CostModel] Address reviewer follow-up on Zen GS
tests
Two test-only nits from review of #199488:
1. The LoopVectorize test had `CHECK-NOT: @llvm.masked.gather` on Case 2
(i64 gather avoided) and Case 3 (unit-stride no gather) without a
positive anchor, so the check would pass vacuously if the loop ever
failed to vectorize at all (rather than vectorizing without a gather).
Adding `CHECK: vector.body` in front of each `CHECK-NOT` distinguishes
the two outcomes; under `-force-vector-width=1` both new CHECKs now
correctly fail.
2. The cost-model test had `scatter_v16{i32,f32}_gep` to exercise the
GEP-index reducibility path for the v16 scatter row but no analogous
case for gather. Added `gather_v16i32_gep` (cost = 30 on znver4/5).
Both i32 and f32 GEP cases for gather would share the same code path,
so one case is sufficient for the v16 gather row; the section comment
is updated to make that explicit.
No behavior change. Both tests pass.
---
.../X86/masked-gather-scatter-amd-zen.ll | 37 +++++++++++++++++--
.../X86/amd-zen-gather-scatter-decisions.ll | 7 ++++
2 files changed, 40 insertions(+), 4 deletions(-)
diff --git a/llvm/test/Analysis/CostModel/X86/masked-gather-scatter-amd-zen.ll b/llvm/test/Analysis/CostModel/X86/masked-gather-scatter-amd-zen.ll
index f2bce4b460f77..cbe588cce0bd8 100644
--- a/llvm/test/Analysis/CostModel/X86/masked-gather-scatter-amd-zen.ll
+++ b/llvm/test/Analysis/CostModel/X86/masked-gather-scatter-amd-zen.ll
@@ -554,10 +554,39 @@ define void @scatter_v8f64(<8 x double> %src, <8 x ptr> %ptrs, <8 x i1> %mask) {
; Coverage for the v16 entries that are only reachable when the index can be
; reduced to i32 (no SplitFactor in getGSVectorCost). Without this stanza, the
-; <16 x ptr>-form scatter splits into two <8 x ...> scatters and the v16 row of
-; the cost table is never consulted, masking any divergence between i32 and f32
-; scatter costs at VF=16 (vpscatterdd vs vscatterdps -- same physical store on
-; Zen, same expected cost).
+; <16 x ptr>-form gather/scatter splits into two <8 x ...> operations and the
+; v16 row of the cost table is never consulted, masking any divergence between
+; i32 and f32 costs at VF=16 (vpgatherdd/vpscatterdd vs vgatherdps/vscatterdps
+; -- same physical load/store on Zen, same expected cost). The scatter pair
+; below pins the scatter v16 row; gather_v16i32_gep does the same for the
+; gather v16 row (the code path that does the GEP-index reducibility check is
+; shared between gather and scatter, so one gather case suffices).
+
+define <16 x i32> @gather_v16i32_gep(ptr %base, <16 x i32> %idx, <16 x i1> %mask) {
+; ZNVER4-LABEL: 'gather_v16i32_gep'
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 30 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> undef)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i32> %v
+;
+; ZNVER5-LABEL: 'gather_v16i32_gep'
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 30 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> undef)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i32> %v
+;
+; ZNVER3-LABEL: 'gather_v16i32_gep'
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 55 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> undef)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i32> %v
+;
+; SKX-LABEL: 'gather_v16i32_gep'
+; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
+; SKX-NEXT: Cost Model: Found an estimated cost of 18 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> undef)
+; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i32> %v
+;
+ %ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
+ %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> %ptrs, i32 4, <16 x i1> %mask, <16 x i32> undef)
+ ret <16 x i32> %v
+}
define void @scatter_v16f32_gep(ptr %base, <16 x i32> %idx, <16 x float> %val, <16 x i1> %mask) {
; ZNVER4-LABEL: 'scatter_v16f32_gep'
diff --git a/llvm/test/Transforms/LoopVectorize/X86/amd-zen-gather-scatter-decisions.ll b/llvm/test/Transforms/LoopVectorize/X86/amd-zen-gather-scatter-decisions.ll
index 0b187f724781d..c1087ead1f2ad 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/amd-zen-gather-scatter-decisions.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/amd-zen-gather-scatter-decisions.ll
@@ -49,7 +49,11 @@ exit:
}
; --- Case 2: i64 indirect-load gather is NOT chosen on znver5 ------------
+; The positive CHECK on vector.body distinguishes "vectorized without a
+; gather" from "did not vectorize at all" -- without it, a future regression
+; that fails to vectorize the loop entirely would pass CHECK-NOT vacuously.
; CHECK-LABEL: define i64 @i64_indirect_gather_avoided
+; CHECK: vector.body
; CHECK-NOT: call <{{[0-9]+}} x i64> @llvm.masked.gather.v{{[0-9]+}}i64
define i64 @i64_indirect_gather_avoided(ptr noundef readonly %data, ptr noundef readonly %idx, i32 noundef %n) {
entry:
@@ -74,7 +78,10 @@ exit:
}
; --- Case 3: unit-stride load must NOT become a gather (#91370 guard) -----
+; Same vector.body anchor as Case 2: ensures the loop did vectorize (to a
+; wide load) rather than failing to vectorize entirely.
; CHECK-LABEL: define void @unit_stride_no_gather
+; CHECK: vector.body
; CHECK-NOT: call <{{[0-9]+}} x double> @llvm.masked.gather
define void @unit_stride_no_gather(ptr noundef writeonly %out, ptr noundef readonly %in, i32 noundef %n) {
entry:
>From e6125e2a5a6befb20d294c6393b77e503fca3f1d Mon Sep 17 00:00:00 2001
From: Sumukh Bharadwaj <Sumukh.Bharadwaj at amd.com>
Date: Thu, 28 May 2026 13:57:07 +0530
Subject: [PATCH 3/6] [X86][CostModel][NFC] Use poison for gather passthru in
Zen GS cost test
CI's code_formatter job runs both clang-format AND the undef-deprecator
check; the latter rejects new uses of `undef` in tests under the
LangRef poison/undef migration. The masked-gather passthru argument
was the only `undef` in the file. Replace all 75 occurrences with
`poison` (purely an operand-printing change -- gather costs and
behaviour are unaffected).
No functional change; both tests still pass.
---
.../X86/masked-gather-scatter-amd-zen.ll | 150 +++++++++---------
1 file changed, 75 insertions(+), 75 deletions(-)
diff --git a/llvm/test/Analysis/CostModel/X86/masked-gather-scatter-amd-zen.ll b/llvm/test/Analysis/CostModel/X86/masked-gather-scatter-amd-zen.ll
index cbe588cce0bd8..84c24429b9b82 100644
--- a/llvm/test/Analysis/CostModel/X86/masked-gather-scatter-amd-zen.ll
+++ b/llvm/test/Analysis/CostModel/X86/masked-gather-scatter-amd-zen.ll
@@ -22,85 +22,85 @@
define <2 x i32> @gather_v2i32(<2 x ptr> %ptrs, <2 x i1> %mask) {
; ZNVER4-LABEL: 'gather_v2i32'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i32> @llvm.masked.gather.v2i32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x i32> undef)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i32> @llvm.masked.gather.v2i32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x i32> poison)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i32> %v
;
; ZNVER5-LABEL: 'gather_v2i32'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i32> @llvm.masked.gather.v2i32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x i32> undef)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i32> @llvm.masked.gather.v2i32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x i32> poison)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i32> %v
;
; ZNVER3-LABEL: 'gather_v2i32'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x i32> @llvm.masked.gather.v2i32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x i32> undef)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x i32> @llvm.masked.gather.v2i32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x i32> poison)
; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i32> %v
;
; SKX-LABEL: 'gather_v2i32'
-; SKX-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i32> @llvm.masked.gather.v2i32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x i32> undef)
+; SKX-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i32> @llvm.masked.gather.v2i32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x i32> poison)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i32> %v
;
- %v = call <2 x i32> @llvm.masked.gather.v2i32.v2p0(<2 x ptr> %ptrs, i32 4, <2 x i1> %mask, <2 x i32> undef)
+ %v = call <2 x i32> @llvm.masked.gather.v2i32.v2p0(<2 x ptr> %ptrs, i32 4, <2 x i1> %mask, <2 x i32> poison)
ret <2 x i32> %v
}
define <4 x i32> @gather_v4i32(<4 x ptr> %ptrs, <4 x i1> %mask) {
; ZNVER4-LABEL: 'gather_v4i32'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 11 for instruction: %v = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x i32> undef)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 11 for instruction: %v = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x i32> poison)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i32> %v
;
; ZNVER5-LABEL: 'gather_v4i32'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 11 for instruction: %v = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x i32> undef)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 11 for instruction: %v = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x i32> poison)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i32> %v
;
; ZNVER3-LABEL: 'gather_v4i32'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 14 for instruction: %v = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x i32> undef)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 14 for instruction: %v = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x i32> poison)
; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i32> %v
;
; SKX-LABEL: 'gather_v4i32'
-; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x i32> undef)
+; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x i32> poison)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i32> %v
;
- %v = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> %ptrs, i32 4, <4 x i1> %mask, <4 x i32> undef)
+ %v = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> %ptrs, i32 4, <4 x i1> %mask, <4 x i32> poison)
ret <4 x i32> %v
}
define <8 x i32> @gather_v8i32(<8 x ptr> %ptrs, <8 x i1> %mask) {
; ZNVER4-LABEL: 'gather_v8i32'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x i32> undef)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x i32> poison)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i32> %v
;
; ZNVER5-LABEL: 'gather_v8i32'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x i32> undef)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x i32> poison)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i32> %v
;
; ZNVER3-LABEL: 'gather_v8i32'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 28 for instruction: %v = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x i32> undef)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 28 for instruction: %v = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x i32> poison)
; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i32> %v
;
; SKX-LABEL: 'gather_v8i32'
-; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x i32> undef)
+; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x i32> poison)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i32> %v
;
- %v = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> %ptrs, i32 4, <8 x i1> %mask, <8 x i32> undef)
+ %v = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> %ptrs, i32 4, <8 x i1> %mask, <8 x i32> poison)
ret <8 x i32> %v
}
define <16 x i32> @gather_v16i32(<16 x ptr> %ptrs, <16 x i1> %mask) {
; ZNVER4-LABEL: 'gather_v16i32'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 50 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> undef)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 50 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> poison)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i32> %v
;
; ZNVER5-LABEL: 'gather_v16i32'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 50 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> undef)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 50 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> poison)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i32> %v
;
; ZNVER3-LABEL: 'gather_v16i32'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 55 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> undef)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 55 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> poison)
; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i32> %v
;
; SKX-LABEL: 'gather_v16i32'
-; SKX-NEXT: Cost Model: Found an estimated cost of 20 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> undef)
+; SKX-NEXT: Cost Model: Found an estimated cost of 20 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> poison)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i32> %v
;
- %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> %ptrs, i32 4, <16 x i1> %mask, <16 x i32> undef)
+ %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> %ptrs, i32 4, <16 x i1> %mask, <16 x i32> poison)
ret <16 x i32> %v
}
@@ -110,64 +110,64 @@ define <16 x i32> @gather_v16i32(<16 x ptr> %ptrs, <16 x i1> %mask) {
define <2 x i64> @gather_v2i64(<2 x ptr> %ptrs, <2 x i1> %mask) {
; ZNVER4-LABEL: 'gather_v2i64'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i64> @llvm.masked.gather.v2i64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x i64> undef)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i64> @llvm.masked.gather.v2i64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x i64> poison)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i64> %v
;
; ZNVER5-LABEL: 'gather_v2i64'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i64> @llvm.masked.gather.v2i64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x i64> undef)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i64> @llvm.masked.gather.v2i64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x i64> poison)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i64> %v
;
; ZNVER3-LABEL: 'gather_v2i64'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x i64> @llvm.masked.gather.v2i64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x i64> undef)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x i64> @llvm.masked.gather.v2i64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x i64> poison)
; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i64> %v
;
; SKX-LABEL: 'gather_v2i64'
-; SKX-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i64> @llvm.masked.gather.v2i64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x i64> undef)
+; SKX-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i64> @llvm.masked.gather.v2i64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x i64> poison)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i64> %v
;
- %v = call <2 x i64> @llvm.masked.gather.v2i64.v2p0(<2 x ptr> %ptrs, i32 8, <2 x i1> %mask, <2 x i64> undef)
+ %v = call <2 x i64> @llvm.masked.gather.v2i64.v2p0(<2 x ptr> %ptrs, i32 8, <2 x i1> %mask, <2 x i64> poison)
ret <2 x i64> %v
}
define <4 x i64> @gather_v4i64(<4 x ptr> %ptrs, <4 x i1> %mask) {
; ZNVER4-LABEL: 'gather_v4i64'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 14 for instruction: %v = call <4 x i64> @llvm.masked.gather.v4i64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x i64> undef)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 14 for instruction: %v = call <4 x i64> @llvm.masked.gather.v4i64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x i64> poison)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i64> %v
;
; ZNVER5-LABEL: 'gather_v4i64'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 14 for instruction: %v = call <4 x i64> @llvm.masked.gather.v4i64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x i64> undef)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 14 for instruction: %v = call <4 x i64> @llvm.masked.gather.v4i64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x i64> poison)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i64> %v
;
; ZNVER3-LABEL: 'gather_v4i64'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 15 for instruction: %v = call <4 x i64> @llvm.masked.gather.v4i64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x i64> undef)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 15 for instruction: %v = call <4 x i64> @llvm.masked.gather.v4i64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x i64> poison)
; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i64> %v
;
; SKX-LABEL: 'gather_v4i64'
-; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x i64> @llvm.masked.gather.v4i64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x i64> undef)
+; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x i64> @llvm.masked.gather.v4i64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x i64> poison)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i64> %v
;
- %v = call <4 x i64> @llvm.masked.gather.v4i64.v4p0(<4 x ptr> %ptrs, i32 8, <4 x i1> %mask, <4 x i64> undef)
+ %v = call <4 x i64> @llvm.masked.gather.v4i64.v4p0(<4 x ptr> %ptrs, i32 8, <4 x i1> %mask, <4 x i64> poison)
ret <4 x i64> %v
}
define <8 x i64> @gather_v8i64(<8 x ptr> %ptrs, <8 x i1> %mask) {
; ZNVER4-LABEL: 'gather_v8i64'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 30 for instruction: %v = call <8 x i64> @llvm.masked.gather.v8i64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x i64> undef)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 30 for instruction: %v = call <8 x i64> @llvm.masked.gather.v8i64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x i64> poison)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i64> %v
;
; ZNVER5-LABEL: 'gather_v8i64'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 30 for instruction: %v = call <8 x i64> @llvm.masked.gather.v8i64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x i64> undef)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 30 for instruction: %v = call <8 x i64> @llvm.masked.gather.v8i64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x i64> poison)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i64> %v
;
; ZNVER3-LABEL: 'gather_v8i64'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 29 for instruction: %v = call <8 x i64> @llvm.masked.gather.v8i64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x i64> undef)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 29 for instruction: %v = call <8 x i64> @llvm.masked.gather.v8i64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x i64> poison)
; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i64> %v
;
; SKX-LABEL: 'gather_v8i64'
-; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x i64> @llvm.masked.gather.v8i64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x i64> undef)
+; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x i64> @llvm.masked.gather.v8i64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x i64> poison)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i64> %v
;
- %v = call <8 x i64> @llvm.masked.gather.v8i64.v8p0(<8 x ptr> %ptrs, i32 8, <8 x i1> %mask, <8 x i64> undef)
+ %v = call <8 x i64> @llvm.masked.gather.v8i64.v8p0(<8 x ptr> %ptrs, i32 8, <8 x i1> %mask, <8 x i64> poison)
ret <8 x i64> %v
}
@@ -177,85 +177,85 @@ define <8 x i64> @gather_v8i64(<8 x ptr> %ptrs, <8 x i1> %mask) {
define <2 x float> @gather_v2f32(<2 x ptr> %ptrs, <2 x i1> %mask) {
; ZNVER4-LABEL: 'gather_v2f32'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x float> @llvm.masked.gather.v2f32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x float> undef)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x float> @llvm.masked.gather.v2f32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x float> poison)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x float> %v
;
; ZNVER5-LABEL: 'gather_v2f32'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x float> @llvm.masked.gather.v2f32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x float> undef)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x float> @llvm.masked.gather.v2f32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x float> poison)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x float> %v
;
; ZNVER3-LABEL: 'gather_v2f32'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <2 x float> @llvm.masked.gather.v2f32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x float> undef)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <2 x float> @llvm.masked.gather.v2f32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x float> poison)
; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x float> %v
;
; SKX-LABEL: 'gather_v2f32'
-; SKX-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x float> @llvm.masked.gather.v2f32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x float> undef)
+; SKX-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x float> @llvm.masked.gather.v2f32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x float> poison)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x float> %v
;
- %v = call <2 x float> @llvm.masked.gather.v2f32.v2p0(<2 x ptr> %ptrs, i32 4, <2 x i1> %mask, <2 x float> undef)
+ %v = call <2 x float> @llvm.masked.gather.v2f32.v2p0(<2 x ptr> %ptrs, i32 4, <2 x i1> %mask, <2 x float> poison)
ret <2 x float> %v
}
define <4 x float> @gather_v4f32(<4 x ptr> %ptrs, <4 x i1> %mask) {
; ZNVER4-LABEL: 'gather_v4f32'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 11 for instruction: %v = call <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x float> undef)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 11 for instruction: %v = call <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x float> poison)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x float> %v
;
; ZNVER5-LABEL: 'gather_v4f32'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 11 for instruction: %v = call <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x float> undef)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 11 for instruction: %v = call <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x float> poison)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x float> %v
;
; ZNVER3-LABEL: 'gather_v4f32'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 13 for instruction: %v = call <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x float> undef)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 13 for instruction: %v = call <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x float> poison)
; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x float> %v
;
; SKX-LABEL: 'gather_v4f32'
-; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x float> undef)
+; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x float> poison)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x float> %v
;
- %v = call <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr> %ptrs, i32 4, <4 x i1> %mask, <4 x float> undef)
+ %v = call <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr> %ptrs, i32 4, <4 x i1> %mask, <4 x float> poison)
ret <4 x float> %v
}
define <8 x float> @gather_v8f32(<8 x ptr> %ptrs, <8 x i1> %mask) {
; ZNVER4-LABEL: 'gather_v8f32'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x float> undef)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x float> poison)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x float> %v
;
; ZNVER5-LABEL: 'gather_v8f32'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x float> undef)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x float> poison)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x float> %v
;
; ZNVER3-LABEL: 'gather_v8f32'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 26 for instruction: %v = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x float> undef)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 26 for instruction: %v = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x float> poison)
; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x float> %v
;
; SKX-LABEL: 'gather_v8f32'
-; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x float> undef)
+; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x float> poison)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x float> %v
;
- %v = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> %ptrs, i32 4, <8 x i1> %mask, <8 x float> undef)
+ %v = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> %ptrs, i32 4, <8 x i1> %mask, <8 x float> poison)
ret <8 x float> %v
}
define <16 x float> @gather_v16f32(<16 x ptr> %ptrs, <16 x i1> %mask) {
; ZNVER4-LABEL: 'gather_v16f32'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 50 for instruction: %v = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x float> undef)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 50 for instruction: %v = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x float> poison)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x float> %v
;
; ZNVER5-LABEL: 'gather_v16f32'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 50 for instruction: %v = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x float> undef)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 50 for instruction: %v = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x float> poison)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x float> %v
;
; ZNVER3-LABEL: 'gather_v16f32'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 51 for instruction: %v = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x float> undef)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 51 for instruction: %v = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x float> poison)
; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x float> %v
;
; SKX-LABEL: 'gather_v16f32'
-; SKX-NEXT: Cost Model: Found an estimated cost of 20 for instruction: %v = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x float> undef)
+; SKX-NEXT: Cost Model: Found an estimated cost of 20 for instruction: %v = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x float> poison)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x float> %v
;
- %v = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> %ptrs, i32 4, <16 x i1> %mask, <16 x float> undef)
+ %v = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> %ptrs, i32 4, <16 x i1> %mask, <16 x float> poison)
ret <16 x float> %v
}
@@ -265,64 +265,64 @@ define <16 x float> @gather_v16f32(<16 x ptr> %ptrs, <16 x i1> %mask) {
define <2 x double> @gather_v2f64(<2 x ptr> %ptrs, <2 x i1> %mask) {
; ZNVER4-LABEL: 'gather_v2f64'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x double> @llvm.masked.gather.v2f64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x double> undef)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x double> @llvm.masked.gather.v2f64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x double> poison)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x double> %v
;
; ZNVER5-LABEL: 'gather_v2f64'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x double> @llvm.masked.gather.v2f64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x double> undef)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x double> @llvm.masked.gather.v2f64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x double> poison)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x double> %v
;
; ZNVER3-LABEL: 'gather_v2f64'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <2 x double> @llvm.masked.gather.v2f64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x double> undef)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <2 x double> @llvm.masked.gather.v2f64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x double> poison)
; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x double> %v
;
; SKX-LABEL: 'gather_v2f64'
-; SKX-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x double> @llvm.masked.gather.v2f64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x double> undef)
+; SKX-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x double> @llvm.masked.gather.v2f64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x double> poison)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x double> %v
;
- %v = call <2 x double> @llvm.masked.gather.v2f64.v2p0(<2 x ptr> %ptrs, i32 8, <2 x i1> %mask, <2 x double> undef)
+ %v = call <2 x double> @llvm.masked.gather.v2f64.v2p0(<2 x ptr> %ptrs, i32 8, <2 x i1> %mask, <2 x double> poison)
ret <2 x double> %v
}
define <4 x double> @gather_v4f64(<4 x ptr> %ptrs, <4 x i1> %mask) {
; ZNVER4-LABEL: 'gather_v4f64'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 11 for instruction: %v = call <4 x double> @llvm.masked.gather.v4f64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x double> undef)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 11 for instruction: %v = call <4 x double> @llvm.masked.gather.v4f64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x double> poison)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x double> %v
;
; ZNVER5-LABEL: 'gather_v4f64'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 11 for instruction: %v = call <4 x double> @llvm.masked.gather.v4f64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x double> undef)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 11 for instruction: %v = call <4 x double> @llvm.masked.gather.v4f64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x double> poison)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x double> %v
;
; ZNVER3-LABEL: 'gather_v4f64'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 13 for instruction: %v = call <4 x double> @llvm.masked.gather.v4f64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x double> undef)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 13 for instruction: %v = call <4 x double> @llvm.masked.gather.v4f64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x double> poison)
; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x double> %v
;
; SKX-LABEL: 'gather_v4f64'
-; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x double> @llvm.masked.gather.v4f64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x double> undef)
+; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x double> @llvm.masked.gather.v4f64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x double> poison)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x double> %v
;
- %v = call <4 x double> @llvm.masked.gather.v4f64.v4p0(<4 x ptr> %ptrs, i32 8, <4 x i1> %mask, <4 x double> undef)
+ %v = call <4 x double> @llvm.masked.gather.v4f64.v4p0(<4 x ptr> %ptrs, i32 8, <4 x i1> %mask, <4 x double> poison)
ret <4 x double> %v
}
define <8 x double> @gather_v8f64(<8 x ptr> %ptrs, <8 x i1> %mask) {
; ZNVER4-LABEL: 'gather_v8f64'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x double> @llvm.masked.gather.v8f64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x double> undef)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x double> @llvm.masked.gather.v8f64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x double> poison)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x double> %v
;
; ZNVER5-LABEL: 'gather_v8f64'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x double> @llvm.masked.gather.v8f64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x double> undef)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x double> @llvm.masked.gather.v8f64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x double> poison)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x double> %v
;
; ZNVER3-LABEL: 'gather_v8f64'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x double> @llvm.masked.gather.v8f64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x double> undef)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x double> @llvm.masked.gather.v8f64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x double> poison)
; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x double> %v
;
; SKX-LABEL: 'gather_v8f64'
-; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x double> @llvm.masked.gather.v8f64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x double> undef)
+; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x double> @llvm.masked.gather.v8f64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x double> poison)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x double> %v
;
- %v = call <8 x double> @llvm.masked.gather.v8f64.v8p0(<8 x ptr> %ptrs, i32 8, <8 x i1> %mask, <8 x double> undef)
+ %v = call <8 x double> @llvm.masked.gather.v8f64.v8p0(<8 x ptr> %ptrs, i32 8, <8 x i1> %mask, <8 x double> poison)
ret <8 x double> %v
}
@@ -565,26 +565,26 @@ define void @scatter_v8f64(<8 x double> %src, <8 x ptr> %ptrs, <8 x i1> %mask) {
define <16 x i32> @gather_v16i32_gep(ptr %base, <16 x i32> %idx, <16 x i1> %mask) {
; ZNVER4-LABEL: 'gather_v16i32_gep'
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 30 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> undef)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 30 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> poison)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i32> %v
;
; ZNVER5-LABEL: 'gather_v16i32_gep'
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 30 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> undef)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 30 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> poison)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i32> %v
;
; ZNVER3-LABEL: 'gather_v16i32_gep'
; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 55 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> undef)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 55 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> poison)
; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i32> %v
;
; SKX-LABEL: 'gather_v16i32_gep'
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
-; SKX-NEXT: Cost Model: Found an estimated cost of 18 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> undef)
+; SKX-NEXT: Cost Model: Found an estimated cost of 18 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> poison)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i32> %v
;
%ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
- %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> %ptrs, i32 4, <16 x i1> %mask, <16 x i32> undef)
+ %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> %ptrs, i32 4, <16 x i1> %mask, <16 x i32> poison)
ret <16 x i32> %v
}
>From 8cc82c40fc3d965efb799c615f2d2a6f3a9b81cf Mon Sep 17 00:00:00 2001
From: Sumukh Bharadwaj <Sumukh.Bharadwaj at amd.com>
Date: Thu, 28 May 2026 23:29:10 +0530
Subject: [PATCH 4/6] [X86][CostModel][NFC] clang-format Zen gather/scatter
cost tables
Apply clang-format to the per-shape Zen gather/scatter tables in
X86TargetTransformInfo.cpp:
- Drop the manual double-space numeric alignment in the rows
(clang-format collapses it and re-packs the rows 2-per-line).
- Re-wrap the `if (const auto *E = CostTableLookup(...))` line
in getGatherOverhead to put the call expression on its own
indented line.
Pure formatting; cost tables and lookup behaviour are unchanged.
---
.../lib/Target/X86/X86TargetTransformInfo.cpp | 22 +++++++++----------
1 file changed, 10 insertions(+), 12 deletions(-)
diff --git a/llvm/lib/Target/X86/X86TargetTransformInfo.cpp b/llvm/lib/Target/X86/X86TargetTransformInfo.cpp
index 455869b233eec..333d573076e2b 100644
--- a/llvm/lib/Target/X86/X86TargetTransformInfo.cpp
+++ b/llvm/lib/Target/X86/X86TargetTransformInfo.cpp
@@ -6399,17 +6399,16 @@ int X86TTIImpl::getGatherOverhead(Type *SrcVTy) const {
// lookup -- corresponding rows are omitted to make the live set
// explicit. Adding rows for them would have no observable effect.
static const CostTblEntry ZenGatherCostTable[] = {
- {ISD::LOAD, MVT::v4i32, 7}, {ISD::LOAD, MVT::v8i32, 17},
- {ISD::LOAD, MVT::v16i32, 14},
- {ISD::LOAD, MVT::v4f32, 7}, {ISD::LOAD, MVT::v8f32, 17},
- {ISD::LOAD, MVT::v16f32, 14},
- {ISD::LOAD, MVT::v4f64, 7}, {ISD::LOAD, MVT::v8f64, 17},
- {ISD::LOAD, MVT::v4i64, 10}, {ISD::LOAD, MVT::v8i64, 22},
+ {ISD::LOAD, MVT::v4i32, 7}, {ISD::LOAD, MVT::v8i32, 17},
+ {ISD::LOAD, MVT::v16i32, 14}, {ISD::LOAD, MVT::v4f32, 7},
+ {ISD::LOAD, MVT::v8f32, 17}, {ISD::LOAD, MVT::v16f32, 14},
+ {ISD::LOAD, MVT::v4f64, 7}, {ISD::LOAD, MVT::v8f64, 17},
+ {ISD::LOAD, MVT::v4i64, 10}, {ISD::LOAD, MVT::v8i64, 22},
};
EVT VT = TLI->getValueType(DL, SrcVTy);
if (VT.isSimple())
- if (const auto *E = CostTableLookup(ZenGatherCostTable, ISD::LOAD,
- VT.getSimpleVT()))
+ if (const auto *E =
+ CostTableLookup(ZenGatherCostTable, ISD::LOAD, VT.getSimpleVT()))
return E->Cost;
}
@@ -6438,10 +6437,9 @@ int X86TTIImpl::getScatterOverhead(Type *SrcVTy) const {
// split via type legalisation, so those rows are omitted.
static const CostTblEntry ZenScatterCostTable[] = {
{ISD::STORE, MVT::v4i32, 12}, {ISD::STORE, MVT::v8i32, 14},
- {ISD::STORE, MVT::v16i32, 6},
- {ISD::STORE, MVT::v4f32, 12}, {ISD::STORE, MVT::v8f32, 14},
- {ISD::STORE, MVT::v16f32, 6},
- {ISD::STORE, MVT::v4f64, 5}, {ISD::STORE, MVT::v8f64, 15},
+ {ISD::STORE, MVT::v16i32, 6}, {ISD::STORE, MVT::v4f32, 12},
+ {ISD::STORE, MVT::v8f32, 14}, {ISD::STORE, MVT::v16f32, 6},
+ {ISD::STORE, MVT::v4f64, 5}, {ISD::STORE, MVT::v8f64, 15},
{ISD::STORE, MVT::v4i64, 10}, {ISD::STORE, MVT::v8i64, 22},
};
EVT VT = TLI->getValueType(DL, SrcVTy);
>From c9e20b6c09e2329d91da2399a02da6ed91e3eb74 Mon Sep 17 00:00:00 2001
From: Sumukh Bharadwaj <Sumukh.Bharadwaj at amd.com>
Date: Sat, 6 Jun 2026 21:43:26 +0530
Subject: [PATCH 5/6] [X86][CostModel] Address Zen GS cost-table review
comments
Resolve feedback from @ganeshgit and @Andarwinux on #199488.
- Rename TuningPreferAMDZenGSCost -> TuningPreferGSCostTable
(accessor HasPreferGSCostTable, mattr prefer-gs-cost-table) as
suggested by reviewers.
- Replace the "shapes are omitted" comments in getGatherOverhead
and getScatterOverhead with asserts inside the table-lookup
branch. VF<4 is force-scalarised on AVX-512+VLX
(forceScalarizeMaskedGather/Scatter) and v16f64 is split by type
legalisation in getGSVectorCost, so neither shape can reach these
functions today. The asserts fail in asserts-builds if either
path is relaxed upstream, naming the exact row to add.
- Reword the i64 paragraph in getGatherOverhead to spell out why
the i64 break-even is HIGHER than f64 (the i64 scalar fallback
runs on the cheaper integer pipeline, so the integer scalar
baseline is harder to beat). Removes the ambiguous "gather has
to be cheaper to win" phrasing.
- Add a -mcpu=x86-64-v4 RUN line to masked-gather-scatter-amd-zen.ll
pinning the glibc-hwcap x86-64-v4 (generic AVX-512) baseline.
Values match SKX, locking in that this change cannot affect
distros shipping x86-64-v4-tier binaries.
No behaviour change for any subtarget. Test-only additions
(x86-64-v4 RUN line) and assertion-only invariants (release builds
unaffected).
---
llvm/docs/ReleaseNotes.md | 9 +-
llvm/lib/Target/X86/X86.td | 17 ++-
.../lib/Target/X86/X86TargetTransformInfo.cpp | 70 ++++++-----
.../X86/masked-gather-scatter-amd-zen.ll | 119 +++++++++++++++++-
.../X86/amd-zen-gather-scatter-decisions.ll | 2 +-
5 files changed, 171 insertions(+), 46 deletions(-)
diff --git a/llvm/docs/ReleaseNotes.md b/llvm/docs/ReleaseNotes.md
index d1b1bef07ebd3..3fc1908322b97 100644
--- a/llvm/docs/ReleaseNotes.md
+++ b/llvm/docs/ReleaseNotes.md
@@ -241,10 +241,11 @@ Makes programs 10x faster by doing Special New Thing.
in use. This matches the behaviour of Intel syntax and aids with
compatibility when changing the default Clang syntax to the Intel syntax.
* Masked gather and scatter cost overheads are now per-shape on AMD znver4
- and znver5 targets via a new `TuningPreferAMDZenGSCost` subtarget
- feature, replacing the single flat overhead inherited from the generic
- AVX-512 path. The per-shape costs use empirical break-even values
- measured on Zen 4 / Zen 5 hardware.
+ and znver5 targets via a new `TuningPreferGSCostTable` subtarget
+ feature (set in ZN4Tuning, inherited by ZN5Tuning), replacing the
+ single flat overhead inherited from the generic AVX-512 path. The
+ per-shape costs use empirical break-even values measured on Zen 4 /
+ Zen 5 hardware.
### Changes to the OCaml bindings
diff --git a/llvm/lib/Target/X86/X86.td b/llvm/lib/Target/X86/X86.td
index 28bbd639649bb..8725f613cb422 100644
--- a/llvm/lib/Target/X86/X86.td
+++ b/llvm/lib/Target/X86/X86.td
@@ -722,15 +722,12 @@ def TuningFastGather
"Indicates if gather is reasonably fast (this is true for Skylake client and all AVX-512 CPUs)">;
// Use AMD Zen-tuned cost tables for masked gather/scatter intrinsics in the
-// X86 TargetTransformInfo cost model. Refines the flat overhead used by other
-// AVX-512 targets with per-element-type/per-VL costs measured on znver4 and
-// znver5. Inherited automatically by every znver4+ CPU via ZN4Tuning; not
-// applied to pre-AVX-512 Zen parts (znver1..3), which take the scalarise
-// path for masked gather anyway.
-def TuningPreferAMDZenGSCost
- : SubtargetFeature<"prefer-amd-zen-gs-cost",
- "HasPreferAMDZenGSCost", "true",
- "Use AMD Zen-tuned gather/scatter cost tables in the cost model">;
+// Enables per-shape gather/scatter cost tables on AMD znver4 and znver5
+// (set in ZN4Tuning, inherited by ZN5Tuning).
+def TuningPreferGSCostTable
+ : SubtargetFeature<"prefer-gs-cost-table",
+ "HasPreferGSCostTable", "true",
+ "Use per-shape gather/scatter cost tables in the cost model">;
// Generate vpdpwssd instead of vpmaddwd+vpaddd sequence.
def TuningFastDPWSSD
@@ -1643,7 +1640,7 @@ def ProcessorFeatures {
!listconcat(ZN2Features, ZN3AdditionalFeatures);
list<SubtargetFeature> ZN4AdditionalTuning = [TuningFastDPWSSD,
- TuningPreferAMDZenGSCost];
+ TuningPreferGSCostTable];
list<SubtargetFeature> ZN4Tuning =
!listconcat(ZN3Tuning, ZN4AdditionalTuning);
list<SubtargetFeature> ZN4AdditionalFeatures = [FeatureAVX512,
diff --git a/llvm/lib/Target/X86/X86TargetTransformInfo.cpp b/llvm/lib/Target/X86/X86TargetTransformInfo.cpp
index 333d573076e2b..13fcacc0fde68 100644
--- a/llvm/lib/Target/X86/X86TargetTransformInfo.cpp
+++ b/llvm/lib/Target/X86/X86TargetTransformInfo.cpp
@@ -6370,11 +6370,9 @@ int X86TTIImpl::getGatherOverhead(Type *SrcVTy) const {
// TODO: Remove the explicit hasAVX512()?, That would mean we would only
// enable gather with a -march.
- // AMD znver4+ targets enable per-shape costs measured on the hardware via
- // TuningPreferAMDZenGSCost (set in ZN4Tuning). Pre-AVX-512 Zen parts
- // (znver1..3) take the scalarise path for masked gather and never reach
- // this code, so the table only needs to cover AVX-512 widths.
- if (ST->hasPreferAMDZenGSCost() && SrcVTy) {
+ // Per-shape gather costs for AMD znver4 and znver5 via
+ // TuningPreferGSCostTable (set in ZN4Tuning, inherited by ZN5Tuning).
+ if (ST->hasPreferGSCostTable() && SrcVTy) {
// Per-shape gather costs for AMD znver4+ targets.
//
// The numbers are the empirical "break-even" (lower-bound) costs
@@ -6386,18 +6384,13 @@ int X86TTIImpl::getGatherOverhead(Type *SrcVTy) const {
// the value tabulated below is the cost at which gather emission
// was the right call for that shape.
//
- // i64 entries were measured separately and intentionally exceed the
- // f64 entries for the same VF: the scalar alternative for i64 runs
- // on the integer pipeline (faster than f64 on the FP pipeline), so
- // gather has to be cheaper to win. At the f64-style break-even, i64
- // gather was 1.7-3.5x slower than the scalarised lowering across
- // stride patterns, so the i64 break-even sits at the minimum cost
- // that suppresses vpgatherqq emission on this microbench.
- // VF=2 is force-scalarised on AVX-512 (forceScalarizeMaskedGather)
- // and v16f64 is split via type legalisation in getGSVectorCost
- // (1024-bit data exceeds zmm), so neither shape ever reaches this
- // lookup -- corresponding rows are omitted to make the live set
- // explicit. Adding rows for them would have no observable effect.
+ // i64 entries are intentionally HIGHER than the f64 entries for the
+ // same VF: the i64 scalar fallback runs on the integer pipeline,
+ // which is cheaper per element than the FP pipeline used by f64
+ // scalars, so the integer scalar baseline is harder to beat. At the
+ // f64-style break-even, vpgatherqq was 1.7-3.5x slower than the
+ // scalarised i64 lowering across stride patterns. The i64 entries
+ // below sit at the minimum cost that suppresses vpgatherqq emission.
static const CostTblEntry ZenGatherCostTable[] = {
{ISD::LOAD, MVT::v4i32, 7}, {ISD::LOAD, MVT::v8i32, 17},
{ISD::LOAD, MVT::v16i32, 14}, {ISD::LOAD, MVT::v4f32, 7},
@@ -6406,10 +6399,20 @@ int X86TTIImpl::getGatherOverhead(Type *SrcVTy) const {
{ISD::LOAD, MVT::v4i64, 10}, {ISD::LOAD, MVT::v8i64, 22},
};
EVT VT = TLI->getValueType(DL, SrcVTy);
- if (VT.isSimple())
+ if (VT.isSimple()) {
+ MVT SimpleVT = VT.getSimpleVT();
+ assert(SimpleVT.getVectorNumElements() >= 4 &&
+ "VF<4 gather should be force-scalarised on AVX-512+VLX before "
+ "reaching here. Add a v{1,2} table row if "
+ "forceScalarizeMaskedGather is relaxed");
+ assert(SimpleVT != MVT::v16f64 &&
+ "v16f64 gather should be split by type legalisation in "
+ "getGSVectorCost before reaching here. Add a v16f64 table row "
+ "if that path changes");
if (const auto *E =
- CostTableLookup(ZenGatherCostTable, ISD::LOAD, VT.getSimpleVT()))
+ CostTableLookup(ZenGatherCostTable, ISD::LOAD, SimpleVT))
return E->Cost;
+ }
}
if (ST->hasAVX512() || (ST->hasAVX2() && ST->hasFastGather()))
@@ -6419,13 +6422,14 @@ int X86TTIImpl::getGatherOverhead(Type *SrcVTy) const {
}
int X86TTIImpl::getScatterOverhead(Type *SrcVTy) const {
- // AMD znver4+ targets use per-shape scatter costs measured on the hardware
- // via TuningPreferAMDZenGSCost (set in ZN4Tuning). Fall through to the
- // generic flat overhead for shapes we have not characterised.
- if (ST->hasPreferAMDZenGSCost() && ST->hasAVX512() && SrcVTy) {
+ // Per-shape scatter costs for AMD znver4 and znver5 via
+ // TuningPreferGSCostTable (set in ZN4Tuning, inherited by ZN5Tuning).
+ // Falls through to the generic flat overhead for shapes we have not
+ // characterised.
+ if (ST->hasPreferGSCostTable() && ST->hasAVX512() && SrcVTy) {
// Per-shape scatter costs for AMD znver4+ targets, measured with the
// same break-even methodology as the gather table above. The
- // original sweep characterised i32 and f64 lanes; the f32 rows
+ // original sweep characterised i32 and f64 lanes. The f32 rows
// mirror i32 because vpscatterdd and vscatterdps are the same
// physical 16-lane 32-bit scatter on Zen and were measured as
// runtime-equivalent (within 3% across VF and stride patterns).
@@ -6433,8 +6437,6 @@ int X86TTIImpl::getScatterOverhead(Type *SrcVTy) const {
// for all stride patterns tested (1.2-1.4x slower than scalarised
// on Zen 5), so the entry is set to the minimum cost that
// suppresses vpscatterqq emission.
- // As with the gather table: VF=2 is force-scalarised and v16f64 is
- // split via type legalisation, so those rows are omitted.
static const CostTblEntry ZenScatterCostTable[] = {
{ISD::STORE, MVT::v4i32, 12}, {ISD::STORE, MVT::v8i32, 14},
{ISD::STORE, MVT::v16i32, 6}, {ISD::STORE, MVT::v4f32, 12},
@@ -6443,10 +6445,20 @@ int X86TTIImpl::getScatterOverhead(Type *SrcVTy) const {
{ISD::STORE, MVT::v4i64, 10}, {ISD::STORE, MVT::v8i64, 22},
};
EVT VT = TLI->getValueType(DL, SrcVTy);
- if (VT.isSimple())
- if (const auto *E = CostTableLookup(ZenScatterCostTable, ISD::STORE,
- VT.getSimpleVT()))
+ if (VT.isSimple()) {
+ MVT SimpleVT = VT.getSimpleVT();
+ assert(SimpleVT.getVectorNumElements() >= 4 &&
+ "VF<4 scatter should be force-scalarised on AVX-512+VLX "
+ "before reaching here. Add a v{1,2} table row if "
+ "forceScalarizeMaskedScatter is relaxed");
+ assert(SimpleVT != MVT::v16f64 &&
+ "v16f64 scatter should be split by type legalisation in "
+ "getGSVectorCost before reaching here. Add a v16f64 table "
+ "row if that path changes");
+ if (const auto *E =
+ CostTableLookup(ZenScatterCostTable, ISD::STORE, SimpleVT))
return E->Cost;
+ }
}
if (ST->hasAVX512())
diff --git a/llvm/test/Analysis/CostModel/X86/masked-gather-scatter-amd-zen.ll b/llvm/test/Analysis/CostModel/X86/masked-gather-scatter-amd-zen.ll
index 84c24429b9b82..6ddf39869dfba 100644
--- a/llvm/test/Analysis/CostModel/X86/masked-gather-scatter-amd-zen.ll
+++ b/llvm/test/Analysis/CostModel/X86/masked-gather-scatter-amd-zen.ll
@@ -2,19 +2,23 @@
; Cost-model coverage for AMD Zen-tuned masked gather/scatter overheads.
;
; ZNVER4 / ZNVER5 enable the per-shape Zen cost tables via
-; TuningPreferAMDZenGSCost (set in ZN4Tuning and inherited by ZN5Tuning) and
+; TuningPreferGSCostTable (set in ZN4Tuning and inherited by ZN5Tuning) and
; have AVX-512, so the new tables are consulted in getGSVectorCost.
-; ZNVER3 does NOT carry TuningPreferAMDZenGSCost and lacks both AVX-512 and
+; ZNVER3 does NOT carry TuningPreferGSCostTable and lacks both AVX-512 and
; TuningFastGather, so isLegalMaskedGather() returns false and the cost model
; walks the scalarise path (getGSScalarCost). The ZNVER3 numbers below are the
; unchanged scalar fallback cost, included here only to lock in that this
; change does not regress pre-AVX-512 Zen targets.
; SKX is a non-Zen AVX-512 baseline showing the generic flat overhead of 2.
+; X8664V4 pins the glibc-hwcap x86-64-v4 (generic AVX-512) baseline. Values
+; match SKX, confirming that this change cannot affect distros shipping
+; x86-64-v4-tier binaries.
;
; RUN: opt < %s -S -mtriple=x86_64-unknown-linux-gnu -passes="print<cost-model>" 2>&1 -disable-output -cost-kind=throughput -mcpu=znver4 | FileCheck %s --check-prefix=ZNVER4
; RUN: opt < %s -S -mtriple=x86_64-unknown-linux-gnu -passes="print<cost-model>" 2>&1 -disable-output -cost-kind=throughput -mcpu=znver5 | FileCheck %s --check-prefix=ZNVER5
; RUN: opt < %s -S -mtriple=x86_64-unknown-linux-gnu -passes="print<cost-model>" 2>&1 -disable-output -cost-kind=throughput -mcpu=znver3 | FileCheck %s --check-prefix=ZNVER3
; RUN: opt < %s -S -mtriple=x86_64-unknown-linux-gnu -passes="print<cost-model>" 2>&1 -disable-output -cost-kind=throughput -mcpu=skx | FileCheck %s --check-prefix=SKX
+; RUN: opt < %s -S -mtriple=x86_64-unknown-linux-gnu -passes="print<cost-model>" 2>&1 -disable-output -cost-kind=throughput -mcpu=x86-64-v4 | FileCheck %s --check-prefix=X8664V4
;------------------------------------------------------------------------------
; Masked gather - i32 element type
@@ -36,6 +40,10 @@ define <2 x i32> @gather_v2i32(<2 x ptr> %ptrs, <2 x i1> %mask) {
; SKX-LABEL: 'gather_v2i32'
; SKX-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i32> @llvm.masked.gather.v2i32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x i32> poison)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i32> %v
+;
+; X8664V4-LABEL: 'gather_v2i32'
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i32> @llvm.masked.gather.v2i32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x i32> poison)
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i32> %v
;
%v = call <2 x i32> @llvm.masked.gather.v2i32.v2p0(<2 x ptr> %ptrs, i32 4, <2 x i1> %mask, <2 x i32> poison)
ret <2 x i32> %v
@@ -57,6 +65,10 @@ define <4 x i32> @gather_v4i32(<4 x ptr> %ptrs, <4 x i1> %mask) {
; SKX-LABEL: 'gather_v4i32'
; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x i32> poison)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i32> %v
+;
+; X8664V4-LABEL: 'gather_v4i32'
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x i32> poison)
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i32> %v
;
%v = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> %ptrs, i32 4, <4 x i1> %mask, <4 x i32> poison)
ret <4 x i32> %v
@@ -78,6 +90,10 @@ define <8 x i32> @gather_v8i32(<8 x ptr> %ptrs, <8 x i1> %mask) {
; SKX-LABEL: 'gather_v8i32'
; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x i32> poison)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i32> %v
+;
+; X8664V4-LABEL: 'gather_v8i32'
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x i32> poison)
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i32> %v
;
%v = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> %ptrs, i32 4, <8 x i1> %mask, <8 x i32> poison)
ret <8 x i32> %v
@@ -99,6 +115,10 @@ define <16 x i32> @gather_v16i32(<16 x ptr> %ptrs, <16 x i1> %mask) {
; SKX-LABEL: 'gather_v16i32'
; SKX-NEXT: Cost Model: Found an estimated cost of 20 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> poison)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i32> %v
+;
+; X8664V4-LABEL: 'gather_v16i32'
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 20 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> poison)
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i32> %v
;
%v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> %ptrs, i32 4, <16 x i1> %mask, <16 x i32> poison)
ret <16 x i32> %v
@@ -124,6 +144,10 @@ define <2 x i64> @gather_v2i64(<2 x ptr> %ptrs, <2 x i1> %mask) {
; SKX-LABEL: 'gather_v2i64'
; SKX-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i64> @llvm.masked.gather.v2i64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x i64> poison)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i64> %v
+;
+; X8664V4-LABEL: 'gather_v2i64'
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i64> @llvm.masked.gather.v2i64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x i64> poison)
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i64> %v
;
%v = call <2 x i64> @llvm.masked.gather.v2i64.v2p0(<2 x ptr> %ptrs, i32 8, <2 x i1> %mask, <2 x i64> poison)
ret <2 x i64> %v
@@ -145,6 +169,10 @@ define <4 x i64> @gather_v4i64(<4 x ptr> %ptrs, <4 x i1> %mask) {
; SKX-LABEL: 'gather_v4i64'
; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x i64> @llvm.masked.gather.v4i64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x i64> poison)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i64> %v
+;
+; X8664V4-LABEL: 'gather_v4i64'
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x i64> @llvm.masked.gather.v4i64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x i64> poison)
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i64> %v
;
%v = call <4 x i64> @llvm.masked.gather.v4i64.v4p0(<4 x ptr> %ptrs, i32 8, <4 x i1> %mask, <4 x i64> poison)
ret <4 x i64> %v
@@ -166,6 +194,10 @@ define <8 x i64> @gather_v8i64(<8 x ptr> %ptrs, <8 x i1> %mask) {
; SKX-LABEL: 'gather_v8i64'
; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x i64> @llvm.masked.gather.v8i64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x i64> poison)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i64> %v
+;
+; X8664V4-LABEL: 'gather_v8i64'
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x i64> @llvm.masked.gather.v8i64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x i64> poison)
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i64> %v
;
%v = call <8 x i64> @llvm.masked.gather.v8i64.v8p0(<8 x ptr> %ptrs, i32 8, <8 x i1> %mask, <8 x i64> poison)
ret <8 x i64> %v
@@ -191,6 +223,10 @@ define <2 x float> @gather_v2f32(<2 x ptr> %ptrs, <2 x i1> %mask) {
; SKX-LABEL: 'gather_v2f32'
; SKX-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x float> @llvm.masked.gather.v2f32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x float> poison)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x float> %v
+;
+; X8664V4-LABEL: 'gather_v2f32'
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x float> @llvm.masked.gather.v2f32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x float> poison)
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x float> %v
;
%v = call <2 x float> @llvm.masked.gather.v2f32.v2p0(<2 x ptr> %ptrs, i32 4, <2 x i1> %mask, <2 x float> poison)
ret <2 x float> %v
@@ -212,6 +248,10 @@ define <4 x float> @gather_v4f32(<4 x ptr> %ptrs, <4 x i1> %mask) {
; SKX-LABEL: 'gather_v4f32'
; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x float> poison)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x float> %v
+;
+; X8664V4-LABEL: 'gather_v4f32'
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x float> poison)
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x float> %v
;
%v = call <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr> %ptrs, i32 4, <4 x i1> %mask, <4 x float> poison)
ret <4 x float> %v
@@ -233,6 +273,10 @@ define <8 x float> @gather_v8f32(<8 x ptr> %ptrs, <8 x i1> %mask) {
; SKX-LABEL: 'gather_v8f32'
; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x float> poison)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x float> %v
+;
+; X8664V4-LABEL: 'gather_v8f32'
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x float> poison)
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x float> %v
;
%v = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> %ptrs, i32 4, <8 x i1> %mask, <8 x float> poison)
ret <8 x float> %v
@@ -254,6 +298,10 @@ define <16 x float> @gather_v16f32(<16 x ptr> %ptrs, <16 x i1> %mask) {
; SKX-LABEL: 'gather_v16f32'
; SKX-NEXT: Cost Model: Found an estimated cost of 20 for instruction: %v = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x float> poison)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x float> %v
+;
+; X8664V4-LABEL: 'gather_v16f32'
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 20 for instruction: %v = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x float> poison)
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x float> %v
;
%v = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> %ptrs, i32 4, <16 x i1> %mask, <16 x float> poison)
ret <16 x float> %v
@@ -279,6 +327,10 @@ define <2 x double> @gather_v2f64(<2 x ptr> %ptrs, <2 x i1> %mask) {
; SKX-LABEL: 'gather_v2f64'
; SKX-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x double> @llvm.masked.gather.v2f64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x double> poison)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x double> %v
+;
+; X8664V4-LABEL: 'gather_v2f64'
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x double> @llvm.masked.gather.v2f64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x double> poison)
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x double> %v
;
%v = call <2 x double> @llvm.masked.gather.v2f64.v2p0(<2 x ptr> %ptrs, i32 8, <2 x i1> %mask, <2 x double> poison)
ret <2 x double> %v
@@ -300,6 +352,10 @@ define <4 x double> @gather_v4f64(<4 x ptr> %ptrs, <4 x i1> %mask) {
; SKX-LABEL: 'gather_v4f64'
; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x double> @llvm.masked.gather.v4f64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x double> poison)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x double> %v
+;
+; X8664V4-LABEL: 'gather_v4f64'
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x double> @llvm.masked.gather.v4f64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x double> poison)
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x double> %v
;
%v = call <4 x double> @llvm.masked.gather.v4f64.v4p0(<4 x ptr> %ptrs, i32 8, <4 x i1> %mask, <4 x double> poison)
ret <4 x double> %v
@@ -321,6 +377,10 @@ define <8 x double> @gather_v8f64(<8 x ptr> %ptrs, <8 x i1> %mask) {
; SKX-LABEL: 'gather_v8f64'
; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x double> @llvm.masked.gather.v8f64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x double> poison)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x double> %v
+;
+; X8664V4-LABEL: 'gather_v8f64'
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x double> @llvm.masked.gather.v8f64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x double> poison)
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x double> %v
;
%v = call <8 x double> @llvm.masked.gather.v8f64.v8p0(<8 x ptr> %ptrs, i32 8, <8 x i1> %mask, <8 x double> poison)
ret <8 x double> %v
@@ -346,6 +406,10 @@ define void @scatter_v4i32(<4 x i32> %src, <4 x ptr> %ptrs, <4 x i1> %mask) {
; SKX-LABEL: 'scatter_v4i32'
; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: call void @llvm.masked.scatter.v4i32.v4p0(<4 x i32> %src, <4 x ptr> align 4 %ptrs, <4 x i1> %mask)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; X8664V4-LABEL: 'scatter_v4i32'
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 6 for instruction: call void @llvm.masked.scatter.v4i32.v4p0(<4 x i32> %src, <4 x ptr> align 4 %ptrs, <4 x i1> %mask)
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
call void @llvm.masked.scatter.v4i32.v4p0(<4 x i32> %src, <4 x ptr> %ptrs, i32 4, <4 x i1> %mask)
ret void
@@ -367,6 +431,10 @@ define void @scatter_v8i32(<8 x i32> %src, <8 x ptr> %ptrs, <8 x i1> %mask) {
; SKX-LABEL: 'scatter_v8i32'
; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: call void @llvm.masked.scatter.v8i32.v8p0(<8 x i32> %src, <8 x ptr> align 4 %ptrs, <8 x i1> %mask)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; X8664V4-LABEL: 'scatter_v8i32'
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 10 for instruction: call void @llvm.masked.scatter.v8i32.v8p0(<8 x i32> %src, <8 x ptr> align 4 %ptrs, <8 x i1> %mask)
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
call void @llvm.masked.scatter.v8i32.v8p0(<8 x i32> %src, <8 x ptr> %ptrs, i32 4, <8 x i1> %mask)
ret void
@@ -388,6 +456,10 @@ define void @scatter_v16i32(<16 x i32> %src, <16 x ptr> %ptrs, <16 x i1> %mask)
; SKX-LABEL: 'scatter_v16i32'
; SKX-NEXT: Cost Model: Found an estimated cost of 20 for instruction: call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32> %src, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; X8664V4-LABEL: 'scatter_v16i32'
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 20 for instruction: call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32> %src, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32> %src, <16 x ptr> %ptrs, i32 4, <16 x i1> %mask)
ret void
@@ -413,6 +485,10 @@ define void @scatter_v4i64(<4 x i64> %src, <4 x ptr> %ptrs, <4 x i1> %mask) {
; SKX-LABEL: 'scatter_v4i64'
; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: call void @llvm.masked.scatter.v4i64.v4p0(<4 x i64> %src, <4 x ptr> align 8 %ptrs, <4 x i1> %mask)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; X8664V4-LABEL: 'scatter_v4i64'
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 6 for instruction: call void @llvm.masked.scatter.v4i64.v4p0(<4 x i64> %src, <4 x ptr> align 8 %ptrs, <4 x i1> %mask)
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
call void @llvm.masked.scatter.v4i64.v4p0(<4 x i64> %src, <4 x ptr> %ptrs, i32 8, <4 x i1> %mask)
ret void
@@ -434,6 +510,10 @@ define void @scatter_v8i64(<8 x i64> %src, <8 x ptr> %ptrs, <8 x i1> %mask) {
; SKX-LABEL: 'scatter_v8i64'
; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: call void @llvm.masked.scatter.v8i64.v8p0(<8 x i64> %src, <8 x ptr> align 8 %ptrs, <8 x i1> %mask)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; X8664V4-LABEL: 'scatter_v8i64'
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 10 for instruction: call void @llvm.masked.scatter.v8i64.v8p0(<8 x i64> %src, <8 x ptr> align 8 %ptrs, <8 x i1> %mask)
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
call void @llvm.masked.scatter.v8i64.v8p0(<8 x i64> %src, <8 x ptr> %ptrs, i32 8, <8 x i1> %mask)
ret void
@@ -459,6 +539,10 @@ define void @scatter_v4f32(<4 x float> %src, <4 x ptr> %ptrs, <4 x i1> %mask) {
; SKX-LABEL: 'scatter_v4f32'
; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: call void @llvm.masked.scatter.v4f32.v4p0(<4 x float> %src, <4 x ptr> align 4 %ptrs, <4 x i1> %mask)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; X8664V4-LABEL: 'scatter_v4f32'
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 6 for instruction: call void @llvm.masked.scatter.v4f32.v4p0(<4 x float> %src, <4 x ptr> align 4 %ptrs, <4 x i1> %mask)
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
call void @llvm.masked.scatter.v4f32.v4p0(<4 x float> %src, <4 x ptr> %ptrs, i32 4, <4 x i1> %mask)
ret void
@@ -480,6 +564,10 @@ define void @scatter_v8f32(<8 x float> %src, <8 x ptr> %ptrs, <8 x i1> %mask) {
; SKX-LABEL: 'scatter_v8f32'
; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: call void @llvm.masked.scatter.v8f32.v8p0(<8 x float> %src, <8 x ptr> align 4 %ptrs, <8 x i1> %mask)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; X8664V4-LABEL: 'scatter_v8f32'
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 10 for instruction: call void @llvm.masked.scatter.v8f32.v8p0(<8 x float> %src, <8 x ptr> align 4 %ptrs, <8 x i1> %mask)
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
call void @llvm.masked.scatter.v8f32.v8p0(<8 x float> %src, <8 x ptr> %ptrs, i32 4, <8 x i1> %mask)
ret void
@@ -501,6 +589,10 @@ define void @scatter_v16f32(<16 x float> %src, <16 x ptr> %ptrs, <16 x i1> %mask
; SKX-LABEL: 'scatter_v16f32'
; SKX-NEXT: Cost Model: Found an estimated cost of 20 for instruction: call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> %src, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; X8664V4-LABEL: 'scatter_v16f32'
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 20 for instruction: call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> %src, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> %src, <16 x ptr> %ptrs, i32 4, <16 x i1> %mask)
ret void
@@ -526,6 +618,10 @@ define void @scatter_v4f64(<4 x double> %src, <4 x ptr> %ptrs, <4 x i1> %mask) {
; SKX-LABEL: 'scatter_v4f64'
; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: call void @llvm.masked.scatter.v4f64.v4p0(<4 x double> %src, <4 x ptr> align 8 %ptrs, <4 x i1> %mask)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; X8664V4-LABEL: 'scatter_v4f64'
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 6 for instruction: call void @llvm.masked.scatter.v4f64.v4p0(<4 x double> %src, <4 x ptr> align 8 %ptrs, <4 x i1> %mask)
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
call void @llvm.masked.scatter.v4f64.v4p0(<4 x double> %src, <4 x ptr> %ptrs, i32 8, <4 x i1> %mask)
ret void
@@ -547,6 +643,10 @@ define void @scatter_v8f64(<8 x double> %src, <8 x ptr> %ptrs, <8 x i1> %mask) {
; SKX-LABEL: 'scatter_v8f64'
; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: call void @llvm.masked.scatter.v8f64.v8p0(<8 x double> %src, <8 x ptr> align 8 %ptrs, <8 x i1> %mask)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; X8664V4-LABEL: 'scatter_v8f64'
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 10 for instruction: call void @llvm.masked.scatter.v8f64.v8p0(<8 x double> %src, <8 x ptr> align 8 %ptrs, <8 x i1> %mask)
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
call void @llvm.masked.scatter.v8f64.v8p0(<8 x double> %src, <8 x ptr> %ptrs, i32 8, <8 x i1> %mask)
ret void
@@ -582,6 +682,11 @@ define <16 x i32> @gather_v16i32_gep(ptr %base, <16 x i32> %idx, <16 x i1> %mask
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
; SKX-NEXT: Cost Model: Found an estimated cost of 18 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> poison)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i32> %v
+;
+; X8664V4-LABEL: 'gather_v16i32_gep'
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 18 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> poison)
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i32> %v
;
%ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
%v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> %ptrs, i32 4, <16 x i1> %mask, <16 x i32> poison)
@@ -608,6 +713,11 @@ define void @scatter_v16f32_gep(ptr %base, <16 x i32> %idx, <16 x float> %val, <
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr float, ptr %base, <16 x i32> %idx
; SKX-NEXT: Cost Model: Found an estimated cost of 18 for instruction: call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> %val, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; X8664V4-LABEL: 'scatter_v16f32_gep'
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr float, ptr %base, <16 x i32> %idx
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 18 for instruction: call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> %val, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
%ptrs = getelementptr float, ptr %base, <16 x i32> %idx
call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> %val, <16 x ptr> %ptrs, i32 4, <16 x i1> %mask)
@@ -634,6 +744,11 @@ define void @scatter_v16i32_gep(ptr %base, <16 x i32> %idx, <16 x i32> %val, <16
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
; SKX-NEXT: Cost Model: Found an estimated cost of 18 for instruction: call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32> %val, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; X8664V4-LABEL: 'scatter_v16i32_gep'
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 18 for instruction: call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32> %val, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
+; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
%ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32> %val, <16 x ptr> %ptrs, i32 4, <16 x i1> %mask)
diff --git a/llvm/test/Transforms/LoopVectorize/X86/amd-zen-gather-scatter-decisions.ll b/llvm/test/Transforms/LoopVectorize/X86/amd-zen-gather-scatter-decisions.ll
index c1087ead1f2ad..4723b2b07f400 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/amd-zen-gather-scatter-decisions.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/amd-zen-gather-scatter-decisions.ll
@@ -1,5 +1,5 @@
; End-to-end loop-vectorize decisions driven by the AMD Zen per-shape
-; gather/scatter cost tables (TuningPreferAMDZenGSCost, set on znver4+).
+; gather/scatter cost tables (TuningPreferGSCostTable, set on znver4+).
;
; The companion cost-model test
; llvm/test/Analysis/CostModel/X86/masked-gather-scatter-amd-zen.ll
>From ea23a9273c392af6c9ddb27698d366ee9ff9e852 Mon Sep 17 00:00:00 2001
From: Sumukh Bharadwaj <Sumukh.Bharadwaj at amd.com>
Date: Thu, 30 Jul 2026 12:27:16 +0530
Subject: [PATCH 6/6] [X86][CostModel] Decouple Zen gather/scatter cost from
the schedule model
Rework the per-shape masked gather/scatter cost model so the hardware
cost and the vectorize-vs-scalarize break-even overhead are kept
separate instead of being baked together into a single TTI table.
Previously the per-shape tables in getGatherOverhead / getScatterOverhead
returned a lumped number (hardware body cost plus break-even margin) that
the LoopVectorizer consumed directly. That duplicated numbers already
described by the schedule model and left llvm-mca / the MachineScheduler
with no honest source of truth for these instructions.
The cost is now computed as:
cost = break-even overhead + instruction body cost
where the break-even overhead remains a (shape, element-type) function
in the TTI tables (it cannot be derived from the schedule model), and the
body cost is read from the schedule model via getModeledGSInstrCost for
Zen AVX-512 parts (getAVX512GSRepresentativeOpcode +
getSchedModelCostForOpcode), falling back to the synthetic
VF * scalar-memory-op estimate elsewhere. The Znver4/Znver5 schedule
model gains honest measured throughput/uops/latency entries for the
masked gather/scatter shapes, so llvm-mca and the scheduler now see the
real hardware numbers while the vectorizer still gets the same overall
cost.
The tuning bit is renamed from TuningPreferAMDZenGSCost to the
vendor-neutral TuningPreferGSCostTable: the mechanism is portable and any
target that provides measured values can opt in; only the tables and the
schedule-model entries are Zen-specific. The bit stays on ZN4Tuning so
znver4/znver5 pick it up automatically; pre-AVX-512 Zen parts still take
the scalarise path and are intentionally excluded.
The tests are renamed to drop the vendor prefix.
---
llvm/lib/Target/X86/X86.td | 7 +-
llvm/lib/Target/X86/X86ScheduleZnver4.td | 70 +++
.../lib/Target/X86/X86TargetTransformInfo.cpp | 179 +++++---
llvm/lib/Target/X86/X86TargetTransformInfo.h | 3 +
...ll => masked-gather-scatter-cost-table.ll} | 404 +++++-------------
...=> gather-scatter-cost-table-decisions.ll} | 31 +-
.../llvm-mca/X86/Znver4/resources-avx512.s | 34 +-
.../llvm-mca/X86/Znver4/resources-avx512vl.s | 50 +--
8 files changed, 362 insertions(+), 416 deletions(-)
rename llvm/test/Analysis/CostModel/X86/{masked-gather-scatter-amd-zen.ll => masked-gather-scatter-cost-table.ll} (57%)
rename llvm/test/Transforms/LoopVectorize/X86/{amd-zen-gather-scatter-decisions.ll => gather-scatter-cost-table-decisions.ll} (78%)
diff --git a/llvm/lib/Target/X86/X86.td b/llvm/lib/Target/X86/X86.td
index 8725f613cb422..2b2c8ce4fcffb 100644
--- a/llvm/lib/Target/X86/X86.td
+++ b/llvm/lib/Target/X86/X86.td
@@ -722,8 +722,11 @@ def TuningFastGather
"Indicates if gather is reasonably fast (this is true for Skylake client and all AVX-512 CPUs)">;
// Use AMD Zen-tuned cost tables for masked gather/scatter intrinsics in the
-// Enables per-shape gather/scatter cost tables on AMD znver4 and znver5
-// (set in ZN4Tuning, inherited by ZN5Tuning).
+// X86 TargetTransformInfo cost model. Refines the flat overhead used by other
+// AVX-512 targets with per-element-type/per-VL costs measured on znver4 and
+// znver5. Inherited automatically by every znver4+ CPU via ZN4Tuning; not
+// applied to pre-AVX-512 Zen parts (znver1..3), which take the scalarise
+// path for masked gather anyway.
def TuningPreferGSCostTable
: SubtargetFeature<"prefer-gs-cost-table",
"HasPreferGSCostTable", "true",
diff --git a/llvm/lib/Target/X86/X86ScheduleZnver4.td b/llvm/lib/Target/X86/X86ScheduleZnver4.td
index ac4d31de8dbfe..83e9931282dee 100644
--- a/llvm/lib/Target/X86/X86ScheduleZnver4.td
+++ b/llvm/lib/Target/X86/X86ScheduleZnver4.td
@@ -509,6 +509,76 @@ defm : Zn4WriteResInt<WriteLoad, [Zn4AGU012, Zn4Load], !add(Znver4Model.LoadLate
// Does not cost anything by itself, only has latency, matching that of the WriteLoad,
defm : Zn4WriteResInt<WriteVecMaskedGatherWriteback, [], !add(Znver4Model.LoadLatency, 1), [], 0>;
+// AVX-512 masked GATHER / SCATTER, per shape.
+//
+// Zen4/Zen5 implement these as microcoded sequences. Throughput, uops and
+// latency were measured on Znver5 (which reuses Znver4Model) with
+// mask-reloading microbenchmarks under perf:
+// shape gather: tput(cyc) uops lat scatter: tput(cyc) uops
+// v4x32 4.0 20 17 6.0 28
+// v8x32 6.2 33 23 9.0 49
+// v16x32 12.7 65 30 17.0 89
+// v4x64 4.0 20 19 5.0 28
+// v8x64 8.0 41 25 9.0 49
+// ReleaseAtCycles = round(tput * NumUnits) over the load/store pipes and AGU.
+//
+// These model the raw hardware only, so llvm-mca / the MachineScheduler stay
+// honest; the break-even overhead the LoopVectorizer needs is applied on top
+// in getGSVectorCost (getGatherOverhead / getScatterOverhead). 64-bit int and
+// FP shapes share one entry; their break-even differs and lives in the TTI
+// tables.
+def Zn4WriteVGATHER_4x32 : SchedWriteRes<[Zn4AGU012, Zn4Load]> {
+ let ReleaseAtCycles = [12, 12]; let Latency = 17; let NumMicroOps = 20;
+}
+def : InstRW<[Zn4WriteVGATHER_4x32, WriteVecMaskedGatherWriteback],
+ (instrs VPGATHERDDZ128rm, VGATHERDPSZ128rm)>;
+def Zn4WriteVGATHER_8x32 : SchedWriteRes<[Zn4AGU012, Zn4Load]> {
+ let ReleaseAtCycles = [19, 19]; let Latency = 23; let NumMicroOps = 33;
+}
+def : InstRW<[Zn4WriteVGATHER_8x32, WriteVecMaskedGatherWriteback],
+ (instrs VPGATHERDDZ256rm, VGATHERDPSZ256rm)>;
+def Zn4WriteVGATHER_16x32 : SchedWriteRes<[Zn4AGU012, Zn4Load]> {
+ let ReleaseAtCycles = [38, 38]; let Latency = 30; let NumMicroOps = 65;
+}
+def : InstRW<[Zn4WriteVGATHER_16x32, WriteVecMaskedGatherWriteback],
+ (instrs VPGATHERDDZrm, VGATHERDPSZrm)>;
+def Zn4WriteVGATHER_4x64 : SchedWriteRes<[Zn4AGU012, Zn4Load]> {
+ let ReleaseAtCycles = [12, 12]; let Latency = 19; let NumMicroOps = 20;
+}
+def : InstRW<[Zn4WriteVGATHER_4x64, WriteVecMaskedGatherWriteback],
+ (instrs VPGATHERQQZ256rm, VGATHERQPDZ256rm)>;
+def Zn4WriteVGATHER_8x64 : SchedWriteRes<[Zn4AGU012, Zn4Load]> {
+ let ReleaseAtCycles = [24, 24]; let Latency = 25; let NumMicroOps = 41;
+}
+def : InstRW<[Zn4WriteVGATHER_8x64, WriteVecMaskedGatherWriteback],
+ (instrs VPGATHERQQZrm, VGATHERQPDZrm)>;
+
+def Zn4WriteVSCATTER_4x32 : SchedWriteRes<[Zn4AGU012, Zn4Store]> {
+ let ReleaseAtCycles = [18, 12]; let Latency = 10; let NumMicroOps = 28;
+}
+def : InstRW<[Zn4WriteVSCATTER_4x32],
+ (instrs VPSCATTERDDZ128mr, VSCATTERDPSZ128mr)>;
+def Zn4WriteVSCATTER_8x32 : SchedWriteRes<[Zn4AGU012, Zn4Store]> {
+ let ReleaseAtCycles = [27, 18]; let Latency = 14; let NumMicroOps = 49;
+}
+def : InstRW<[Zn4WriteVSCATTER_8x32],
+ (instrs VPSCATTERDDZ256mr, VSCATTERDPSZ256mr)>;
+def Zn4WriteVSCATTER_16x32 : SchedWriteRes<[Zn4AGU012, Zn4Store]> {
+ let ReleaseAtCycles = [51, 34]; let Latency = 24; let NumMicroOps = 89;
+}
+def : InstRW<[Zn4WriteVSCATTER_16x32],
+ (instrs VPSCATTERDDZmr, VSCATTERDPSZmr)>;
+def Zn4WriteVSCATTER_4x64 : SchedWriteRes<[Zn4AGU012, Zn4Store]> {
+ let ReleaseAtCycles = [15, 10]; let Latency = 9; let NumMicroOps = 28;
+}
+def : InstRW<[Zn4WriteVSCATTER_4x64],
+ (instrs VPSCATTERQQZ256mr, VSCATTERQPDZ256mr)>;
+def Zn4WriteVSCATTER_8x64 : SchedWriteRes<[Zn4AGU012, Zn4Store]> {
+ let ReleaseAtCycles = [27, 18]; let Latency = 14; let NumMicroOps = 49;
+}
+def : InstRW<[Zn4WriteVSCATTER_8x64],
+ (instrs VPSCATTERQQZmr, VSCATTERQPDZmr)>;
+
def Zn4WriteMOVSlow : SchedWriteRes<[Zn4AGU012, Zn4Load]> {
let Latency = !add(Znver4Model.LoadLatency, 1);
let ReleaseAtCycles = [3, 1];
diff --git a/llvm/lib/Target/X86/X86TargetTransformInfo.cpp b/llvm/lib/Target/X86/X86TargetTransformInfo.cpp
index 13fcacc0fde68..3b1c16c602d04 100644
--- a/llvm/lib/Target/X86/X86TargetTransformInfo.cpp
+++ b/llvm/lib/Target/X86/X86TargetTransformInfo.cpp
@@ -56,6 +56,8 @@
#include "llvm/CodeGen/TargetLowering.h"
#include "llvm/IR/InstIterator.h"
#include "llvm/IR/IntrinsicInst.h"
+#include "llvm/MC/MCSchedule.h"
+#include <cmath>
#include <optional>
using namespace llvm;
@@ -6370,33 +6372,24 @@ int X86TTIImpl::getGatherOverhead(Type *SrcVTy) const {
// TODO: Remove the explicit hasAVX512()?, That would mean we would only
// enable gather with a -march.
- // Per-shape gather costs for AMD znver4 and znver5 via
- // TuningPreferGSCostTable (set in ZN4Tuning, inherited by ZN5Tuning).
+ // AMD znver4+ targets enable per-shape costs measured on the hardware via
+ // TuningPreferGSCostTable. Pre-AVX-512 Zen parts take the scalarise path
+ // for masked gather instead.
if (ST->hasPreferGSCostTable() && SrcVTy) {
- // Per-shape gather costs for AMD znver4+ targets.
- //
- // The numbers are the empirical "break-even" (lower-bound) costs
- // measured by sweeping a forced gather cost while compiling a
- // controlled gather micro-benchmark and observing the point at which
- // the LoopVectorizer still chose the gather lowering over the scalar
- // fallback. The sweep was run independently for every (data type,
- // VF) combination on Genoa / Milan / Turin and re-validated on Zen 5;
- // the value tabulated below is the cost at which gather emission
- // was the right call for that shape.
- //
- // i64 entries are intentionally HIGHER than the f64 entries for the
- // same VF: the i64 scalar fallback runs on the integer pipeline,
- // which is cheaper per element than the FP pipeline used by f64
- // scalars, so the integer scalar baseline is harder to beat. At the
- // f64-style break-even, vpgatherqq was 1.7-3.5x slower than the
- // scalarised i64 lowering across stride patterns. The i64 entries
- // below sit at the minimum cost that suppresses vpgatherqq emission.
+ // Per-shape gather break-even overhead: the empirical cost at which the
+ // LoopVectorizer stops preferring the gather over the scalar fallback,
+ // measured by sweeping a forced overhead over a gather micro-benchmark.
+ // For shapes the znver4+ schedule model carries it is added on top of the
+ // measured hardware reciprocal-throughput in getGSVectorCost; otherwise it
+ // is the flat overhead of the generic formula. Values are non-monotonic in
+ // VF, and the i64 overhead exceeds f64 because the i64 scalar fallback runs
+ // on the cheaper integer pipeline and is harder to beat.
static const CostTblEntry ZenGatherCostTable[] = {
- {ISD::LOAD, MVT::v4i32, 7}, {ISD::LOAD, MVT::v8i32, 17},
- {ISD::LOAD, MVT::v16i32, 14}, {ISD::LOAD, MVT::v4f32, 7},
- {ISD::LOAD, MVT::v8f32, 17}, {ISD::LOAD, MVT::v16f32, 14},
- {ISD::LOAD, MVT::v4f64, 7}, {ISD::LOAD, MVT::v8f64, 17},
- {ISD::LOAD, MVT::v4i64, 10}, {ISD::LOAD, MVT::v8i64, 22},
+ {ISD::LOAD, MVT::v4i32, 7}, {ISD::LOAD, MVT::v8i32, 17},
+ {ISD::LOAD, MVT::v16i32, 14}, {ISD::LOAD, MVT::v4f32, 7},
+ {ISD::LOAD, MVT::v8f32, 17}, {ISD::LOAD, MVT::v16f32, 14},
+ {ISD::LOAD, MVT::v4f64, 7}, {ISD::LOAD, MVT::v8f64, 17},
+ {ISD::LOAD, MVT::v4i64, 10}, {ISD::LOAD, MVT::v8i64, 22},
};
EVT VT = TLI->getValueType(DL, SrcVTy);
if (VT.isSimple()) {
@@ -6422,21 +6415,7 @@ int X86TTIImpl::getGatherOverhead(Type *SrcVTy) const {
}
int X86TTIImpl::getScatterOverhead(Type *SrcVTy) const {
- // Per-shape scatter costs for AMD znver4 and znver5 via
- // TuningPreferGSCostTable (set in ZN4Tuning, inherited by ZN5Tuning).
- // Falls through to the generic flat overhead for shapes we have not
- // characterised.
if (ST->hasPreferGSCostTable() && ST->hasAVX512() && SrcVTy) {
- // Per-shape scatter costs for AMD znver4+ targets, measured with the
- // same break-even methodology as the gather table above. The
- // original sweep characterised i32 and f64 lanes. The f32 rows
- // mirror i32 because vpscatterdd and vscatterdps are the same
- // physical 16-lane 32-bit scatter on Zen and were measured as
- // runtime-equivalent (within 3% across VF and stride patterns).
- // i64 entries match the i64 gather rationale: scatter is harmful
- // for all stride patterns tested (1.2-1.4x slower than scalarised
- // on Zen 5), so the entry is set to the minimum cost that
- // suppresses vpscatterqq emission.
static const CostTblEntry ZenScatterCostTable[] = {
{ISD::STORE, MVT::v4i32, 12}, {ISD::STORE, MVT::v8i32, 14},
{ISD::STORE, MVT::v16i32, 6}, {ISD::STORE, MVT::v4f32, 12},
@@ -6448,13 +6427,13 @@ int X86TTIImpl::getScatterOverhead(Type *SrcVTy) const {
if (VT.isSimple()) {
MVT SimpleVT = VT.getSimpleVT();
assert(SimpleVT.getVectorNumElements() >= 4 &&
- "VF<4 scatter should be force-scalarised on AVX-512+VLX "
- "before reaching here. Add a v{1,2} table row if "
+ "VF<4 scatter should be force-scalarised on AVX-512+VLX before "
+ "reaching here. Add a v{1,2} table row if "
"forceScalarizeMaskedScatter is relaxed");
assert(SimpleVT != MVT::v16f64 &&
"v16f64 scatter should be split by type legalisation in "
- "getGSVectorCost before reaching here. Add a v16f64 table "
- "row if that path changes");
+ "getGSVectorCost before reaching here. Add a v16f64 table row "
+ "if that path changes");
if (const auto *E =
CostTableLookup(ZenScatterCostTable, ISD::STORE, SimpleVT))
return E->Cost;
@@ -6467,6 +6446,91 @@ int X86TTIImpl::getScatterOverhead(Type *SrcVTy) const {
return 1024;
}
+// Map a non-split masked gather/scatter shape to a representative AVX-512
+// opcode for the schedule-model query. Uses the matched index/data-width form
+// (not the mixed VPGATHERDQ/QD forms, which not all AVX-512 models carry) and
+// distinguishes int vs FP because Zen's 64-bit entries are element-type
+// dependent. Returns 0 for shapes we do not model.
+static unsigned getAVX512GSRepresentativeOpcode(bool IsLoad, MVT VT) {
+ if (!VT.isVector())
+ return 0;
+ unsigned NumElts = VT.getVectorNumElements();
+ unsigned EltBits = VT.getScalarSizeInBits();
+ const bool IsFP = VT.getScalarType().isFloatingPoint();
+ if (IsLoad) {
+ if (EltBits == 32)
+ return NumElts == 4 ? (IsFP ? X86::VGATHERDPSZ128rm : X86::VPGATHERDDZ128rm)
+ : NumElts == 8 ? (IsFP ? X86::VGATHERDPSZ256rm : X86::VPGATHERDDZ256rm)
+ : NumElts == 16 ? (IsFP ? X86::VGATHERDPSZrm : X86::VPGATHERDDZrm)
+ : 0;
+ if (EltBits == 64)
+ return NumElts == 4 ? (IsFP ? X86::VGATHERQPDZ256rm : X86::VPGATHERQQZ256rm)
+ : NumElts == 8 ? (IsFP ? X86::VGATHERQPDZrm : X86::VPGATHERQQZrm)
+ : 0;
+ return 0;
+ }
+ if (EltBits == 32)
+ return NumElts == 4 ? (IsFP ? X86::VSCATTERDPSZ128mr : X86::VPSCATTERDDZ128mr)
+ : NumElts == 8 ? (IsFP ? X86::VSCATTERDPSZ256mr : X86::VPSCATTERDDZ256mr)
+ : NumElts == 16 ? (IsFP ? X86::VSCATTERDPSZmr : X86::VPSCATTERDDZmr)
+ : 0;
+ if (EltBits == 64)
+ return NumElts == 4 ? (IsFP ? X86::VSCATTERQPDZ256mr : X86::VPSCATTERQQZ256mr)
+ : NumElts == 8 ? (IsFP ? X86::VSCATTERQPDZmr : X86::VPSCATTERQQZmr)
+ : 0;
+ return 0;
+}
+
+// Read the whole-instruction cost of a single opcode from the subtarget's
+// schedule model for the requested cost kind, or nullopt so the caller can
+// fall back.
+static std::optional<unsigned>
+getSchedModelCostForOpcode(unsigned Opc, TTI::TargetCostKind CostKind,
+ const X86Subtarget *ST) {
+ const MCSchedModel &SM = ST->getSchedModel();
+ if (!SM.hasInstrSchedModel())
+ return std::nullopt;
+
+ unsigned SClassID = ST->getInstrInfo()->get(Opc).getSchedClass();
+ const MCSchedClassDesc *SCDesc = SM.getSchedClassDesc(SClassID);
+ if (!SCDesc || !SCDesc->isValid() || SCDesc->isVariant())
+ return std::nullopt;
+
+ switch (CostKind) {
+ case TTI::TCK_RecipThroughput: {
+ double RThru = MCSchedModel::getReciprocalThroughput(*ST, *SCDesc);
+ if (RThru < 0.0)
+ return std::nullopt;
+ return static_cast<unsigned>(std::lround(RThru));
+ }
+ case TTI::TCK_Latency:
+ case TTI::TCK_SizeAndLatency: {
+ int Lat = MCSchedModel::computeInstrLatency(*ST, *SCDesc);
+ if (Lat < 0)
+ return std::nullopt;
+ return static_cast<unsigned>(Lat);
+ }
+ case TTI::TCK_CodeSize:
+ // Encoded size is not a schedule-model property; let the caller decide.
+ return std::nullopt;
+ }
+ llvm_unreachable("Unknown TargetCostKind");
+}
+
+std::optional<unsigned>
+X86TTIImpl::getModeledGSInstrCost(bool IsLoad, Type *SrcVTy,
+ TTI::TargetCostKind CostKind) const {
+ if (!ST->hasAVX512() || !ST->hasPreferGSCostTable())
+ return std::nullopt;
+ EVT VT = TLI->getValueType(DL, SrcVTy);
+ if (!VT.isSimple())
+ return std::nullopt;
+ unsigned Opc = getAVX512GSRepresentativeOpcode(IsLoad, VT.getSimpleVT());
+ if (!Opc)
+ return std::nullopt;
+ return getSchedModelCostForOpcode(Opc, CostKind, ST);
+}
+
// Return an average cost of Gather / Scatter instruction, maybe improved later.
InstructionCost X86TTIImpl::getGSVectorCost(unsigned Opcode,
TTI::TargetCostKind CostKind,
@@ -6530,13 +6594,28 @@ InstructionCost X86TTIImpl::getGSVectorCost(unsigned Opcode,
if (CostKind == TTI::TCK_CodeSize)
return 1;
- // The gather / scatter cost is given by Intel architects. It is a rough
- // number since we are looking at one instruction in a time.
- const int GSOverhead = (Opcode == Instruction::Load)
- ? getGatherOverhead(SrcVTy)
- : getScatterOverhead(SrcVTy);
- return GSOverhead + VF * getMemoryOpCost(Opcode, SrcVTy->getScalarType(),
- Alignment, AddressSpace, CostKind);
+ const bool IsLoad = Opcode == Instruction::Load;
+
+ // A masked gather/scatter cost is a fixed vectorize-vs-scalarize break-even
+ // overhead plus the cost of the instruction body:
+ //
+ // cost = break-even overhead + instruction body cost
+ //
+ // The overhead is a (shape, element type) function that cannot be derived
+ // from the schedule model, so it lives in the per-shape TTI tables (flat
+ // default otherwise). The body cost is the measured schedule-model cost where
+ // we have one (getModeledGSInstrCost, Zen AVX-512 today, keeping the .td
+ // honest for llvm-mca), else the synthetic VF * scalar-memory-op estimate.
+ const int GSOverhead =
+ IsLoad ? getGatherOverhead(SrcVTy) : getScatterOverhead(SrcVTy);
+ InstructionCost BodyCost;
+ if (std::optional<unsigned> HW =
+ getModeledGSInstrCost(IsLoad, SrcVTy, CostKind))
+ BodyCost = *HW;
+ else
+ BodyCost = VF * getMemoryOpCost(Opcode, SrcVTy->getScalarType(), Alignment,
+ AddressSpace, CostKind);
+ return GSOverhead + BodyCost;
}
/// Calculate the cost of Gather / Scatter operation
diff --git a/llvm/lib/Target/X86/X86TargetTransformInfo.h b/llvm/lib/Target/X86/X86TargetTransformInfo.h
index f44caa769be64..60d4b2745cf2e 100644
--- a/llvm/lib/Target/X86/X86TargetTransformInfo.h
+++ b/llvm/lib/Target/X86/X86TargetTransformInfo.h
@@ -350,6 +350,9 @@ class X86TTIImpl final : public BasicTTIImplBase<X86TTIImpl> {
int getGatherOverhead(Type *SrcVTy) const;
int getScatterOverhead(Type *SrcVTy) const;
+ std::optional<unsigned>
+ getModeledGSInstrCost(bool IsLoad, Type *SrcVTy,
+ TTI::TargetCostKind CostKind) const;
/// @}
};
diff --git a/llvm/test/Analysis/CostModel/X86/masked-gather-scatter-amd-zen.ll b/llvm/test/Analysis/CostModel/X86/masked-gather-scatter-cost-table.ll
similarity index 57%
rename from llvm/test/Analysis/CostModel/X86/masked-gather-scatter-amd-zen.ll
rename to llvm/test/Analysis/CostModel/X86/masked-gather-scatter-cost-table.ll
index 6ddf39869dfba..eb8951d288b98 100644
--- a/llvm/test/Analysis/CostModel/X86/masked-gather-scatter-amd-zen.ll
+++ b/llvm/test/Analysis/CostModel/X86/masked-gather-scatter-cost-table.ll
@@ -1,24 +1,19 @@
; NOTE: Assertions have been autogenerated by utils/update_analyze_test_checks.py
-; Cost-model coverage for AMD Zen-tuned masked gather/scatter overheads.
-;
-; ZNVER4 / ZNVER5 enable the per-shape Zen cost tables via
-; TuningPreferGSCostTable (set in ZN4Tuning and inherited by ZN5Tuning) and
-; have AVX-512, so the new tables are consulted in getGSVectorCost.
-; ZNVER3 does NOT carry TuningPreferGSCostTable and lacks both AVX-512 and
-; TuningFastGather, so isLegalMaskedGather() returns false and the cost model
-; walks the scalarise path (getGSScalarCost). The ZNVER3 numbers below are the
-; unchanged scalar fallback cost, included here only to lock in that this
-; change does not regress pre-AVX-512 Zen targets.
-; SKX is a non-Zen AVX-512 baseline showing the generic flat overhead of 2.
-; X8664V4 pins the glibc-hwcap x86-64-v4 (generic AVX-512) baseline. Values
-; match SKX, confirming that this change cannot affect distros shipping
-; x86-64-v4-tier binaries.
+; Cost-model coverage for the per-shape masked gather/scatter overheads.
+;
+; ZNVER4 / ZNVER5 carry TuningPreferGSCostTable and have AVX-512, so
+; getGSVectorCost reads the measured hardware cost from the schedule model
+; (which stays honest for llvm-mca) and adds the per-shape break-even overhead
+; from the TTI tables on top.
+; ZNVER3 lacks AVX-512, so masked gather is scalarised; its numbers are the
+; unchanged scalar fallback, kept only to show no regression on pre-AVX-512 Zen.
+; SKX is a non-Zen AVX-512 baseline: not gated into the schedule-model path, so
+; it keeps the generic flat overhead and its numbers are unchanged.
;
; RUN: opt < %s -S -mtriple=x86_64-unknown-linux-gnu -passes="print<cost-model>" 2>&1 -disable-output -cost-kind=throughput -mcpu=znver4 | FileCheck %s --check-prefix=ZNVER4
; RUN: opt < %s -S -mtriple=x86_64-unknown-linux-gnu -passes="print<cost-model>" 2>&1 -disable-output -cost-kind=throughput -mcpu=znver5 | FileCheck %s --check-prefix=ZNVER5
; RUN: opt < %s -S -mtriple=x86_64-unknown-linux-gnu -passes="print<cost-model>" 2>&1 -disable-output -cost-kind=throughput -mcpu=znver3 | FileCheck %s --check-prefix=ZNVER3
; RUN: opt < %s -S -mtriple=x86_64-unknown-linux-gnu -passes="print<cost-model>" 2>&1 -disable-output -cost-kind=throughput -mcpu=skx | FileCheck %s --check-prefix=SKX
-; RUN: opt < %s -S -mtriple=x86_64-unknown-linux-gnu -passes="print<cost-model>" 2>&1 -disable-output -cost-kind=throughput -mcpu=x86-64-v4 | FileCheck %s --check-prefix=X8664V4
;------------------------------------------------------------------------------
; Masked gather - i32 element type
@@ -26,101 +21,85 @@
define <2 x i32> @gather_v2i32(<2 x ptr> %ptrs, <2 x i1> %mask) {
; ZNVER4-LABEL: 'gather_v2i32'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i32> @llvm.masked.gather.v2i32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x i32> poison)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i32> @llvm.masked.gather.v2i32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x i32> undef)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i32> %v
;
; ZNVER5-LABEL: 'gather_v2i32'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i32> @llvm.masked.gather.v2i32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x i32> poison)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i32> @llvm.masked.gather.v2i32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x i32> undef)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i32> %v
;
; ZNVER3-LABEL: 'gather_v2i32'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x i32> @llvm.masked.gather.v2i32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x i32> poison)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x i32> @llvm.masked.gather.v2i32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x i32> undef)
; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i32> %v
;
; SKX-LABEL: 'gather_v2i32'
-; SKX-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i32> @llvm.masked.gather.v2i32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x i32> poison)
+; SKX-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i32> @llvm.masked.gather.v2i32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x i32> undef)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i32> %v
;
-; X8664V4-LABEL: 'gather_v2i32'
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i32> @llvm.masked.gather.v2i32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x i32> poison)
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i32> %v
-;
- %v = call <2 x i32> @llvm.masked.gather.v2i32.v2p0(<2 x ptr> %ptrs, i32 4, <2 x i1> %mask, <2 x i32> poison)
+ %v = call <2 x i32> @llvm.masked.gather.v2i32.v2p0(<2 x ptr> %ptrs, i32 4, <2 x i1> %mask, <2 x i32> undef)
ret <2 x i32> %v
}
define <4 x i32> @gather_v4i32(<4 x ptr> %ptrs, <4 x i1> %mask) {
; ZNVER4-LABEL: 'gather_v4i32'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 11 for instruction: %v = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x i32> poison)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 11 for instruction: %v = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x i32> undef)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i32> %v
;
; ZNVER5-LABEL: 'gather_v4i32'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 11 for instruction: %v = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x i32> poison)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 11 for instruction: %v = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x i32> undef)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i32> %v
;
; ZNVER3-LABEL: 'gather_v4i32'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 14 for instruction: %v = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x i32> poison)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 14 for instruction: %v = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x i32> undef)
; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i32> %v
;
; SKX-LABEL: 'gather_v4i32'
-; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x i32> poison)
+; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x i32> undef)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i32> %v
;
-; X8664V4-LABEL: 'gather_v4i32'
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x i32> poison)
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i32> %v
-;
- %v = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> %ptrs, i32 4, <4 x i1> %mask, <4 x i32> poison)
+ %v = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> %ptrs, i32 4, <4 x i1> %mask, <4 x i32> undef)
ret <4 x i32> %v
}
define <8 x i32> @gather_v8i32(<8 x ptr> %ptrs, <8 x i1> %mask) {
; ZNVER4-LABEL: 'gather_v8i32'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x i32> poison)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 23 for instruction: %v = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x i32> undef)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i32> %v
;
; ZNVER5-LABEL: 'gather_v8i32'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x i32> poison)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 23 for instruction: %v = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x i32> undef)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i32> %v
;
; ZNVER3-LABEL: 'gather_v8i32'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 28 for instruction: %v = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x i32> poison)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 28 for instruction: %v = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x i32> undef)
; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i32> %v
;
; SKX-LABEL: 'gather_v8i32'
-; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x i32> poison)
+; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x i32> undef)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i32> %v
;
-; X8664V4-LABEL: 'gather_v8i32'
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x i32> poison)
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i32> %v
-;
- %v = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> %ptrs, i32 4, <8 x i1> %mask, <8 x i32> poison)
+ %v = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> %ptrs, i32 4, <8 x i1> %mask, <8 x i32> undef)
ret <8 x i32> %v
}
define <16 x i32> @gather_v16i32(<16 x ptr> %ptrs, <16 x i1> %mask) {
; ZNVER4-LABEL: 'gather_v16i32'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 50 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> poison)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 46 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> undef)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i32> %v
;
; ZNVER5-LABEL: 'gather_v16i32'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 50 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> poison)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 46 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> undef)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i32> %v
;
; ZNVER3-LABEL: 'gather_v16i32'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 55 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> poison)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 55 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> undef)
; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i32> %v
;
; SKX-LABEL: 'gather_v16i32'
-; SKX-NEXT: Cost Model: Found an estimated cost of 20 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> poison)
+; SKX-NEXT: Cost Model: Found an estimated cost of 20 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> undef)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i32> %v
;
-; X8664V4-LABEL: 'gather_v16i32'
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 20 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> poison)
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i32> %v
-;
- %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> %ptrs, i32 4, <16 x i1> %mask, <16 x i32> poison)
+ %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> %ptrs, i32 4, <16 x i1> %mask, <16 x i32> undef)
ret <16 x i32> %v
}
@@ -130,76 +109,64 @@ define <16 x i32> @gather_v16i32(<16 x ptr> %ptrs, <16 x i1> %mask) {
define <2 x i64> @gather_v2i64(<2 x ptr> %ptrs, <2 x i1> %mask) {
; ZNVER4-LABEL: 'gather_v2i64'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i64> @llvm.masked.gather.v2i64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x i64> poison)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i64> @llvm.masked.gather.v2i64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x i64> undef)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i64> %v
;
; ZNVER5-LABEL: 'gather_v2i64'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i64> @llvm.masked.gather.v2i64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x i64> poison)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i64> @llvm.masked.gather.v2i64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x i64> undef)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i64> %v
;
; ZNVER3-LABEL: 'gather_v2i64'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x i64> @llvm.masked.gather.v2i64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x i64> poison)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x i64> @llvm.masked.gather.v2i64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x i64> undef)
; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i64> %v
;
; SKX-LABEL: 'gather_v2i64'
-; SKX-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i64> @llvm.masked.gather.v2i64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x i64> poison)
+; SKX-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i64> @llvm.masked.gather.v2i64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x i64> undef)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i64> %v
;
-; X8664V4-LABEL: 'gather_v2i64'
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v = call <2 x i64> @llvm.masked.gather.v2i64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x i64> poison)
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x i64> %v
-;
- %v = call <2 x i64> @llvm.masked.gather.v2i64.v2p0(<2 x ptr> %ptrs, i32 8, <2 x i1> %mask, <2 x i64> poison)
+ %v = call <2 x i64> @llvm.masked.gather.v2i64.v2p0(<2 x ptr> %ptrs, i32 8, <2 x i1> %mask, <2 x i64> undef)
ret <2 x i64> %v
}
define <4 x i64> @gather_v4i64(<4 x ptr> %ptrs, <4 x i1> %mask) {
; ZNVER4-LABEL: 'gather_v4i64'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 14 for instruction: %v = call <4 x i64> @llvm.masked.gather.v4i64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x i64> poison)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 14 for instruction: %v = call <4 x i64> @llvm.masked.gather.v4i64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x i64> undef)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i64> %v
;
; ZNVER5-LABEL: 'gather_v4i64'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 14 for instruction: %v = call <4 x i64> @llvm.masked.gather.v4i64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x i64> poison)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 14 for instruction: %v = call <4 x i64> @llvm.masked.gather.v4i64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x i64> undef)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i64> %v
;
; ZNVER3-LABEL: 'gather_v4i64'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 15 for instruction: %v = call <4 x i64> @llvm.masked.gather.v4i64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x i64> poison)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 15 for instruction: %v = call <4 x i64> @llvm.masked.gather.v4i64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x i64> undef)
; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i64> %v
;
; SKX-LABEL: 'gather_v4i64'
-; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x i64> @llvm.masked.gather.v4i64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x i64> poison)
+; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x i64> @llvm.masked.gather.v4i64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x i64> undef)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i64> %v
;
-; X8664V4-LABEL: 'gather_v4i64'
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x i64> @llvm.masked.gather.v4i64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x i64> poison)
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x i64> %v
-;
- %v = call <4 x i64> @llvm.masked.gather.v4i64.v4p0(<4 x ptr> %ptrs, i32 8, <4 x i1> %mask, <4 x i64> poison)
+ %v = call <4 x i64> @llvm.masked.gather.v4i64.v4p0(<4 x ptr> %ptrs, i32 8, <4 x i1> %mask, <4 x i64> undef)
ret <4 x i64> %v
}
define <8 x i64> @gather_v8i64(<8 x ptr> %ptrs, <8 x i1> %mask) {
; ZNVER4-LABEL: 'gather_v8i64'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 30 for instruction: %v = call <8 x i64> @llvm.masked.gather.v8i64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x i64> poison)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 30 for instruction: %v = call <8 x i64> @llvm.masked.gather.v8i64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x i64> undef)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i64> %v
;
; ZNVER5-LABEL: 'gather_v8i64'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 30 for instruction: %v = call <8 x i64> @llvm.masked.gather.v8i64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x i64> poison)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 30 for instruction: %v = call <8 x i64> @llvm.masked.gather.v8i64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x i64> undef)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i64> %v
;
; ZNVER3-LABEL: 'gather_v8i64'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 29 for instruction: %v = call <8 x i64> @llvm.masked.gather.v8i64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x i64> poison)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 29 for instruction: %v = call <8 x i64> @llvm.masked.gather.v8i64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x i64> undef)
; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i64> %v
;
; SKX-LABEL: 'gather_v8i64'
-; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x i64> @llvm.masked.gather.v8i64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x i64> poison)
+; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x i64> @llvm.masked.gather.v8i64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x i64> undef)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i64> %v
;
-; X8664V4-LABEL: 'gather_v8i64'
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x i64> @llvm.masked.gather.v8i64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x i64> poison)
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x i64> %v
-;
- %v = call <8 x i64> @llvm.masked.gather.v8i64.v8p0(<8 x ptr> %ptrs, i32 8, <8 x i1> %mask, <8 x i64> poison)
+ %v = call <8 x i64> @llvm.masked.gather.v8i64.v8p0(<8 x ptr> %ptrs, i32 8, <8 x i1> %mask, <8 x i64> undef)
ret <8 x i64> %v
}
@@ -209,101 +176,85 @@ define <8 x i64> @gather_v8i64(<8 x ptr> %ptrs, <8 x i1> %mask) {
define <2 x float> @gather_v2f32(<2 x ptr> %ptrs, <2 x i1> %mask) {
; ZNVER4-LABEL: 'gather_v2f32'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x float> @llvm.masked.gather.v2f32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x float> poison)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x float> @llvm.masked.gather.v2f32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x float> undef)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x float> %v
;
; ZNVER5-LABEL: 'gather_v2f32'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x float> @llvm.masked.gather.v2f32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x float> poison)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x float> @llvm.masked.gather.v2f32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x float> undef)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x float> %v
;
; ZNVER3-LABEL: 'gather_v2f32'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <2 x float> @llvm.masked.gather.v2f32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x float> poison)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <2 x float> @llvm.masked.gather.v2f32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x float> undef)
; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x float> %v
;
; SKX-LABEL: 'gather_v2f32'
-; SKX-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x float> @llvm.masked.gather.v2f32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x float> poison)
+; SKX-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x float> @llvm.masked.gather.v2f32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x float> undef)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x float> %v
;
-; X8664V4-LABEL: 'gather_v2f32'
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x float> @llvm.masked.gather.v2f32.v2p0(<2 x ptr> align 4 %ptrs, <2 x i1> %mask, <2 x float> poison)
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x float> %v
-;
- %v = call <2 x float> @llvm.masked.gather.v2f32.v2p0(<2 x ptr> %ptrs, i32 4, <2 x i1> %mask, <2 x float> poison)
+ %v = call <2 x float> @llvm.masked.gather.v2f32.v2p0(<2 x ptr> %ptrs, i32 4, <2 x i1> %mask, <2 x float> undef)
ret <2 x float> %v
}
define <4 x float> @gather_v4f32(<4 x ptr> %ptrs, <4 x i1> %mask) {
; ZNVER4-LABEL: 'gather_v4f32'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 11 for instruction: %v = call <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x float> poison)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 11 for instruction: %v = call <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x float> undef)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x float> %v
;
; ZNVER5-LABEL: 'gather_v4f32'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 11 for instruction: %v = call <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x float> poison)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 11 for instruction: %v = call <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x float> undef)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x float> %v
;
; ZNVER3-LABEL: 'gather_v4f32'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 13 for instruction: %v = call <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x float> poison)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 13 for instruction: %v = call <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x float> undef)
; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x float> %v
;
; SKX-LABEL: 'gather_v4f32'
-; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x float> poison)
+; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x float> undef)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x float> %v
;
-; X8664V4-LABEL: 'gather_v4f32'
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr> align 4 %ptrs, <4 x i1> %mask, <4 x float> poison)
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x float> %v
-;
- %v = call <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr> %ptrs, i32 4, <4 x i1> %mask, <4 x float> poison)
+ %v = call <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr> %ptrs, i32 4, <4 x i1> %mask, <4 x float> undef)
ret <4 x float> %v
}
define <8 x float> @gather_v8f32(<8 x ptr> %ptrs, <8 x i1> %mask) {
; ZNVER4-LABEL: 'gather_v8f32'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x float> poison)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 23 for instruction: %v = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x float> undef)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x float> %v
;
; ZNVER5-LABEL: 'gather_v8f32'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x float> poison)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 23 for instruction: %v = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x float> undef)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x float> %v
;
; ZNVER3-LABEL: 'gather_v8f32'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 26 for instruction: %v = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x float> poison)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 26 for instruction: %v = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x float> undef)
; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x float> %v
;
; SKX-LABEL: 'gather_v8f32'
-; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x float> poison)
+; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x float> undef)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x float> %v
;
-; X8664V4-LABEL: 'gather_v8f32'
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> align 4 %ptrs, <8 x i1> %mask, <8 x float> poison)
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x float> %v
-;
- %v = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> %ptrs, i32 4, <8 x i1> %mask, <8 x float> poison)
+ %v = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> %ptrs, i32 4, <8 x i1> %mask, <8 x float> undef)
ret <8 x float> %v
}
define <16 x float> @gather_v16f32(<16 x ptr> %ptrs, <16 x i1> %mask) {
; ZNVER4-LABEL: 'gather_v16f32'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 50 for instruction: %v = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x float> poison)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 46 for instruction: %v = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x float> undef)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x float> %v
;
; ZNVER5-LABEL: 'gather_v16f32'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 50 for instruction: %v = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x float> poison)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 46 for instruction: %v = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x float> undef)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x float> %v
;
; ZNVER3-LABEL: 'gather_v16f32'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 51 for instruction: %v = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x float> poison)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 51 for instruction: %v = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x float> undef)
; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x float> %v
;
; SKX-LABEL: 'gather_v16f32'
-; SKX-NEXT: Cost Model: Found an estimated cost of 20 for instruction: %v = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x float> poison)
+; SKX-NEXT: Cost Model: Found an estimated cost of 20 for instruction: %v = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x float> undef)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x float> %v
;
-; X8664V4-LABEL: 'gather_v16f32'
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 20 for instruction: %v = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x float> poison)
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x float> %v
-;
- %v = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> %ptrs, i32 4, <16 x i1> %mask, <16 x float> poison)
+ %v = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> %ptrs, i32 4, <16 x i1> %mask, <16 x float> undef)
ret <16 x float> %v
}
@@ -313,76 +264,64 @@ define <16 x float> @gather_v16f32(<16 x ptr> %ptrs, <16 x i1> %mask) {
define <2 x double> @gather_v2f64(<2 x ptr> %ptrs, <2 x i1> %mask) {
; ZNVER4-LABEL: 'gather_v2f64'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x double> @llvm.masked.gather.v2f64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x double> poison)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x double> @llvm.masked.gather.v2f64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x double> undef)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x double> %v
;
; ZNVER5-LABEL: 'gather_v2f64'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x double> @llvm.masked.gather.v2f64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x double> poison)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x double> @llvm.masked.gather.v2f64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x double> undef)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x double> %v
;
; ZNVER3-LABEL: 'gather_v2f64'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <2 x double> @llvm.masked.gather.v2f64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x double> poison)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <2 x double> @llvm.masked.gather.v2f64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x double> undef)
; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x double> %v
;
; SKX-LABEL: 'gather_v2f64'
-; SKX-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x double> @llvm.masked.gather.v2f64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x double> poison)
+; SKX-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x double> @llvm.masked.gather.v2f64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x double> undef)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x double> %v
;
-; X8664V4-LABEL: 'gather_v2f64'
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 7 for instruction: %v = call <2 x double> @llvm.masked.gather.v2f64.v2p0(<2 x ptr> align 8 %ptrs, <2 x i1> %mask, <2 x double> poison)
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <2 x double> %v
-;
- %v = call <2 x double> @llvm.masked.gather.v2f64.v2p0(<2 x ptr> %ptrs, i32 8, <2 x i1> %mask, <2 x double> poison)
+ %v = call <2 x double> @llvm.masked.gather.v2f64.v2p0(<2 x ptr> %ptrs, i32 8, <2 x i1> %mask, <2 x double> undef)
ret <2 x double> %v
}
define <4 x double> @gather_v4f64(<4 x ptr> %ptrs, <4 x i1> %mask) {
; ZNVER4-LABEL: 'gather_v4f64'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 11 for instruction: %v = call <4 x double> @llvm.masked.gather.v4f64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x double> poison)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 11 for instruction: %v = call <4 x double> @llvm.masked.gather.v4f64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x double> undef)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x double> %v
;
; ZNVER5-LABEL: 'gather_v4f64'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 11 for instruction: %v = call <4 x double> @llvm.masked.gather.v4f64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x double> poison)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 11 for instruction: %v = call <4 x double> @llvm.masked.gather.v4f64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x double> undef)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x double> %v
;
; ZNVER3-LABEL: 'gather_v4f64'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 13 for instruction: %v = call <4 x double> @llvm.masked.gather.v4f64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x double> poison)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 13 for instruction: %v = call <4 x double> @llvm.masked.gather.v4f64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x double> undef)
; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x double> %v
;
; SKX-LABEL: 'gather_v4f64'
-; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x double> @llvm.masked.gather.v4f64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x double> poison)
+; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x double> @llvm.masked.gather.v4f64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x double> undef)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x double> %v
;
-; X8664V4-LABEL: 'gather_v4f64'
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %v = call <4 x double> @llvm.masked.gather.v4f64.v4p0(<4 x ptr> align 8 %ptrs, <4 x i1> %mask, <4 x double> poison)
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <4 x double> %v
-;
- %v = call <4 x double> @llvm.masked.gather.v4f64.v4p0(<4 x ptr> %ptrs, i32 8, <4 x i1> %mask, <4 x double> poison)
+ %v = call <4 x double> @llvm.masked.gather.v4f64.v4p0(<4 x ptr> %ptrs, i32 8, <4 x i1> %mask, <4 x double> undef)
ret <4 x double> %v
}
define <8 x double> @gather_v8f64(<8 x ptr> %ptrs, <8 x i1> %mask) {
; ZNVER4-LABEL: 'gather_v8f64'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x double> @llvm.masked.gather.v8f64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x double> poison)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x double> @llvm.masked.gather.v8f64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x double> undef)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x double> %v
;
; ZNVER5-LABEL: 'gather_v8f64'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x double> @llvm.masked.gather.v8f64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x double> poison)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x double> @llvm.masked.gather.v8f64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x double> undef)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x double> %v
;
; ZNVER3-LABEL: 'gather_v8f64'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x double> @llvm.masked.gather.v8f64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x double> poison)
+; ZNVER3-NEXT: Cost Model: Found an estimated cost of 25 for instruction: %v = call <8 x double> @llvm.masked.gather.v8f64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x double> undef)
; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x double> %v
;
; SKX-LABEL: 'gather_v8f64'
-; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x double> @llvm.masked.gather.v8f64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x double> poison)
+; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x double> @llvm.masked.gather.v8f64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x double> undef)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x double> %v
;
-; X8664V4-LABEL: 'gather_v8f64'
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v = call <8 x double> @llvm.masked.gather.v8f64.v8p0(<8 x ptr> align 8 %ptrs, <8 x i1> %mask, <8 x double> poison)
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <8 x double> %v
-;
- %v = call <8 x double> @llvm.masked.gather.v8f64.v8p0(<8 x ptr> %ptrs, i32 8, <8 x i1> %mask, <8 x double> poison)
+ %v = call <8 x double> @llvm.masked.gather.v8f64.v8p0(<8 x ptr> %ptrs, i32 8, <8 x i1> %mask, <8 x double> undef)
ret <8 x double> %v
}
@@ -392,11 +331,11 @@ define <8 x double> @gather_v8f64(<8 x ptr> %ptrs, <8 x i1> %mask) {
define void @scatter_v4i32(<4 x i32> %src, <4 x ptr> %ptrs, <4 x i1> %mask) {
; ZNVER4-LABEL: 'scatter_v4i32'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 16 for instruction: call void @llvm.masked.scatter.v4i32.v4p0(<4 x i32> %src, <4 x ptr> align 4 %ptrs, <4 x i1> %mask)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 18 for instruction: call void @llvm.masked.scatter.v4i32.v4p0(<4 x i32> %src, <4 x ptr> align 4 %ptrs, <4 x i1> %mask)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; ZNVER5-LABEL: 'scatter_v4i32'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 16 for instruction: call void @llvm.masked.scatter.v4i32.v4p0(<4 x i32> %src, <4 x ptr> align 4 %ptrs, <4 x i1> %mask)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 18 for instruction: call void @llvm.masked.scatter.v4i32.v4p0(<4 x i32> %src, <4 x ptr> align 4 %ptrs, <4 x i1> %mask)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; ZNVER3-LABEL: 'scatter_v4i32'
@@ -406,10 +345,6 @@ define void @scatter_v4i32(<4 x i32> %src, <4 x ptr> %ptrs, <4 x i1> %mask) {
; SKX-LABEL: 'scatter_v4i32'
; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: call void @llvm.masked.scatter.v4i32.v4p0(<4 x i32> %src, <4 x ptr> align 4 %ptrs, <4 x i1> %mask)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
-;
-; X8664V4-LABEL: 'scatter_v4i32'
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 6 for instruction: call void @llvm.masked.scatter.v4i32.v4p0(<4 x i32> %src, <4 x ptr> align 4 %ptrs, <4 x i1> %mask)
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
call void @llvm.masked.scatter.v4i32.v4p0(<4 x i32> %src, <4 x ptr> %ptrs, i32 4, <4 x i1> %mask)
ret void
@@ -417,11 +352,11 @@ define void @scatter_v4i32(<4 x i32> %src, <4 x ptr> %ptrs, <4 x i1> %mask) {
define void @scatter_v8i32(<8 x i32> %src, <8 x ptr> %ptrs, <8 x i1> %mask) {
; ZNVER4-LABEL: 'scatter_v8i32'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 22 for instruction: call void @llvm.masked.scatter.v8i32.v8p0(<8 x i32> %src, <8 x ptr> align 4 %ptrs, <8 x i1> %mask)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 23 for instruction: call void @llvm.masked.scatter.v8i32.v8p0(<8 x i32> %src, <8 x ptr> align 4 %ptrs, <8 x i1> %mask)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; ZNVER5-LABEL: 'scatter_v8i32'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 22 for instruction: call void @llvm.masked.scatter.v8i32.v8p0(<8 x i32> %src, <8 x ptr> align 4 %ptrs, <8 x i1> %mask)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 23 for instruction: call void @llvm.masked.scatter.v8i32.v8p0(<8 x i32> %src, <8 x ptr> align 4 %ptrs, <8 x i1> %mask)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; ZNVER3-LABEL: 'scatter_v8i32'
@@ -431,10 +366,6 @@ define void @scatter_v8i32(<8 x i32> %src, <8 x ptr> %ptrs, <8 x i1> %mask) {
; SKX-LABEL: 'scatter_v8i32'
; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: call void @llvm.masked.scatter.v8i32.v8p0(<8 x i32> %src, <8 x ptr> align 4 %ptrs, <8 x i1> %mask)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
-;
-; X8664V4-LABEL: 'scatter_v8i32'
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 10 for instruction: call void @llvm.masked.scatter.v8i32.v8p0(<8 x i32> %src, <8 x ptr> align 4 %ptrs, <8 x i1> %mask)
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
call void @llvm.masked.scatter.v8i32.v8p0(<8 x i32> %src, <8 x ptr> %ptrs, i32 4, <8 x i1> %mask)
ret void
@@ -442,11 +373,11 @@ define void @scatter_v8i32(<8 x i32> %src, <8 x ptr> %ptrs, <8 x i1> %mask) {
define void @scatter_v16i32(<16 x i32> %src, <16 x ptr> %ptrs, <16 x i1> %mask) {
; ZNVER4-LABEL: 'scatter_v16i32'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 44 for instruction: call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32> %src, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 46 for instruction: call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32> %src, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; ZNVER5-LABEL: 'scatter_v16i32'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 44 for instruction: call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32> %src, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 46 for instruction: call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32> %src, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; ZNVER3-LABEL: 'scatter_v16i32'
@@ -456,10 +387,6 @@ define void @scatter_v16i32(<16 x i32> %src, <16 x ptr> %ptrs, <16 x i1> %mask)
; SKX-LABEL: 'scatter_v16i32'
; SKX-NEXT: Cost Model: Found an estimated cost of 20 for instruction: call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32> %src, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
-;
-; X8664V4-LABEL: 'scatter_v16i32'
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 20 for instruction: call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32> %src, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32> %src, <16 x ptr> %ptrs, i32 4, <16 x i1> %mask)
ret void
@@ -471,11 +398,11 @@ define void @scatter_v16i32(<16 x i32> %src, <16 x ptr> %ptrs, <16 x i1> %mask)
define void @scatter_v4i64(<4 x i64> %src, <4 x ptr> %ptrs, <4 x i1> %mask) {
; ZNVER4-LABEL: 'scatter_v4i64'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 14 for instruction: call void @llvm.masked.scatter.v4i64.v4p0(<4 x i64> %src, <4 x ptr> align 8 %ptrs, <4 x i1> %mask)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 15 for instruction: call void @llvm.masked.scatter.v4i64.v4p0(<4 x i64> %src, <4 x ptr> align 8 %ptrs, <4 x i1> %mask)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; ZNVER5-LABEL: 'scatter_v4i64'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 14 for instruction: call void @llvm.masked.scatter.v4i64.v4p0(<4 x i64> %src, <4 x ptr> align 8 %ptrs, <4 x i1> %mask)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 15 for instruction: call void @llvm.masked.scatter.v4i64.v4p0(<4 x i64> %src, <4 x ptr> align 8 %ptrs, <4 x i1> %mask)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; ZNVER3-LABEL: 'scatter_v4i64'
@@ -485,10 +412,6 @@ define void @scatter_v4i64(<4 x i64> %src, <4 x ptr> %ptrs, <4 x i1> %mask) {
; SKX-LABEL: 'scatter_v4i64'
; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: call void @llvm.masked.scatter.v4i64.v4p0(<4 x i64> %src, <4 x ptr> align 8 %ptrs, <4 x i1> %mask)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
-;
-; X8664V4-LABEL: 'scatter_v4i64'
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 6 for instruction: call void @llvm.masked.scatter.v4i64.v4p0(<4 x i64> %src, <4 x ptr> align 8 %ptrs, <4 x i1> %mask)
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
call void @llvm.masked.scatter.v4i64.v4p0(<4 x i64> %src, <4 x ptr> %ptrs, i32 8, <4 x i1> %mask)
ret void
@@ -496,11 +419,11 @@ define void @scatter_v4i64(<4 x i64> %src, <4 x ptr> %ptrs, <4 x i1> %mask) {
define void @scatter_v8i64(<8 x i64> %src, <8 x ptr> %ptrs, <8 x i1> %mask) {
; ZNVER4-LABEL: 'scatter_v8i64'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 30 for instruction: call void @llvm.masked.scatter.v8i64.v8p0(<8 x i64> %src, <8 x ptr> align 8 %ptrs, <8 x i1> %mask)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 31 for instruction: call void @llvm.masked.scatter.v8i64.v8p0(<8 x i64> %src, <8 x ptr> align 8 %ptrs, <8 x i1> %mask)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; ZNVER5-LABEL: 'scatter_v8i64'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 30 for instruction: call void @llvm.masked.scatter.v8i64.v8p0(<8 x i64> %src, <8 x ptr> align 8 %ptrs, <8 x i1> %mask)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 31 for instruction: call void @llvm.masked.scatter.v8i64.v8p0(<8 x i64> %src, <8 x ptr> align 8 %ptrs, <8 x i1> %mask)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; ZNVER3-LABEL: 'scatter_v8i64'
@@ -510,10 +433,6 @@ define void @scatter_v8i64(<8 x i64> %src, <8 x ptr> %ptrs, <8 x i1> %mask) {
; SKX-LABEL: 'scatter_v8i64'
; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: call void @llvm.masked.scatter.v8i64.v8p0(<8 x i64> %src, <8 x ptr> align 8 %ptrs, <8 x i1> %mask)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
-;
-; X8664V4-LABEL: 'scatter_v8i64'
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 10 for instruction: call void @llvm.masked.scatter.v8i64.v8p0(<8 x i64> %src, <8 x ptr> align 8 %ptrs, <8 x i1> %mask)
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
call void @llvm.masked.scatter.v8i64.v8p0(<8 x i64> %src, <8 x ptr> %ptrs, i32 8, <8 x i1> %mask)
ret void
@@ -525,11 +444,11 @@ define void @scatter_v8i64(<8 x i64> %src, <8 x ptr> %ptrs, <8 x i1> %mask) {
define void @scatter_v4f32(<4 x float> %src, <4 x ptr> %ptrs, <4 x i1> %mask) {
; ZNVER4-LABEL: 'scatter_v4f32'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 16 for instruction: call void @llvm.masked.scatter.v4f32.v4p0(<4 x float> %src, <4 x ptr> align 4 %ptrs, <4 x i1> %mask)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 18 for instruction: call void @llvm.masked.scatter.v4f32.v4p0(<4 x float> %src, <4 x ptr> align 4 %ptrs, <4 x i1> %mask)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; ZNVER5-LABEL: 'scatter_v4f32'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 16 for instruction: call void @llvm.masked.scatter.v4f32.v4p0(<4 x float> %src, <4 x ptr> align 4 %ptrs, <4 x i1> %mask)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 18 for instruction: call void @llvm.masked.scatter.v4f32.v4p0(<4 x float> %src, <4 x ptr> align 4 %ptrs, <4 x i1> %mask)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; ZNVER3-LABEL: 'scatter_v4f32'
@@ -539,10 +458,6 @@ define void @scatter_v4f32(<4 x float> %src, <4 x ptr> %ptrs, <4 x i1> %mask) {
; SKX-LABEL: 'scatter_v4f32'
; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: call void @llvm.masked.scatter.v4f32.v4p0(<4 x float> %src, <4 x ptr> align 4 %ptrs, <4 x i1> %mask)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
-;
-; X8664V4-LABEL: 'scatter_v4f32'
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 6 for instruction: call void @llvm.masked.scatter.v4f32.v4p0(<4 x float> %src, <4 x ptr> align 4 %ptrs, <4 x i1> %mask)
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
call void @llvm.masked.scatter.v4f32.v4p0(<4 x float> %src, <4 x ptr> %ptrs, i32 4, <4 x i1> %mask)
ret void
@@ -550,11 +465,11 @@ define void @scatter_v4f32(<4 x float> %src, <4 x ptr> %ptrs, <4 x i1> %mask) {
define void @scatter_v8f32(<8 x float> %src, <8 x ptr> %ptrs, <8 x i1> %mask) {
; ZNVER4-LABEL: 'scatter_v8f32'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 22 for instruction: call void @llvm.masked.scatter.v8f32.v8p0(<8 x float> %src, <8 x ptr> align 4 %ptrs, <8 x i1> %mask)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 23 for instruction: call void @llvm.masked.scatter.v8f32.v8p0(<8 x float> %src, <8 x ptr> align 4 %ptrs, <8 x i1> %mask)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; ZNVER5-LABEL: 'scatter_v8f32'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 22 for instruction: call void @llvm.masked.scatter.v8f32.v8p0(<8 x float> %src, <8 x ptr> align 4 %ptrs, <8 x i1> %mask)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 23 for instruction: call void @llvm.masked.scatter.v8f32.v8p0(<8 x float> %src, <8 x ptr> align 4 %ptrs, <8 x i1> %mask)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; ZNVER3-LABEL: 'scatter_v8f32'
@@ -564,10 +479,6 @@ define void @scatter_v8f32(<8 x float> %src, <8 x ptr> %ptrs, <8 x i1> %mask) {
; SKX-LABEL: 'scatter_v8f32'
; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: call void @llvm.masked.scatter.v8f32.v8p0(<8 x float> %src, <8 x ptr> align 4 %ptrs, <8 x i1> %mask)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
-;
-; X8664V4-LABEL: 'scatter_v8f32'
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 10 for instruction: call void @llvm.masked.scatter.v8f32.v8p0(<8 x float> %src, <8 x ptr> align 4 %ptrs, <8 x i1> %mask)
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
call void @llvm.masked.scatter.v8f32.v8p0(<8 x float> %src, <8 x ptr> %ptrs, i32 4, <8 x i1> %mask)
ret void
@@ -575,11 +486,11 @@ define void @scatter_v8f32(<8 x float> %src, <8 x ptr> %ptrs, <8 x i1> %mask) {
define void @scatter_v16f32(<16 x float> %src, <16 x ptr> %ptrs, <16 x i1> %mask) {
; ZNVER4-LABEL: 'scatter_v16f32'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 44 for instruction: call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> %src, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 46 for instruction: call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> %src, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; ZNVER5-LABEL: 'scatter_v16f32'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 44 for instruction: call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> %src, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 46 for instruction: call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> %src, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; ZNVER3-LABEL: 'scatter_v16f32'
@@ -589,10 +500,6 @@ define void @scatter_v16f32(<16 x float> %src, <16 x ptr> %ptrs, <16 x i1> %mask
; SKX-LABEL: 'scatter_v16f32'
; SKX-NEXT: Cost Model: Found an estimated cost of 20 for instruction: call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> %src, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
-;
-; X8664V4-LABEL: 'scatter_v16f32'
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 20 for instruction: call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> %src, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> %src, <16 x ptr> %ptrs, i32 4, <16 x i1> %mask)
ret void
@@ -604,11 +511,11 @@ define void @scatter_v16f32(<16 x float> %src, <16 x ptr> %ptrs, <16 x i1> %mask
define void @scatter_v4f64(<4 x double> %src, <4 x ptr> %ptrs, <4 x i1> %mask) {
; ZNVER4-LABEL: 'scatter_v4f64'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 9 for instruction: call void @llvm.masked.scatter.v4f64.v4p0(<4 x double> %src, <4 x ptr> align 8 %ptrs, <4 x i1> %mask)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 10 for instruction: call void @llvm.masked.scatter.v4f64.v4p0(<4 x double> %src, <4 x ptr> align 8 %ptrs, <4 x i1> %mask)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; ZNVER5-LABEL: 'scatter_v4f64'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 9 for instruction: call void @llvm.masked.scatter.v4f64.v4p0(<4 x double> %src, <4 x ptr> align 8 %ptrs, <4 x i1> %mask)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 10 for instruction: call void @llvm.masked.scatter.v4f64.v4p0(<4 x double> %src, <4 x ptr> align 8 %ptrs, <4 x i1> %mask)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; ZNVER3-LABEL: 'scatter_v4f64'
@@ -618,10 +525,6 @@ define void @scatter_v4f64(<4 x double> %src, <4 x ptr> %ptrs, <4 x i1> %mask) {
; SKX-LABEL: 'scatter_v4f64'
; SKX-NEXT: Cost Model: Found an estimated cost of 6 for instruction: call void @llvm.masked.scatter.v4f64.v4p0(<4 x double> %src, <4 x ptr> align 8 %ptrs, <4 x i1> %mask)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
-;
-; X8664V4-LABEL: 'scatter_v4f64'
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 6 for instruction: call void @llvm.masked.scatter.v4f64.v4p0(<4 x double> %src, <4 x ptr> align 8 %ptrs, <4 x i1> %mask)
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
call void @llvm.masked.scatter.v4f64.v4p0(<4 x double> %src, <4 x ptr> %ptrs, i32 8, <4 x i1> %mask)
ret void
@@ -629,11 +532,11 @@ define void @scatter_v4f64(<4 x double> %src, <4 x ptr> %ptrs, <4 x i1> %mask) {
define void @scatter_v8f64(<8 x double> %src, <8 x ptr> %ptrs, <8 x i1> %mask) {
; ZNVER4-LABEL: 'scatter_v8f64'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 23 for instruction: call void @llvm.masked.scatter.v8f64.v8p0(<8 x double> %src, <8 x ptr> align 8 %ptrs, <8 x i1> %mask)
+; ZNVER4-NEXT: Cost Model: Found an estimated cost of 24 for instruction: call void @llvm.masked.scatter.v8f64.v8p0(<8 x double> %src, <8 x ptr> align 8 %ptrs, <8 x i1> %mask)
; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; ZNVER5-LABEL: 'scatter_v8f64'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 23 for instruction: call void @llvm.masked.scatter.v8f64.v8p0(<8 x double> %src, <8 x ptr> align 8 %ptrs, <8 x i1> %mask)
+; ZNVER5-NEXT: Cost Model: Found an estimated cost of 24 for instruction: call void @llvm.masked.scatter.v8f64.v8p0(<8 x double> %src, <8 x ptr> align 8 %ptrs, <8 x i1> %mask)
; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; ZNVER3-LABEL: 'scatter_v8f64'
@@ -643,114 +546,7 @@ define void @scatter_v8f64(<8 x double> %src, <8 x ptr> %ptrs, <8 x i1> %mask) {
; SKX-LABEL: 'scatter_v8f64'
; SKX-NEXT: Cost Model: Found an estimated cost of 10 for instruction: call void @llvm.masked.scatter.v8f64.v8p0(<8 x double> %src, <8 x ptr> align 8 %ptrs, <8 x i1> %mask)
; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
-;
-; X8664V4-LABEL: 'scatter_v8f64'
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 10 for instruction: call void @llvm.masked.scatter.v8f64.v8p0(<8 x double> %src, <8 x ptr> align 8 %ptrs, <8 x i1> %mask)
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
call void @llvm.masked.scatter.v8f64.v8p0(<8 x double> %src, <8 x ptr> %ptrs, i32 8, <8 x i1> %mask)
ret void
}
-
-; Coverage for the v16 entries that are only reachable when the index can be
-; reduced to i32 (no SplitFactor in getGSVectorCost). Without this stanza, the
-; <16 x ptr>-form gather/scatter splits into two <8 x ...> operations and the
-; v16 row of the cost table is never consulted, masking any divergence between
-; i32 and f32 costs at VF=16 (vpgatherdd/vpscatterdd vs vgatherdps/vscatterdps
-; -- same physical load/store on Zen, same expected cost). The scatter pair
-; below pins the scatter v16 row; gather_v16i32_gep does the same for the
-; gather v16 row (the code path that does the GEP-index reducibility check is
-; shared between gather and scatter, so one gather case suffices).
-
-define <16 x i32> @gather_v16i32_gep(ptr %base, <16 x i32> %idx, <16 x i1> %mask) {
-; ZNVER4-LABEL: 'gather_v16i32_gep'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 30 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> poison)
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i32> %v
-;
-; ZNVER5-LABEL: 'gather_v16i32_gep'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 30 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> poison)
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i32> %v
-;
-; ZNVER3-LABEL: 'gather_v16i32_gep'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 55 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> poison)
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i32> %v
-;
-; SKX-LABEL: 'gather_v16i32_gep'
-; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
-; SKX-NEXT: Cost Model: Found an estimated cost of 18 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> poison)
-; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i32> %v
-;
-; X8664V4-LABEL: 'gather_v16i32_gep'
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 18 for instruction: %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> align 4 %ptrs, <16 x i1> %mask, <16 x i32> poison)
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret <16 x i32> %v
-;
- %ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
- %v = call <16 x i32> @llvm.masked.gather.v16i32.v16p0(<16 x ptr> %ptrs, i32 4, <16 x i1> %mask, <16 x i32> poison)
- ret <16 x i32> %v
-}
-
-define void @scatter_v16f32_gep(ptr %base, <16 x i32> %idx, <16 x float> %val, <16 x i1> %mask) {
-; ZNVER4-LABEL: 'scatter_v16f32_gep'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr float, ptr %base, <16 x i32> %idx
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 22 for instruction: call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> %val, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
-;
-; ZNVER5-LABEL: 'scatter_v16f32_gep'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr float, ptr %base, <16 x i32> %idx
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 22 for instruction: call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> %val, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
-;
-; ZNVER3-LABEL: 'scatter_v16f32_gep'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr float, ptr %base, <16 x i32> %idx
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 51 for instruction: call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> %val, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
-;
-; SKX-LABEL: 'scatter_v16f32_gep'
-; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr float, ptr %base, <16 x i32> %idx
-; SKX-NEXT: Cost Model: Found an estimated cost of 18 for instruction: call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> %val, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
-; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
-;
-; X8664V4-LABEL: 'scatter_v16f32_gep'
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr float, ptr %base, <16 x i32> %idx
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 18 for instruction: call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> %val, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
-;
- %ptrs = getelementptr float, ptr %base, <16 x i32> %idx
- call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> %val, <16 x ptr> %ptrs, i32 4, <16 x i1> %mask)
- ret void
-}
-
-define void @scatter_v16i32_gep(ptr %base, <16 x i32> %idx, <16 x i32> %val, <16 x i1> %mask) {
-; ZNVER4-LABEL: 'scatter_v16i32_gep'
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 22 for instruction: call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32> %val, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
-; ZNVER4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
-;
-; ZNVER5-LABEL: 'scatter_v16i32_gep'
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 22 for instruction: call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32> %val, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
-; ZNVER5-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
-;
-; ZNVER3-LABEL: 'scatter_v16i32_gep'
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 55 for instruction: call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32> %val, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
-; ZNVER3-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
-;
-; SKX-LABEL: 'scatter_v16i32_gep'
-; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
-; SKX-NEXT: Cost Model: Found an estimated cost of 18 for instruction: call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32> %val, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
-; SKX-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
-;
-; X8664V4-LABEL: 'scatter_v16i32_gep'
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 18 for instruction: call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32> %val, <16 x ptr> align 4 %ptrs, <16 x i1> %mask)
-; X8664V4-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
-;
- %ptrs = getelementptr i32, ptr %base, <16 x i32> %idx
- call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32> %val, <16 x ptr> %ptrs, i32 4, <16 x i1> %mask)
- ret void
-}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/amd-zen-gather-scatter-decisions.ll b/llvm/test/Transforms/LoopVectorize/X86/gather-scatter-cost-table-decisions.ll
similarity index 78%
rename from llvm/test/Transforms/LoopVectorize/X86/amd-zen-gather-scatter-decisions.ll
rename to llvm/test/Transforms/LoopVectorize/X86/gather-scatter-cost-table-decisions.ll
index 4723b2b07f400..2474aa4bcb755 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/amd-zen-gather-scatter-decisions.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/gather-scatter-cost-table-decisions.ll
@@ -1,22 +1,17 @@
-; End-to-end loop-vectorize decisions driven by the AMD Zen per-shape
-; gather/scatter cost tables (TuningPreferGSCostTable, set on znver4+).
+; End-to-end loop-vectorize decisions driven by the per-shape gather/scatter
+; cost tables (TuningPreferGSCostTable, set on znver4+). The companion
+; cost-model test masked-gather-scatter-cost-table.ll pins the individual cost
+; numbers; this test pins the resulting vectorizer decisions so cost-model
+; refactors that re-enable harmful gathers (or suppress profitable ones) are
+; caught here.
;
-; The companion cost-model test
-; llvm/test/Analysis/CostModel/X86/masked-gather-scatter-amd-zen.ll
-; pins individual cost numbers; this test pins the resulting vectorizer
-; decisions so future cost-model refactors that accidentally re-enable
-; harmful gathers (or suppress profitable ones) are caught here.
-;
-; The three cases below correspond to:
-; 1. f64 indirect-load reduction -- gather IS chosen on znver5
-; (the lbm-style win the cost table exists to enable).
-; 2. i64 indirect-load reduction -- gather is NOT chosen on znver5;
-; the i64 entry was measured separately and deliberately set above
-; the break-even so vpgatherqq is suppressed for harmful patterns
-; (cf. PR #198850 / libquantum regression).
-; 3. Unit-stride load -- the vectorizer must emit a plain wide load
-; (not @llvm.masked.gather) regardless of cost-table values.
-; Regression guard for issue #91370.
+; The three cases below:
+; 1. f64 indirect-load reduction -- gather IS chosen on znver5.
+; 2. i64 indirect-load reduction -- gather is NOT chosen on znver5 (the i64
+; entry is set above the break-even to suppress vpgatherqq for harmful
+; patterns, cf. PR llvm#198850).
+; 3. Unit-stride load -- must stay a plain wide load, not a gather.
+; Regression guard for issue llvm#91370.
;
; RUN: opt < %s -S -passes=loop-vectorize -mtriple=x86_64-unknown-linux-gnu -mcpu=znver5 | FileCheck %s
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512.s
index 14b8e5f36c666..b84743abd6ab4 100644
--- a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512.s
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512.s
@@ -1496,8 +1496,8 @@ vunpcklps (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
# CHECK-NEXT: 1 11 1.00 * vfmadd231ps (%rax), %zmm17, %zmm19 {%k1} {z}
# CHECK-NEXT: 1 11 1.00 * vfmadd231ps (%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
# CHECK-NEXT: 1 5 0.33 * vgatherdpd (%rax,%ymm1,2), %zmm2 {%k1}
-# CHECK-NEXT: 1 5 0.33 * vgatherdps (%rax,%zmm1,2), %zmm2 {%k1}
-# CHECK-NEXT: 1 5 0.33 * vgatherqpd (%rax,%zmm1,2), %zmm2 {%k1}
+# CHECK-NEXT: 65 30 12.67 * vgatherdps (%rax,%zmm1,2), %zmm2 {%k1}
+# CHECK-NEXT: 41 25 8.00 * vgatherqpd (%rax,%zmm1,2), %zmm2 {%k1}
# CHECK-NEXT: 1 5 0.33 * vgatherqps (%rax,%zmm1,2), %ymm2 {%k1}
# CHECK-NEXT: 1 2 1.00 vmaxpd %zmm16, %zmm17, %zmm19
# CHECK-NEXT: 1 9 1.00 * vmaxpd (%rax), %zmm17, %zmm19
@@ -1733,8 +1733,8 @@ vunpcklps (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
# CHECK-NEXT: 1 8 0.50 * vpcmpequq (%rax), %zmm1, %k2 {%k3}
# CHECK-NEXT: 1 8 0.50 * vpcmpequq (%rax){1to8}, %zmm1, %k2 {%k3}
# CHECK-NEXT: 1 5 0.33 * vpgatherdq (%rax,%ymm1,2), %zmm2 {%k1}
-# CHECK-NEXT: 1 5 0.33 * vpgatherdd (%rax,%zmm1,2), %zmm2 {%k1}
-# CHECK-NEXT: 1 5 0.33 * vpgatherqq (%rax,%zmm1,2), %zmm2 {%k1}
+# CHECK-NEXT: 65 30 12.67 * vpgatherdd (%rax,%zmm1,2), %zmm2 {%k1}
+# CHECK-NEXT: 41 25 8.00 * vpgatherqq (%rax,%zmm1,2), %zmm2 {%k1}
# CHECK-NEXT: 1 5 0.33 * vpgatherqd (%rax,%zmm1,2), %ymm2 {%k1}
# CHECK-NEXT: 1 5 1.00 vpmovdb %zmm19, %xmm16
# CHECK-NEXT: 1 11 1.50 * vpmovdb %zmm19, (%rax)
@@ -1970,10 +1970,10 @@ vunpcklps (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
# CHECK-NEXT: 2 1 0.50 vpermq %zmm16, %zmm17, %zmm19 {%k1} {z}
# CHECK-NEXT: 2 8 0.50 * vpermq (%rax), %zmm17, %zmm19 {%k1} {z}
# CHECK-NEXT: 2 8 0.50 * vpermq (%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
-# CHECK-NEXT: 1 1 1.00 * vpscatterdd %zmm1, (%rdx,%zmm0,4) {%k1}
+# CHECK-NEXT: 89 24 17.00 * vpscatterdd %zmm1, (%rdx,%zmm0,4) {%k1}
# CHECK-NEXT: 1 1 1.00 * vpscatterdq %zmm1, (%rdx,%ymm0,4) {%k1}
# CHECK-NEXT: 1 1 1.00 * vpscatterqd %ymm1, (%rdx,%zmm0,4) {%k1}
-# CHECK-NEXT: 1 1 1.00 * vpscatterqq %zmm1, (%rdx,%zmm0,4) {%k1}
+# CHECK-NEXT: 49 14 9.00 * vpscatterqq %zmm1, (%rdx,%zmm0,4) {%k1}
# CHECK-NEXT: 1 1 1.00 vpshufd $0, %zmm16, %zmm19
# CHECK-NEXT: 1 8 1.00 * vpshufd $0, (%rax), %zmm19
# CHECK-NEXT: 1 8 1.00 * vpshufd $0, (%rax){1to16}, %zmm19
@@ -2037,10 +2037,10 @@ vunpcklps (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
# CHECK-NEXT: 1 1 1.00 vpunpcklqdq %zmm16, %zmm17, %zmm19 {%k1} {z}
# CHECK-NEXT: 1 8 1.00 * vpunpcklqdq (%rax), %zmm17, %zmm19 {%k1} {z}
# CHECK-NEXT: 1 8 1.00 * vpunpcklqdq (%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
-# CHECK-NEXT: 1 1 1.00 * vscatterdps %zmm1, (%rdx,%zmm0,4) {%k1}
+# CHECK-NEXT: 89 24 17.00 * vscatterdps %zmm1, (%rdx,%zmm0,4) {%k1}
# CHECK-NEXT: 1 1 1.00 * vscatterdpd %zmm1, (%rdx,%ymm0,4) {%k1}
# CHECK-NEXT: 1 1 1.00 * vscatterqps %ymm1, (%rdx,%zmm0,4) {%k1}
-# CHECK-NEXT: 1 1 1.00 * vscatterqpd %zmm1, (%rdx,%zmm0,4) {%k1}
+# CHECK-NEXT: 49 14 9.00 * vscatterqpd %zmm1, (%rdx,%zmm0,4) {%k1}
# CHECK-NEXT: 1 2 1.00 vshuff32x4 $0, %zmm16, %zmm17, %zmm19
# CHECK-NEXT: 3 9 1.00 * vshuff32x4 $0, (%rax), %zmm17, %zmm19
# CHECK-NEXT: 3 9 1.00 * vshuff32x4 $0, (%rax){1to16}, %zmm17, %zmm19
@@ -2233,7 +2233,7 @@ vunpcklps (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
# CHECK: Resource pressure per iteration:
# CHECK-NEXT: [0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12.0] [12.1] [13] [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
-# CHECK-NEXT: 5.33 5.33 5.33 - - - - - 219.50 1119.00 676.50 351.00 312.50 312.50 17.00 215.67 215.67 215.67 204.67 204.67 204.67 16.50 16.50
+# CHECK-NEXT: 96.00 96.00 96.00 - - - - - 219.50 1119.00 676.50 351.00 312.50 312.50 17.00 287.67 287.67 287.67 244.67 244.67 244.67 64.50 64.50
# CHECK: Resource pressure by instruction:
# CHECK-NEXT: [0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12.0] [12.1] [13] [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
@@ -2553,8 +2553,8 @@ vunpcklps (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
# CHECK-NEXT: - - - - - - - - 1.00 1.00 - - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vfmadd231ps (%rax), %zmm17, %zmm19 {%k1} {z}
# CHECK-NEXT: - - - - - - - - 1.00 1.00 - - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vfmadd231ps (%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vgatherdpd (%rax,%ymm1,2), %zmm2 {%k1}
-# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vgatherdps (%rax,%zmm1,2), %zmm2 {%k1}
-# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vgatherqpd (%rax,%zmm1,2), %zmm2 {%k1}
+# CHECK-NEXT: 12.67 12.67 12.67 - - - - - - - - - - - - 12.67 12.67 12.67 12.67 12.67 12.67 - - vgatherdps (%rax,%zmm1,2), %zmm2 {%k1}
+# CHECK-NEXT: 8.00 8.00 8.00 - - - - - - - - - - - - 8.00 8.00 8.00 8.00 8.00 8.00 - - vgatherqpd (%rax,%zmm1,2), %zmm2 {%k1}
# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vgatherqps (%rax,%zmm1,2), %ymm2 {%k1}
# CHECK-NEXT: - - - - - - - - 1.00 1.00 - - - - - - - - - - - - - vmaxpd %zmm16, %zmm17, %zmm19
# CHECK-NEXT: - - - - - - - - 1.00 1.00 - - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vmaxpd (%rax), %zmm17, %zmm19
@@ -2790,8 +2790,8 @@ vunpcklps (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpcmpequq (%rax), %zmm1, %k2 {%k3}
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpcmpequq (%rax){1to8}, %zmm1, %k2 {%k3}
# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpgatherdq (%rax,%ymm1,2), %zmm2 {%k1}
-# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpgatherdd (%rax,%zmm1,2), %zmm2 {%k1}
-# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpgatherqq (%rax,%zmm1,2), %zmm2 {%k1}
+# CHECK-NEXT: 12.67 12.67 12.67 - - - - - - - - - - - - 12.67 12.67 12.67 12.67 12.67 12.67 - - vpgatherdd (%rax,%zmm1,2), %zmm2 {%k1}
+# CHECK-NEXT: 8.00 8.00 8.00 - - - - - - - - - - - - 8.00 8.00 8.00 8.00 8.00 8.00 - - vpgatherqq (%rax,%zmm1,2), %zmm2 {%k1}
# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpgatherqd (%rax,%zmm1,2), %ymm2 {%k1}
# CHECK-NEXT: - - - - - - - - - 1.00 1.00 - - - - - - - - - - - - vpmovdb %zmm19, %xmm16
# CHECK-NEXT: - - - - - - - - - 1.50 1.50 - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpmovdb %zmm19, (%rax)
@@ -3027,10 +3027,10 @@ vunpcklps (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
# CHECK-NEXT: - - - - - - - - - 0.50 0.50 - - - - - - - - - - - - vpermq %zmm16, %zmm17, %zmm19 {%k1} {z}
# CHECK-NEXT: - - - - - - - - - 0.50 0.50 - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpermq (%rax), %zmm17, %zmm19 {%k1} {z}
# CHECK-NEXT: - - - - - - - - - 0.50 0.50 - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpermq (%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
-# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.67 0.67 0.67 - - - 1.00 1.00 vpscatterdd %zmm1, (%rdx,%zmm0,4) {%k1}
+# CHECK-NEXT: 17.00 17.00 17.00 - - - - - - - - - - - - 11.33 11.33 11.33 - - - 17.00 17.00 vpscatterdd %zmm1, (%rdx,%zmm0,4) {%k1}
# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.67 0.67 0.67 - - - 1.00 1.00 vpscatterdq %zmm1, (%rdx,%ymm0,4) {%k1}
# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.67 0.67 0.67 - - - 1.00 1.00 vpscatterqd %ymm1, (%rdx,%zmm0,4) {%k1}
-# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.67 0.67 0.67 - - - 1.00 1.00 vpscatterqq %zmm1, (%rdx,%zmm0,4) {%k1}
+# CHECK-NEXT: 9.00 9.00 9.00 - - - - - - - - - - - - 6.00 6.00 6.00 - - - 9.00 9.00 vpscatterqq %zmm1, (%rdx,%zmm0,4) {%k1}
# CHECK-NEXT: - - - - - - - - - 1.00 1.00 - - - - - - - - - - - - vpshufd $0, %zmm16, %zmm19
# CHECK-NEXT: - - - - - - - - - 1.00 1.00 - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpshufd $0, (%rax), %zmm19
# CHECK-NEXT: - - - - - - - - - 1.00 1.00 - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpshufd $0, (%rax){1to16}, %zmm19
@@ -3094,10 +3094,10 @@ vunpcklps (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
# CHECK-NEXT: - - - - - - - - - 1.00 1.00 - - - - - - - - - - - - vpunpcklqdq %zmm16, %zmm17, %zmm19 {%k1} {z}
# CHECK-NEXT: - - - - - - - - - 1.00 1.00 - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpunpcklqdq (%rax), %zmm17, %zmm19 {%k1} {z}
# CHECK-NEXT: - - - - - - - - - 1.00 1.00 - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpunpcklqdq (%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
-# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.67 0.67 0.67 - - - 1.00 1.00 vscatterdps %zmm1, (%rdx,%zmm0,4) {%k1}
+# CHECK-NEXT: 17.00 17.00 17.00 - - - - - - - - - - - - 11.33 11.33 11.33 - - - 17.00 17.00 vscatterdps %zmm1, (%rdx,%zmm0,4) {%k1}
# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.67 0.67 0.67 - - - 1.00 1.00 vscatterdpd %zmm1, (%rdx,%ymm0,4) {%k1}
# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.67 0.67 0.67 - - - 1.00 1.00 vscatterqps %ymm1, (%rdx,%zmm0,4) {%k1}
-# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.67 0.67 0.67 - - - 1.00 1.00 vscatterqpd %zmm1, (%rdx,%zmm0,4) {%k1}
+# CHECK-NEXT: 9.00 9.00 9.00 - - - - - - - - - - - - 6.00 6.00 6.00 - - - 9.00 9.00 vscatterqpd %zmm1, (%rdx,%zmm0,4) {%k1}
# CHECK-NEXT: - - - - - - - - - 1.00 - - - - - - - - - - - - - vshuff32x4 $0, %zmm16, %zmm17, %zmm19
# CHECK-NEXT: - - - - - - - - - 1.00 - - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vshuff32x4 $0, (%rax), %zmm17, %zmm19
# CHECK-NEXT: - - - - - - - - - 1.00 - - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vshuff32x4 $0, (%rax){1to16}, %zmm17, %zmm19
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vl.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vl.s
index ead609e33da4d..3fce1ca8b962d 100644
--- a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vl.s
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vl.s
@@ -2393,11 +2393,11 @@ vunpcklps (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
# CHECK-NEXT: 1 11 0.50 * vfmadd231ps (%rax), %ymm17, %ymm19 {%k1} {z}
# CHECK-NEXT: 1 11 0.50 * vfmadd231ps (%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
# CHECK-NEXT: 1 5 0.33 * vgatherdpd (%rax,%xmm1,2), %ymm2 {%k1}
-# CHECK-NEXT: 1 5 0.33 * vgatherdps (%rax,%ymm1,2), %ymm2 {%k1}
-# CHECK-NEXT: 1 5 0.33 * vgatherqpd (%rax,%ymm1,2), %ymm2 {%k1}
+# CHECK-NEXT: 33 23 6.33 * vgatherdps (%rax,%ymm1,2), %ymm2 {%k1}
+# CHECK-NEXT: 20 19 4.00 * vgatherqpd (%rax,%ymm1,2), %ymm2 {%k1}
# CHECK-NEXT: 1 5 0.33 * vgatherqps (%rax,%ymm1,2), %xmm2 {%k1}
# CHECK-NEXT: 1 5 0.33 * vgatherdpd (%rax,%xmm1,2), %xmm2 {%k1}
-# CHECK-NEXT: 1 5 0.33 * vgatherdps (%rax,%xmm1,2), %xmm2 {%k1}
+# CHECK-NEXT: 20 17 4.00 * vgatherdps (%rax,%xmm1,2), %xmm2 {%k1}
# CHECK-NEXT: 1 5 0.33 * vgatherqpd (%rax,%xmm1,2), %xmm2 {%k1}
# CHECK-NEXT: 1 5 0.33 * vgatherqps (%rax,%xmm1,2), %xmm2 {%k1}
# CHECK-NEXT: 1 2 1.00 vmaxpd %xmm16, %xmm17, %xmm19
@@ -2957,11 +2957,11 @@ vunpcklps (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
# CHECK-NEXT: 2 8 0.50 * vpermq (%rax), %ymm17, %ymm19 {%k1} {z}
# CHECK-NEXT: 2 8 0.50 * vpermq (%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
# CHECK-NEXT: 1 5 0.33 * vpgatherdq (%rax,%xmm1,2), %ymm2 {%k1}
-# CHECK-NEXT: 1 5 0.33 * vpgatherdd (%rax,%ymm1,2), %ymm2 {%k1}
-# CHECK-NEXT: 1 5 0.33 * vpgatherqq (%rax,%ymm1,2), %ymm2 {%k1}
+# CHECK-NEXT: 33 23 6.33 * vpgatherdd (%rax,%ymm1,2), %ymm2 {%k1}
+# CHECK-NEXT: 20 19 4.00 * vpgatherqq (%rax,%ymm1,2), %ymm2 {%k1}
# CHECK-NEXT: 1 5 0.33 * vpgatherqd (%rax,%ymm1,2), %xmm2 {%k1}
# CHECK-NEXT: 1 5 0.33 * vpgatherdq (%rax,%xmm1,2), %xmm2 {%k1}
-# CHECK-NEXT: 1 5 0.33 * vpgatherdd (%rax,%xmm1,2), %xmm2 {%k1}
+# CHECK-NEXT: 20 17 4.00 * vpgatherdd (%rax,%xmm1,2), %xmm2 {%k1}
# CHECK-NEXT: 1 5 0.33 * vpgatherqq (%rax,%xmm1,2), %xmm2 {%k1}
# CHECK-NEXT: 1 5 0.33 * vpgatherqd (%rax,%xmm1,2), %xmm2 {%k1}
# CHECK-NEXT: 1 2 0.50 vpmovdb %xmm19, %xmm16
@@ -3252,14 +3252,14 @@ vunpcklps (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
# CHECK-NEXT: 1 3 0.50 vpmulld %ymm16, %ymm17, %ymm19 {%k1} {z}
# CHECK-NEXT: 1 10 0.50 * vpmulld (%rax), %ymm17, %ymm19 {%k1} {z}
# CHECK-NEXT: 1 10 0.50 * vpmulld (%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
-# CHECK-NEXT: 1 1 1.00 * vpscatterdd %xmm1, (%rdx,%xmm0,4) {%k1}
+# CHECK-NEXT: 28 10 6.00 * vpscatterdd %xmm1, (%rdx,%xmm0,4) {%k1}
# CHECK-NEXT: 1 1 1.00 * vpscatterdq %xmm1, (%rdx,%xmm0,4) {%k1}
# CHECK-NEXT: 1 1 1.00 * vpscatterqd %xmm1, (%rdx,%xmm0,4) {%k1}
# CHECK-NEXT: 1 1 1.00 * vpscatterqq %xmm1, (%rdx,%xmm0,4) {%k1}
-# CHECK-NEXT: 1 1 1.00 * vpscatterdd %ymm1, (%rdx,%ymm0,4) {%k1}
+# CHECK-NEXT: 49 14 9.00 * vpscatterdd %ymm1, (%rdx,%ymm0,4) {%k1}
# CHECK-NEXT: 1 1 1.00 * vpscatterdq %ymm1, (%rdx,%xmm0,4) {%k1}
# CHECK-NEXT: 1 1 1.00 * vpscatterqd %xmm1, (%rdx,%ymm0,4) {%k1}
-# CHECK-NEXT: 1 1 1.00 * vpscatterqq %ymm1, (%rdx,%ymm0,4) {%k1}
+# CHECK-NEXT: 28 9 5.00 * vpscatterqq %ymm1, (%rdx,%ymm0,4) {%k1}
# CHECK-NEXT: 1 1 0.50 vpshufd $0, %xmm16, %xmm19
# CHECK-NEXT: 1 8 0.50 * vpshufd $0, (%rax), %xmm19
# CHECK-NEXT: 1 8 0.50 * vpshufd $0, (%rax){1to4}, %xmm19
@@ -3398,14 +3398,14 @@ vunpcklps (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
# CHECK-NEXT: 1 1 0.50 vpunpckldq %ymm16, %ymm17, %ymm19 {%k1} {z}
# CHECK-NEXT: 1 8 0.50 * vpunpckldq (%rax), %ymm17, %ymm19 {%k1} {z}
# CHECK-NEXT: 1 8 0.50 * vpunpckldq (%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
-# CHECK-NEXT: 1 1 1.00 * vscatterdps %xmm1, (%rdx,%xmm0,4) {%k1}
+# CHECK-NEXT: 28 10 6.00 * vscatterdps %xmm1, (%rdx,%xmm0,4) {%k1}
# CHECK-NEXT: 1 1 1.00 * vscatterdpd %xmm1, (%rdx,%xmm0,4) {%k1}
# CHECK-NEXT: 1 1 1.00 * vscatterqps %xmm1, (%rdx,%xmm0,4) {%k1}
# CHECK-NEXT: 1 1 1.00 * vscatterqpd %xmm1, (%rdx,%xmm0,4) {%k1}
-# CHECK-NEXT: 1 1 1.00 * vscatterdps %ymm1, (%rdx,%ymm0,4) {%k1}
+# CHECK-NEXT: 49 14 9.00 * vscatterdps %ymm1, (%rdx,%ymm0,4) {%k1}
# CHECK-NEXT: 1 1 1.00 * vscatterdpd %ymm1, (%rdx,%xmm0,4) {%k1}
# CHECK-NEXT: 1 1 1.00 * vscatterqps %xmm1, (%rdx,%ymm0,4) {%k1}
-# CHECK-NEXT: 1 1 1.00 * vscatterqpd %ymm1, (%rdx,%ymm0,4) {%k1}
+# CHECK-NEXT: 28 9 5.00 * vscatterqpd %ymm1, (%rdx,%ymm0,4) {%k1}
# CHECK-NEXT: 1 2 1.00 vshuff32x4 $0, %ymm16, %ymm17, %ymm19
# CHECK-NEXT: 3 9 1.00 * vshuff32x4 $0, (%rax), %ymm17, %ymm19
# CHECK-NEXT: 3 9 1.00 * vshuff32x4 $0, (%rax){1to8}, %ymm17, %ymm19
@@ -3614,7 +3614,7 @@ vunpcklps (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
# CHECK: Resource pressure per iteration:
# CHECK-NEXT: [0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12.0] [12.1] [13] [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
-# CHECK-NEXT: 10.67 10.67 10.67 - - - - - 208.00 1084.00 637.50 261.50 509.50 509.50 32.00 355.67 355.67 355.67 334.33 334.33 334.33 32.00 32.00
+# CHECK-NEXT: 75.33 75.33 75.33 - - - - - 208.00 1084.00 637.50 261.50 509.50 509.50 32.00 405.00 405.00 405.00 361.00 361.00 361.00 66.00 66.00
# CHECK: Resource pressure by instruction:
# CHECK-NEXT: [0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12.0] [12.1] [13] [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
@@ -4099,11 +4099,11 @@ vunpcklps (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
# CHECK-NEXT: - - - - - - - - 0.50 0.50 - - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vfmadd231ps (%rax), %ymm17, %ymm19 {%k1} {z}
# CHECK-NEXT: - - - - - - - - 0.50 0.50 - - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vfmadd231ps (%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vgatherdpd (%rax,%xmm1,2), %ymm2 {%k1}
-# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vgatherdps (%rax,%ymm1,2), %ymm2 {%k1}
-# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vgatherqpd (%rax,%ymm1,2), %ymm2 {%k1}
+# CHECK-NEXT: 6.33 6.33 6.33 - - - - - - - - - - - - 6.33 6.33 6.33 6.33 6.33 6.33 - - vgatherdps (%rax,%ymm1,2), %ymm2 {%k1}
+# CHECK-NEXT: 4.00 4.00 4.00 - - - - - - - - - - - - 4.00 4.00 4.00 4.00 4.00 4.00 - - vgatherqpd (%rax,%ymm1,2), %ymm2 {%k1}
# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vgatherqps (%rax,%ymm1,2), %xmm2 {%k1}
# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vgatherdpd (%rax,%xmm1,2), %xmm2 {%k1}
-# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vgatherdps (%rax,%xmm1,2), %xmm2 {%k1}
+# CHECK-NEXT: 4.00 4.00 4.00 - - - - - - - - - - - - 4.00 4.00 4.00 4.00 4.00 4.00 - - vgatherdps (%rax,%xmm1,2), %xmm2 {%k1}
# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vgatherqpd (%rax,%xmm1,2), %xmm2 {%k1}
# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vgatherqps (%rax,%xmm1,2), %xmm2 {%k1}
# CHECK-NEXT: - - - - - - - - 1.00 1.00 - - - - - - - - - - - - - vmaxpd %xmm16, %xmm17, %xmm19
@@ -4663,11 +4663,11 @@ vunpcklps (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
# CHECK-NEXT: - - - - - - - - - 0.50 0.50 - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpermq (%rax), %ymm17, %ymm19 {%k1} {z}
# CHECK-NEXT: - - - - - - - - - 0.50 0.50 - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpermq (%rax){1to4}, %ymm17, %ymm19 {%k1} {z}
# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpgatherdq (%rax,%xmm1,2), %ymm2 {%k1}
-# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpgatherdd (%rax,%ymm1,2), %ymm2 {%k1}
-# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpgatherqq (%rax,%ymm1,2), %ymm2 {%k1}
+# CHECK-NEXT: 6.33 6.33 6.33 - - - - - - - - - - - - 6.33 6.33 6.33 6.33 6.33 6.33 - - vpgatherdd (%rax,%ymm1,2), %ymm2 {%k1}
+# CHECK-NEXT: 4.00 4.00 4.00 - - - - - - - - - - - - 4.00 4.00 4.00 4.00 4.00 4.00 - - vpgatherqq (%rax,%ymm1,2), %ymm2 {%k1}
# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpgatherqd (%rax,%ymm1,2), %xmm2 {%k1}
# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpgatherdq (%rax,%xmm1,2), %xmm2 {%k1}
-# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpgatherdd (%rax,%xmm1,2), %xmm2 {%k1}
+# CHECK-NEXT: 4.00 4.00 4.00 - - - - - - - - - - - - 4.00 4.00 4.00 4.00 4.00 4.00 - - vpgatherdd (%rax,%xmm1,2), %xmm2 {%k1}
# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpgatherqq (%rax,%xmm1,2), %xmm2 {%k1}
# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpgatherqd (%rax,%xmm1,2), %xmm2 {%k1}
# CHECK-NEXT: - - - - - - - - - 0.50 0.50 - - - - - - - - - - - - vpmovdb %xmm19, %xmm16
@@ -4958,14 +4958,14 @@ vunpcklps (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
# CHECK-NEXT: - - - - - - - - 0.50 - - 0.50 - - - - - - - - - - - vpmulld %ymm16, %ymm17, %ymm19 {%k1} {z}
# CHECK-NEXT: - - - - - - - - 0.50 - - 0.50 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpmulld (%rax), %ymm17, %ymm19 {%k1} {z}
# CHECK-NEXT: - - - - - - - - 0.50 - - 0.50 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpmulld (%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
-# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.67 0.67 0.67 - - - 1.00 1.00 vpscatterdd %xmm1, (%rdx,%xmm0,4) {%k1}
+# CHECK-NEXT: 6.00 6.00 6.00 - - - - - - - - - - - - 4.00 4.00 4.00 - - - 6.00 6.00 vpscatterdd %xmm1, (%rdx,%xmm0,4) {%k1}
# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.67 0.67 0.67 - - - 1.00 1.00 vpscatterdq %xmm1, (%rdx,%xmm0,4) {%k1}
# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.67 0.67 0.67 - - - 1.00 1.00 vpscatterqd %xmm1, (%rdx,%xmm0,4) {%k1}
# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.67 0.67 0.67 - - - 1.00 1.00 vpscatterqq %xmm1, (%rdx,%xmm0,4) {%k1}
-# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.67 0.67 0.67 - - - 1.00 1.00 vpscatterdd %ymm1, (%rdx,%ymm0,4) {%k1}
+# CHECK-NEXT: 9.00 9.00 9.00 - - - - - - - - - - - - 6.00 6.00 6.00 - - - 9.00 9.00 vpscatterdd %ymm1, (%rdx,%ymm0,4) {%k1}
# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.67 0.67 0.67 - - - 1.00 1.00 vpscatterdq %ymm1, (%rdx,%xmm0,4) {%k1}
# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.67 0.67 0.67 - - - 1.00 1.00 vpscatterqd %xmm1, (%rdx,%ymm0,4) {%k1}
-# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.67 0.67 0.67 - - - 1.00 1.00 vpscatterqq %ymm1, (%rdx,%ymm0,4) {%k1}
+# CHECK-NEXT: 5.00 5.00 5.00 - - - - - - - - - - - - 3.33 3.33 3.33 - - - 5.00 5.00 vpscatterqq %ymm1, (%rdx,%ymm0,4) {%k1}
# CHECK-NEXT: - - - - - - - - - 0.50 0.50 - - - - - - - - - - - - vpshufd $0, %xmm16, %xmm19
# CHECK-NEXT: - - - - - - - - - 0.50 0.50 - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpshufd $0, (%rax), %xmm19
# CHECK-NEXT: - - - - - - - - - 0.50 0.50 - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpshufd $0, (%rax){1to4}, %xmm19
@@ -5104,14 +5104,14 @@ vunpcklps (%rax){1to8}, %ymm17, %ymm19 {z}{k1}
# CHECK-NEXT: - - - - - - - - - 0.50 0.50 - - - - - - - - - - - - vpunpckldq %ymm16, %ymm17, %ymm19 {%k1} {z}
# CHECK-NEXT: - - - - - - - - - 0.50 0.50 - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpunpckldq (%rax), %ymm17, %ymm19 {%k1} {z}
# CHECK-NEXT: - - - - - - - - - 0.50 0.50 - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpunpckldq (%rax){1to8}, %ymm17, %ymm19 {%k1} {z}
-# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.67 0.67 0.67 - - - 1.00 1.00 vscatterdps %xmm1, (%rdx,%xmm0,4) {%k1}
+# CHECK-NEXT: 6.00 6.00 6.00 - - - - - - - - - - - - 4.00 4.00 4.00 - - - 6.00 6.00 vscatterdps %xmm1, (%rdx,%xmm0,4) {%k1}
# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.67 0.67 0.67 - - - 1.00 1.00 vscatterdpd %xmm1, (%rdx,%xmm0,4) {%k1}
# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.67 0.67 0.67 - - - 1.00 1.00 vscatterqps %xmm1, (%rdx,%xmm0,4) {%k1}
# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.67 0.67 0.67 - - - 1.00 1.00 vscatterqpd %xmm1, (%rdx,%xmm0,4) {%k1}
-# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.67 0.67 0.67 - - - 1.00 1.00 vscatterdps %ymm1, (%rdx,%ymm0,4) {%k1}
+# CHECK-NEXT: 9.00 9.00 9.00 - - - - - - - - - - - - 6.00 6.00 6.00 - - - 9.00 9.00 vscatterdps %ymm1, (%rdx,%ymm0,4) {%k1}
# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.67 0.67 0.67 - - - 1.00 1.00 vscatterdpd %ymm1, (%rdx,%xmm0,4) {%k1}
# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.67 0.67 0.67 - - - 1.00 1.00 vscatterqps %xmm1, (%rdx,%ymm0,4) {%k1}
-# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.67 0.67 0.67 - - - 1.00 1.00 vscatterqpd %ymm1, (%rdx,%ymm0,4) {%k1}
+# CHECK-NEXT: 5.00 5.00 5.00 - - - - - - - - - - - - 3.33 3.33 3.33 - - - 5.00 5.00 vscatterqpd %ymm1, (%rdx,%ymm0,4) {%k1}
# CHECK-NEXT: - - - - - - - - - 1.00 - - - - - - - - - - - - - vshuff32x4 $0, %ymm16, %ymm17, %ymm19
# CHECK-NEXT: - - - - - - - - - 1.00 - - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vshuff32x4 $0, (%rax), %ymm17, %ymm19
# CHECK-NEXT: - - - - - - - - - 1.00 - - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vshuff32x4 $0, (%rax){1to8}, %ymm17, %ymm19
More information about the llvm-commits
mailing list