[llvm] [X86][CostModel] Decouple AVX512 from fast-gather in getGatherOverhead() (PR #206506)
Sourabh Singh Tomar via llvm-commits
llvm-commits at lists.llvm.org
Mon Jun 29 08:46:37 PDT 2026
https://github.com/SouraVX updated https://github.com/llvm/llvm-project/pull/206506
>From 5f173bcbe8646adf22a86503dbd796e919261994 Mon Sep 17 00:00:00 2001
From: Sourabh Singh Tomar <SourabhSingh.tomar at amd.com>
Date: Mon, 29 Jun 2026 19:50:16 +0530
Subject: [PATCH] [X86][CostModel] Decouple AVX512 from fast-gather in
getGatherOverhead()
A fast hardware gather is a micro-architectural property, not an ISA one.
getGatherOverhead() previously treated the presence of AVX512 as proof of a
fast gather, returning the small overhead (2) for every AVX512 target. This
keys the decision solely off the fast-gather tuning, so the cost model honors
it uniformly for AVX2 and AVX512.
Mainstream Intel AVX512 CPUs are unaffected: they already declare
TuningFastGather (x86-64-v4, SKL, SKX, CNL, ICL, KNL). Only targets that have
AVX512 without declaring fast-gather change -- a bare -mattr=+avx512f and AMD
Zen parts now receive the accurate slow-gather overhead instead of being
assumed fast.
Tests:
- New Analysis/CostModel/X86/gather-fast-vs-slow.ll: two RUN lines on the same
+avx512f ISA differing only by +fast-gather, asserting the slow vs fast
gather cost.
- CodeGen tests whose prefixes model real fast-gather CPUs (KNL/SKX) get
+fast-gather added to their RUN lines.
- LoopVectorize tests that model an Intel AVX512 CPU via bare +avx512f/bw/vl
get +fast-gather added to preserve intent (including the AVX512 function in
cost-conditional-branches.ll, keeping its gather-based vectorization).
---
.../lib/Target/X86/X86TargetTransformInfo.cpp | 9 ++++--
.../CostModel/X86/gather-fast-vs-slow.ll | 28 +++++++++++++++++++
llvm/test/CodeGen/X86/avx512-mask-set-opt.ll | 8 +++---
.../test/CodeGen/X86/masked_gather_scatter.ll | 10 +++----
.../X86/masked_gather_scatter_widen.ll | 4 +--
.../X86/x86-prefer-no-gather-no-scatter.ll | 4 +--
.../X86/CostModel/gather-i32-with-i8-index.ll | 28 +++++++++++++------
.../X86/CostModel/gather-i64-with-i8-index.ll | 28 +++++++++++++------
.../interleaved-load-f64-stride-5.ll | 2 +-
.../interleaved-load-f64-stride-6.ll | 2 +-
.../interleaved-load-f64-stride-7.ll | 2 +-
.../interleaved-load-f64-stride-8.ll | 2 +-
...erleaved-load-i32-stride-4-indices-0uuu.ll | 2 +-
.../interleaved-load-i64-stride-2.ll | 2 +-
.../interleaved-load-i64-stride-4.ll | 2 +-
.../interleaved-load-i64-stride-5.ll | 2 +-
.../interleaved-load-i64-stride-6.ll | 2 +-
.../interleaved-load-i64-stride-7.ll | 2 +-
.../interleaved-load-i64-stride-8.ll | 2 +-
.../masked-gather-i32-with-i8-index.ll | 28 +++++++++++++------
.../masked-gather-i64-with-i8-index.ll | 28 +++++++++++++------
.../X86/cost-conditional-branches.ll | 2 +-
.../LoopVectorize/X86/gather-vs-interleave.ll | 2 +-
.../X86/invariant-load-gather.ll | 2 +-
24 files changed, 137 insertions(+), 66 deletions(-)
create mode 100644 llvm/test/Analysis/CostModel/X86/gather-fast-vs-slow.ll
diff --git a/llvm/lib/Target/X86/X86TargetTransformInfo.cpp b/llvm/lib/Target/X86/X86TargetTransformInfo.cpp
index 1bc0ff397e476..6938aafe31297 100644
--- a/llvm/lib/Target/X86/X86TargetTransformInfo.cpp
+++ b/llvm/lib/Target/X86/X86TargetTransformInfo.cpp
@@ -6224,9 +6224,12 @@ int X86TTIImpl::getGatherOverhead() const {
// to the Load operation. "2" is the number provided by Intel architects. This
// parameter is used for cost estimation of Gather Op and comparison with
// other alternatives.
- // TODO: Remove the explicit hasAVX512()?, That would mean we would only
- // enable gather with a -march.
- if (ST->hasAVX512() || (ST->hasAVX2() && ST->hasFastGather()))
+ // Honor the "fast-gather" tuning uniformly: a fast hardware gather is a
+ // micro-architectural property, not an ISA one, so do not assume every
+ // AVX512 target has one. Targets that genuinely have a fast gather declare
+ // TuningFastGather (e.g. SKX, KNL, x86-64-v4); others (bare +avx512f, AMD
+ // Zen) get the slow-gather overhead.
+ if (ST->hasFastGather())
return 2;
return 1024;
diff --git a/llvm/test/Analysis/CostModel/X86/gather-fast-vs-slow.ll b/llvm/test/Analysis/CostModel/X86/gather-fast-vs-slow.ll
new file mode 100644
index 0000000000000..9aa63f2c47bbd
--- /dev/null
+++ b/llvm/test/Analysis/CostModel/X86/gather-fast-vs-slow.ll
@@ -0,0 +1,28 @@
+; NOTE: Assertions have been autogenerated by utils/update_analyze_test_checks.py
+; Check that the X86 gather cost model honors the "fast-gather" tuning
+; independently of the AVX512 ISA level. The two RUN lines use the *same*
+; AVX512 ISA and differ only by +fast-gather: without it the gather is costed
+; as a slow (scalarized) operation, with it the gather is cheap. This locks in
+; the decoupling of "target has AVX512" from "target has a fast hardware
+; gather".
+; RUN: opt < %s -S -mtriple=x86_64-- -passes="print<cost-model>" 2>&1 -disable-output -cost-kind=throughput -mattr=+avx512f | FileCheck %s --check-prefixes=AVX512-SLOWGATHER
+; RUN: opt < %s -S -mtriple=x86_64-- -passes="print<cost-model>" 2>&1 -disable-output -cost-kind=throughput -mattr=+avx512f,+fast-gather | FileCheck %s --check-prefixes=AVX512-FASTGATHER
+
+define void @gather_costs(<8 x ptr> %ptrs8, <8 x i1> %mask8, <16 x ptr> %ptrs16, <16 x i1> %mask16) {
+; AVX512-SLOWGATHER-LABEL: 'gather_costs'
+; AVX512-SLOWGATHER-NEXT: Cost Model: Found an estimated cost of 1032 for instruction: %v8 = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> align 4 %ptrs8, <8 x i1> %mask8, <8 x float> poison)
+; AVX512-SLOWGATHER-NEXT: Cost Model: Found an estimated cost of 2064 for instruction: %v16 = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> align 4 %ptrs16, <16 x i1> %mask16, <16 x float> poison)
+; AVX512-SLOWGATHER-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+; AVX512-FASTGATHER-LABEL: 'gather_costs'
+; AVX512-FASTGATHER-NEXT: Cost Model: Found an estimated cost of 10 for instruction: %v8 = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> align 4 %ptrs8, <8 x i1> %mask8, <8 x float> poison)
+; AVX512-FASTGATHER-NEXT: Cost Model: Found an estimated cost of 20 for instruction: %v16 = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> align 4 %ptrs16, <16 x i1> %mask16, <16 x float> poison)
+; AVX512-FASTGATHER-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+ %v8 = call <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr> %ptrs8, i32 4, <8 x i1> %mask8, <8 x float> poison)
+ %v16 = call <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr> %ptrs16, i32 4, <16 x i1> %mask16, <16 x float> poison)
+ ret void
+}
+
+declare <8 x float> @llvm.masked.gather.v8f32.v8p0(<8 x ptr>, i32, <8 x i1>, <8 x float>)
+declare <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr>, i32, <16 x i1>, <16 x float>)
diff --git a/llvm/test/CodeGen/X86/avx512-mask-set-opt.ll b/llvm/test/CodeGen/X86/avx512-mask-set-opt.ll
index ca5f3192d7b97..2f8c660217129 100644
--- a/llvm/test/CodeGen/X86/avx512-mask-set-opt.ll
+++ b/llvm/test/CodeGen/X86/avx512-mask-set-opt.ll
@@ -1,8 +1,8 @@
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx512f | FileCheck %s --check-prefixes=AVX512,AVX512F
-; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx512f,+avx512dq | FileCheck %s --check-prefixes=AVX512,AVX512DQ
-; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx512f,+avx512bw | FileCheck %s --check-prefixes=AVX512,AVX512BW
-; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx512f,+avx512dq,+avx512bw | FileCheck %s --check-prefixes=AVX512,AVX512DQBW
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx512f,+fast-gather | FileCheck %s --check-prefixes=AVX512,AVX512F
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx512f,+avx512dq,+fast-gather | FileCheck %s --check-prefixes=AVX512,AVX512DQ
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx512f,+avx512bw,+fast-gather | FileCheck %s --check-prefixes=AVX512,AVX512BW
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx512f,+avx512dq,+avx512bw,+fast-gather | FileCheck %s --check-prefixes=AVX512,AVX512DQBW
declare <16 x float> @llvm.masked.gather.v16f32.v16p0(<16 x ptr>, i32, <16 x i1>, <16 x float>)
declare <16 x float> @llvm.masked.expandload.v16f32(ptr, <16 x i1>, <16 x float>)
diff --git a/llvm/test/CodeGen/X86/masked_gather_scatter.ll b/llvm/test/CodeGen/X86/masked_gather_scatter.ll
index 65402bdaf4fae..7ba430c1c70de 100644
--- a/llvm/test/CodeGen/X86/masked_gather_scatter.ll
+++ b/llvm/test/CodeGen/X86/masked_gather_scatter.ll
@@ -1,9 +1,9 @@
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx512f < %s | FileCheck %s --check-prefixes=ALL,X64,X64-KNL
-; RUN: llc -mtriple=i386-unknown-linux-gnu -mattr=+avx512f < %s | FileCheck %s --check-prefixes=ALL,X86,X86-KNL
-; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx512vl -mattr=+avx512dq < %s | FileCheck %s --check-prefixes=ALL,X64,X64-SKX,X64-SKX-SMALL
-; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx512vl -mattr=+avx512dq -code-model=large < %s | FileCheck %s --check-prefixes=ALL,X64,X64-SKX,X64-SKX-LARGE
-; RUN: llc -mtriple=i386-unknown-linux-gnu -mattr=+avx512vl -mattr=+avx512dq < %s | FileCheck %s --check-prefixes=ALL,X86,X86-SKX
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx512f,+fast-gather < %s | FileCheck %s --check-prefixes=ALL,X64,X64-KNL
+; RUN: llc -mtriple=i386-unknown-linux-gnu -mattr=+avx512f,+fast-gather < %s | FileCheck %s --check-prefixes=ALL,X86,X86-KNL
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx512vl -mattr=+avx512dq,+fast-gather < %s | FileCheck %s --check-prefixes=ALL,X64,X64-SKX,X64-SKX-SMALL
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx512vl -mattr=+avx512dq,+fast-gather -code-model=large < %s | FileCheck %s --check-prefixes=ALL,X64,X64-SKX,X64-SKX-LARGE
+; RUN: llc -mtriple=i386-unknown-linux-gnu -mattr=+avx512vl -mattr=+avx512dq,+fast-gather < %s | FileCheck %s --check-prefixes=ALL,X86,X86-SKX
; RUN: opt -mtriple=x86_64-apple-darwin -passes=scalarize-masked-mem-intrin -mcpu=corei7-avx -S < %s | FileCheck %s -check-prefixes=SCALAR
; RUN: llc -O0 -mtriple=x86_64-unknown-linux-gnu -mcpu=skx < %s -o /dev/null
diff --git a/llvm/test/CodeGen/X86/masked_gather_scatter_widen.ll b/llvm/test/CodeGen/X86/masked_gather_scatter_widen.ll
index 5b5280601ea71..3a6b33b0f7003 100644
--- a/llvm/test/CodeGen/X86/masked_gather_scatter_widen.ll
+++ b/llvm/test/CodeGen/X86/masked_gather_scatter_widen.ll
@@ -1,6 +1,6 @@
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx512vl -mattr=+avx512dq < %s | FileCheck %s --check-prefix=WIDEN_SKX
-; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx512f < %s | FileCheck %s --check-prefix=WIDEN_KNL
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx512vl -mattr=+avx512dq,+fast-gather < %s | FileCheck %s --check-prefix=WIDEN_SKX
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx512f,+fast-gather < %s | FileCheck %s --check-prefix=WIDEN_KNL
; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mcpu=skylake < %s | FileCheck %s --check-prefix=WIDEN_AVX2
define <2 x double> @test_gather_v2i32_index(ptr %base, <2 x i32> %ind, <2 x i1> %mask, <2 x double> %src0) {
diff --git a/llvm/test/CodeGen/X86/x86-prefer-no-gather-no-scatter.ll b/llvm/test/CodeGen/X86/x86-prefer-no-gather-no-scatter.ll
index 33250b3495a00..9d614d8cb8e79 100644
--- a/llvm/test/CodeGen/X86/x86-prefer-no-gather-no-scatter.ll
+++ b/llvm/test/CodeGen/X86/x86-prefer-no-gather-no-scatter.ll
@@ -1,9 +1,9 @@
; Check that if option prefer-no-gather/scatter can disable gather/scatter instructions.
; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx2,+fast-gather %s -o - | FileCheck %s --check-prefixes=GATHER
-; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx2,+fast-gather,+prefer-no-gather %s -o - | FileCheck %s --check-prefixes=NO-GATHER
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx2,+prefer-no-gather %s -o - | FileCheck %s --check-prefixes=NO-GATHER
; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx512vl,+avx512dq < %s | FileCheck %s --check-prefix=SCATTER
; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx512vl,+avx512dq,+prefer-no-gather < %s | FileCheck %s --check-prefix=SCATTER-NO-GATHER
-; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx512vl,+avx512dq,+prefer-no-scatter < %s | FileCheck %s --check-prefix=GATHER-NO-SCATTER
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx512vl,+avx512dq,+prefer-no-scatter,+fast-gather < %s | FileCheck %s --check-prefix=GATHER-NO-SCATTER
; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx512vl,+avx512dq,+prefer-no-gather,+prefer-no-scatter < %s | FileCheck %s --check-prefix=NO-SCATTER-GATHER
@A = global [1024 x i8] zeroinitializer, align 128
diff --git a/llvm/test/Transforms/LoopVectorize/X86/CostModel/gather-i32-with-i8-index.ll b/llvm/test/Transforms/LoopVectorize/X86/CostModel/gather-i32-with-i8-index.ll
index a23d57cfd4448..48b3326b5efea 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/CostModel/gather-i32-with-i8-index.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/CostModel/gather-i32-with-i8-index.ll
@@ -4,7 +4,8 @@
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefixes=AVX1
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx2,-fast-gather --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefixes=AVX2-SLOWGATHER
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx2,+fast-gather --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefixes=AVX2-FASTGATHER
-; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512bw --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefixes=AVX512
+; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512bw --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefixes=AVX512-SLOWGATHER
+; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512bw,+fast-gather --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefixes=AVX512-FASTGATHER
; REQUIRES: asserts
@@ -54,14 +55,23 @@ define void @test() {
; AVX2-FASTGATHER: Cost of 24 for VF 16: {{.*}}ir<%valB> = load
; AVX2-FASTGATHER: Cost of 48 for VF 32: {{.*}}ir<%valB> = load
;
-; AVX512-LABEL: 'test'
-; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: %valB = load i32, ptr %inB, align 4
-; AVX512: Cost of 6 for VF 2: {{.*}}ir<%valB> = load
-; AVX512: Cost of 13 for VF 4: {{.*}}ir<%valB> = load
-; AVX512: Cost of 10 for VF 8: {{.*}}ir<%valB> = load
-; AVX512: Cost of 18 for VF 16: {{.*}}ir<%valB> = load
-; AVX512: Cost of 36 for VF 32: {{.*}}ir<%valB> = load
-; AVX512: Cost of 72 for VF 64: {{.*}}ir<%valB> = load
+; AVX512-SLOWGATHER-LABEL: 'test'
+; AVX512-SLOWGATHER: LV: Found an estimated cost of 1 for VF 1 For instruction: %valB = load i32, ptr %inB, align 4
+; AVX512-SLOWGATHER: Cost of 6 for VF 2: {{.*}}ir<%valB> = load
+; AVX512-SLOWGATHER: Cost of 13 for VF 4: {{.*}}ir<%valB> = load
+; AVX512-SLOWGATHER: Cost of 28 for VF 8: {{.*}}ir<%valB> = load
+; AVX512-SLOWGATHER: Cost of 57 for VF 16: {{.*}}ir<%valB> = load
+; AVX512-SLOWGATHER: Cost of 114 for VF 32: {{.*}}ir<%valB> = load
+; AVX512-SLOWGATHER: Cost of 228 for VF 64: {{.*}}ir<%valB> = load
+;
+; AVX512-FASTGATHER-LABEL: 'test'
+; AVX512-FASTGATHER: LV: Found an estimated cost of 1 for VF 1 For instruction: %valB = load i32, ptr %inB, align 4
+; AVX512-FASTGATHER: Cost of 6 for VF 2: {{.*}}ir<%valB> = load
+; AVX512-FASTGATHER: Cost of 13 for VF 4: {{.*}}ir<%valB> = load
+; AVX512-FASTGATHER: Cost of 10 for VF 8: {{.*}}ir<%valB> = load
+; AVX512-FASTGATHER: Cost of 18 for VF 16: {{.*}}ir<%valB> = load
+; AVX512-FASTGATHER: Cost of 36 for VF 32: {{.*}}ir<%valB> = load
+; AVX512-FASTGATHER: Cost of 72 for VF 64: {{.*}}ir<%valB> = load
;
entry:
br label %for.body
diff --git a/llvm/test/Transforms/LoopVectorize/X86/CostModel/gather-i64-with-i8-index.ll b/llvm/test/Transforms/LoopVectorize/X86/CostModel/gather-i64-with-i8-index.ll
index 5e52483aa93b3..d251be286a7ca 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/CostModel/gather-i64-with-i8-index.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/CostModel/gather-i64-with-i8-index.ll
@@ -4,7 +4,8 @@
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefixes=AVX1
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx2,-fast-gather --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefixes=AVX2-SLOWGATHER
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx2,+fast-gather --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefixes=AVX2-FASTGATHER
-; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512bw --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefixes=AVX512
+; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512bw --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefixes=AVX512-SLOWGATHER
+; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512bw,+fast-gather --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefixes=AVX512-FASTGATHER
; REQUIRES: asserts
@@ -54,14 +55,23 @@ define void @test() {
; AVX2-FASTGATHER: Cost of 24 for VF 16: {{.*}}ir<%valB> = load
; AVX2-FASTGATHER: Cost of 48 for VF 32: {{.*}}ir<%valB> = load
;
-; AVX512-LABEL: 'test'
-; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: %valB = load i64, ptr %inB, align 8
-; AVX512: Cost of 6 for VF 2: {{.*}}ir<%valB> = load
-; AVX512: Cost of 14 for VF 4: {{.*}}ir<%valB> = load
-; AVX512: Cost of 10 for VF 8: {{.*}}ir<%valB> = load
-; AVX512: Cost of 20 for VF 16: {{.*}}ir<%valB> = load
-; AVX512: Cost of 40 for VF 32: {{.*}}ir<%valB> = load
-; AVX512: Cost of 80 for VF 64: {{.*}}ir<%valB> = load
+; AVX512-SLOWGATHER-LABEL: 'test'
+; AVX512-SLOWGATHER: LV: Found an estimated cost of 1 for VF 1 For instruction: %valB = load i64, ptr %inB, align 8
+; AVX512-SLOWGATHER: Cost of 6 for VF 2: {{.*}}ir<%valB> = load
+; AVX512-SLOWGATHER: Cost of 14 for VF 4: {{.*}}ir<%valB> = load
+; AVX512-SLOWGATHER: Cost of 30 for VF 8: {{.*}}ir<%valB> = load
+; AVX512-SLOWGATHER: Cost of 60 for VF 16: {{.*}}ir<%valB> = load
+; AVX512-SLOWGATHER: Cost of 120 for VF 32: {{.*}}ir<%valB> = load
+; AVX512-SLOWGATHER: Cost of 240 for VF 64: {{.*}}ir<%valB> = load
+;
+; AVX512-FASTGATHER-LABEL: 'test'
+; AVX512-FASTGATHER: LV: Found an estimated cost of 1 for VF 1 For instruction: %valB = load i64, ptr %inB, align 8
+; AVX512-FASTGATHER: Cost of 6 for VF 2: {{.*}}ir<%valB> = load
+; AVX512-FASTGATHER: Cost of 14 for VF 4: {{.*}}ir<%valB> = load
+; AVX512-FASTGATHER: Cost of 10 for VF 8: {{.*}}ir<%valB> = load
+; AVX512-FASTGATHER: Cost of 20 for VF 16: {{.*}}ir<%valB> = load
+; AVX512-FASTGATHER: Cost of 40 for VF 32: {{.*}}ir<%valB> = load
+; AVX512-FASTGATHER: Cost of 80 for VF 64: {{.*}}ir<%valB> = load
;
entry:
br label %for.body
diff --git a/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-f64-stride-5.ll b/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-f64-stride-5.ll
index f0d72a118d2f2..d94d8e1cfab2c 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-f64-stride-5.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-f64-stride-5.ll
@@ -2,7 +2,7 @@
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+sse2 --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=SSE2
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX1
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx2 --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX2
-; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512vl --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX512
+; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512vl,+fast-gather --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX512
; REQUIRES: asserts
target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
diff --git a/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-f64-stride-6.ll b/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-f64-stride-6.ll
index d43a3b3746783..67b5b9f817d3c 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-f64-stride-6.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-f64-stride-6.ll
@@ -2,7 +2,7 @@
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+sse2 --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=SSE2
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX1
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx2 --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX2
-; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512vl --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX512
+; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512vl,+fast-gather --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX512
; REQUIRES: asserts
target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
diff --git a/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-f64-stride-7.ll b/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-f64-stride-7.ll
index 0d3f3b708ce08..eed344d2e0238 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-f64-stride-7.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-f64-stride-7.ll
@@ -2,7 +2,7 @@
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+sse2 --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=SSE2
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX1
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx2 --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX2
-; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512vl --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX512
+; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512vl,+fast-gather --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX512
; REQUIRES: asserts
target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
diff --git a/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-f64-stride-8.ll b/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-f64-stride-8.ll
index f739aadf7e138..6e4e3a983c569 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-f64-stride-8.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-f64-stride-8.ll
@@ -2,7 +2,7 @@
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+sse2 --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=SSE2
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX1
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx2 --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX2
-; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512vl --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX512
+; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512vl,+fast-gather --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX512
; REQUIRES: asserts
target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
diff --git a/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-i32-stride-4-indices-0uuu.ll b/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-i32-stride-4-indices-0uuu.ll
index 0bdefbcb269bf..6c11e5cd26d08 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-i32-stride-4-indices-0uuu.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-i32-stride-4-indices-0uuu.ll
@@ -2,7 +2,7 @@
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+sse2 --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=SSE2
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX1
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx2 --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX2
-; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512vl --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX512
+; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512vl,+fast-gather --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX512
; REQUIRES: asserts
target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
diff --git a/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-i64-stride-2.ll b/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-i64-stride-2.ll
index 52276bce225eb..a582425c8c245 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-i64-stride-2.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-i64-stride-2.ll
@@ -2,7 +2,7 @@
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+sse2 --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=SSE2
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX1
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx2 --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX2
-; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512vl --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX512
+; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512vl,+fast-gather --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX512
; REQUIRES: asserts
target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
diff --git a/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-i64-stride-4.ll b/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-i64-stride-4.ll
index 328d0d6f8cef8..c9b2073f133c9 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-i64-stride-4.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-i64-stride-4.ll
@@ -2,7 +2,7 @@
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+sse2 --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=SSE2
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX1
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx2 --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX2
-; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512vl --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX512
+; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512vl,+fast-gather --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX512
; REQUIRES: asserts
target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
diff --git a/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-i64-stride-5.ll b/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-i64-stride-5.ll
index e534038b2e795..67026aff11d3c 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-i64-stride-5.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-i64-stride-5.ll
@@ -2,7 +2,7 @@
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+sse2 --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=SSE2
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX1
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx2 --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX2
-; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512vl --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX512
+; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512vl,+fast-gather --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX512
; REQUIRES: asserts
target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
diff --git a/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-i64-stride-6.ll b/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-i64-stride-6.ll
index 8647841feeaab..f8b883106b5fd 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-i64-stride-6.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-i64-stride-6.ll
@@ -2,7 +2,7 @@
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+sse2 --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=SSE2
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX1
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx2 --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX2
-; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512vl --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX512
+; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512vl,+fast-gather --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX512
; REQUIRES: asserts
target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
diff --git a/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-i64-stride-7.ll b/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-i64-stride-7.ll
index 972ebc51fdeec..8b58913e99690 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-i64-stride-7.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-i64-stride-7.ll
@@ -2,7 +2,7 @@
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+sse2 --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=SSE2
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX1
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx2 --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX2
-; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512vl --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX512
+; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512vl,+fast-gather --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX512
; REQUIRES: asserts
target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
diff --git a/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-i64-stride-8.ll b/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-i64-stride-8.ll
index e166fd5296d4b..9fed42272b5da 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-i64-stride-8.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/CostModel/interleaved-load-i64-stride-8.ll
@@ -2,7 +2,7 @@
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+sse2 --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=SSE2
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX1
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx2 --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX2
-; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512vl --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX512
+; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512vl,+fast-gather --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefix=AVX512
; REQUIRES: asserts
target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
diff --git a/llvm/test/Transforms/LoopVectorize/X86/CostModel/masked-gather-i32-with-i8-index.ll b/llvm/test/Transforms/LoopVectorize/X86/CostModel/masked-gather-i32-with-i8-index.ll
index caf8f10d1169c..a5c68bc6a574e 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/CostModel/masked-gather-i32-with-i8-index.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/CostModel/masked-gather-i32-with-i8-index.ll
@@ -4,7 +4,8 @@
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefixes=AVX1
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx2,-fast-gather --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefixes=AVX2-SLOWGATHER
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx2,+fast-gather --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefixes=AVX2-FASTGATHER
-; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512bw --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefixes=AVX512
+; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512bw --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefixes=AVX512-SLOWGATHER
+; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512bw,+fast-gather --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefixes=AVX512-FASTGATHER
; REQUIRES: asserts
@@ -47,14 +48,23 @@ define void @test() {
; AVX2-FASTGATHER: Cost of 24 for VF 16: {{.*}}ir<%valB.loaded> = load
; AVX2-FASTGATHER: Cost of 48 for VF 32: {{.*}}ir<%valB.loaded> = load
;
-; AVX512-LABEL: 'test'
-; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: %valB.loaded = load i32, ptr %inB, align 4
-; AVX512: Cost of 8 for VF 2: {{.*}}ir<%valB.loaded> = load
-; AVX512: Cost of 17 for VF 4: {{.*}}ir<%valB.loaded> = load
-; AVX512: Cost of 10 for VF 8: {{.*}}ir<%valB.loaded> = load
-; AVX512: Cost of 18 for VF 16: {{.*}}ir<%valB.loaded> = load
-; AVX512: Cost of 36 for VF 32: {{.*}}ir<%valB.loaded> = load
-; AVX512: Cost of 72 for VF 64: {{.*}}ir<%valB.loaded> = load
+; AVX512-SLOWGATHER-LABEL: 'test'
+; AVX512-SLOWGATHER: LV: Found an estimated cost of 1 for VF 1 For instruction: %valB.loaded = load i32, ptr %inB, align 4
+; AVX512-SLOWGATHER: Cost of 8 for VF 2: {{.*}}ir<%valB.loaded> = load
+; AVX512-SLOWGATHER: Cost of 17 for VF 4: {{.*}}ir<%valB.loaded> = load
+; AVX512-SLOWGATHER: Cost of 1032 for VF 8: {{.*}}ir<%valB.loaded> = load
+; AVX512-SLOWGATHER: Cost of 1040 for VF 16: {{.*}}ir<%valB.loaded> = load
+; AVX512-SLOWGATHER: Cost of 2080 for VF 32: {{.*}}ir<%valB.loaded> = load
+; AVX512-SLOWGATHER: Cost of 4160 for VF 64: {{.*}}ir<%valB.loaded> = load
+;
+; AVX512-FASTGATHER-LABEL: 'test'
+; AVX512-FASTGATHER: LV: Found an estimated cost of 1 for VF 1 For instruction: %valB.loaded = load i32, ptr %inB, align 4
+; AVX512-FASTGATHER: Cost of 8 for VF 2: {{.*}}ir<%valB.loaded> = load
+; AVX512-FASTGATHER: Cost of 17 for VF 4: {{.*}}ir<%valB.loaded> = load
+; AVX512-FASTGATHER: Cost of 10 for VF 8: {{.*}}ir<%valB.loaded> = load
+; AVX512-FASTGATHER: Cost of 18 for VF 16: {{.*}}ir<%valB.loaded> = load
+; AVX512-FASTGATHER: Cost of 36 for VF 32: {{.*}}ir<%valB.loaded> = load
+; AVX512-FASTGATHER: Cost of 72 for VF 64: {{.*}}ir<%valB.loaded> = load
;
entry:
br label %for.body
diff --git a/llvm/test/Transforms/LoopVectorize/X86/CostModel/masked-gather-i64-with-i8-index.ll b/llvm/test/Transforms/LoopVectorize/X86/CostModel/masked-gather-i64-with-i8-index.ll
index c70ab9962785b..f4b5ff440a262 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/CostModel/masked-gather-i64-with-i8-index.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/CostModel/masked-gather-i64-with-i8-index.ll
@@ -4,7 +4,8 @@
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefixes=AVX1
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx2,-fast-gather --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefixes=AVX2-SLOWGATHER
; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx2,+fast-gather --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefixes=AVX2-FASTGATHER
-; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512bw --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefixes=AVX512
+; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512bw --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefixes=AVX512-SLOWGATHER
+; RUN: opt -passes=loop-vectorize -vectorizer-maximize-bandwidth -S -mattr=+avx512bw,+fast-gather --debug-only=loop-vectorize --disable-output < %s 2>&1 | FileCheck %s --check-prefixes=AVX512-FASTGATHER
; REQUIRES: asserts
@@ -47,14 +48,23 @@ define void @test() {
; AVX2-FASTGATHER: Cost of 24 for VF 16: {{.*}}ir<%valB.loaded> = load
; AVX2-FASTGATHER: Cost of 48 for VF 32: {{.*}}ir<%valB.loaded> = load
;
-; AVX512-LABEL: 'test'
-; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: %valB.loaded = load i64, ptr %inB, align 8
-; AVX512: Cost of 8 for VF 2: {{.*}}ir<%valB.loaded> = load
-; AVX512: Cost of 18 for VF 4: {{.*}}ir<%valB.loaded> = load
-; AVX512: Cost of 10 for VF 8: {{.*}}ir<%valB.loaded> = load
-; AVX512: Cost of 20 for VF 16: {{.*}}ir<%valB.loaded> = load
-; AVX512: Cost of 40 for VF 32: {{.*}}ir<%valB.loaded> = load
-; AVX512: Cost of 80 for VF 64: {{.*}}ir<%valB.loaded> = load
+; AVX512-SLOWGATHER-LABEL: 'test'
+; AVX512-SLOWGATHER: LV: Found an estimated cost of 1 for VF 1 For instruction: %valB.loaded = load i64, ptr %inB, align 8
+; AVX512-SLOWGATHER: Cost of 8 for VF 2: {{.*}}ir<%valB.loaded> = load
+; AVX512-SLOWGATHER: Cost of 18 for VF 4: {{.*}}ir<%valB.loaded> = load
+; AVX512-SLOWGATHER: Cost of 1032 for VF 8: {{.*}}ir<%valB.loaded> = load
+; AVX512-SLOWGATHER: Cost of 2064 for VF 16: {{.*}}ir<%valB.loaded> = load
+; AVX512-SLOWGATHER: Cost of 4128 for VF 32: {{.*}}ir<%valB.loaded> = load
+; AVX512-SLOWGATHER: Cost of 8256 for VF 64: {{.*}}ir<%valB.loaded> = load
+;
+; AVX512-FASTGATHER-LABEL: 'test'
+; AVX512-FASTGATHER: LV: Found an estimated cost of 1 for VF 1 For instruction: %valB.loaded = load i64, ptr %inB, align 8
+; AVX512-FASTGATHER: Cost of 8 for VF 2: {{.*}}ir<%valB.loaded> = load
+; AVX512-FASTGATHER: Cost of 18 for VF 4: {{.*}}ir<%valB.loaded> = load
+; AVX512-FASTGATHER: Cost of 10 for VF 8: {{.*}}ir<%valB.loaded> = load
+; AVX512-FASTGATHER: Cost of 20 for VF 16: {{.*}}ir<%valB.loaded> = load
+; AVX512-FASTGATHER: Cost of 40 for VF 32: {{.*}}ir<%valB.loaded> = load
+; AVX512-FASTGATHER: Cost of 80 for VF 64: {{.*}}ir<%valB.loaded> = load
;
entry:
br label %for.body
diff --git a/llvm/test/Transforms/LoopVectorize/X86/cost-conditional-branches.ll b/llvm/test/Transforms/LoopVectorize/X86/cost-conditional-branches.ll
index 314118a308034..9f28858ab60a4 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/cost-conditional-branches.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/cost-conditional-branches.ll
@@ -1280,5 +1280,5 @@ exit:
}
attributes #0 = { "target-cpu"="znver4" }
-attributes #1 = { "target-features"="+avx512bw,+avx512cd,+avx512dq,+avx512f,+avx512vl" }
+attributes #1 = { "target-features"="+avx512bw,+avx512cd,+avx512dq,+avx512f,+avx512vl,+fast-gather" }
attributes #2 = { "target-cpu"="znver3" }
diff --git a/llvm/test/Transforms/LoopVectorize/X86/gather-vs-interleave.ll b/llvm/test/Transforms/LoopVectorize/X86/gather-vs-interleave.ll
index 28af1465c54e2..5ad9bbb1fa452 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/gather-vs-interleave.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/gather-vs-interleave.ll
@@ -1,4 +1,4 @@
-; RUN: opt -passes=loop-vectorize -S -mattr=avx512f < %s | FileCheck %s
+; RUN: opt -passes=loop-vectorize -S -mattr=+avx512f,+fast-gather < %s | FileCheck %s
target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-unknown-linux-gnu"
diff --git a/llvm/test/Transforms/LoopVectorize/X86/invariant-load-gather.ll b/llvm/test/Transforms/LoopVectorize/X86/invariant-load-gather.ll
index fd9e4bc8d95f1..aa052d957e1e5 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/invariant-load-gather.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/invariant-load-gather.ll
@@ -1,5 +1,5 @@
; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -passes=loop-vectorize -mattr=avx512f -S %s | FileCheck %s
+; RUN: opt -passes=loop-vectorize -mattr=+avx512f,+fast-gather -S %s | FileCheck %s
target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-unknown-linux-gnu"
More information about the llvm-commits
mailing list