[llvm] 94bb1b0 - [AArch64][TTI] Increase cost for masked memory operations requiring splitting (#191417)
via llvm-commits
llvm-commits at lists.llvm.org
Tue Apr 14 07:47:16 PDT 2026
Author: Graham Hunter
Date: 2026-04-14T14:47:10Z
New Revision: 94bb1b0590f43b517f290053b74e72e38101aa9b
URL: https://github.com/llvm/llvm-project/commit/94bb1b0590f43b517f290053b74e72e38101aa9b
DIFF: https://github.com/llvm/llvm-project/commit/94bb1b0590f43b517f290053b74e72e38101aa9b.diff
LOG: [AArch64][TTI] Increase cost for masked memory operations requiring splitting (#191417)
If we need to split the memory operation, we'll also need to split the
mask.
This has a performance benefit in some cases when the loop vectorizer is
asked to maximize bandwidth and ends up choosing a VF that's too high
when tail folding. The costs of splitting the masks are not accounted
for in the current model, so this is something of a brute-force approach
to avoiding the wider VFs.
Added:
Modified:
llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp
llvm/test/Analysis/CostModel/AArch64/masked_ldst.ll
llvm/test/Analysis/CostModel/AArch64/masked_ldst_vls.ll
Removed:
################################################################################
diff --git a/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp b/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp
index c97121a83fa08..ff99eeefde6e3 100644
--- a/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp
+++ b/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp
@@ -4904,6 +4904,17 @@ AArch64TTIImpl::getMaskedMemoryOpCost(const MemIntrinsicCostAttributes &MICA,
if (VT->getElementCount() == ElementCount::getScalable(1))
return InstructionCost::getInvalid();
+ // If we need to split the memory operation, we will also need to split the
+ // mask. This will likely lead to overestimating the cost in some cases if
+ // multiple memory operations use the same mask, but we often don't have
+ // enough context to figure that out here.
+ //
+ // If the elements being loaded are bytes then the mask will already be split,
+ // since the number of bits in a P register matches the number of bytes in a
+ // Z register.
+ if (LT.first > 1 && LT.second.getScalarSizeInBits() > 8)
+ return LT.first * 2;
+
return LT.first;
}
diff --git a/llvm/test/Analysis/CostModel/AArch64/masked_ldst.ll b/llvm/test/Analysis/CostModel/AArch64/masked_ldst.ll
index 780db0920bbbd..25426227e4e9e 100644
--- a/llvm/test/Analysis/CostModel/AArch64/masked_ldst.ll
+++ b/llvm/test/Analysis/CostModel/AArch64/masked_ldst.ll
@@ -73,8 +73,8 @@ define void @scalable() {
; CHECK-NEXT: Cost Model: Found costs of 1 for: %nxv4f32 = call <vscale x 4 x float> @llvm.masked.load.nxv4f32.p0(ptr align 8 undef, <vscale x 4 x i1> undef, <vscale x 4 x float> undef)
; CHECK-NEXT: Cost Model: Found costs of 1 for: %nxv2f64 = call <vscale x 2 x double> @llvm.masked.load.nxv2f64.p0(ptr align 8 undef, <vscale x 2 x i1> undef, <vscale x 2 x double> undef)
; CHECK-NEXT: Cost Model: Found costs of Invalid for: %nxv1i64 = call <vscale x 1 x i64> @llvm.masked.load.nxv1i64.p0(ptr align 8 undef, <vscale x 1 x i1> undef, <vscale x 1 x i64> undef)
-; CHECK-NEXT: Cost Model: Found costs of 2 for: %nxv4i64 = call <vscale x 4 x i64> @llvm.masked.load.nxv4i64.p0(ptr align 8 undef, <vscale x 4 x i1> undef, <vscale x 4 x i64> undef)
-; CHECK-NEXT: Cost Model: Found costs of 4 for: %nxv32f16 = call <vscale x 32 x half> @llvm.masked.load.nxv32f16.p0(ptr align 8 undef, <vscale x 32 x i1> undef, <vscale x 32 x half> undef)
+; CHECK-NEXT: Cost Model: Found costs of 4 for: %nxv4i64 = call <vscale x 4 x i64> @llvm.masked.load.nxv4i64.p0(ptr align 8 undef, <vscale x 4 x i1> undef, <vscale x 4 x i64> undef)
+; CHECK-NEXT: Cost Model: Found costs of 8 for: %nxv32f16 = call <vscale x 32 x half> @llvm.masked.load.nxv32f16.p0(ptr align 8 undef, <vscale x 32 x i1> undef, <vscale x 32 x half> undef)
; CHECK-NEXT: Cost Model: Found costs of Invalid for: %nxv4i1 = call <vscale x 4 x i1> @llvm.masked.load.nxv4i1.p0(ptr align 16 undef, <vscale x 4 x i1> undef, <vscale x 4 x i1> undef)
; CHECK-NEXT: Cost Model: Found costs of RThru:0 CodeSize:1 Lat:1 SizeLat:1 for: ret void
;
diff --git a/llvm/test/Analysis/CostModel/AArch64/masked_ldst_vls.ll b/llvm/test/Analysis/CostModel/AArch64/masked_ldst_vls.ll
index 6f3977ea4948f..74d6ce3b3ff6d 100644
--- a/llvm/test/Analysis/CostModel/AArch64/masked_ldst_vls.ll
+++ b/llvm/test/Analysis/CostModel/AArch64/masked_ldst_vls.ll
@@ -10,42 +10,42 @@ target triple = "aarch64-unknown-linux-gnu"
define void @fixed_sve_vls() #0 {
; VL256-LABEL: 'fixed_sve_vls'
; VL256-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v256i8 = call <256 x i8> @llvm.masked.load.v256i8.p0(ptr poison, <256 x i1> poison, <256 x i8> poison)
-; VL256-NEXT: Cost Model: Found an estimated cost of 16 for instruction: %v256i16 = call <256 x i16> @llvm.masked.load.v256i16.p0(ptr poison, <256 x i1> poison, <256 x i16> poison)
-; VL256-NEXT: Cost Model: Found an estimated cost of 2 for instruction: %v16i32 = call <16 x i32> @llvm.masked.load.v16i32.p0(ptr poison, <16 x i1> poison, <16 x i32> poison)
-; VL256-NEXT: Cost Model: Found an estimated cost of 4 for instruction: %v16i64 = call <16 x i64> @llvm.masked.load.v16i64.p0(ptr poison, <16 x i1> poison, <16 x i64> poison)
-; VL256-NEXT: Cost Model: Found an estimated cost of 32 for instruction: %v512f16 = call <512 x half> @llvm.masked.load.v512f16.p0(ptr poison, <512 x i1> poison, <512 x half> poison)
-; VL256-NEXT: Cost Model: Found an estimated cost of 32 for instruction: %v256f32 = call <256 x float> @llvm.masked.load.v256f32.p0(ptr poison, <256 x i1> poison, <256 x float> poison)
-; VL256-NEXT: Cost Model: Found an estimated cost of 32 for instruction: %v128f64 = call <128 x double> @llvm.masked.load.v128f64.p0(ptr poison, <128 x i1> poison, <128 x double> poison)
+; VL256-NEXT: Cost Model: Found an estimated cost of 32 for instruction: %v256i16 = call <256 x i16> @llvm.masked.load.v256i16.p0(ptr poison, <256 x i1> poison, <256 x i16> poison)
+; VL256-NEXT: Cost Model: Found an estimated cost of 4 for instruction: %v16i32 = call <16 x i32> @llvm.masked.load.v16i32.p0(ptr poison, <16 x i1> poison, <16 x i32> poison)
+; VL256-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v16i64 = call <16 x i64> @llvm.masked.load.v16i64.p0(ptr poison, <16 x i1> poison, <16 x i64> poison)
+; VL256-NEXT: Cost Model: Found an estimated cost of 64 for instruction: %v512f16 = call <512 x half> @llvm.masked.load.v512f16.p0(ptr poison, <512 x i1> poison, <512 x half> poison)
+; VL256-NEXT: Cost Model: Found an estimated cost of 64 for instruction: %v256f32 = call <256 x float> @llvm.masked.load.v256f32.p0(ptr poison, <256 x i1> poison, <256 x float> poison)
+; VL256-NEXT: Cost Model: Found an estimated cost of 64 for instruction: %v128f64 = call <128 x double> @llvm.masked.load.v128f64.p0(ptr poison, <128 x i1> poison, <128 x double> poison)
; VL256-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; VL512-LABEL: 'fixed_sve_vls'
; VL512-NEXT: Cost Model: Found an estimated cost of 4 for instruction: %v256i8 = call <256 x i8> @llvm.masked.load.v256i8.p0(ptr poison, <256 x i1> poison, <256 x i8> poison)
-; VL512-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v256i16 = call <256 x i16> @llvm.masked.load.v256i16.p0(ptr poison, <256 x i1> poison, <256 x i16> poison)
+; VL512-NEXT: Cost Model: Found an estimated cost of 16 for instruction: %v256i16 = call <256 x i16> @llvm.masked.load.v256i16.p0(ptr poison, <256 x i1> poison, <256 x i16> poison)
; VL512-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %v16i32 = call <16 x i32> @llvm.masked.load.v16i32.p0(ptr poison, <16 x i1> poison, <16 x i32> poison)
-; VL512-NEXT: Cost Model: Found an estimated cost of 2 for instruction: %v16i64 = call <16 x i64> @llvm.masked.load.v16i64.p0(ptr poison, <16 x i1> poison, <16 x i64> poison)
-; VL512-NEXT: Cost Model: Found an estimated cost of 16 for instruction: %v512f16 = call <512 x half> @llvm.masked.load.v512f16.p0(ptr poison, <512 x i1> poison, <512 x half> poison)
-; VL512-NEXT: Cost Model: Found an estimated cost of 16 for instruction: %v256f32 = call <256 x float> @llvm.masked.load.v256f32.p0(ptr poison, <256 x i1> poison, <256 x float> poison)
-; VL512-NEXT: Cost Model: Found an estimated cost of 16 for instruction: %v128f64 = call <128 x double> @llvm.masked.load.v128f64.p0(ptr poison, <128 x i1> poison, <128 x double> poison)
+; VL512-NEXT: Cost Model: Found an estimated cost of 4 for instruction: %v16i64 = call <16 x i64> @llvm.masked.load.v16i64.p0(ptr poison, <16 x i1> poison, <16 x i64> poison)
+; VL512-NEXT: Cost Model: Found an estimated cost of 32 for instruction: %v512f16 = call <512 x half> @llvm.masked.load.v512f16.p0(ptr poison, <512 x i1> poison, <512 x half> poison)
+; VL512-NEXT: Cost Model: Found an estimated cost of 32 for instruction: %v256f32 = call <256 x float> @llvm.masked.load.v256f32.p0(ptr poison, <256 x i1> poison, <256 x float> poison)
+; VL512-NEXT: Cost Model: Found an estimated cost of 32 for instruction: %v128f64 = call <128 x double> @llvm.masked.load.v128f64.p0(ptr poison, <128 x i1> poison, <128 x double> poison)
; VL512-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; VL1024-LABEL: 'fixed_sve_vls'
; VL1024-NEXT: Cost Model: Found an estimated cost of 2 for instruction: %v256i8 = call <256 x i8> @llvm.masked.load.v256i8.p0(ptr poison, <256 x i1> poison, <256 x i8> poison)
-; VL1024-NEXT: Cost Model: Found an estimated cost of 4 for instruction: %v256i16 = call <256 x i16> @llvm.masked.load.v256i16.p0(ptr poison, <256 x i1> poison, <256 x i16> poison)
+; VL1024-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v256i16 = call <256 x i16> @llvm.masked.load.v256i16.p0(ptr poison, <256 x i1> poison, <256 x i16> poison)
; VL1024-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %v16i32 = call <16 x i32> @llvm.masked.load.v16i32.p0(ptr poison, <16 x i1> poison, <16 x i32> poison)
; VL1024-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %v16i64 = call <16 x i64> @llvm.masked.load.v16i64.p0(ptr poison, <16 x i1> poison, <16 x i64> poison)
-; VL1024-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v512f16 = call <512 x half> @llvm.masked.load.v512f16.p0(ptr poison, <512 x i1> poison, <512 x half> poison)
-; VL1024-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v256f32 = call <256 x float> @llvm.masked.load.v256f32.p0(ptr poison, <256 x i1> poison, <256 x float> poison)
-; VL1024-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v128f64 = call <128 x double> @llvm.masked.load.v128f64.p0(ptr poison, <128 x i1> poison, <128 x double> poison)
+; VL1024-NEXT: Cost Model: Found an estimated cost of 16 for instruction: %v512f16 = call <512 x half> @llvm.masked.load.v512f16.p0(ptr poison, <512 x i1> poison, <512 x half> poison)
+; VL1024-NEXT: Cost Model: Found an estimated cost of 16 for instruction: %v256f32 = call <256 x float> @llvm.masked.load.v256f32.p0(ptr poison, <256 x i1> poison, <256 x float> poison)
+; VL1024-NEXT: Cost Model: Found an estimated cost of 16 for instruction: %v128f64 = call <128 x double> @llvm.masked.load.v128f64.p0(ptr poison, <128 x i1> poison, <128 x double> poison)
; VL1024-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; VL2048-LABEL: 'fixed_sve_vls'
; VL2048-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %v256i8 = call <256 x i8> @llvm.masked.load.v256i8.p0(ptr poison, <256 x i1> poison, <256 x i8> poison)
-; VL2048-NEXT: Cost Model: Found an estimated cost of 2 for instruction: %v256i16 = call <256 x i16> @llvm.masked.load.v256i16.p0(ptr poison, <256 x i1> poison, <256 x i16> poison)
+; VL2048-NEXT: Cost Model: Found an estimated cost of 4 for instruction: %v256i16 = call <256 x i16> @llvm.masked.load.v256i16.p0(ptr poison, <256 x i1> poison, <256 x i16> poison)
; VL2048-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %v16i32 = call <16 x i32> @llvm.masked.load.v16i32.p0(ptr poison, <16 x i1> poison, <16 x i32> poison)
; VL2048-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %v16i64 = call <16 x i64> @llvm.masked.load.v16i64.p0(ptr poison, <16 x i1> poison, <16 x i64> poison)
-; VL2048-NEXT: Cost Model: Found an estimated cost of 4 for instruction: %v512f16 = call <512 x half> @llvm.masked.load.v512f16.p0(ptr poison, <512 x i1> poison, <512 x half> poison)
-; VL2048-NEXT: Cost Model: Found an estimated cost of 4 for instruction: %v256f32 = call <256 x float> @llvm.masked.load.v256f32.p0(ptr poison, <256 x i1> poison, <256 x float> poison)
-; VL2048-NEXT: Cost Model: Found an estimated cost of 4 for instruction: %v128f64 = call <128 x double> @llvm.masked.load.v128f64.p0(ptr poison, <128 x i1> poison, <128 x double> poison)
+; VL2048-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v512f16 = call <512 x half> @llvm.masked.load.v512f16.p0(ptr poison, <512 x i1> poison, <512 x half> poison)
+; VL2048-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v256f32 = call <256 x float> @llvm.masked.load.v256f32.p0(ptr poison, <256 x i1> poison, <256 x float> poison)
+; VL2048-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %v128f64 = call <128 x double> @llvm.masked.load.v128f64.p0(ptr poison, <128 x i1> poison, <128 x double> poison)
; VL2048-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
entry:
More information about the llvm-commits
mailing list