[llvm] [RISCV] Don't overcost wide load in optimized segment load/store (PR #207146)
Luke Lau via llvm-commits
llvm-commits at lists.llvm.org
Mon Jul 13 03:07:02 PDT 2026
https://github.com/lukel97 updated https://github.com/llvm/llvm-project/pull/207146
>From 47e6b17a1e4261d2ecfa44fa566a73db9dec0cc9 Mon Sep 17 00:00:00 2001
From: Luke Lau <luke at igalia.com>
Date: Thu, 2 Jul 2026 17:07:42 +0800
Subject: [PATCH 1/2] [RISCV] Don't overcost wide load in optimized segment
load store
With the +optimized-nfX-segment-load-store tuning flag, we cost a segmented store as a single wide load + some shuffle ops.
However for e.g. a `<vscale x 5 x i64>` Factor=5 segmented load, a wide `<vscale x 5 x i64>` load gets costed as a full LMUL 8 load.
>From what I can see on https://camel-cdr.github.io/rvv-bench-results/spacemit_x100/index.html and on my own measurements on the spacemit-x60, uarchs likely don't do a full LMUL 8 load under the hood and instead dispatch the minimum number of DLEN sized ops needed for the full segment.
This changes the wide load cost to be divideCeil(vector size, DLEN) ops so we don't overcost it.
Whilst we're here, this also removes the LT.first legalization multiplier. We're computing the cost in terms of the unlegalized type so we shouldn't be scaling it by the legalization cost.
---
.../Target/RISCV/RISCVTargetTransformInfo.cpp | 12 ++--
.../LoopVectorize/RISCV/interleaved-cost.ll | 56 +++++++++----------
2 files changed, 35 insertions(+), 33 deletions(-)
diff --git a/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp b/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
index 07a962f7dd03d..bf6b37c187965 100644
--- a/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
+++ b/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
@@ -1124,14 +1124,16 @@ InstructionCost RISCVTTIImpl::getInterleavedMemoryOpCost(
TLI->isLegalInterleavedAccessType(SubVecTy, Factor, Alignment,
AddressSpace, DL)) {
- // Some processors optimize segment loads/stores as one wide memory op +
- // Factor * LMUL shuffle ops.
+ // Some processors optimize segment loads/stores as N * DLEN sized
+ // load ops + Factor * LMUL shuffle ops.
if (ST->hasOptimizedSegmentLoadStore(Factor)) {
- InstructionCost Cost =
- getMemoryOpCost(Opcode, VTy, Alignment, AddressSpace, CostKind);
+ unsigned VecSizeInBits =
+ getEstimatedVLFor(VTy) * VTy->getScalarSizeInBits();
+ unsigned DLEN = ST->getRealMinVLen() / ST->getDLenFactor();
+ InstructionCost Cost = divideCeil(VecSizeInBits, DLEN);
MVT SubVecVT = getTLI()->getValueType(DL, SubVecTy).getSimpleVT();
Cost += Factor * TLI->getLMULCost(SubVecVT);
- return LT.first * Cost;
+ return Cost;
}
// Otherwise, the cost is proportional to the number of elements (VL *
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/interleaved-cost.ll b/llvm/test/Transforms/LoopVectorize/RISCV/interleaved-cost.ll
index 62c73635e419e..1d452536c8d0b 100644
--- a/llvm/test/Transforms/LoopVectorize/RISCV/interleaved-cost.ll
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/interleaved-cost.ll
@@ -96,10 +96,10 @@ define void @i8_factor_3(ptr %data, i64 %n) {
; OPT: Cost of 4 for VF vscale x 2: INTERLEAVE-GROUP with factor 3, ir<%p0>, vp<%evl>
; OPT: Cost of 5 for VF vscale x 4: INTERLEAVE-GROUP with factor 3, ir<%p0>, vp<%evl>
; OPT: Cost of 5 for VF vscale x 4: INTERLEAVE-GROUP with factor 3, ir<%p0>, vp<%evl>
-; OPT: Cost of 7 for VF vscale x 8: INTERLEAVE-GROUP with factor 3, ir<%p0>, vp<%evl>
-; OPT: Cost of 7 for VF vscale x 8: INTERLEAVE-GROUP with factor 3, ir<%p0>, vp<%evl>
-; OPT: Cost of 14 for VF vscale x 16: INTERLEAVE-GROUP with factor 3, ir<%p0>, vp<%evl>
-; OPT: Cost of 14 for VF vscale x 16: INTERLEAVE-GROUP with factor 3, ir<%p0>, vp<%evl>
+; OPT: Cost of 6 for VF vscale x 8: INTERLEAVE-GROUP with factor 3, ir<%p0>, vp<%evl>
+; OPT: Cost of 6 for VF vscale x 8: INTERLEAVE-GROUP with factor 3, ir<%p0>, vp<%evl>
+; OPT: Cost of 12 for VF vscale x 16: INTERLEAVE-GROUP with factor 3, ir<%p0>, vp<%evl>
+; OPT: Cost of 12 for VF vscale x 16: INTERLEAVE-GROUP with factor 3, ir<%p0>, vp<%evl>
;
; FIXED-NO-OPT-LABEL: 'i8_factor_3'
; FIXED-NO-OPT: Cost of 6 for VF 2: INTERLEAVE-GROUP with factor 3, ir<%p0>
@@ -120,10 +120,10 @@ define void @i8_factor_3(ptr %data, i64 %n) {
; FIXED-OPT: Cost of 4 for VF 4: INTERLEAVE-GROUP with factor 3, ir<%p0>
; FIXED-OPT: Cost of 5 for VF 8: INTERLEAVE-GROUP with factor 3, ir<%p0>
; FIXED-OPT: Cost of 5 for VF 8: INTERLEAVE-GROUP with factor 3, ir<%p0>
-; FIXED-OPT: Cost of 7 for VF 16: INTERLEAVE-GROUP with factor 3, ir<%p0>
-; FIXED-OPT: Cost of 7 for VF 16: INTERLEAVE-GROUP with factor 3, ir<%p0>
-; FIXED-OPT: Cost of 14 for VF 32: INTERLEAVE-GROUP with factor 3, ir<%p0>
-; FIXED-OPT: Cost of 14 for VF 32: INTERLEAVE-GROUP with factor 3, ir<%p0>
+; FIXED-OPT: Cost of 6 for VF 16: INTERLEAVE-GROUP with factor 3, ir<%p0>
+; FIXED-OPT: Cost of 6 for VF 16: INTERLEAVE-GROUP with factor 3, ir<%p0>
+; FIXED-OPT: Cost of 12 for VF 32: INTERLEAVE-GROUP with factor 3, ir<%p0>
+; FIXED-OPT: Cost of 12 for VF 32: INTERLEAVE-GROUP with factor 3, ir<%p0>
;
entry:
br label %for.body
@@ -244,10 +244,10 @@ define void @i8_factor_5(ptr %data, i64 %n) {
; OPT: Cost of 6 for VF vscale x 1: INTERLEAVE-GROUP with factor 5, ir<%p0>, vp<%evl>
; OPT: Cost of 7 for VF vscale x 2: INTERLEAVE-GROUP with factor 5, ir<%p0>, vp<%evl>
; OPT: Cost of 7 for VF vscale x 2: INTERLEAVE-GROUP with factor 5, ir<%p0>, vp<%evl>
-; OPT: Cost of 9 for VF vscale x 4: INTERLEAVE-GROUP with factor 5, ir<%p0>, vp<%evl>
-; OPT: Cost of 9 for VF vscale x 4: INTERLEAVE-GROUP with factor 5, ir<%p0>, vp<%evl>
-; OPT: Cost of 13 for VF vscale x 8: INTERLEAVE-GROUP with factor 5, ir<%p0>, vp<%evl>
-; OPT: Cost of 13 for VF vscale x 8: INTERLEAVE-GROUP with factor 5, ir<%p0>, vp<%evl>
+; OPT: Cost of 8 for VF vscale x 4: INTERLEAVE-GROUP with factor 5, ir<%p0>, vp<%evl>
+; OPT: Cost of 8 for VF vscale x 4: INTERLEAVE-GROUP with factor 5, ir<%p0>, vp<%evl>
+; OPT: Cost of 10 for VF vscale x 8: INTERLEAVE-GROUP with factor 5, ir<%p0>, vp<%evl>
+; OPT: Cost of 10 for VF vscale x 8: INTERLEAVE-GROUP with factor 5, ir<%p0>, vp<%evl>
;
; FIXED-NO-OPT-LABEL: 'i8_factor_5'
; FIXED-NO-OPT: Cost of 10 for VF 2: INTERLEAVE-GROUP with factor 5, ir<%p0>
@@ -264,10 +264,10 @@ define void @i8_factor_5(ptr %data, i64 %n) {
; FIXED-OPT: Cost of 6 for VF 2: INTERLEAVE-GROUP with factor 5, ir<%p0>
; FIXED-OPT: Cost of 7 for VF 4: INTERLEAVE-GROUP with factor 5, ir<%p0>
; FIXED-OPT: Cost of 7 for VF 4: INTERLEAVE-GROUP with factor 5, ir<%p0>
-; FIXED-OPT: Cost of 9 for VF 8: INTERLEAVE-GROUP with factor 5, ir<%p0>
-; FIXED-OPT: Cost of 9 for VF 8: INTERLEAVE-GROUP with factor 5, ir<%p0>
-; FIXED-OPT: Cost of 13 for VF 16: INTERLEAVE-GROUP with factor 5, ir<%p0>
-; FIXED-OPT: Cost of 13 for VF 16: INTERLEAVE-GROUP with factor 5, ir<%p0>
+; FIXED-OPT: Cost of 8 for VF 8: INTERLEAVE-GROUP with factor 5, ir<%p0>
+; FIXED-OPT: Cost of 8 for VF 8: INTERLEAVE-GROUP with factor 5, ir<%p0>
+; FIXED-OPT: Cost of 10 for VF 16: INTERLEAVE-GROUP with factor 5, ir<%p0>
+; FIXED-OPT: Cost of 10 for VF 16: INTERLEAVE-GROUP with factor 5, ir<%p0>
;
entry:
br label %for.body
@@ -318,10 +318,10 @@ define void @i8_factor_6(ptr %data, i64 %n) {
; OPT: Cost of 7 for VF vscale x 1: INTERLEAVE-GROUP with factor 6, ir<%p0>, vp<%evl>
; OPT: Cost of 8 for VF vscale x 2: INTERLEAVE-GROUP with factor 6, ir<%p0>, vp<%evl>
; OPT: Cost of 8 for VF vscale x 2: INTERLEAVE-GROUP with factor 6, ir<%p0>, vp<%evl>
-; OPT: Cost of 10 for VF vscale x 4: INTERLEAVE-GROUP with factor 6, ir<%p0>, vp<%evl>
-; OPT: Cost of 10 for VF vscale x 4: INTERLEAVE-GROUP with factor 6, ir<%p0>, vp<%evl>
-; OPT: Cost of 14 for VF vscale x 8: INTERLEAVE-GROUP with factor 6, ir<%p0>, vp<%evl>
-; OPT: Cost of 14 for VF vscale x 8: INTERLEAVE-GROUP with factor 6, ir<%p0>, vp<%evl>
+; OPT: Cost of 9 for VF vscale x 4: INTERLEAVE-GROUP with factor 6, ir<%p0>, vp<%evl>
+; OPT: Cost of 9 for VF vscale x 4: INTERLEAVE-GROUP with factor 6, ir<%p0>, vp<%evl>
+; OPT: Cost of 12 for VF vscale x 8: INTERLEAVE-GROUP with factor 6, ir<%p0>, vp<%evl>
+; OPT: Cost of 12 for VF vscale x 8: INTERLEAVE-GROUP with factor 6, ir<%p0>, vp<%evl>
;
; FIXED-NO-OPT-LABEL: 'i8_factor_6'
; FIXED-NO-OPT: Cost of 12 for VF 2: INTERLEAVE-GROUP with factor 6, ir<%p0>
@@ -338,10 +338,10 @@ define void @i8_factor_6(ptr %data, i64 %n) {
; FIXED-OPT: Cost of 7 for VF 2: INTERLEAVE-GROUP with factor 6, ir<%p0>
; FIXED-OPT: Cost of 8 for VF 4: INTERLEAVE-GROUP with factor 6, ir<%p0>
; FIXED-OPT: Cost of 8 for VF 4: INTERLEAVE-GROUP with factor 6, ir<%p0>
-; FIXED-OPT: Cost of 10 for VF 8: INTERLEAVE-GROUP with factor 6, ir<%p0>
-; FIXED-OPT: Cost of 10 for VF 8: INTERLEAVE-GROUP with factor 6, ir<%p0>
-; FIXED-OPT: Cost of 14 for VF 16: INTERLEAVE-GROUP with factor 6, ir<%p0>
-; FIXED-OPT: Cost of 14 for VF 16: INTERLEAVE-GROUP with factor 6, ir<%p0>
+; FIXED-OPT: Cost of 9 for VF 8: INTERLEAVE-GROUP with factor 6, ir<%p0>
+; FIXED-OPT: Cost of 9 for VF 8: INTERLEAVE-GROUP with factor 6, ir<%p0>
+; FIXED-OPT: Cost of 12 for VF 16: INTERLEAVE-GROUP with factor 6, ir<%p0>
+; FIXED-OPT: Cost of 12 for VF 16: INTERLEAVE-GROUP with factor 6, ir<%p0>
;
entry:
br label %for.body
@@ -398,8 +398,8 @@ define void @i8_factor_7(ptr %data, i64 %n) {
; OPT: Cost of 9 for VF vscale x 2: INTERLEAVE-GROUP with factor 7, ir<%p0>, vp<%evl>
; OPT: Cost of 11 for VF vscale x 4: INTERLEAVE-GROUP with factor 7, ir<%p0>, vp<%evl>
; OPT: Cost of 11 for VF vscale x 4: INTERLEAVE-GROUP with factor 7, ir<%p0>, vp<%evl>
-; OPT: Cost of 15 for VF vscale x 8: INTERLEAVE-GROUP with factor 7, ir<%p0>, vp<%evl>
-; OPT: Cost of 15 for VF vscale x 8: INTERLEAVE-GROUP with factor 7, ir<%p0>, vp<%evl>
+; OPT: Cost of 14 for VF vscale x 8: INTERLEAVE-GROUP with factor 7, ir<%p0>, vp<%evl>
+; OPT: Cost of 14 for VF vscale x 8: INTERLEAVE-GROUP with factor 7, ir<%p0>, vp<%evl>
;
; FIXED-NO-OPT-LABEL: 'i8_factor_7'
; FIXED-NO-OPT: Cost of 14 for VF 2: INTERLEAVE-GROUP with factor 7, ir<%p0>
@@ -418,8 +418,8 @@ define void @i8_factor_7(ptr %data, i64 %n) {
; FIXED-OPT: Cost of 9 for VF 4: INTERLEAVE-GROUP with factor 7, ir<%p0>
; FIXED-OPT: Cost of 11 for VF 8: INTERLEAVE-GROUP with factor 7, ir<%p0>
; FIXED-OPT: Cost of 11 for VF 8: INTERLEAVE-GROUP with factor 7, ir<%p0>
-; FIXED-OPT: Cost of 15 for VF 16: INTERLEAVE-GROUP with factor 7, ir<%p0>
-; FIXED-OPT: Cost of 15 for VF 16: INTERLEAVE-GROUP with factor 7, ir<%p0>
+; FIXED-OPT: Cost of 14 for VF 16: INTERLEAVE-GROUP with factor 7, ir<%p0>
+; FIXED-OPT: Cost of 14 for VF 16: INTERLEAVE-GROUP with factor 7, ir<%p0>
;
entry:
br label %for.body
>From 7f422e1a65025413b84d6d328c01826bcc816a58 Mon Sep 17 00:00:00 2001
From: Luke Lau <luke at igalia.com>
Date: Mon, 13 Jul 2026 18:06:07 +0800
Subject: [PATCH 2/2] Rewrite DLEN in terms of getVScaleForTuning
---
llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp b/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
index 7f24c2fe5631c..93d001689ef87 100644
--- a/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
+++ b/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
@@ -1132,8 +1132,10 @@ InstructionCost RISCVTTIImpl::getInterleavedMemoryOpCost(
if (ST->hasOptimizedSegmentLoadStore(Factor)) {
unsigned VecSizeInBits =
getEstimatedVLFor(VTy) * VTy->getScalarSizeInBits();
- unsigned DLEN = ST->getRealMinVLen() / ST->getDLenFactor();
- InstructionCost Cost = divideCeil(VecSizeInBits, DLEN);
+ unsigned VLENForTuning =
+ *getVScaleForTuning() * RISCV::RVVBitsPerBlock;
+ unsigned DLENForTuning = VLENForTuning / ST->getDLenFactor();
+ InstructionCost Cost = divideCeil(VecSizeInBits, DLENForTuning);
MVT SubVecVT = getTLI()->getValueType(DL, SubVecTy).getSimpleVT();
Cost += Factor * TLI->getLMULCost(SubVecVT);
return Cost;
More information about the llvm-commits
mailing list