[llvm] [RISCV] Don't overcost wide load in optimized segment load/store (PR #207146)
via llvm-commits
llvm-commits at lists.llvm.org
Thu Jul 2 02:26:29 PDT 2026
llvmorg-github-actions[bot] wrote:
<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-backend-risc-v
Author: Luke Lau (lukel97)
<details>
<summary>Changes</summary>
With the +optimized-nfX-segment-load-store tuning flag, we cost a segmented store as a single wide load + some shuffle ops.
However for e.g. a `<vscale x 5 x i64>` Factor=5 segmented load, a wide `<vscale x 5 x i64>` load gets costed as a full LMUL 8 load.
>From what I can see on https://camel-cdr.github.io/rvv-bench-results/spacemit_x100/index.html and on my own measurements on the spacemit-x60, uarchs likely don't do a full LMUL 8 load under the hood and instead dispatch the minimum number of DLEN sized ops needed for the full segment.
This changes the wide load cost to be divideCeil(vector size, DLEN) ops so we don't overcost it.
Whilst we're here, this also removes the LT.first legalization multiplier. We're computing the cost in terms of the unlegalized type so we shouldn't be scaling it by the legalization cost.
---
Full diff: https://github.com/llvm/llvm-project/pull/207146.diff
2 Files Affected:
- (modified) llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp (+7-5)
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/interleaved-cost.ll (+28-28)
``````````diff
diff --git a/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp b/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
index 07a962f7dd03d..bf6b37c187965 100644
--- a/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
+++ b/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
@@ -1124,14 +1124,16 @@ InstructionCost RISCVTTIImpl::getInterleavedMemoryOpCost(
TLI->isLegalInterleavedAccessType(SubVecTy, Factor, Alignment,
AddressSpace, DL)) {
- // Some processors optimize segment loads/stores as one wide memory op +
- // Factor * LMUL shuffle ops.
+ // Some processors optimize segment loads/stores as N * DLEN sized
+ // load ops + Factor * LMUL shuffle ops.
if (ST->hasOptimizedSegmentLoadStore(Factor)) {
- InstructionCost Cost =
- getMemoryOpCost(Opcode, VTy, Alignment, AddressSpace, CostKind);
+ unsigned VecSizeInBits =
+ getEstimatedVLFor(VTy) * VTy->getScalarSizeInBits();
+ unsigned DLEN = ST->getRealMinVLen() / ST->getDLenFactor();
+ InstructionCost Cost = divideCeil(VecSizeInBits, DLEN);
MVT SubVecVT = getTLI()->getValueType(DL, SubVecTy).getSimpleVT();
Cost += Factor * TLI->getLMULCost(SubVecVT);
- return LT.first * Cost;
+ return Cost;
}
// Otherwise, the cost is proportional to the number of elements (VL *
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/interleaved-cost.ll b/llvm/test/Transforms/LoopVectorize/RISCV/interleaved-cost.ll
index 62c73635e419e..1d452536c8d0b 100644
--- a/llvm/test/Transforms/LoopVectorize/RISCV/interleaved-cost.ll
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/interleaved-cost.ll
@@ -96,10 +96,10 @@ define void @i8_factor_3(ptr %data, i64 %n) {
; OPT: Cost of 4 for VF vscale x 2: INTERLEAVE-GROUP with factor 3, ir<%p0>, vp<%evl>
; OPT: Cost of 5 for VF vscale x 4: INTERLEAVE-GROUP with factor 3, ir<%p0>, vp<%evl>
; OPT: Cost of 5 for VF vscale x 4: INTERLEAVE-GROUP with factor 3, ir<%p0>, vp<%evl>
-; OPT: Cost of 7 for VF vscale x 8: INTERLEAVE-GROUP with factor 3, ir<%p0>, vp<%evl>
-; OPT: Cost of 7 for VF vscale x 8: INTERLEAVE-GROUP with factor 3, ir<%p0>, vp<%evl>
-; OPT: Cost of 14 for VF vscale x 16: INTERLEAVE-GROUP with factor 3, ir<%p0>, vp<%evl>
-; OPT: Cost of 14 for VF vscale x 16: INTERLEAVE-GROUP with factor 3, ir<%p0>, vp<%evl>
+; OPT: Cost of 6 for VF vscale x 8: INTERLEAVE-GROUP with factor 3, ir<%p0>, vp<%evl>
+; OPT: Cost of 6 for VF vscale x 8: INTERLEAVE-GROUP with factor 3, ir<%p0>, vp<%evl>
+; OPT: Cost of 12 for VF vscale x 16: INTERLEAVE-GROUP with factor 3, ir<%p0>, vp<%evl>
+; OPT: Cost of 12 for VF vscale x 16: INTERLEAVE-GROUP with factor 3, ir<%p0>, vp<%evl>
;
; FIXED-NO-OPT-LABEL: 'i8_factor_3'
; FIXED-NO-OPT: Cost of 6 for VF 2: INTERLEAVE-GROUP with factor 3, ir<%p0>
@@ -120,10 +120,10 @@ define void @i8_factor_3(ptr %data, i64 %n) {
; FIXED-OPT: Cost of 4 for VF 4: INTERLEAVE-GROUP with factor 3, ir<%p0>
; FIXED-OPT: Cost of 5 for VF 8: INTERLEAVE-GROUP with factor 3, ir<%p0>
; FIXED-OPT: Cost of 5 for VF 8: INTERLEAVE-GROUP with factor 3, ir<%p0>
-; FIXED-OPT: Cost of 7 for VF 16: INTERLEAVE-GROUP with factor 3, ir<%p0>
-; FIXED-OPT: Cost of 7 for VF 16: INTERLEAVE-GROUP with factor 3, ir<%p0>
-; FIXED-OPT: Cost of 14 for VF 32: INTERLEAVE-GROUP with factor 3, ir<%p0>
-; FIXED-OPT: Cost of 14 for VF 32: INTERLEAVE-GROUP with factor 3, ir<%p0>
+; FIXED-OPT: Cost of 6 for VF 16: INTERLEAVE-GROUP with factor 3, ir<%p0>
+; FIXED-OPT: Cost of 6 for VF 16: INTERLEAVE-GROUP with factor 3, ir<%p0>
+; FIXED-OPT: Cost of 12 for VF 32: INTERLEAVE-GROUP with factor 3, ir<%p0>
+; FIXED-OPT: Cost of 12 for VF 32: INTERLEAVE-GROUP with factor 3, ir<%p0>
;
entry:
br label %for.body
@@ -244,10 +244,10 @@ define void @i8_factor_5(ptr %data, i64 %n) {
; OPT: Cost of 6 for VF vscale x 1: INTERLEAVE-GROUP with factor 5, ir<%p0>, vp<%evl>
; OPT: Cost of 7 for VF vscale x 2: INTERLEAVE-GROUP with factor 5, ir<%p0>, vp<%evl>
; OPT: Cost of 7 for VF vscale x 2: INTERLEAVE-GROUP with factor 5, ir<%p0>, vp<%evl>
-; OPT: Cost of 9 for VF vscale x 4: INTERLEAVE-GROUP with factor 5, ir<%p0>, vp<%evl>
-; OPT: Cost of 9 for VF vscale x 4: INTERLEAVE-GROUP with factor 5, ir<%p0>, vp<%evl>
-; OPT: Cost of 13 for VF vscale x 8: INTERLEAVE-GROUP with factor 5, ir<%p0>, vp<%evl>
-; OPT: Cost of 13 for VF vscale x 8: INTERLEAVE-GROUP with factor 5, ir<%p0>, vp<%evl>
+; OPT: Cost of 8 for VF vscale x 4: INTERLEAVE-GROUP with factor 5, ir<%p0>, vp<%evl>
+; OPT: Cost of 8 for VF vscale x 4: INTERLEAVE-GROUP with factor 5, ir<%p0>, vp<%evl>
+; OPT: Cost of 10 for VF vscale x 8: INTERLEAVE-GROUP with factor 5, ir<%p0>, vp<%evl>
+; OPT: Cost of 10 for VF vscale x 8: INTERLEAVE-GROUP with factor 5, ir<%p0>, vp<%evl>
;
; FIXED-NO-OPT-LABEL: 'i8_factor_5'
; FIXED-NO-OPT: Cost of 10 for VF 2: INTERLEAVE-GROUP with factor 5, ir<%p0>
@@ -264,10 +264,10 @@ define void @i8_factor_5(ptr %data, i64 %n) {
; FIXED-OPT: Cost of 6 for VF 2: INTERLEAVE-GROUP with factor 5, ir<%p0>
; FIXED-OPT: Cost of 7 for VF 4: INTERLEAVE-GROUP with factor 5, ir<%p0>
; FIXED-OPT: Cost of 7 for VF 4: INTERLEAVE-GROUP with factor 5, ir<%p0>
-; FIXED-OPT: Cost of 9 for VF 8: INTERLEAVE-GROUP with factor 5, ir<%p0>
-; FIXED-OPT: Cost of 9 for VF 8: INTERLEAVE-GROUP with factor 5, ir<%p0>
-; FIXED-OPT: Cost of 13 for VF 16: INTERLEAVE-GROUP with factor 5, ir<%p0>
-; FIXED-OPT: Cost of 13 for VF 16: INTERLEAVE-GROUP with factor 5, ir<%p0>
+; FIXED-OPT: Cost of 8 for VF 8: INTERLEAVE-GROUP with factor 5, ir<%p0>
+; FIXED-OPT: Cost of 8 for VF 8: INTERLEAVE-GROUP with factor 5, ir<%p0>
+; FIXED-OPT: Cost of 10 for VF 16: INTERLEAVE-GROUP with factor 5, ir<%p0>
+; FIXED-OPT: Cost of 10 for VF 16: INTERLEAVE-GROUP with factor 5, ir<%p0>
;
entry:
br label %for.body
@@ -318,10 +318,10 @@ define void @i8_factor_6(ptr %data, i64 %n) {
; OPT: Cost of 7 for VF vscale x 1: INTERLEAVE-GROUP with factor 6, ir<%p0>, vp<%evl>
; OPT: Cost of 8 for VF vscale x 2: INTERLEAVE-GROUP with factor 6, ir<%p0>, vp<%evl>
; OPT: Cost of 8 for VF vscale x 2: INTERLEAVE-GROUP with factor 6, ir<%p0>, vp<%evl>
-; OPT: Cost of 10 for VF vscale x 4: INTERLEAVE-GROUP with factor 6, ir<%p0>, vp<%evl>
-; OPT: Cost of 10 for VF vscale x 4: INTERLEAVE-GROUP with factor 6, ir<%p0>, vp<%evl>
-; OPT: Cost of 14 for VF vscale x 8: INTERLEAVE-GROUP with factor 6, ir<%p0>, vp<%evl>
-; OPT: Cost of 14 for VF vscale x 8: INTERLEAVE-GROUP with factor 6, ir<%p0>, vp<%evl>
+; OPT: Cost of 9 for VF vscale x 4: INTERLEAVE-GROUP with factor 6, ir<%p0>, vp<%evl>
+; OPT: Cost of 9 for VF vscale x 4: INTERLEAVE-GROUP with factor 6, ir<%p0>, vp<%evl>
+; OPT: Cost of 12 for VF vscale x 8: INTERLEAVE-GROUP with factor 6, ir<%p0>, vp<%evl>
+; OPT: Cost of 12 for VF vscale x 8: INTERLEAVE-GROUP with factor 6, ir<%p0>, vp<%evl>
;
; FIXED-NO-OPT-LABEL: 'i8_factor_6'
; FIXED-NO-OPT: Cost of 12 for VF 2: INTERLEAVE-GROUP with factor 6, ir<%p0>
@@ -338,10 +338,10 @@ define void @i8_factor_6(ptr %data, i64 %n) {
; FIXED-OPT: Cost of 7 for VF 2: INTERLEAVE-GROUP with factor 6, ir<%p0>
; FIXED-OPT: Cost of 8 for VF 4: INTERLEAVE-GROUP with factor 6, ir<%p0>
; FIXED-OPT: Cost of 8 for VF 4: INTERLEAVE-GROUP with factor 6, ir<%p0>
-; FIXED-OPT: Cost of 10 for VF 8: INTERLEAVE-GROUP with factor 6, ir<%p0>
-; FIXED-OPT: Cost of 10 for VF 8: INTERLEAVE-GROUP with factor 6, ir<%p0>
-; FIXED-OPT: Cost of 14 for VF 16: INTERLEAVE-GROUP with factor 6, ir<%p0>
-; FIXED-OPT: Cost of 14 for VF 16: INTERLEAVE-GROUP with factor 6, ir<%p0>
+; FIXED-OPT: Cost of 9 for VF 8: INTERLEAVE-GROUP with factor 6, ir<%p0>
+; FIXED-OPT: Cost of 9 for VF 8: INTERLEAVE-GROUP with factor 6, ir<%p0>
+; FIXED-OPT: Cost of 12 for VF 16: INTERLEAVE-GROUP with factor 6, ir<%p0>
+; FIXED-OPT: Cost of 12 for VF 16: INTERLEAVE-GROUP with factor 6, ir<%p0>
;
entry:
br label %for.body
@@ -398,8 +398,8 @@ define void @i8_factor_7(ptr %data, i64 %n) {
; OPT: Cost of 9 for VF vscale x 2: INTERLEAVE-GROUP with factor 7, ir<%p0>, vp<%evl>
; OPT: Cost of 11 for VF vscale x 4: INTERLEAVE-GROUP with factor 7, ir<%p0>, vp<%evl>
; OPT: Cost of 11 for VF vscale x 4: INTERLEAVE-GROUP with factor 7, ir<%p0>, vp<%evl>
-; OPT: Cost of 15 for VF vscale x 8: INTERLEAVE-GROUP with factor 7, ir<%p0>, vp<%evl>
-; OPT: Cost of 15 for VF vscale x 8: INTERLEAVE-GROUP with factor 7, ir<%p0>, vp<%evl>
+; OPT: Cost of 14 for VF vscale x 8: INTERLEAVE-GROUP with factor 7, ir<%p0>, vp<%evl>
+; OPT: Cost of 14 for VF vscale x 8: INTERLEAVE-GROUP with factor 7, ir<%p0>, vp<%evl>
;
; FIXED-NO-OPT-LABEL: 'i8_factor_7'
; FIXED-NO-OPT: Cost of 14 for VF 2: INTERLEAVE-GROUP with factor 7, ir<%p0>
@@ -418,8 +418,8 @@ define void @i8_factor_7(ptr %data, i64 %n) {
; FIXED-OPT: Cost of 9 for VF 4: INTERLEAVE-GROUP with factor 7, ir<%p0>
; FIXED-OPT: Cost of 11 for VF 8: INTERLEAVE-GROUP with factor 7, ir<%p0>
; FIXED-OPT: Cost of 11 for VF 8: INTERLEAVE-GROUP with factor 7, ir<%p0>
-; FIXED-OPT: Cost of 15 for VF 16: INTERLEAVE-GROUP with factor 7, ir<%p0>
-; FIXED-OPT: Cost of 15 for VF 16: INTERLEAVE-GROUP with factor 7, ir<%p0>
+; FIXED-OPT: Cost of 14 for VF 16: INTERLEAVE-GROUP with factor 7, ir<%p0>
+; FIXED-OPT: Cost of 14 for VF 16: INTERLEAVE-GROUP with factor 7, ir<%p0>
;
entry:
br label %for.body
``````````
</details>
https://github.com/llvm/llvm-project/pull/207146
More information about the llvm-commits
mailing list