[llvm] [VPlan] Fix register usage of VPExpressionRecipe (PR #176971)
John Brawn via llvm-commits
llvm-commits at lists.llvm.org
Wed Jan 28 06:30:02 PST 2026
john-brawn-arm wrote:
> > For the function dotp_unrolled in reg-usage.ll the code we generate (using -force-vector-width=16 -force-vector-interleave=1 so we select VF 16 with no interleaving) is
>
> What is the VPlan for the test case? The result should only have a scale factor, if we formed a partial reduction, and in that case, all recipes in the expression should be folded into the partial reduction IIRC.
>
> But from the assembly, it looks like we don't create a partial reduction. is it possible that we incorrectly formed a VPExpression recipe for a dot product that we then cannot lower to a dot product?
The vector loop in the vplan is
```
<x1> vector loop: {
vector.body:
EMIT vp<%7> = CANONICAL-INDUCTION ir<0>, vp<%index.next>
WIDEN-REDUCTION-PHI ir<%accum3> = phi vp<%3>, vp<%20> (VF scaled by 1/4)
WIDEN-REDUCTION-PHI ir<%accum2> = phi vp<%4>, vp<%17> (VF scaled by 1/4)
WIDEN-REDUCTION-PHI ir<%accum1> = phi vp<%5>, vp<%14> (VF scaled by 1/4)
WIDEN-REDUCTION-PHI ir<%accum0> = phi vp<%6>, vp<%11> (VF scaled by 1/4)
vp<%8> = SCALAR-STEPS vp<%7>, ir<1>, vp<%0>
CLONE ir<%gep.a0> = getelementptr inbounds ir<%a>, vp<%8>
CLONE ir<%gep.b0> = getelementptr inbounds ir<%b>, vp<%8>
CLONE ir<%offset.1> = or disjoint vp<%8>, ir<1>
CLONE ir<%gep.a1> = getelementptr inbounds ir<%a>, ir<%offset.1>
CLONE ir<%gep.b1> = getelementptr inbounds ir<%b>, ir<%offset.1>
CLONE ir<%offset.2> = or disjoint vp<%8>, ir<2>
CLONE ir<%gep.a2> = getelementptr inbounds ir<%a>, ir<%offset.2>
CLONE ir<%gep.b2> = getelementptr inbounds ir<%b>, ir<%offset.2>
CLONE ir<%offset.3> = or disjoint vp<%8>, ir<3>
CLONE ir<%gep.a3> = getelementptr inbounds ir<%a>, ir<%offset.3>
CLONE ir<%gep.b3> = getelementptr inbounds ir<%b>, ir<%offset.3>
vp<%9> = vector-pointer inbounds ir<%gep.a0>
WIDEN ir<%load.a0> = load vp<%9>
vp<%10> = vector-pointer inbounds ir<%gep.b0>
WIDEN ir<%load.b0> = load vp<%10>
EXPRESSION vp<%11> = ir<%accum0> + partial.reduce.add (mul nsw (ir<%load.b0> sext to i32), (ir<%load.a0> sext to i32))
vp<%12> = vector-pointer inbounds ir<%gep.a1>
WIDEN ir<%load.a1> = load vp<%12>
vp<%13> = vector-pointer inbounds ir<%gep.b1>
WIDEN ir<%load.b1> = load vp<%13>
EXPRESSION vp<%14> = ir<%accum1> + partial.reduce.add (mul nsw (ir<%load.a1> sext to i32), (ir<%load.b1> sext to i32))
vp<%15> = vector-pointer inbounds ir<%gep.a2>
WIDEN ir<%load.a2> = load vp<%15>
vp<%16> = vector-pointer inbounds ir<%gep.b2>
WIDEN ir<%load.b2> = load vp<%16>
EXPRESSION vp<%17> = ir<%accum2> + partial.reduce.add (mul nsw (ir<%load.a2> sext to i32), (ir<%load.b2> sext to i32))
vp<%18> = vector-pointer inbounds ir<%gep.a3>
WIDEN ir<%load.a3> = load vp<%18>
vp<%19> = vector-pointer inbounds ir<%gep.b3>
WIDEN ir<%load.b3> = load vp<%19>
EXPRESSION vp<%20> = ir<%accum3> + partial.reduce.add (mul nsw (ir<%load.a3> sext to i32), (ir<%load.b3> sext to i32))
EMIT vp<%index.next> = add nuw vp<%7>, vp<%1>
EMIT branch-on-count vp<%index.next>, vp<%2>
No successors
}
```
The IR output for this is
```
vector.body: ; preds = %vector.body, %vector.ph
%index = phi i64 [ 0, %vector.ph ], [ %index.next, %vector.body ]
%vec.phi = phi <4 x i32> [ zeroinitializer, %vector.ph ], [ %partial.reduce13, %vector.body ]
%vec.phi1 = phi <4 x i32> [ zeroinitializer, %vector.ph ], [ %partial.reduce10, %vector.body ]
%vec.phi2 = phi <4 x i32> [ zeroinitializer, %vector.ph ], [ %partial.reduce7, %vector.body ]
%vec.phi3 = phi <4 x i32> [ zeroinitializer, %vector.ph ], [ %partial.reduce, %vector.body ]
%0 = getelementptr inbounds i8, ptr %a, i64 %index
%1 = getelementptr inbounds i8, ptr %b, i64 %index
%2 = or disjoint i64 %index, 1
%3 = getelementptr inbounds i8, ptr %a, i64 %2
%4 = getelementptr inbounds i8, ptr %b, i64 %2
%5 = or disjoint i64 %index, 2
%6 = getelementptr inbounds i8, ptr %a, i64 %5
%7 = getelementptr inbounds i8, ptr %b, i64 %5
%8 = or disjoint i64 %index, 3
%9 = getelementptr inbounds i8, ptr %a, i64 %8
%10 = getelementptr inbounds i8, ptr %b, i64 %8
%wide.load = load <16 x i8>, ptr %0, align 1
%wide.load4 = load <16 x i8>, ptr %1, align 1
%11 = sext <16 x i8> %wide.load4 to <16 x i32>
%12 = sext <16 x i8> %wide.load to <16 x i32>
%13 = mul nsw <16 x i32> %11, %12
%partial.reduce = call <4 x i32> @llvm.vector.partial.reduce.add.v4i32.v16i32(<4 x i32> %vec.phi3, <16 x i32> %13)
%wide.load5 = load <16 x i8>, ptr %3, align 1
%wide.load6 = load <16 x i8>, ptr %4, align 1
%14 = sext <16 x i8> %wide.load5 to <16 x i32>
%15 = sext <16 x i8> %wide.load6 to <16 x i32>
%16 = mul nsw <16 x i32> %14, %15
%partial.reduce7 = call <4 x i32> @llvm.vector.partial.reduce.add.v4i32.v16i32(<4 x i32> %vec.phi2, <16 x i32> %16)
%wide.load8 = load <16 x i8>, ptr %6, align 1
%wide.load9 = load <16 x i8>, ptr %7, align 1
%17 = sext <16 x i8> %wide.load8 to <16 x i32>
%18 = sext <16 x i8> %wide.load9 to <16 x i32>
%19 = mul nsw <16 x i32> %17, %18
%partial.reduce10 = call <4 x i32> @llvm.vector.partial.reduce.add.v4i32.v16i32(<4 x i32> %vec.phi1, <16 x i32> %19)
%wide.load11 = load <16 x i8>, ptr %9, align 1
%wide.load12 = load <16 x i8>, ptr %10, align 1
%20 = sext <16 x i8> %wide.load11 to <16 x i32>
%21 = sext <16 x i8> %wide.load12 to <16 x i32>
%22 = mul nsw <16 x i32> %20, %21
%partial.reduce13 = call <4 x i32> @llvm.vector.partial.reduce.add.v4i32.v16i32(<4 x i32> %vec.phi, <16 x i32> %22)
%index.next = add nuw i64 %index, 16
%23 = icmp eq i64 %index.next, %n.vec
br i1 %23, label %middle.block, label %vector.body, !llvm.loop !13
```
The important thing here for the register usage is the argument to the partial reduction. The ``mul nsw (ir<%load.b0> sext to i32), (ir<%load.a0> sext to i32)`` in the vplan becomes in IR
```
%11 = sext <16 x i8> %wide.load4 to <16 x i32>
%12 = sext <16 x i8> %wide.load to <16 x i32>
%13 = mul nsw <16 x i32> %11, %12
```
The 16 x i32 vectors get split up into four 4 x i32 vectors, so the register usage here is 4.
Though actually, looking again at the generated assembly that's not exactly what ends up happening. The instructions corresponding to the first partial reduction expression are
```
smull v20.8h, v5.8b, v4.8b
smull2 v4.8h, v5.16b, v4.16b
saddw v1.4s, v1.4s, v20.4h
saddw2 v1.4s, v1.4s, v20.8h
saddw v1.4s, v1.4s, v4.4h
saddw2 v1.4s, v1.4s, v4.8h
```
It looks like during ISel an optimization has happened causing the multiply to be done using i16 values, meaning it uses half as many (i.e. 2 instead of 4) registers. This would result in a lower register usage, but instruction scheduling has transformed things so that more registers are used. Probably the best way to handle this would be for the VPlan to implement whatever ISel transformation is happening here. But regardless, the new register usage does match what's in the output IR.
https://github.com/llvm/llvm-project/pull/176971
More information about the llvm-commits
mailing list