[llvm] [LV] Improve cost model for some replicating recipes (PR #192056)

[David Sherwood via llvm-commits]

================
@@ -380,22 +380,70 @@ exit:
 define void @sdiv_power_of_2_divisor_in_replicate_region(i32 %x, ptr %dst, i64 %n) {
 ; CHECK-LABEL: define void @sdiv_power_of_2_divisor_in_replicate_region(
 ; CHECK-SAME: i32 [[X:%.*]], ptr [[DST:%.*]], i64 [[N:%.*]]) {
-; CHECK-NEXT:  [[ENTRY:.*]]:
+; CHECK-NEXT:  [[ENTRY:.*:]]
 ; CHECK-NEXT:    [[C_1:%.*]] = icmp sgt i64 [[N]], 0
 ; CHECK-NEXT:    [[C_2:%.*]] = icmp slt i64 [[N]], 9
 ; CHECK-NEXT:    call void @llvm.assume(i1 [[C_1]])
 ; CHECK-NEXT:    call void @llvm.assume(i1 [[C_2]])
 ; CHECK-NEXT:    br label %[[LOOP:.*]]
 ; CHECK:       [[LOOP]]:
-; CHECK-NEXT:    [[IV:%.*]] = phi i64 [ 0, %[[ENTRY]] ], [ [[IV_NEXT:%.*]], %[[LOOP]] ]
+; CHECK-NEXT:    [[N_RND_UP:%.*]] = add i64 [[N]], 3
+; CHECK-NEXT:    [[N_MOD_VF:%.*]] = urem i64 [[N_RND_UP]], 4
+; CHECK-NEXT:    [[N_VEC:%.*]] = sub i64 [[N_RND_UP]], [[N_MOD_VF]]
+; CHECK-NEXT:    [[TRIP_COUNT_MINUS_1:%.*]] = sub i64 [[N]], 1
+; CHECK-NEXT:    [[BROADCAST_SPLATINSERT:%.*]] = insertelement <4 x i64> poison, i64 [[TRIP_COUNT_MINUS_1]], i64 0
+; CHECK-NEXT:    [[BROADCAST_SPLAT:%.*]] = shufflevector <4 x i64> [[BROADCAST_SPLATINSERT]], <4 x i64> poison, <4 x i32> zeroinitializer
+; CHECK-NEXT:    br label %[[VECTOR_BODY:.*]]
+; CHECK:       [[VECTOR_BODY]]:
----------------
david-arm wrote:

I have a downstream PR for adding the cost of the SCALAR-STEPS recipe, but it still doesn't help with this case. However, after looking at this again today I think the original test case also doesn't represent what happens in reality. For a start, if you take the output from this test the sdiv is completely hoisted out anyway leaving just some predicated stores. In practice, loop invariants would probably be hoisted out prior to loop vectorisation although I understand sometimes weird things can happen depending upon the pipeline. In `sdiv_power_of_2_divisor_in_replicate_region` the first sdiv has a very cheap cost, which is presumably because it's a loop invariant. I think this is what causes us to vectorise, which actually makes sense.

Secondly, if you wrote a loop with a loop-varying divide that tried to use tail-folding, e.g.

```
void foo(int * __restrict__ dst, int * __restrict__ src, int n) {
  if (n <= 0 || n > 9)
    return;

  for (int i = 0; i < n; i++) {
    dst[i] = 3 / src[i];
  }

  return;
}
```

then on a non-SVE AArch64 CPU we simply wouldn't vectorise due to the high cost of masked loads/stores. On a SVE-enabled core we would widen (not replicate) using masked load/sdiv/store operations.

If I try to compile a loop that requires tail-folding without loads/stores:

```
void foo(int * __restrict__ dst, int * __restrict__ src, int x, int n) {
  if (n <= 0 || n > 9)
    return;

  for (int i = 0; i < n; i++) {
    dst[0] += x / (3 / ((i + 1) & x));
  }

  return;
}
```

and target something like neoverse-n1 (no SVE) then the cost model says it's not worth vectorising.


https://github.com/llvm/llvm-project/pull/192056