[llvm] LoopVectorize: guard marking iv as scalar; fix bug (PR #88730)

[Ramkumar Ramachandra via llvm-commits]

================
@@ -3875,6 +3875,13 @@ void LoopVectorizationCostModel::collectLoopScalars(ElementCount VF) {
     if (!ScalarInd)
       continue;
 
+    // If the induction variable update is a fixed-order recurrence, neither the
+    // induction variable or its update should be marked scalar after
+    // vectorization.
+    auto *IndUpdatePhi = dyn_cast<PHINode>(IndUpdate);
+    if (IndUpdatePhi && Legal->isFixedOrderRecurrence(IndUpdatePhi))
----------------
artagnon wrote:

> Does being a fixed-order recurrence guarantee that it will be vectorised?

I think when you rephrase this as "a first-order recurrence can never be scalarized", it always holds.

> Also, it looks like we're treating this as scalar-after-vectorisation because a user of the update is in the WorkList. I must admit to finding this code very complicated so I don't fully understand it, but I wonder if there is a way to stop it from appearing in the WorkList early on?

Actually, it's not the user of the update that's in the Worklist. The underlying problem is that the following condition gets matched: `I == Ind` in

```
    auto ScalarIndUpdate =
        llvm::all_of(IndUpdate->users(), [&](User *U) -> bool {
          auto *I = cast<Instruction>(U);
          return I == Ind || !TheLoop->contains(I) || Worklist.count(I) ||
                 IsDirectLoadStoreFromPtrIndvar(IndUpdate, I);
        });
```

for the iv `%idx.merge = phi i64 [ 0, %entry ], [ %idx.ext.merge, %for.body ]`. Now, this condition is technically correct, because if the user of a phi is the same phi, we should consider it as scalar. The case at hand is when the iv is considered scalar in isolation, without considering the iv update, but when the iv update is actually a first-order recurrence, neither the iv nor the iv update should be marked as scalar-after-vectorization. I think my patch is correct, and that we can't do better.

https://github.com/llvm/llvm-project/pull/88730