[llvm] [LV] Support tail-folded epilogue loops (PR #208764)

[David Sherwood via llvm-commits]

================
@@ -7554,6 +7697,23 @@ static SmallVector<Instruction *> preparePlanForEpilogueVectorLoop(
           continue;
         }
       }
+    } else if (isa<VPActiveLaneMaskPHIRecipe>(&R)) {
+      // The active-lane-mask phi's entry value was computed assuming the
+      // epilogue vector loop starts at 0. Rebuild it using the resume value
+      // instead, so the mask reflects how many elements the main vector loop
+      // already processed.
+      VPBuilder EntryBuilder(Plan.getVectorPreheader());
+      Type *CanIVTy = VectorLoop->getCanonicalIVType();
+      VPValue *ALMMultiplier = Plan.getConstantInt(CanIVTy, 1);
+      auto *EntryIncrement = EntryBuilder.createOverflowingOp(
+          VPInstruction::CanonicalIVIncrementForPart, {VPV, &Plan.getVF()}, {},
+          R.getDebugLoc(), "index.part.next");
+      auto *EntryALM = EntryBuilder.createNaryOp(
+          VPInstruction::WideActiveLaneMask,
----------------
david-arm wrote:

Not all tail-folding forms use `WideActiveLaneMask`. According to the documentation:

```
    // As above, but takes an additional operand (Multiplier). The size of
    // the mask returned is VF * Multiplier (UF, op2).
    // WideActiveLaneMask is used for control flow and is unrolled by widening,
    // with one extract vector created per unroll part.
    WideActiveLaneMask,
```

so it's only use for the `DataAndControlFlow` tail-folding style. I think you might need to check the `TailFoldingStyle` to know which one to use. However, given we don't unroll the epilogue loop you might just be able to use `ActiveLaneMask`.

Also, we were actually checking for `VPActiveLaneMaskPHIRecipe` in if statement above:

```
  } else if (isa<VPActiveLaneMaskPHIRecipe>(&R)) {
```

In general, it would be good to make sure that epilogue tail-folding works for other targets, e.g. MVE and RISCV. Or, if those targets cannot be supported then it would be good to explain why, add some tests, and possibly bail out early. For example, can we perform EVL tail-folding in the epilogue? Can we use the simpler `Data` tail-folding style, etc? They don't have to all be suppored in this first PR - it's just about making sure we do the right thing and have appropriate testing.

https://github.com/llvm/llvm-project/pull/208764