[llvm] [VPlan] Support masked VPInsts, use for predication (NFC) (PR #142285)

[Florian Hahn via llvm-commits]

https://github.com/fhahn commented:

> > The mask will then be dropped or handled when creating wide recipes.
> 
> This sounds like it's less of a proper operand but rather an auxiliary data (like a basic block number) that we want to keep inside `VPInstruction` just for convenience and for use by a few transformations only. Why not do it outside operands? Also, how often would `addMask` in its current version cause operands vector reallocation?

Having it as proper operand has multiple advantages: 
1) properly part of def-use chain means everything works correctly when (parts) oft the mask are RAUW'd (e.g happens when we replace the VPInstruction for the mask with a different recipe. This removes a complexity where previously we had to keep track of all possible updates. 
2) Making it part of VPInstructions gives us flexibility for additional optimizations, as it allows moving the masked instructions to different blocks.
3) it is consistent with how mask for all other recipes are handled, for similar reasons to the above

In some cases we will need to reallocate for when adding the mask, for some we won't, but it should be less complexity than the current tracking logic, and we only have to pay the price when something needs to be masked (as compared to storing it separately in VPInstruction).

After this change, we should also be able to move the predicator to only run once on VPlan0, which will further reduce the overhead.

https://github.com/llvm/llvm-project/pull/142285