[llvm] [LV][AArch64] LoopVectorizer allows scalable frem instructions (PR #76247)

[Maciej Gabka via llvm-commits]

================
@@ -6951,10 +6951,26 @@ LoopVectorizationCostModel::getInstructionCost(Instruction *I, ElementCount VF,
       Op2Info.Kind = TargetTransformInfo::OK_UniformValue;
 
     SmallVector<const Value *, 4> Operands(I->operand_values());
-    return TTI.getArithmeticInstrCost(
+    auto InstrCost = TTI.getArithmeticInstrCost(
         I->getOpcode(), VectorTy, CostKind,
         {TargetTransformInfo::OK_AnyValue, TargetTransformInfo::OP_None},
         Op2Info, Operands, I);
+
+    // Some targets replace frem with vector library calls.
+    if (I->getOpcode() == Instruction::FRem) {
+      LibFunc Func;
+      if (TLI->getLibFunc(I->getOpcode(), I->getType(), Func)) {
+        if (TLI->isFunctionVectorizable(TLI->getName(Func))) {
+          SmallVector<Type *, 4> OpTypes;
+          for (auto &Op : I->operands())
+            OpTypes.push_back(Op->getType());
+          auto CallCost =
+              TTI.getCallInstrCost(nullptr, VectorTy, OpTypes, CostKind);
+          return std::min(InstrCost, CallCost);
----------------
mgabka wrote:

I am not sure if that is correct.
As there is no logic in the ReplaceWithVeclib pass deciding when the frem with vector operands gets replaced to call to fmod/fmodf.
I think the idea implemented at the moment is assuming that if there are mappings in the TLI to vector functions they are worth to use always.

Otherwise we can have a patological case when:

1. **CallCost** is Invalid or very high
2. frem gets vectorised because of cheap cost of **InstrCost** (vector frem)
3. replaceWIthVeclib still decided to replace vector frem with call to vector fmod

In my opinion that is more misleading then just returning here CallCost.

@huntergr-arm what do you think?


https://github.com/llvm/llvm-project/pull/76247