[llvm] [LV] Add vector library call legalization via VPlan transform (PR #197661)
Ashutosh Nema via llvm-commits
llvm-commits at lists.llvm.org
Thu May 14 05:02:38 PDT 2026
https://github.com/nema-ashutosh created https://github.com/llvm/llvm-project/pull/197661
When the loop vectorizer picks a VF for which no usable vector-library variant exists -- either because the call exceeds the target's register width, or because the library has no variant at that VF -- the vectorizer today either emits an unsupported call (forcing backend scalarization) or falls back to scalar calls outright.
This patch introduces a VPlan-to-VPlan transformation that legalizes such calls by splitting them into multiple calls at a width the library actually provides. For example, sin(double) at VF=8 on AVX (256-bit) is split into 2 x VF=4 calls to __svml_sin4 (or amd_vrd4_sin, etc.).
The transformation is gated behind a hidden flag
-legalize-vector-library-calls (default off).
The cost model finds the largest TLI-known variant that fits in registers and evenly divides the requested VF, and estimates the split cost (N calls + extract/insert shuffles). Legalization is chosen only when it beats all alternatives.
A new VPlan pass, legalizeVecLibCalls(), expands marked recipes into ExtractSubvector + legal-width call + ConcatVectors sequences before IR generation.
This works with any vector library that has TLI mappings (SVML, AMDLIBM, LIBMVEC, etc.) and supports multi-argument functions like pow().
>From 1c604db7b1e9a5f9a3db855b48ef925c31fc4cde Mon Sep 17 00:00:00 2001
From: Ashutosh Nema <ashu1212 at gmail.com>
Date: Thu, 14 May 2026 17:30:05 +0530
Subject: [PATCH] [LV] Add vector library call legalization via VPlan transform
When the loop vectorizer picks a VF for which no usable vector-library
variant exists -- either because the call exceeds the target's register
width, or because the library has no variant at that VF -- the vectorizer
today either emits an unsupported call (forcing backend scalarization) or
falls back to scalar calls outright.
This patch introduces a VPlan-to-VPlan transformation that legalizes
such calls by splitting them into multiple calls at a width the library
actually provides. For example, sin(double) at VF=8 on AVX (256-bit) is
split into 2 x VF=4 calls to __svml_sin4 (or amd_vrd4_sin, etc.).
The transformation is gated behind a hidden flag
-legalize-vector-library-calls (default off).
The cost model finds the largest TLI-known variant that fits in
registers and evenly divides the requested VF, and estimates the split
cost (N calls + extract/insert shuffles). Legalization is chosen only
when it beats all alternatives.
A new VPlan pass, legalizeVecLibCalls(), expands marked recipes into
ExtractSubvector + legal-width call + ConcatVectors sequences before
IR generation.
This works with any vector library that has TLI mappings (SVML, AMDLIBM,
LIBMVEC, etc.) and supports multi-argument functions like pow().
---
.../Transforms/Vectorize/LoopVectorize.cpp | 213 +++++++++++++++++-
llvm/lib/Transforms/Vectorize/VPlan.h | 37 ++-
.../Transforms/Vectorize/VPlanAnalysis.cpp | 2 +
.../lib/Transforms/Vectorize/VPlanRecipes.cpp | 54 ++++-
.../Transforms/Vectorize/VPlanTransforms.cpp | 101 +++++++++
.../Transforms/Vectorize/VPlanTransforms.h | 7 +
.../VPlan/vplan-print-after-all.ll | 1 +
.../X86/veclib-legalize-amdlibm-float.ll | 74 ++++++
.../X86/veclib-legalize-amdlibm.ll | 103 +++++++++
.../X86/veclib-legalize-basic.ll | 73 ++++++
.../LoopVectorize/X86/veclib-legalize-pow.ll | 52 +++++
11 files changed, 700 insertions(+), 17 deletions(-)
create mode 100644 llvm/test/Transforms/LoopVectorize/X86/veclib-legalize-amdlibm-float.ll
create mode 100644 llvm/test/Transforms/LoopVectorize/X86/veclib-legalize-amdlibm.ll
create mode 100644 llvm/test/Transforms/LoopVectorize/X86/veclib-legalize-basic.ll
create mode 100644 llvm/test/Transforms/LoopVectorize/X86/veclib-legalize-pow.ll
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index 61c2d3cd228ec..ddfba6d4730f4 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -316,6 +316,11 @@ static cl::opt<bool> PreferPredicatedReductionSelect(
cl::desc(
"Prefer predicating a reduction operation over an after loop select."));
+static cl::opt<bool> LegalizeVectorLibraryCalls(
+ "legalize-vector-library-calls", cl::init(false), cl::Hidden,
+ cl::desc("Legalize vector library calls by splitting oversized vector "
+ "calls into multiple calls with a legal vector factor"));
+
cl::opt<bool> llvm::EnableVPlanNativePath(
"enable-vplan-native-path", cl::Hidden,
cl::desc("Enable VPlan-native vectorization path with "
@@ -854,6 +859,26 @@ class LoopVectorizationCostModel {
/// decision in a map for use in planning and plan execution.
void setVectorizedCallDecision(ElementCount VF);
+ /// Returns true if \p VF elements of the given scalar type exceed the
+ /// target's vector register width.
+ bool exceedsRegisterWidth(ElementCount VF, Type *ScalarRetTy,
+ ArrayRef<Type *> ScalarParamTys) const;
+
+ /// Find the largest TLI-known vector variant of \p CI's callee whose VF
+ /// evenly divides \p VF. Returns {nullptr, 0} if none found.
+ std::pair<Function *, ElementCount>
+ findLargestLegalVariant(CallInst *CI, ElementCount VF, Type *ScalarRetTy,
+ ArrayRef<Type *> ScalarParamTys) const;
+
+ /// Compute the cost of splitting a vector call at \p VF into multiple
+ /// legal-width calls. Populates \p LegalVecFunc and \p LegalVecVF.
+ InstructionCost getLegalizedCallCost(CallInst *CI, ElementCount VF,
+ Type *ScalarRetTy,
+ ArrayRef<Type *> ScalarTys,
+ bool NeedsLegalization,
+ Function *&LegalVecFunc,
+ ElementCount &LegalVecVF) const;
+
/// Collect values we want to ignore in the cost model.
void collectValuesToIgnore();
@@ -929,7 +954,8 @@ class LoopVectorizationCostModel {
CM_GatherScatter,
CM_Scalarize,
CM_VectorCall,
- CM_IntrinsicCall
+ CM_IntrinsicCall,
+ CM_VectorCallLegalized
};
/// Save vectorization decision \p W and \p Cost taken by the cost model for
@@ -996,14 +1022,19 @@ class LoopVectorizationCostModel {
Intrinsic::ID IID;
std::optional<unsigned> MaskPos;
InstructionCost Cost;
+ Function *LegalVariant = nullptr;
+ ElementCount LegalVF = ElementCount::getFixed(0);
};
void setCallWideningDecision(CallInst *CI, ElementCount VF, InstWidening Kind,
Function *Variant, Intrinsic::ID IID,
std::optional<unsigned> MaskPos,
- InstructionCost Cost) {
+ InstructionCost Cost,
+ Function *LegalVariant = nullptr,
+ ElementCount LegalVF = ElementCount::getFixed(0)) {
assert(!VF.isScalar() && "Expected vector VF");
- CallWideningDecisions[{CI, VF}] = {Kind, Variant, IID, MaskPos, Cost};
+ CallWideningDecisions[{CI, VF}] = {Kind, Variant, IID, MaskPos,
+ Cost, LegalVariant, LegalVF};
}
CallWideningDecision getCallWideningDecision(CallInst *CI,
@@ -4921,6 +4952,109 @@ void LoopVectorizationCostModel::setCostBasedWideningDecision(ElementCount VF) {
}
}
+/// Returns true if VF elements of the given scalar type exceed the target's
+/// fixed-width vector register.
+bool LoopVectorizationCostModel::exceedsRegisterWidth(
+ ElementCount VF, Type *ScalarRetTy, ArrayRef<Type *> ScalarParamTys) const {
+ unsigned RegWidth =
+ TTI.getRegisterBitWidth(TargetTransformInfo::RGK_FixedWidthVector)
+ .getFixedValue();
+ Type *ElemTy = ScalarRetTy->isVoidTy() ? ScalarParamTys[0] : ScalarRetTy;
+ unsigned ElemBits =
+ TheLoop->getHeader()->getDataLayout().getTypeSizeInBits(ElemTy);
+ return VF.getKnownMinValue() * ElemBits > RegWidth;
+}
+
+/// Search TLI for the largest vector variant of \p CI's callee whose VF is
+/// smaller than \p VF and evenly divides it. Returns {nullptr, 0} if none.
+std::pair<Function *, ElementCount>
+LoopVectorizationCostModel::findLargestLegalVariant(
+ CallInst *CI, ElementCount VF, Type *ScalarRetTy,
+ ArrayRef<Type *> ScalarParamTys) const {
+ Function *ScalarFn = CI->getCalledFunction();
+ if (!ScalarFn || !TLI)
+ return {nullptr, ElementCount::getFixed(0)};
+
+ StringRef ScalarName = ScalarFn->getName();
+ for (unsigned CandVF = VF.getKnownMinValue() / 2; CandVF >= 2;
+ CandVF /= 2) {
+ if (VF.getKnownMinValue() % CandVF != 0)
+ continue;
+ ElementCount CandEC = ElementCount::getFixed(CandVF);
+ if (exceedsRegisterWidth(CandEC, ScalarRetTy, ScalarParamTys))
+ continue;
+ StringRef VecName = TLI->getVectorizedFunction(ScalarName, CandEC);
+ if (VecName.empty())
+ continue;
+
+ Function *VecFunc = CI->getModule()->getFunction(VecName);
+ if (!VecFunc) {
+ Type *RetTy = toVectorizedTy(ScalarRetTy, CandEC);
+ SmallVector<Type *, 4> ParamTys;
+ for (Type *T : ScalarParamTys)
+ ParamTys.push_back(toVectorizedTy(T, CandEC));
+ VecFunc = Function::Create(FunctionType::get(RetTy, ParamTys, false),
+ Function::ExternalLinkage, VecName,
+ CI->getModule());
+ VecFunc->copyAttributesFrom(ScalarFn);
+ }
+ return {VecFunc, CandEC};
+ }
+ return {nullptr, ElementCount::getFixed(0)};
+}
+
+/// Compute the cost of legalizing a vector call at \p VF by splitting it into
+/// multiple calls to a smaller legal-width variant. Populates \p LegalVecFunc
+/// and \p LegalVecVF with the chosen variant. Returns Invalid if legalization
+/// is not applicable or no suitable variant exists.
+InstructionCost LoopVectorizationCostModel::getLegalizedCallCost(
+ CallInst *CI, ElementCount VF, Type *ScalarRetTy,
+ ArrayRef<Type *> ScalarTys, bool NeedsLegalization,
+ Function *&LegalVecFunc, ElementCount &LegalVecVF) const {
+ if (!NeedsLegalization || !LegalizeVectorLibraryCalls || !VF.isFixed() ||
+ !TLI || CI->isNoBuiltin())
+ return InstructionCost::getInvalid();
+
+ std::tie(LegalVecFunc, LegalVecVF) =
+ findLargestLegalVariant(CI, VF, ScalarRetTy, ScalarTys);
+ if (!LegalVecFunc)
+ return InstructionCost::getInvalid();
+
+ unsigned WideLen = VF.getKnownMinValue();
+ unsigned LegalLen = LegalVecVF.getKnownMinValue();
+ unsigned NumParts = WideLen / LegalLen;
+
+ InstructionCost CallCost =
+ NumParts *
+ TTI.getCallInstrCost(nullptr, LegalVecFunc->getReturnType(),
+ LegalVecFunc->getFunctionType()->params(),
+ Config.CostKind);
+
+ // Extract-subvector cost per argument, using each argument's actual type.
+ InstructionCost ExtractCost = 0;
+ for (Type *ScalarTy : ScalarTys) {
+ auto *WideArgTy = FixedVectorType::get(ScalarTy, WideLen);
+ auto *SubArgTy = FixedVectorType::get(ScalarTy, LegalLen);
+ ExtractCost +=
+ NumParts * TTI.getShuffleCost(TargetTransformInfo::SK_ExtractSubvector,
+ WideArgTy, WideArgTy, {}, Config.CostKind,
+ 0, SubArgTy);
+ }
+
+ // Reassembly cost: NumParts-1 insert-subvector shuffles on the return type.
+ InstructionCost InsertCost = 0;
+ if (!ScalarRetTy->isVoidTy()) {
+ auto *WideRetTy = FixedVectorType::get(ScalarRetTy, WideLen);
+ auto *SubRetTy = FixedVectorType::get(ScalarRetTy, LegalLen);
+ InsertCost =
+ (NumParts - 1) *
+ TTI.getShuffleCost(TargetTransformInfo::SK_InsertSubvector, WideRetTy,
+ WideRetTy, {}, Config.CostKind, 0, SubRetTy);
+ }
+
+ return CallCost + ExtractCost + InsertCost;
+}
+
void LoopVectorizationCostModel::setVectorizedCallDecision(ElementCount VF) {
assert(!VF.isScalar() &&
"Trying to set a vectorization decision for a scalar VF");
@@ -5059,6 +5193,37 @@ void LoopVectorizationCostModel::setVectorizedCallDecision(ElementCount VF) {
if (IID != Intrinsic::not_intrinsic)
IntrinsicCost = getVectorIntrinsicCost(CI, VF);
+ // When legalization is enabled, invalidate exact-match vector variants
+ // that exceed the target's register width (e.g., __svml_sin8 on AVX).
+ if (VecFunc && LegalizeVectorLibraryCalls && VF.isFixed() &&
+ exceedsRegisterWidth(VF, ScalarRetTy, ScalarTys)) {
+ VecFunc = nullptr;
+ VectorCost = InstructionCost::getInvalid();
+ }
+
+ // If no usable variant exists, try to find a smaller legal-width variant
+ // that the call can be split into (e.g., 2 x __svml_sin4 for VF=8).
+ Function *LegalVecFunc = nullptr;
+ ElementCount LegalVecVF = ElementCount::getFixed(0);
+ InstructionCost LegalizedCallCost =
+ getLegalizedCallCost(CI, VF, ScalarRetTy, ScalarTys,
+ !VecFunc, LegalVecFunc, LegalVecVF);
+
+ LLVM_DEBUG({
+ dbgs() << "LV(CallCost): VF=" << VF << " "
+ << CI->getCalledFunction()->getName()
+ << ": ScalarCost=" << ScalarCost
+ << " VectorCost=" << VectorCost
+ << " IntrinsicCost=" << IntrinsicCost
+ << " LegalizedCallCost=" << LegalizedCallCost;
+ if (LegalVecFunc)
+ dbgs() << " (via " << LegalVecFunc->getName()
+ << ", LegalVF=" << LegalVecVF << ", NumParts="
+ << VF.getKnownMinValue() / LegalVecVF.getKnownMinValue()
+ << ")";
+ dbgs() << "\n";
+ });
+
InstructionCost Cost = ScalarCost;
InstWidening Decision = CM_Scalarize;
@@ -5072,8 +5237,17 @@ void LoopVectorizationCostModel::setVectorizedCallDecision(ElementCount VF) {
Decision = CM_IntrinsicCall;
}
- setCallWideningDecision(CI, VF, Decision, VecFunc, IID,
- FuncInfo.getParamIndexForOptionalMask(), Cost);
+ if (LegalizedCallCost.isValid() && LegalizedCallCost <= Cost) {
+ Cost = LegalizedCallCost;
+ Decision = CM_VectorCallLegalized;
+ }
+
+ if (Decision == CM_VectorCallLegalized)
+ setCallWideningDecision(CI, VF, Decision, nullptr, IID, std::nullopt,
+ Cost, LegalVecFunc, LegalVecVF);
+ else
+ setCallWideningDecision(CI, VF, Decision, VecFunc, IID,
+ FuncInfo.getParamIndexForOptionalMask(), Cost);
}
}
}
@@ -6141,6 +6315,7 @@ DenseMap<const SCEV *, Value *> LoopVectorizationPlanner::executePlan(
RUN_VPLAN_PASS(VPlanTransforms::removeDeadRecipes, BestVPlan);
+ RUN_VPLAN_PASS(VPlanTransforms::legalizeVecLibCalls, BestVPlan, BestVF);
RUN_VPLAN_PASS(VPlanTransforms::convertToConcreteRecipes, BestVPlan);
// Convert the exit condition to AVLNext == 0 for EVL tail folded loops.
RUN_VPLAN_PASS(VPlanTransforms::convertEVLExitCond, BestVPlan);
@@ -6513,6 +6688,34 @@ VPSingleDefRecipe *VPRecipeBuilder::tryToWidenCall(VPInstruction *VPI,
VPI->getDebugLoc());
}
+ // Check if we should legalize (split) the vector call into multiple
+ // calls at a smaller legal VF.
+ Function *LegalVariant = nullptr;
+ ElementCount LegalVF = ElementCount::getFixed(0);
+ auto ShouldLegalizeCall =
+ LoopVectorizationPlanner::getDecisionAndClampRange(
+ [&](ElementCount VF) -> bool {
+ if (LegalVariant)
+ return false;
+ LoopVectorizationCostModel::CallWideningDecision Decision =
+ CM.getCallWideningDecision(CI, VF);
+ if (Decision.Kind ==
+ LoopVectorizationCostModel::CM_VectorCallLegalized) {
+ LegalVariant = Decision.LegalVariant;
+ LegalVF = Decision.LegalVF;
+ return true;
+ }
+ return false;
+ },
+ Range);
+ if (ShouldLegalizeCall) {
+ Ops.push_back(VPI->getOperand(VPI->getNumOperandsWithoutMask() - 1));
+ auto *Recipe = new VPWidenCallRecipe(CI, LegalVariant, Ops, *VPI, *VPI,
+ VPI->getDebugLoc());
+ Recipe->setLegalizationInfo(LegalVariant, LegalVF);
+ return Recipe;
+ }
+
return nullptr;
}
diff --git a/llvm/lib/Transforms/Vectorize/VPlan.h b/llvm/lib/Transforms/Vectorize/VPlan.h
index 63436c79e9a98..c222c9dd0fad1 100644
--- a/llvm/lib/Transforms/Vectorize/VPlan.h
+++ b/llvm/lib/Transforms/Vectorize/VPlan.h
@@ -1338,7 +1338,16 @@ class LLVM_ABI_FOR_TEST VPInstruction : public VPRecipeWithIRFlags,
/// backedge value). Has the wide induction recipe as operand.
ExitingIVValue,
MaskedCond,
- OpsEnd = MaskedCond,
+
+ /// Extracts a sub-vector from a wider vector. Operands: (wide_vec,
+ /// start_idx_constant, subvec_len_constant). The start index and
+ /// sub-vector length are VPValues wrapping ConstantInt live-ins.
+ ExtractSubvector,
+ /// Concatenates two vectors into a single wider vector by creating a
+ /// shufflevector with an identity mask. Operands: (vec_a, vec_b).
+ ConcatVectors,
+
+ OpsEnd = ConcatVectors,
};
/// Returns true if this VPInstruction generates scalar values for all lanes.
@@ -1991,6 +2000,12 @@ class LLVM_ABI_FOR_TEST VPWidenCallRecipe : public VPRecipeWithIRFlags,
/// different VPlan for each VF with a valid variant.
Function *Variant;
+ /// When legalization is needed, LegalVariant stores the legal-width vector
+ /// function and LegalVF stores the legal vectorization factor.
+ Function *LegalVariant = nullptr;
+ ElementCount LegalVF = ElementCount::getFixed(0);
+ bool NeedsLegalization = false;
+
public:
VPWidenCallRecipe(Value *UV, Function *Variant,
ArrayRef<VPValue *> CallArguments,
@@ -2008,8 +2023,12 @@ class LLVM_ABI_FOR_TEST VPWidenCallRecipe : public VPRecipeWithIRFlags,
~VPWidenCallRecipe() override = default;
VPWidenCallRecipe *clone() override {
- return new VPWidenCallRecipe(getUnderlyingValue(), Variant, operands(),
- *this, *this, getDebugLoc());
+ auto *R = new VPWidenCallRecipe(getUnderlyingValue(), Variant, operands(),
+ *this, *this, getDebugLoc());
+ R->LegalVariant = LegalVariant;
+ R->LegalVF = LegalVF;
+ R->NeedsLegalization = NeedsLegalization;
+ return R;
}
VP_CLASSOF_IMPL(VPRecipeBase::VPWidenCallSC)
@@ -2025,6 +2044,18 @@ class LLVM_ABI_FOR_TEST VPWidenCallRecipe : public VPRecipeWithIRFlags,
return cast<Function>(getOperand(getNumOperands() - 1)->getLiveInIRValue());
}
+ Function *getVariant() const { return Variant; }
+
+ void setLegalizationInfo(Function *LegalFunc, ElementCount VF) {
+ LegalVariant = LegalFunc;
+ LegalVF = VF;
+ NeedsLegalization = true;
+ }
+
+ bool needsLegalization() const { return NeedsLegalization; }
+ Function *getLegalVariant() const { return LegalVariant; }
+ ElementCount getLegalVF() const { return LegalVF; }
+
operand_range args() { return drop_end(operands()); }
const_operand_range args() const { return drop_end(operands()); }
diff --git a/llvm/lib/Transforms/Vectorize/VPlanAnalysis.cpp b/llvm/lib/Transforms/Vectorize/VPlanAnalysis.cpp
index a42b631cd3304..08b19763e602a 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanAnalysis.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanAnalysis.cpp
@@ -95,6 +95,8 @@ Type *VPTypeAnalysis::inferScalarTypeForRecipe(const VPInstruction *R) {
case VPInstruction::ReductionStartVector:
case VPInstruction::ResumeForEpilogue:
case VPInstruction::Reverse:
+ case VPInstruction::ExtractSubvector:
+ case VPInstruction::ConcatVectors:
return inferScalarType(R->getOperand(0));
case Instruction::Select: {
Type *ResTy = inferScalarType(R->getOperand(1));
diff --git a/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp b/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
index 11a91dcd46867..f4eb7c9f77ea0 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
@@ -516,6 +516,7 @@ unsigned VPInstruction::getNumOperandsForOpcode() const {
case Instruction::Store:
case VPInstruction::BranchOnCount:
case VPInstruction::BranchOnTwoConds:
+ case VPInstruction::ConcatVectors:
case VPInstruction::FirstOrderRecurrenceSplice:
case VPInstruction::LogicalAnd:
case VPInstruction::LogicalOr:
@@ -527,6 +528,7 @@ unsigned VPInstruction::getNumOperandsForOpcode() const {
case Instruction::InsertElement:
case Instruction::Select:
case VPInstruction::ActiveLaneMask:
+ case VPInstruction::ExtractSubvector:
case VPInstruction::ReductionStartVector:
return 3;
case Instruction::Call:
@@ -926,6 +928,27 @@ Value *VPInstruction::generate(VPTransformState &State) {
return State.get(getOperand(0), true);
case VPInstruction::Reverse:
return Builder.CreateVectorReverse(State.get(getOperand(0)), "reverse");
+ case VPInstruction::ExtractSubvector: {
+ Value *WideVec = State.get(getOperand(0));
+ unsigned StartIdx =
+ cast<ConstantInt>(getOperand(1)->getLiveInIRValue())->getZExtValue();
+ unsigned SubVecLen =
+ cast<ConstantInt>(getOperand(2)->getLiveInIRValue())->getZExtValue();
+ SmallVector<int, 16> Mask;
+ for (unsigned K = 0; K < SubVecLen; ++K)
+ Mask.push_back(StartIdx + K);
+ return Builder.CreateShuffleVector(WideVec, Mask, "extract.subvec");
+ }
+ case VPInstruction::ConcatVectors: {
+ Value *VecA = State.get(getOperand(0));
+ Value *VecB = State.get(getOperand(1));
+ unsigned LenA = cast<FixedVectorType>(VecA->getType())->getNumElements();
+ unsigned LenB = cast<FixedVectorType>(VecB->getType())->getNumElements();
+ SmallVector<int, 16> Mask;
+ for (unsigned K = 0; K < LenA + LenB; ++K)
+ Mask.push_back(K);
+ return Builder.CreateShuffleVector(VecA, VecB, Mask, "concat.vec");
+ }
case VPInstruction::ExtractLastActive: {
Value *Result = State.get(getOperand(0), /*IsScalar=*/true);
for (unsigned Idx = 1; Idx < getNumOperands(); Idx += 2) {
@@ -1412,6 +1435,8 @@ bool VPInstruction::opcodeMayReadOrWriteFromMemory() const {
case VPInstruction::Reverse:
case VPInstruction::VScale:
case VPInstruction::Unpack:
+ case VPInstruction::ExtractSubvector:
+ case VPInstruction::ConcatVectors:
return false;
case Instruction::Call:
return !getCalledFunction(*this)->doesNotAccessMemory();
@@ -1601,6 +1626,12 @@ void VPInstruction::printRecipe(raw_ostream &O, const Twine &Indent,
case VPInstruction::ExtractLastActive:
O << "extract-last-active";
break;
+ case VPInstruction::ExtractSubvector:
+ O << "extract-subvector";
+ break;
+ case VPInstruction::ConcatVectors:
+ O << "concat-vectors";
+ break;
default:
O << Instruction::getOpcodeName(getOpcode());
}
@@ -1847,14 +1878,15 @@ void VPWidenCallRecipe::execute(VPTransformState &State) {
assert(State.VF.isVector() && "not widening");
assert(Variant != nullptr && "Can't create vector function.");
+ auto *CI = cast_or_null<CallInst>(getUnderlyingValue());
+ SmallVector<OperandBundleDef, 1> OpBundles;
+ if (CI)
+ CI->getOperandBundlesAsDefs(OpBundles);
+
FunctionType *VFTy = Variant->getFunctionType();
- // Add return type if intrinsic is overloaded on it.
SmallVector<Value *, 4> Args;
for (const auto &I : enumerate(args())) {
Value *Arg;
- // Some vectorized function variants may also take a scalar argument,
- // e.g. linear parameters for pointers. This needs to be the scalar value
- // from the start of the respective part when interleaving.
if (!VFTy->getParamType(I.index())->isVectorTy())
Arg = State.get(I.value(), VPLane(0));
else
@@ -1862,11 +1894,6 @@ void VPWidenCallRecipe::execute(VPTransformState &State) {
Args.push_back(Arg);
}
- auto *CI = cast_or_null<CallInst>(getUnderlyingValue());
- SmallVector<OperandBundleDef, 1> OpBundles;
- if (CI)
- CI->getOperandBundlesAsDefs(OpBundles);
-
CallInst *V = State.Builder.CreateCall(Variant, Args, OpBundles);
applyFlags(*V);
applyMetadata(*V);
@@ -1878,6 +1905,15 @@ void VPWidenCallRecipe::execute(VPTransformState &State) {
InstructionCost VPWidenCallRecipe::computeCost(ElementCount VF,
VPCostContext &Ctx) const {
+ if (NeedsLegalization && LegalVariant) {
+ unsigned WideVF = VF.getKnownMinValue();
+ unsigned LegalWidth = LegalVF.getKnownMinValue();
+ unsigned NumParts = WideVF / LegalWidth;
+ InstructionCost PerPartCost = Ctx.TTI.getCallInstrCost(
+ nullptr, LegalVariant->getReturnType(),
+ LegalVariant->getFunctionType()->params(), Ctx.CostKind);
+ return NumParts * PerPartCost;
+ }
return Ctx.TTI.getCallInstrCost(nullptr, Variant->getReturnType(),
Variant->getFunctionType()->params(),
Ctx.CostKind);
diff --git a/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp b/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
index 32d89a34105a4..a5e4cd6953e60 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
@@ -3862,6 +3862,107 @@ void VPlanTransforms::expandBranchOnTwoConds(VPlan &Plan) {
}
}
+void VPlanTransforms::legalizeVecLibCalls(VPlan &Plan, ElementCount VF) {
+ if (!VF.isFixed())
+ return;
+
+ SmallVector<VPWidenCallRecipe *> ToLegalize;
+ for (VPBasicBlock *VPBB : VPBlockUtils::blocksOnly<VPBasicBlock>(
+ vp_depth_first_deep(Plan.getEntry()))) {
+ for (VPRecipeBase &R : *VPBB) {
+ auto *WC = dyn_cast<VPWidenCallRecipe>(&R);
+ if (WC && WC->needsLegalization())
+ ToLegalize.push_back(WC);
+ }
+ }
+
+ for (VPWidenCallRecipe *WC : ToLegalize) {
+ Function *LegalVariant = WC->getLegalVariant();
+ ElementCount LegalVFEC = WC->getLegalVF();
+ assert(LegalVariant && "Legalization requires a legal variant");
+ assert(LegalVFEC.isFixed() && "Legalization only supports fixed-width VFs");
+
+ FunctionType *LegalFTy = LegalVariant->getFunctionType();
+ LLVMContext &Ctx = LegalVariant->getContext();
+ unsigned LegalWidth = LegalVFEC.getKnownMinValue();
+ unsigned WideVF = VF.getKnownMinValue();
+
+ if (WideVF <= LegalWidth)
+ continue;
+
+ unsigned NumParts = WideVF / LegalWidth;
+ assert(NumParts > 1 && "Legalization requires at least 2 parts");
+ assert(WideVF % LegalWidth == 0 && "VF must be a multiple of LegalVF");
+
+ // Collect the wide argument VPValues (excluding the last callee operand).
+ SmallVector<VPValue *, 4> WideArgVPVs;
+ for (VPValue *Arg : WC->args())
+ WideArgVPVs.push_back(Arg);
+
+ // Get the callee function operand (last operand).
+ VPValue *CalleeVPV = WC->getOperand(WC->getNumOperands() - 1);
+
+ // Create the sequence: for each Part, extract sub-vectors from wide args,
+ // call the legal variant, then combine partial results.
+ SmallVector<VPValue *, 4> PartialResults;
+ VPBuilder Builder(WC);
+
+ for (unsigned Part = 0; Part < NumParts; ++Part) {
+ SmallVector<VPValue *, 4> PartOps;
+ unsigned StartIdx = Part * LegalWidth;
+
+ for (unsigned ArgIdx = 0; ArgIdx < WideArgVPVs.size(); ++ArgIdx) {
+ if (!LegalFTy->getParamType(ArgIdx)->isVectorTy()) {
+ PartOps.push_back(WideArgVPVs[ArgIdx]);
+ } else {
+ VPValue *StartIdxVPV = Plan.getOrAddLiveIn(
+ ConstantInt::get(Type::getInt32Ty(Ctx), StartIdx));
+ VPValue *SubVecLenVPV = Plan.getOrAddLiveIn(
+ ConstantInt::get(Type::getInt32Ty(Ctx), LegalWidth));
+ auto *Extract = new VPInstruction(VPInstruction::ExtractSubvector,
+ {WideArgVPVs[ArgIdx], StartIdxVPV,
+ SubVecLenVPV});
+ Extract->insertBefore(WC);
+ PartOps.push_back(Extract);
+ }
+ }
+
+ // Add the callee function operand.
+ PartOps.push_back(CalleeVPV);
+
+ auto *PartCall = new VPWidenCallRecipe(
+ WC->getUnderlyingValue(), LegalVariant, PartOps, *WC, *WC,
+ WC->getDebugLoc());
+ PartCall->insertBefore(WC);
+
+ if (!LegalVariant->getReturnType()->isVoidTy())
+ PartialResults.push_back(PartCall);
+ }
+
+ // Combine partial results using a tree of ConcatVectors.
+ if (!PartialResults.empty()) {
+ while (PartialResults.size() > 1) {
+ SmallVector<VPValue *, 4> NewResults;
+ for (unsigned I = 0; I < PartialResults.size(); I += 2) {
+ if (I + 1 < PartialResults.size()) {
+ auto *Concat = new VPInstruction(VPInstruction::ConcatVectors,
+ {PartialResults[I],
+ PartialResults[I + 1]});
+ Concat->insertBefore(WC);
+ NewResults.push_back(Concat);
+ } else {
+ NewResults.push_back(PartialResults[I]);
+ }
+ }
+ PartialResults = std::move(NewResults);
+ }
+ WC->replaceAllUsesWith(PartialResults[0]);
+ }
+
+ WC->eraseFromParent();
+ }
+}
+
void VPlanTransforms::convertToConcreteRecipes(VPlan &Plan) {
VPTypeAnalysis TypeInfo(Plan);
SmallVector<VPRecipeBase *> ToRemove;
diff --git a/llvm/lib/Transforms/Vectorize/VPlanTransforms.h b/llvm/lib/Transforms/Vectorize/VPlanTransforms.h
index 75fc549167e03..1047f3ac95b2c 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanTransforms.h
+++ b/llvm/lib/Transforms/Vectorize/VPlanTransforms.h
@@ -354,6 +354,13 @@ struct VPlanTransforms {
// * Removes CanonicalIV and increment.
static void convertToVariableLengthStep(VPlan &Plan);
+ /// Legalize vector library calls by replacing VPWidenCallRecipes marked as
+ /// needing legalization with a sequence of ExtractSubvector + legal-width
+ /// VPWidenCallRecipe + ConcatVectors recipes. This splits oversized vector
+ /// calls (e.g., VF=8 sin on AVX) into multiple legal-width calls (e.g.,
+ /// 2 x VF=4 sin calls). \p VF is the chosen vectorization factor.
+ static void legalizeVecLibCalls(VPlan &Plan, ElementCount VF);
+
/// Lower abstract recipes to concrete ones, that can be codegen'd.
static void convertToConcreteRecipes(VPlan &Plan);
diff --git a/llvm/test/Transforms/LoopVectorize/VPlan/vplan-print-after-all.ll b/llvm/test/Transforms/LoopVectorize/VPlan/vplan-print-after-all.ll
index 061588317eba7..12f5cb721adbe 100644
--- a/llvm/test/Transforms/LoopVectorize/VPlan/vplan-print-after-all.ll
+++ b/llvm/test/Transforms/LoopVectorize/VPlan/vplan-print-after-all.ll
@@ -58,6 +58,7 @@
; CHECK: VPlan for loop in 'foo' after VPlanTransforms::simplifyRecipes
; CHECK: VPlan for loop in 'foo' after VPlanTransforms::removeBranchOnConst
; CHECK: VPlan for loop in 'foo' after VPlanTransforms::removeDeadRecipes
+; CHECK: VPlan for loop in 'foo' after VPlanTransforms::legalizeVecLibCalls
; CHECK: VPlan for loop in 'foo' after VPlanTransforms::convertToConcreteRecipes
; CHECK: VPlan for loop in 'foo' after VPlanTransforms::convertEVLExitCond
; CHECK: VPlan for loop in 'foo' after printFinalVPlan
diff --git a/llvm/test/Transforms/LoopVectorize/X86/veclib-legalize-amdlibm-float.ll b/llvm/test/Transforms/LoopVectorize/X86/veclib-legalize-amdlibm-float.ll
new file mode 100644
index 0000000000000..a490c92d1c1ec
--- /dev/null
+++ b/llvm/test/Transforms/LoopVectorize/X86/veclib-legalize-amdlibm-float.ll
@@ -0,0 +1,74 @@
+; RUN: opt -vector-library=AMDLIBM -passes=inject-tli-mappings,loop-vectorize \
+; RUN: -force-vector-width=16 -force-vector-interleave=1 -mattr=avx \
+; RUN: -legalize-vector-library-calls -S < %s | FileCheck %s
+
+; Verify that vector library call legalization works with AMDLIBM for single-
+; precision (float) functions. At VF=16 on AVX (256-bit), <16 x float> = 512
+; bits exceeds the register width. Each call is split into 2 x VF=8 calls
+; using amd_vrs8_* variants.
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+define void @amdlibm_sinf(ptr nocapture %varray) {
+; CHECK-LABEL: @amdlibm_sinf(
+; CHECK: vector.body:
+; CHECK: [[WIDE:%.*]] = sitofp <16 x i32> {{.*}} to <16 x float>
+; CHECK: [[EXT0:%.*]] = shufflevector <16 x float> [[WIDE]], <16 x float> poison, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
+; CHECK: [[CALL0:%.*]] = call <8 x float> @amd_vrs8_sinf(<8 x float> [[EXT0]])
+; CHECK: [[EXT1:%.*]] = shufflevector <16 x float> [[WIDE]], <16 x float> poison, <8 x i32> <i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
+; CHECK: [[CALL1:%.*]] = call <8 x float> @amd_vrs8_sinf(<8 x float> [[EXT1]])
+; CHECK: [[CONCAT:%.*]] = shufflevector <8 x float> [[CALL0]], <8 x float> [[CALL1]], <16 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
+; CHECK: store <16 x float> [[CONCAT]], ptr
+;
+entry:
+ br label %for.body
+
+for.body:
+ %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+ %tmp = trunc i64 %iv to i32
+ %conv = sitofp i32 %tmp to float
+ %call = tail call float @sinf(float %conv)
+ %arrayidx = getelementptr inbounds float, ptr %varray, i64 %iv
+ store float %call, ptr %arrayidx, align 4
+ %iv.next = add nuw nsw i64 %iv, 1
+ %exitcond = icmp eq i64 %iv.next, 1000
+ br i1 %exitcond, label %for.end, label %for.body
+
+for.end:
+ ret void
+}
+
+define void @amdlibm_expf(ptr nocapture %varray) {
+; CHECK-LABEL: @amdlibm_expf(
+; CHECK: vector.body:
+; CHECK: [[WIDE2:%.*]] = sitofp <16 x i32> {{.*}} to <16 x float>
+; CHECK: [[EXT2_0:%.*]] = shufflevector <16 x float> [[WIDE2]], <16 x float> poison, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
+; CHECK: [[ECALL0:%.*]] = call <8 x float> @amd_vrs8_expf(<8 x float> [[EXT2_0]])
+; CHECK: [[EXT2_1:%.*]] = shufflevector <16 x float> [[WIDE2]], <16 x float> poison, <8 x i32> <i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
+; CHECK: [[ECALL1:%.*]] = call <8 x float> @amd_vrs8_expf(<8 x float> [[EXT2_1]])
+; CHECK: [[ECONCAT:%.*]] = shufflevector <8 x float> [[ECALL0]], <8 x float> [[ECALL1]], <16 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
+; CHECK: store <16 x float> [[ECONCAT]], ptr
+;
+entry:
+ br label %for.body
+
+for.body:
+ %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+ %tmp = trunc i64 %iv to i32
+ %conv = sitofp i32 %tmp to float
+ %call = tail call float @expf(float %conv)
+ %arrayidx = getelementptr inbounds float, ptr %varray, i64 %iv
+ store float %call, ptr %arrayidx, align 4
+ %iv.next = add nuw nsw i64 %iv, 1
+ %exitcond = icmp eq i64 %iv.next, 1000
+ br i1 %exitcond, label %for.end, label %for.body
+
+for.end:
+ ret void
+}
+
+attributes #0 = { nounwind readnone }
+
+declare float @sinf(float) #0
+declare float @expf(float) #0
diff --git a/llvm/test/Transforms/LoopVectorize/X86/veclib-legalize-amdlibm.ll b/llvm/test/Transforms/LoopVectorize/X86/veclib-legalize-amdlibm.ll
new file mode 100644
index 0000000000000..843d0a303ae95
--- /dev/null
+++ b/llvm/test/Transforms/LoopVectorize/X86/veclib-legalize-amdlibm.ll
@@ -0,0 +1,103 @@
+; RUN: opt -vector-library=AMDLIBM -passes=inject-tli-mappings,loop-vectorize \
+; RUN: -force-vector-width=8 -force-vector-interleave=1 -mattr=avx \
+; RUN: -legalize-vector-library-calls -S < %s | FileCheck %s
+
+; Verify that vector library call legalization works with AMDLIBM.
+; At VF=8 on AVX (256-bit), <8 x double> = 512 bits exceeds the register
+; width. Each call is split into 2 x VF=4 calls using amd_vrd4_* variants.
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+define void @amdlibm_sin_f64(ptr nocapture %varray) {
+; CHECK-LABEL: @amdlibm_sin_f64(
+; CHECK: vector.body:
+; CHECK: [[WIDE:%.*]] = sitofp <8 x i32> {{.*}} to <8 x double>
+; CHECK: [[EXT0:%.*]] = shufflevector <8 x double> [[WIDE]], <8 x double> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; CHECK: [[CALL0:%.*]] = call <4 x double> @amd_vrd4_sin(<4 x double> [[EXT0]])
+; CHECK: [[EXT1:%.*]] = shufflevector <8 x double> [[WIDE]], <8 x double> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
+; CHECK: [[CALL1:%.*]] = call <4 x double> @amd_vrd4_sin(<4 x double> [[EXT1]])
+; CHECK: [[CONCAT:%.*]] = shufflevector <4 x double> [[CALL0]], <4 x double> [[CALL1]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
+; CHECK: store <8 x double> [[CONCAT]], ptr
+;
+entry:
+ br label %for.body
+
+for.body:
+ %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+ %tmp = trunc i64 %iv to i32
+ %conv = sitofp i32 %tmp to double
+ %call = tail call double @sin(double %conv)
+ %arrayidx = getelementptr inbounds double, ptr %varray, i64 %iv
+ store double %call, ptr %arrayidx, align 8
+ %iv.next = add nuw nsw i64 %iv, 1
+ %exitcond = icmp eq i64 %iv.next, 1000
+ br i1 %exitcond, label %for.end, label %for.body
+
+for.end:
+ ret void
+}
+
+define void @amdlibm_cos_f64(ptr nocapture %varray) {
+; CHECK-LABEL: @amdlibm_cos_f64(
+; CHECK: vector.body:
+; CHECK: [[WIDE2:%.*]] = sitofp <8 x i32> {{.*}} to <8 x double>
+; CHECK: [[EXT2_0:%.*]] = shufflevector <8 x double> [[WIDE2]], <8 x double> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; CHECK: [[CCALL0:%.*]] = call <4 x double> @amd_vrd4_cos(<4 x double> [[EXT2_0]])
+; CHECK: [[EXT2_1:%.*]] = shufflevector <8 x double> [[WIDE2]], <8 x double> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
+; CHECK: [[CCALL1:%.*]] = call <4 x double> @amd_vrd4_cos(<4 x double> [[EXT2_1]])
+; CHECK: [[CCONCAT:%.*]] = shufflevector <4 x double> [[CCALL0]], <4 x double> [[CCALL1]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
+; CHECK: store <8 x double> [[CCONCAT]], ptr
+;
+entry:
+ br label %for.body
+
+for.body:
+ %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+ %tmp = trunc i64 %iv to i32
+ %conv = sitofp i32 %tmp to double
+ %call = tail call double @cos(double %conv)
+ %arrayidx = getelementptr inbounds double, ptr %varray, i64 %iv
+ store double %call, ptr %arrayidx, align 8
+ %iv.next = add nuw nsw i64 %iv, 1
+ %exitcond = icmp eq i64 %iv.next, 1000
+ br i1 %exitcond, label %for.end, label %for.body
+
+for.end:
+ ret void
+}
+
+define void @amdlibm_exp_f64(ptr nocapture %varray) {
+; CHECK-LABEL: @amdlibm_exp_f64(
+; CHECK: vector.body:
+; CHECK: [[WIDE3:%.*]] = sitofp <8 x i32> {{.*}} to <8 x double>
+; CHECK: [[EXT3_0:%.*]] = shufflevector <8 x double> [[WIDE3]], <8 x double> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; CHECK: [[ECALL0:%.*]] = call <4 x double> @amd_vrd4_exp(<4 x double> [[EXT3_0]])
+; CHECK: [[EXT3_1:%.*]] = shufflevector <8 x double> [[WIDE3]], <8 x double> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
+; CHECK: [[ECALL1:%.*]] = call <4 x double> @amd_vrd4_exp(<4 x double> [[EXT3_1]])
+; CHECK: [[ECONCAT:%.*]] = shufflevector <4 x double> [[ECALL0]], <4 x double> [[ECALL1]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
+; CHECK: store <8 x double> [[ECONCAT]], ptr
+;
+entry:
+ br label %for.body
+
+for.body:
+ %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+ %tmp = trunc i64 %iv to i32
+ %conv = sitofp i32 %tmp to double
+ %call = tail call double @exp(double %conv)
+ %arrayidx = getelementptr inbounds double, ptr %varray, i64 %iv
+ store double %call, ptr %arrayidx, align 8
+ %iv.next = add nuw nsw i64 %iv, 1
+ %exitcond = icmp eq i64 %iv.next, 1000
+ br i1 %exitcond, label %for.end, label %for.body
+
+for.end:
+ ret void
+}
+
+attributes #0 = { nounwind readnone }
+
+declare double @sin(double) #0
+declare double @cos(double) #0
+declare double @exp(double) #0
diff --git a/llvm/test/Transforms/LoopVectorize/X86/veclib-legalize-basic.ll b/llvm/test/Transforms/LoopVectorize/X86/veclib-legalize-basic.ll
new file mode 100644
index 0000000000000..3c62e3ee3aa85
--- /dev/null
+++ b/llvm/test/Transforms/LoopVectorize/X86/veclib-legalize-basic.ll
@@ -0,0 +1,73 @@
+; RUN: opt -vector-library=SVML -passes=inject-tli-mappings,loop-vectorize \
+; RUN: -force-vector-width=8 -force-vector-interleave=1 -mattr=avx \
+; RUN: -legalize-vector-library-calls -S < %s | FileCheck %s
+
+; Verify that vector library call legalization splits an oversized sin(double)
+; call at VF=8 on AVX (256-bit) into two VF=4 calls to __svml_sin4, with the
+; appropriate extract-subvector and concat-vectors shuffles.
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+define void @sin_f64(ptr nocapture %varray) {
+; CHECK-LABEL: @sin_f64(
+; CHECK: vector.body:
+; CHECK: [[WIDE:%.*]] = sitofp <8 x i32> {{.*}} to <8 x double>
+; CHECK: [[EXT0:%.*]] = shufflevector <8 x double> [[WIDE]], <8 x double> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; CHECK: [[CALL0:%.*]] = call <4 x double> @__svml_sin4(<4 x double> [[EXT0]])
+; CHECK: [[EXT1:%.*]] = shufflevector <8 x double> [[WIDE]], <8 x double> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
+; CHECK: [[CALL1:%.*]] = call <4 x double> @__svml_sin4(<4 x double> [[EXT1]])
+; CHECK: [[CONCAT:%.*]] = shufflevector <4 x double> [[CALL0]], <4 x double> [[CALL1]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
+; CHECK: store <8 x double> [[CONCAT]], ptr
+;
+entry:
+ br label %for.body
+
+for.body:
+ %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+ %tmp = trunc i64 %iv to i32
+ %conv = sitofp i32 %tmp to double
+ %call = tail call double @sin(double %conv)
+ %arrayidx = getelementptr inbounds double, ptr %varray, i64 %iv
+ store double %call, ptr %arrayidx, align 8
+ %iv.next = add nuw nsw i64 %iv, 1
+ %exitcond = icmp eq i64 %iv.next, 1000
+ br i1 %exitcond, label %for.end, label %for.body
+
+for.end:
+ ret void
+}
+
+define void @cos_f64(ptr nocapture %varray) {
+; CHECK-LABEL: @cos_f64(
+; CHECK: vector.body:
+; CHECK: [[WIDE2:%.*]] = sitofp <8 x i32> {{.*}} to <8 x double>
+; CHECK: [[EXT2_0:%.*]] = shufflevector <8 x double> [[WIDE2]], <8 x double> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; CHECK: [[CCALL0:%.*]] = call <4 x double> @__svml_cos4(<4 x double> [[EXT2_0]])
+; CHECK: [[EXT2_1:%.*]] = shufflevector <8 x double> [[WIDE2]], <8 x double> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
+; CHECK: [[CCALL1:%.*]] = call <4 x double> @__svml_cos4(<4 x double> [[EXT2_1]])
+; CHECK: [[CCONCAT:%.*]] = shufflevector <4 x double> [[CCALL0]], <4 x double> [[CCALL1]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
+; CHECK: store <8 x double> [[CCONCAT]], ptr
+;
+entry:
+ br label %for.body
+
+for.body:
+ %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+ %tmp = trunc i64 %iv to i32
+ %conv = sitofp i32 %tmp to double
+ %call = tail call double @cos(double %conv)
+ %arrayidx = getelementptr inbounds double, ptr %varray, i64 %iv
+ store double %call, ptr %arrayidx, align 8
+ %iv.next = add nuw nsw i64 %iv, 1
+ %exitcond = icmp eq i64 %iv.next, 1000
+ br i1 %exitcond, label %for.end, label %for.body
+
+for.end:
+ ret void
+}
+
+attributes #0 = { nounwind readnone }
+
+declare double @sin(double) #0
+declare double @cos(double) #0
diff --git a/llvm/test/Transforms/LoopVectorize/X86/veclib-legalize-pow.ll b/llvm/test/Transforms/LoopVectorize/X86/veclib-legalize-pow.ll
new file mode 100644
index 0000000000000..c7c8c8973b2e2
--- /dev/null
+++ b/llvm/test/Transforms/LoopVectorize/X86/veclib-legalize-pow.ll
@@ -0,0 +1,52 @@
+; RUN: opt -vector-library=SVML -passes=inject-tli-mappings,loop-vectorize \
+; RUN: -force-vector-width=8 -force-vector-interleave=1 -mattr=avx \
+; RUN: -legalize-vector-library-calls -S < %s | FileCheck %s
+
+; Verify legalization of a two-argument function: pow(double, double) at VF=8
+; on AVX (256-bit) is split into 2 x VF=4 calls to __svml_pow4, with both
+; arguments properly extracted via sub-vector shuffles.
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+define void @pow_f64(ptr nocapture %varray, ptr nocapture readonly %exponents) {
+; CHECK-LABEL: @pow_f64(
+; CHECK: vector.body:
+;
+; First part: extract [0..3] from both wide args, call __svml_pow4.
+; CHECK: [[EXT_X0:%.*]] = shufflevector <8 x double> %{{.*}}, <8 x double> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; CHECK: [[EXT_Y0:%.*]] = shufflevector <8 x double> %{{.*}}, <8 x double> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
+; CHECK: [[CALL0:%.*]] = call fast <4 x double> @__svml_pow4(<4 x double> [[EXT_X0]], <4 x double> [[EXT_Y0]])
+;
+; Second part: extract [4..7] from both wide args, call __svml_pow4.
+; CHECK: [[EXT_X1:%.*]] = shufflevector <8 x double> %{{.*}}, <8 x double> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
+; CHECK: [[EXT_Y1:%.*]] = shufflevector <8 x double> %{{.*}}, <8 x double> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
+; CHECK: [[CALL1:%.*]] = call fast <4 x double> @__svml_pow4(<4 x double> [[EXT_X1]], <4 x double> [[EXT_Y1]])
+;
+; Concat partial results.
+; CHECK: [[CONCAT:%.*]] = shufflevector <4 x double> [[CALL0]], <4 x double> [[CALL1]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
+; CHECK: store <8 x double> [[CONCAT]], ptr
+;
+entry:
+ br label %for.body
+
+for.body:
+ %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+ %tmp = trunc i64 %iv to i32
+ %conv = sitofp i32 %tmp to double
+ %arrayidx.exp = getelementptr inbounds double, ptr %exponents, i64 %iv
+ %exp = load double, ptr %arrayidx.exp, align 8
+ %call = tail call fast double @pow(double %conv, double %exp)
+ %arrayidx = getelementptr inbounds double, ptr %varray, i64 %iv
+ store double %call, ptr %arrayidx, align 8
+ %iv.next = add nuw nsw i64 %iv, 1
+ %exitcond = icmp eq i64 %iv.next, 1000
+ br i1 %exitcond, label %for.end, label %for.body
+
+for.end:
+ ret void
+}
+
+attributes #0 = { nounwind readnone }
+
+declare double @pow(double, double) #0
More information about the llvm-commits
mailing list