[llvm] [AArch64] - Lower composite interleaved loads (factor 6, 8) to ld4+ld2/ld4+ld4. (PR #204871)
Eli Friedman via llvm-commits
llvm-commits at lists.llvm.org
Fri Jun 19 12:54:26 PDT 2026
================
@@ -5262,6 +5184,33 @@ InstructionCost AArch64TTIImpl::getInterleavedMemoryOpCost(
return Factor * TLI->getNumInterleavedAccesses(SubVecTy, DL, UseScalable);
}
+ // For composite factors (e.g. factor-6 = 2 x ld3), decompose into multiple
+ // ldN groups and return the cost of those loads. This is load-only for now
+ // composite factor store lowering (st3+st3) is not yet implemented, so we
+ // let stores fall through to BaseT:: which returns a conservative cost and
+ // prevents the LV from emitting interleaved store IR it can't lower.
+ // TODO: Extend support for composite factor stores as well.
+ if (Opcode == Instruction::Load && !UseMaskForGaps &&
+ !VecTy->isScalableTy() &&
+ Factor > TLI->getMaxSupportedInterleaveFactor()) {
+ unsigned MaxSupported = TLI->getMaxSupportedInterleaveFactor();
+ unsigned MinElts = VecVTy->getElementCount().getKnownMinValue();
+
+ for (unsigned i = MaxSupported; i >= 2; --i) {
+ if (Factor % i != 0)
+ continue;
+ unsigned NumGroups = Factor / i;
+ auto *SubVecTy = VectorType::get(
+ VecVTy->getElementType(),
+ VecVTy->getElementCount().divideCoefficientBy(Factor));
+ bool UseScalable;
+ if (MinElts % Factor == 0 &&
+ TLI->isLegalInterleavedAccessType(SubVecTy, DL, UseScalable))
+ return NumGroups * i *
----------------
efriedma-quic wrote:
This math could use a comment explaining it.
`NumGroups * i` is just `Factor`. getNumInterleavedAccesses() is, for NEON, basically the size of SubVectorTy divided by 128. So the cost of an "ld8" is 8, exactly twice the cost of an "ld4" according to this function. This seems overly optimistic, given that an ld4 is one instruction, and an "ld8" involves a bunch of shuffles.
https://github.com/llvm/llvm-project/pull/204871
More information about the llvm-commits
mailing list