[llvm] [SLP]Fix fmul/fadd fusion costs and retry FMA seeds after all blocks (PR #226117)

[via llvm-commits]

llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-backend-nvptx

Author: Alexey Bataev (alexey-bataev)

<details>
<summary>Changes</summary>

Fix the cost for the scalar of fmul+fadd pairs: fused fmul lanes add
the full fmul cost, fadd lanes with gathered fmuls were costed as
fmuladd, and the FMulAdd combine missed fmuls in the second operand,
c - a*b and constant lanes. Cost fused fmul lanes at (fmuladd - fadd),
such fadd lanes at (fmuladd - fmul), form the combine for either
operand, and treat a multi-use reduction root as fused. Extract nodes
take no credit for extracts the target folds into their users.


---
Full diff: https://github.com/llvm/llvm-project/pull/226117.diff


16 Files Affected:

- (modified) llvm/include/llvm/Transforms/Vectorize/SLPVectorizer.h (+4-2) 
- (modified) llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp (+274-67) 
- (modified) llvm/test/Transforms/PhaseOrdering/AArch64/reassociate-fma-pairs.ll (+17-17) 
- (modified) llvm/test/Transforms/SLPVectorizer/AArch64/extracts-folded-into-fmul-users-no-credit.ll (+5-6) 
- (modified) llvm/test/Transforms/SLPVectorizer/AArch64/fadd-with-gathered-fmul-operands.ll (+15-7) 
- (modified) llvm/test/Transforms/SLPVectorizer/AArch64/fma-candidates-after-store-chains.ll (+10-10) 
- (modified) llvm/test/Transforms/SLPVectorizer/AArch64/fma-chain-no-alt-node-reduction.ll (+31-30) 
- (modified) llvm/test/Transforms/SLPVectorizer/AArch64/fmul-constant-lane-fmuladd-combine.ll (+25-23) 
- (modified) llvm/test/Transforms/SLPVectorizer/AArch64/loop-accumulator-reduction.ll (+98-110) 
- (modified) llvm/test/Transforms/SLPVectorizer/AArch64/reduction-root-multi-use-fma.ll (+8-7) 
- (modified) llvm/test/Transforms/SLPVectorizer/AArch64/vec3-reorder-reshuffle.ll (+3-1) 
- (modified) llvm/test/Transforms/SLPVectorizer/NVPTX/ordered-reduction-fma-fusion.ll (+8-4) 
- (modified) llvm/test/Transforms/SLPVectorizer/X86/fmul-fused-into-scalar-fadd.ll (+13-13) 
- (modified) llvm/test/Transforms/SLPVectorizer/X86/fsub-fmul-rhs-combine.ll (+10-8) 
- (modified) llvm/test/Transforms/SLPVectorizer/X86/slp-fma-loss-ordered.ll (+6-12) 
- (modified) llvm/test/Transforms/SLPVectorizer/consecutive-access.ll (+22-41) 


``````````diff
The server is unavailable at this time. Please wait a few minutes before you try again.
``````````

</details>


https://github.com/llvm/llvm-project/pull/226117