[PATCH] D118538: [SLP] Schedule only sub-graph of vectorizable instructions

Sat Jan 29 07:50:42 PST 2022

reames created this revision.
reames added reviewers: ABataev, fhahn, nikic.
Herald added subscribers: asavonic, kerbowa, bollu, hiraditya, jvesely, mcrosier, qcolombet, jholewinski.
reames requested review of this revision.
Herald added a project: LLVM.

SLP currently schedules all instructions within a scheduling window which stretches from the first instruction potentially vectorized to the last.  This window can include a very large number of unrelated instructions which are not being considered for vectorization.  This change switches the code to only schedule the sub-graph consisting of the instructions being vectorized and their transitive users.

This has the effect of greatly reducing the amount of work performed in large basic blocks, and thus greatly improves compile time on degenerate examples.  To understand the effects, I added some statistics (not planned for upstream contribution).  Here's an illustration from my motivating example:

Before this patch:

704357 SLP                          - Number of calcDeps actions
 699021 SLP                          - Number of schedule calls

   5598 SLP                          - Number of ReSchedule actions
     59 SLP                          - Number of ReScheduleOnFail actions
  10084 SLP                          - Number of schedule resets
   8523 SLP                          - Number of vector instructions generated

After this patch:

102895 SLP                          - Number of calcDeps actions
 161916 SLP                          - Number of schedule calls

   5637 SLP                          - Number of ReSchedule actions
     55 SLP                          - Number of ReScheduleOnFail actions
  10083 SLP                          - Number of schedule resets
   8403 SLP                          - Number of vector instructions generated

I do want to highlight that there is a small difference in number of generated vector instructions.  I have to admit I'm confused by this, as in theory, the scheduling should not change this at all.  I have not been able to reduce an example that differs, and I don't see such a change in the tests.  Any ideas what might be going on here?

The downside of this change can be seen in the large test diff.  We group all vectorizable instructions together at the bottom of the scheduling region.  This means that vector instructions can move quite far from their original point in code.  While maybe undesirable, I don't see this as being a major problem as this pass is not intended to be a general scheduling pass.

For context, it's worth noting that the pre-scheduling that SLP does while building the vector tree is exactly the sub-graph scheduling implemented by this patch.

Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D118538

Files:
  llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
  llvm/test/Transforms/SLPVectorizer/AArch64/64-bit-vector.ll
  llvm/test/Transforms/SLPVectorizer/AArch64/commute.ll
  llvm/test/Transforms/SLPVectorizer/AArch64/gather-reduce.ll
  llvm/test/Transforms/SLPVectorizer/AArch64/horizontal.ll
  llvm/test/Transforms/SLPVectorizer/AArch64/loadi8.ll
  llvm/test/Transforms/SLPVectorizer/AArch64/matmul.ll
  llvm/test/Transforms/SLPVectorizer/AArch64/memory-runtime-checks.ll
  llvm/test/Transforms/SLPVectorizer/AArch64/sdiv-pow2.ll
  llvm/test/Transforms/SLPVectorizer/AArch64/slp-and-reduction.ll
  llvm/test/Transforms/SLPVectorizer/AArch64/slp-or-reduction.ll
  llvm/test/Transforms/SLPVectorizer/AArch64/slp-xor-reduction.ll
  llvm/test/Transforms/SLPVectorizer/AArch64/spillcost-di.ll
  llvm/test/Transforms/SLPVectorizer/AArch64/spillcost-order.ll
  llvm/test/Transforms/SLPVectorizer/AArch64/transpose-inseltpoison.ll
  llvm/test/Transforms/SLPVectorizer/AArch64/transpose.ll
  llvm/test/Transforms/SLPVectorizer/AArch64/tsc-s352.ll
  llvm/test/Transforms/SLPVectorizer/AArch64/widen.ll
  llvm/test/Transforms/SLPVectorizer/AMDGPU/packed-math.ll
  llvm/test/Transforms/SLPVectorizer/NVPTX/v2f16.ll
  llvm/test/Transforms/SLPVectorizer/SystemZ/pr34619.ll
  llvm/test/Transforms/SLPVectorizer/X86/PR32086.ll
  llvm/test/Transforms/SLPVectorizer/X86/PR39774.ll
  llvm/test/Transforms/SLPVectorizer/X86/addsub.ll
  llvm/test/Transforms/SLPVectorizer/X86/align.ll
  llvm/test/Transforms/SLPVectorizer/X86/arith-abs.ll
  llvm/test/Transforms/SLPVectorizer/X86/arith-add-ssat.ll
  llvm/test/Transforms/SLPVectorizer/X86/arith-add-usat.ll
  llvm/test/Transforms/SLPVectorizer/X86/arith-add.ll
  llvm/test/Transforms/SLPVectorizer/X86/arith-div.ll
  llvm/test/Transforms/SLPVectorizer/X86/arith-fix.ll
  llvm/test/Transforms/SLPVectorizer/X86/arith-mul.ll
  llvm/test/Transforms/SLPVectorizer/X86/arith-smax.ll
  llvm/test/Transforms/SLPVectorizer/X86/arith-smin.ll
  llvm/test/Transforms/SLPVectorizer/X86/arith-sub-ssat.ll
  llvm/test/Transforms/SLPVectorizer/X86/arith-sub-usat.ll
  llvm/test/Transforms/SLPVectorizer/X86/arith-sub.ll
  llvm/test/Transforms/SLPVectorizer/X86/arith-umax.ll
  llvm/test/Transforms/SLPVectorizer/X86/arith-umin.ll
  llvm/test/Transforms/SLPVectorizer/X86/bitreverse.ll
  llvm/test/Transforms/SLPVectorizer/X86/broadcast.ll
  llvm/test/Transforms/SLPVectorizer/X86/bswap.ll
  llvm/test/Transforms/SLPVectorizer/X86/cmp_commute-inseltpoison.ll
  llvm/test/Transforms/SLPVectorizer/X86/cmp_commute.ll
  llvm/test/Transforms/SLPVectorizer/X86/combined-stores-chains.ll
  llvm/test/Transforms/SLPVectorizer/X86/consecutive-access.ll
  llvm/test/Transforms/SLPVectorizer/X86/continue_vectorizing.ll
  llvm/test/Transforms/SLPVectorizer/X86/crash_exceed_scheduling.ll
  llvm/test/Transforms/SLPVectorizer/X86/crash_mandeltext.ll
  llvm/test/Transforms/SLPVectorizer/X86/crash_smallpt.ll
  llvm/test/Transforms/SLPVectorizer/X86/cse.ll
  llvm/test/Transforms/SLPVectorizer/X86/ctlz.ll
  llvm/test/Transforms/SLPVectorizer/X86/ctpop.ll
  llvm/test/Transforms/SLPVectorizer/X86/cttz.ll
  llvm/test/Transforms/SLPVectorizer/X86/diamond.ll
  llvm/test/Transforms/SLPVectorizer/X86/diamond_broadcast.ll
  llvm/test/Transforms/SLPVectorizer/X86/diamond_broadcast_extra_shuffle.ll
  llvm/test/Transforms/SLPVectorizer/X86/different-vec-widths.ll
  llvm/test/Transforms/SLPVectorizer/X86/dot-product.ll
  llvm/test/Transforms/SLPVectorizer/X86/extract_in_tree_user.ll
  llvm/test/Transforms/SLPVectorizer/X86/fabs.ll
  llvm/test/Transforms/SLPVectorizer/X86/fcopysign.ll
  llvm/test/Transforms/SLPVectorizer/X86/fma.ll
  llvm/test/Transforms/SLPVectorizer/X86/fmaxnum.ll
  llvm/test/Transforms/SLPVectorizer/X86/fminnum.ll
  llvm/test/Transforms/SLPVectorizer/X86/fmuladd.ll
  llvm/test/Transforms/SLPVectorizer/X86/fptosi-inseltpoison.ll
  llvm/test/Transforms/SLPVectorizer/X86/fptosi.ll
  llvm/test/Transforms/SLPVectorizer/X86/fptoui.ll
  llvm/test/Transforms/SLPVectorizer/X86/fround.ll
  llvm/test/Transforms/SLPVectorizer/X86/funclet.ll
  llvm/test/Transforms/SLPVectorizer/X86/gep.ll
  llvm/test/Transforms/SLPVectorizer/X86/horizontal-list.ll
  llvm/test/Transforms/SLPVectorizer/X86/horizontal-minmax.ll
  llvm/test/Transforms/SLPVectorizer/X86/horizontal.ll
  llvm/test/Transforms/SLPVectorizer/X86/insert-after-bundle.ll
  llvm/test/Transforms/SLPVectorizer/X86/insert-element-build-vector-inseltpoison.ll
  llvm/test/Transforms/SLPVectorizer/X86/insert-element-build-vector.ll
  llvm/test/Transforms/SLPVectorizer/X86/insert-shuffle.ll
  llvm/test/Transforms/SLPVectorizer/X86/insertvalue.ll
  llvm/test/Transforms/SLPVectorizer/X86/inst_size_bug.ll
  llvm/test/Transforms/SLPVectorizer/X86/intrinsic_with_scalar_param.ll
  llvm/test/Transforms/SLPVectorizer/X86/jumbled-load-shuffle-placement.ll
  llvm/test/Transforms/SLPVectorizer/X86/jumbled-load.ll
  llvm/test/Transforms/SLPVectorizer/X86/jumbled_store_crash.ll
  llvm/test/Transforms/SLPVectorizer/X86/load-merge-inseltpoison.ll
  llvm/test/Transforms/SLPVectorizer/X86/load-merge.ll
  llvm/test/Transforms/SLPVectorizer/X86/lookahead.ll
  llvm/test/Transforms/SLPVectorizer/X86/metadata.ll
  llvm/test/Transforms/SLPVectorizer/X86/multi_block.ll
  llvm/test/Transforms/SLPVectorizer/X86/phi_overalignedtype.ll
  llvm/test/Transforms/SLPVectorizer/X86/powof2div.ll
  llvm/test/Transforms/SLPVectorizer/X86/powof2mul.ll
  llvm/test/Transforms/SLPVectorizer/X86/pr35497.ll
  llvm/test/Transforms/SLPVectorizer/X86/pr47629-inseltpoison.ll
  llvm/test/Transforms/SLPVectorizer/X86/pr47629.ll
  llvm/test/Transforms/SLPVectorizer/X86/remark_horcost.ll
  llvm/test/Transforms/SLPVectorizer/X86/reorder_diamond_match.ll
  llvm/test/Transforms/SLPVectorizer/X86/resched.ll
  llvm/test/Transforms/SLPVectorizer/X86/return.ll
  llvm/test/Transforms/SLPVectorizer/X86/reuse-extracts-in-wider-vect.ll
  llvm/test/Transforms/SLPVectorizer/X86/schedule_budget.ll
  llvm/test/Transforms/SLPVectorizer/X86/scheduling.ll
  llvm/test/Transforms/SLPVectorizer/X86/shift-ashr.ll
  llvm/test/Transforms/SLPVectorizer/X86/shift-lshr.ll
  llvm/test/Transforms/SLPVectorizer/X86/shift-shl.ll
  llvm/test/Transforms/SLPVectorizer/X86/shrink_after_reorder.ll
  llvm/test/Transforms/SLPVectorizer/X86/simple-loop.ll
  llvm/test/Transforms/SLPVectorizer/X86/simplebb.ll
  llvm/test/Transforms/SLPVectorizer/X86/sitofp-inseltpoison.ll
  llvm/test/Transforms/SLPVectorizer/X86/sitofp.ll
  llvm/test/Transforms/SLPVectorizer/X86/split-load8_2-unord.ll
  llvm/test/Transforms/SLPVectorizer/X86/sqrt.ll
  llvm/test/Transforms/SLPVectorizer/X86/store-jumbled.ll
  llvm/test/Transforms/SLPVectorizer/X86/stores-non-ordered.ll
  llvm/test/Transforms/SLPVectorizer/X86/stores_vectorize.ll
  llvm/test/Transforms/SLPVectorizer/X86/tiny-tree.ll
  llvm/test/Transforms/SLPVectorizer/X86/uitofp.ll
  llvm/test/Transforms/SLPVectorizer/X86/vectorize-reorder-alt-shuffle.ll
  llvm/test/Transforms/SLPVectorizer/X86/vectorize-reordered-list.ll
  llvm/test/Transforms/SLPVectorizer/X86/vectorize-widest-phis.ll
  llvm/test/Transforms/SLPVectorizer/int_sideeffect.ll