[llvm] [LoopVectorize] Add metadata to distinguish vectorized loop body from scalar remainder (PR #190258)
via llvm-commits
llvm-commits at lists.llvm.org
Fri Apr 3 08:17:40 PDT 2026
llvmbot wrote:
<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-backend-powerpc
@llvm/pr-subscribers-backend-amdgpu
Author: Adel Ejjeh (adelejjeh)
<details>
<summary>Changes</summary>
Add two new loop metadata attributes — `llvm.loop.vectorize.vector_body` and `llvm.loop.vectorize.scalar_remainder` — that the loop vectorizer sets on the generated vector loop and scalar remainder loop respectively.
These enable downstream passes (LoopUnrollPass, WarnMissedTransforms, LoopUnroll) to produce more precise optimization remarks. Instead of the generic "loop not unrolled" warning on a source line that was vectorized, the unroller can now report:
- **"vectorized loop"** for the main vector body
- **"scalar remainder loop after vectorization"** for the scalar cleanup
- **"epilogue vectorized remainder loop"** for a remainder that was itself vectorized during epilogue vectorization (has both attributes)
### Four-way classification
The two independent boolean metadata provide a clean four-way classification:
| `vector_body` | `scalar_remainder` | Meaning | Remark wording |
|---|---|---|---|
| false | false | Plain loop | "loop" |
| true | false | Main vectorized loop | "vectorized loop" |
| false | true | Scalar remainder | "scalar remainder loop after vectorization" |
| true | true | Epilogue vectorized remainder | "epilogue vectorized remainder loop" |
During epilogue vectorization, pass 1 tags the remainder with `scalar_remainder`, then pass 2 vectorizes that remainder and tags the new vector loop with `vector_body`. The epilogue vector loop inherits `scalar_remainder` from its predecessor, giving it both flags — which uniquely identifies it.
### Motivation
When a loop with `#pragma unroll` gets vectorized, the vectorized main body and the scalar remainder both inherit the unroll metadata. The unroller then emits confusing diagnostics: "loop unrolled" for the main body and "loop not unrolled" for the remainder — both referencing the same source line with no indication of which is which. With this change, the user sees clearly that the "vectorized loop" was unrolled and the "scalar remainder loop after vectorization" was not.
### Files changed
- **Source (4 files):** `VPlan.cpp` (metadata tagging), `WarnMissedTransforms.cpp`, `LoopUnroll.cpp`, `LoopUnrollPass.cpp` (four-way remark classification)
- **Docs (2 files):** `LangRef.rst`, `TransformMetadata.rst`
- **Tests (97 files):** 5 new hand-written tests + 92 auto-updated CHECK linesf
Made with [Cursor](https://cursor.com)
---
Patch is 1.17 MiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/190258.diff
172 Files Affected:
- (modified) llvm/docs/LangRef.rst (+42)
- (modified) llvm/docs/TransformMetadata.rst (+21)
- (modified) llvm/lib/Transforms/Scalar/WarnMissedTransforms.cpp (+19-7)
- (modified) llvm/lib/Transforms/Utils/LoopUnroll.cpp (+26-5)
- (modified) llvm/lib/Transforms/Vectorize/VPlan.cpp (+25)
- (modified) llvm/test/CodeGen/AArch64/aggressive-interleaving.ll (+14-12)
- (modified) llvm/test/CodeGen/AMDGPU/vectorize-unroll-metadata.ll (+47-12)
- (added) llvm/test/Transforms/LoopTransformWarning/amdgpu-vectorized-remainder-unroll-warning.ll (+59)
- (added) llvm/test/Transforms/LoopTransformWarning/vectorized-remainder-unroll-warning.ll (+69)
- (added) llvm/test/Transforms/LoopUnroll/remainder-loop-remarks.ll (+61)
- (added) llvm/test/Transforms/LoopUnroll/vectorize-then-unroll-remarks.ll (+42)
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/blend-costs.ll (+12-10)
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/call-costs.ll (+6-4)
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/check-prof-info.ll (+127-119)
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/clmul.ll (+6-4)
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/divs-with-scalable-vfs.ll (+14-12)
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/f128-fmuladd-reduction.ll (+18-16)
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/fixed-wide-lane-mask.ll (+6-4)
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/fminimumnum.ll (+26-24)
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/gather-cost.ll (+6-4)
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/invariant-replicate-region.ll (+3-2)
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/load-cast-context.ll (+16-14)
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/low_trip_count_predicates.ll (+33-29)
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/masked_ldst_sme.ll (+6-4)
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/mul-simplification.ll (+3-2)
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-interleave.ll (+12-8)
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/pr162009.ll (+6-4)
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/pr73894.ll (+3-2)
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/predicated-costs.ll (+13-11)
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/simple_early_exit.ll (+20-18)
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/store-costs-sve.ll (+48-44)
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-fixed-width-inorder-core.ll (+18-12)
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-vscale-based-trip-counts.ll (+25-23)
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/transform-narrow-interleave-to-widen-memory-metadata.ll (+6-4)
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/vector-loop-backedge-elimination-epilogue.ll (+5-4)
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/widen-gep-all-indices-invariant.ll (+3-2)
- (modified) llvm/test/Transforms/LoopVectorize/ARM/active-lane-mask.ll (+3-2)
- (modified) llvm/test/Transforms/LoopVectorize/ARM/gather-cost.ll (+6-4)
- (modified) llvm/test/Transforms/LoopVectorize/ARM/tail-folding-loop-hint.ll (+6-4)
- (modified) llvm/test/Transforms/LoopVectorize/LoongArch/defaults.ll (+3-2)
- (modified) llvm/test/Transforms/LoopVectorize/PowerPC/exit-branch-cost.ll (+10-8)
- (modified) llvm/test/Transforms/LoopVectorize/PowerPC/large-loop-rdx.ll (+10-8)
- (modified) llvm/test/Transforms/LoopVectorize/PowerPC/small-loop-rdx.ll (+12-10)
- (modified) llvm/test/Transforms/LoopVectorize/PowerPC/vplan-scalarivsext-crash.ll (+6-4)
- (modified) llvm/test/Transforms/LoopVectorize/PowerPC/vsx-tsvc-s173.ll (+3-2)
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/blocks-with-dead-instructions.ll (+29-27)
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/cse.ll (+3-2)
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/dbg-tail-folding-by-evl.ll (+3-2)
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/dead-ops-cost.ll (+37-35)
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/evl-compatible-loops.ll (+5-4)
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/first-order-recurrence-scalable-vf1.ll (+3-2)
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/fminimumnum.ll (+52-48)
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/induction-costs.ll (+10-8)
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/pr87378-vpinstruction-or-drop-poison-generating-flags.ll (+3-2)
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/pr88802.ll (+3-2)
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/remark-reductions.ll (+3-2)
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-bin-unary-ops-args.ll (+148-144)
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-call-intrinsics.ll (+80-76)
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-cast-intrinsics.ll (+146-142)
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-cond-reduction.ll (+54-48)
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-div.ll (+51-48)
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-fixed-order-recurrence.ll (+33-30)
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-intermediate-store.ll (+36-28)
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-known-no-overflow.ll (+7-6)
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/truncate-to-minimal-bitwidth-cost.ll (+9-8)
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/truncate-to-minimal-bitwidth-evl-crash.ll (+8-6)
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/type-info-cache-evl-crash.ll (+6-4)
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/veclib-function-calls.ll (+243-241)
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/vf-will-not-generate-any-vector-insts.ll (+6-4)
- (modified) llvm/test/Transforms/LoopVectorize/SystemZ/force-target-instruction-cost.ll (+3-2)
- (modified) llvm/test/Transforms/LoopVectorize/WebAssembly/induction-branch-cost.ll (+6-4)
- (modified) llvm/test/Transforms/LoopVectorize/X86/already-vectorized.ll (+4-2)
- (modified) llvm/test/Transforms/LoopVectorize/X86/clmul.ll (+6-4)
- (modified) llvm/test/Transforms/LoopVectorize/X86/cost-constant-known-via-scev.ll (+8-6)
- (modified) llvm/test/Transforms/LoopVectorize/X86/divs-with-tail-folding.ll (+10-8)
- (modified) llvm/test/Transforms/LoopVectorize/X86/fminimumnum.ll (+26-24)
- (modified) llvm/test/Transforms/LoopVectorize/X86/gep-use-outside-loop.ll (+10-8)
- (modified) llvm/test/Transforms/LoopVectorize/X86/induction-costs.ll (+79-77)
- (modified) llvm/test/Transforms/LoopVectorize/X86/interleave-cost.ll (+43-41)
- (modified) llvm/test/Transforms/LoopVectorize/X86/interleave-ptradd-with-replicated-operand.ll (+6-4)
- (modified) llvm/test/Transforms/LoopVectorize/X86/masked-store-cost.ll (+18-16)
- (modified) llvm/test/Transforms/LoopVectorize/X86/multi-exit-cost.ll (+16-14)
- (modified) llvm/test/Transforms/LoopVectorize/X86/nontemporal.ll (+2-2)
- (modified) llvm/test/Transforms/LoopVectorize/X86/pr109581-unused-blend.ll (+6-4)
- (modified) llvm/test/Transforms/LoopVectorize/X86/pr54634.ll (+12-10)
- (modified) llvm/test/Transforms/LoopVectorize/X86/pr55096-scalarize-add.ll (+3-2)
- (modified) llvm/test/Transforms/LoopVectorize/X86/pr72969.ll (+6-4)
- (modified) llvm/test/Transforms/LoopVectorize/X86/pr81872.ll (+4-3)
- (modified) llvm/test/Transforms/LoopVectorize/X86/predicate-switch.ll (+56-52)
- (modified) llvm/test/Transforms/LoopVectorize/X86/replicating-load-store-costs-max-bandwidth.ll (+6-4)
- (modified) llvm/test/Transforms/LoopVectorize/X86/strided_load_cost.ll (+28-24)
- (modified) llvm/test/Transforms/LoopVectorize/X86/transform-narrow-interleave-to-widen-memory.ll (+42-40)
- (modified) llvm/test/Transforms/LoopVectorize/X86/vplan-native-inner-loop-only.ll (+6-4)
- (modified) llvm/test/Transforms/LoopVectorize/X86/widened-value-used-as-scalar-and-first-lane.ll (+5-4)
- (modified) llvm/test/Transforms/LoopVectorize/blend-in-header.ll (+18-16)
- (modified) llvm/test/Transforms/LoopVectorize/branch-weights.ll (+37-33)
- (modified) llvm/test/Transforms/LoopVectorize/dbg-outer-loop-vect.ll (+10-8)
- (modified) llvm/test/Transforms/LoopVectorize/dead_instructions.ll (+27-25)
- (modified) llvm/test/Transforms/LoopVectorize/epilog-vectorization-any-of-reductions.ll (+31-29)
- (modified) llvm/test/Transforms/LoopVectorize/epilog-vectorization-trunc-induction-steps.ll (+10-8)
- (modified) llvm/test/Transforms/LoopVectorize/exact.ll (+4-3)
- (modified) llvm/test/Transforms/LoopVectorize/extract-from-end-vector-constant.ll (+10-8)
- (modified) llvm/test/Transforms/LoopVectorize/first-order-recurrence-interleave-only.ll (+3-2)
- (modified) llvm/test/Transforms/LoopVectorize/first-order-recurrence-scalable-vf1.ll (+10-8)
- (modified) llvm/test/Transforms/LoopVectorize/followup.ll (+5-3)
- (modified) llvm/test/Transforms/LoopVectorize/hints-trans.ll (+8-6)
- (modified) llvm/test/Transforms/LoopVectorize/if-reduction.ll (+62-60)
- (modified) llvm/test/Transforms/LoopVectorize/induction-step.ll (+27-25)
- (modified) llvm/test/Transforms/LoopVectorize/interleave-with-i65-induction.ll (+3-2)
- (modified) llvm/test/Transforms/LoopVectorize/interleaved-accesses-2.ll (+6-4)
- (modified) llvm/test/Transforms/LoopVectorize/interleaved-accesses-different-insert-position.ll (+12-10)
- (modified) llvm/test/Transforms/LoopVectorize/interleaved-accesses-metadata.ll (+26-24)
- (modified) llvm/test/Transforms/LoopVectorize/invalidate-scev-at-scope-after-vectorization.ll (+10-8)
- (modified) llvm/test/Transforms/LoopVectorize/iv-select-cmp-nested-loop.ll (+18-12)
- (modified) llvm/test/Transforms/LoopVectorize/iv-select-cmp-no-wrap.ll (+18-16)
- (modified) llvm/test/Transforms/LoopVectorize/iv-select-cmp-non-const-iv-start.ll (+30-24)
- (modified) llvm/test/Transforms/LoopVectorize/make-followup-loop-id.ll (+8-6)
- (modified) llvm/test/Transforms/LoopVectorize/metadata.ll (+68-64)
- (modified) llvm/test/Transforms/LoopVectorize/minimumnum-maximumnum-reductions.ll (+9-8)
- (modified) llvm/test/Transforms/LoopVectorize/multi_early_exit.ll (+3-2)
- (modified) llvm/test/Transforms/LoopVectorize/no-fold-tail-by-masking-iv-external-uses.ll (+6-4)
- (modified) llvm/test/Transforms/LoopVectorize/no_outside_user.ll (+30-28)
- (modified) llvm/test/Transforms/LoopVectorize/noalias-scope-decl.ll (+16-14)
- (modified) llvm/test/Transforms/LoopVectorize/non-widenable-intrinsics-outer-loop.ll (+22-20)
- (modified) llvm/test/Transforms/LoopVectorize/optimal-epilog-vectorization.ll (+89-85)
- (modified) llvm/test/Transforms/LoopVectorize/optsize.ll (+56-50)
- (modified) llvm/test/Transforms/LoopVectorize/predicate-switch.ll (+37-33)
- (modified) llvm/test/Transforms/LoopVectorize/preserve-dbg-loc-and-loop-metadata.ll (+276-272)
- (modified) llvm/test/Transforms/LoopVectorize/preserve-or-disjoint.ll (+6-4)
- (modified) llvm/test/Transforms/LoopVectorize/remarks-reduction-inloop.ll (+3-2)
- (modified) llvm/test/Transforms/LoopVectorize/remove_metadata.ll (+8-3)
- (modified) llvm/test/Transforms/LoopVectorize/reverse_induction.ll (+14-12)
- (modified) llvm/test/Transforms/LoopVectorize/runtime-check-needed-but-empty.ll (+10-8)
- (modified) llvm/test/Transforms/LoopVectorize/runtime-check-threshold-with-force-metadata.ll (+6-4)
- (modified) llvm/test/Transforms/LoopVectorize/scalable-iv-outside-user.ll (+6-4)
- (modified) llvm/test/Transforms/LoopVectorize/select-cmp-multiuse.ll (+102-96)
- (modified) llvm/test/Transforms/LoopVectorize/select-cmp-predicated.ll (+12-8)
- (modified) llvm/test/Transforms/LoopVectorize/select-cmp.ll (+102-96)
- (modified) llvm/test/Transforms/LoopVectorize/select-folds.ll (+6-4)
- (modified) llvm/test/Transforms/LoopVectorize/select-with-fastflags.ll (+6-4)
- (modified) llvm/test/Transforms/LoopVectorize/single-early-exit-interleave-hint.ll (+3-2)
- (modified) llvm/test/Transforms/LoopVectorize/single_early_exit.ll (+24-22)
- (modified) llvm/test/Transforms/LoopVectorize/single_early_exit_live_outs.ll (+56-54)
- (modified) llvm/test/Transforms/LoopVectorize/store-reduction-results-in-tail-folded-loop.ll (+3-2)
- (modified) llvm/test/Transforms/LoopVectorize/tail-folding-alloca-in-loop.ll (+3-2)
- (modified) llvm/test/Transforms/LoopVectorize/tail-folding-div.ll (+15-14)
- (modified) llvm/test/Transforms/LoopVectorize/tail-folding-switch.ll (+3-2)
- (modified) llvm/test/Transforms/LoopVectorize/trip-count-expansion-may-introduce-ub.ll (+62-60)
- (modified) llvm/test/Transforms/LoopVectorize/tripcount.ll (+21-19)
- (modified) llvm/test/Transforms/LoopVectorize/trunc-extended-icmps.ll (+12-10)
- (modified) llvm/test/Transforms/LoopVectorize/trunc-loads-p16.ll (+3-2)
- (modified) llvm/test/Transforms/LoopVectorize/uniform-blend.ll (+13-12)
- (modified) llvm/test/Transforms/LoopVectorize/unused-blend-mask-for-first-operand.ll (+7-6)
- (modified) llvm/test/Transforms/LoopVectorize/use-iv-start-value.ll (+6-4)
- (modified) llvm/test/Transforms/LoopVectorize/vector-loop-backedge-elimination.ll (+22-18)
- (modified) llvm/test/Transforms/LoopVectorize/vectorize-once.ll (+78-15)
- (modified) llvm/test/Transforms/LoopVectorize/version-stride-with-integer-casts.ll (+26-24)
- (modified) llvm/test/Transforms/LoopVectorize/vplan-native-path-inner-loop-with-runtime-checks.ll (+6-4)
- (modified) llvm/test/Transforms/LoopVectorize/vplan-widen-call-instruction.ll (+9-7)
- (modified) llvm/test/Transforms/PhaseOrdering/AArch64/hoist-runtime-checks.ll (+13-11)
- (modified) llvm/test/Transforms/PhaseOrdering/AArch64/indvars-vectorization.ll (+6-4)
- (modified) llvm/test/Transforms/PhaseOrdering/AArch64/interleave_vec.ll (+20-18)
- (modified) llvm/test/Transforms/PhaseOrdering/AArch64/loop-rotate-to-enable-unrolling-and-vectorization.ll (+3-2)
- (modified) llvm/test/Transforms/PhaseOrdering/AArch64/predicated-reduction.ll (+10-8)
- (modified) llvm/test/Transforms/PhaseOrdering/AArch64/std-find.ll (+6-4)
- (modified) llvm/test/Transforms/PhaseOrdering/X86/SROA-after-final-loop-unrolling-2.ll (+2-1)
- (modified) llvm/test/Transforms/PhaseOrdering/X86/hoist-load-of-baseptr.ll (+16-12)
- (modified) llvm/test/Transforms/PhaseOrdering/X86/pr88239.ll (+3-2)
- (modified) llvm/test/Transforms/PhaseOrdering/X86/preserve-access-group.ll (+6-4)
- (modified) llvm/test/Transforms/PhaseOrdering/X86/speculation-vs-tbaa.ll (+6-4)
- (modified) llvm/test/Transforms/PhaseOrdering/X86/vdiv-nounroll.ll (+3-2)
- (modified) llvm/test/Transforms/PhaseOrdering/X86/vdiv.ll (+13-11)
``````````diff
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 5584e0828d3cd..c7423028174c0 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -7936,6 +7936,48 @@ Attributes in the metadata will be added to both the vectorized and
epilogue loop.
See :ref:`Transformation Metadata <transformation-metadata>` for details.
+'``llvm.loop.vectorize.vector_body``' Metadata
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This metadata is automatically added by the loop vectorizer to the
+vectorized loop body. It is used by subsequent optimization passes
+(such as the loop unroller and ``WarnMissedTransforms``) to emit more
+precise optimization remarks that identify the loop as a vector loop.
+
+The first operand is the string
+``llvm.loop.vectorize.vector_body`` and the second operand is an
+integer. A value of 1 indicates this is a vectorized loop body:
+
+.. code-block:: llvm
+
+ !0 = !{!"llvm.loop.vectorize.vector_body", i32 1}
+
+'``llvm.loop.vectorize.scalar_remainder``' Metadata
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This metadata is automatically added by the loop vectorizer to the
+scalar remainder (epilogue) loop to identify it as such. It is used by
+subsequent optimization passes (such as the loop unroller and
+``WarnMissedTransforms``) to emit more precise optimization remarks
+that distinguish between the vectorized loop and its scalar remainder.
+
+Together these two attributes provide a four-way classification:
+
+- ``vector_body`` only: main vectorized loop body
+- ``scalar_remainder`` only: scalar remainder loop after vectorization
+- Both ``vector_body`` and ``scalar_remainder``: epilogue vectorized
+ remainder (a remainder loop that was itself vectorized during epilogue
+ vectorization)
+- Neither: a plain loop not produced by the vectorizer
+
+The first operand is the string
+``llvm.loop.vectorize.scalar_remainder`` and the second operand is
+an integer. A value of 1 indicates the loop is a remainder loop:
+
+.. code-block:: llvm
+
+ !0 = !{!"llvm.loop.vectorize.scalar_remainder", i32 1}
+
'``llvm.loop.unroll``'
^^^^^^^^^^^^^^^^^^^^^^
diff --git a/llvm/docs/TransformMetadata.rst b/llvm/docs/TransformMetadata.rst
index 1f9f50f1f33bc..161648c1b2a9f 100644
--- a/llvm/docs/TransformMetadata.rst
+++ b/llvm/docs/TransformMetadata.rst
@@ -168,6 +168,27 @@ has unroll metadata).
The attributes specified by ``llvm.loop.vectorize.followup_all`` are
added to both loops.
+In addition to the above, the vectorizer automatically annotates the
+generated loops with two metadata attributes for remark quality:
+
+- ``llvm.loop.vectorize.vector_body`` is added to the vectorized loop.
+- ``llvm.loop.vectorize.scalar_remainder`` is added to the scalar
+ remainder (epilogue) loop.
+
+Together these provide a four-way classification:
+
+- ``vector_body`` only: main vectorized loop body
+- ``scalar_remainder`` only: scalar remainder loop after vectorization
+- Both: epilogue vectorized remainder (a remainder that was itself
+ vectorized during epilogue vectorization)
+- Neither: a plain loop not produced by the vectorizer
+
+This is used by subsequent passes (e.g., the loop unroller and
+``WarnMissedTransforms``) to produce more precise optimization remarks.
+For instance, instead of reporting both "loop unrolled" and "loop not
+unrolled" for the same source line, the unroller can clarify that a
+*vectorized* loop was unrolled while its *scalar remainder* was not.
+
When using a follow-up attribute, it replaces any automatically deduced
attributes for the generated loop in question. Therefore it is
recommended to add ``llvm.loop.isvectorized`` to
diff --git a/llvm/lib/Transforms/Scalar/WarnMissedTransforms.cpp b/llvm/lib/Transforms/Scalar/WarnMissedTransforms.cpp
index e53019768e881..4e51a7f64e5a6 100644
--- a/llvm/lib/Transforms/Scalar/WarnMissedTransforms.cpp
+++ b/llvm/lib/Transforms/Scalar/WarnMissedTransforms.cpp
@@ -25,13 +25,25 @@ static void warnAboutLeftoverTransformations(Loop *L,
OptimizationRemarkEmitter *ORE) {
if (hasUnrollTransformation(L) == TM_ForcedByUser) {
LLVM_DEBUG(dbgs() << "Leftover unroll transformation\n");
- ORE->emit(
- DiagnosticInfoOptimizationFailure(DEBUG_TYPE,
- "FailedRequestedUnrolling",
- L->getStartLoc(), L->getHeader())
- << "loop not unrolled: the optimizer was unable to perform the "
- "requested transformation; the transformation might be disabled or "
- "specified as part of an unsupported transformation ordering");
+ bool IsVectorBody =
+ getBooleanLoopAttribute(L, "llvm.loop.vectorize.vector_body");
+ bool IsScalarRemainder =
+ getBooleanLoopAttribute(L, "llvm.loop.vectorize.scalar_remainder");
+ const char *LoopKind = (IsVectorBody && IsScalarRemainder)
+ ? "epilogue vectorized remainder loop"
+ : IsScalarRemainder
+ ? "scalar remainder loop after vectorization"
+ : IsVectorBody ? "vectorized loop"
+ : "loop";
+ std::string Msg =
+ std::string(LoopKind) +
+ " not unrolled: the optimizer was unable to perform the "
+ "requested transformation; the transformation might be disabled "
+ "or specified as part of an unsupported transformation ordering";
+ ORE->emit(DiagnosticInfoOptimizationFailure(
+ DEBUG_TYPE, "FailedRequestedUnrolling", L->getStartLoc(),
+ L->getHeader())
+ << Msg);
}
if (hasUnrollAndJamTransformation(L) == TM_ForcedByUser) {
diff --git a/llvm/lib/Transforms/Utils/LoopUnroll.cpp b/llvm/lib/Transforms/Utils/LoopUnroll.cpp
index ab35d217f0d93..d135989f73f42 100644
--- a/llvm/lib/Transforms/Utils/LoopUnroll.cpp
+++ b/llvm/lib/Transforms/Utils/LoopUnroll.cpp
@@ -608,16 +608,36 @@ llvm::UnrollLoop(Loop *L, UnrollLoopOptions ULO, LoopInfo *LI,
}
using namespace ore;
+
+ // Determine whether this loop originated from the vectorizer so we can
+ // produce more informative remarks.
+ bool IsVectorBody =
+ getBooleanLoopAttribute(L, "llvm.loop.vectorize.vector_body");
+ bool IsScalarRemainder =
+ getBooleanLoopAttribute(L, "llvm.loop.vectorize.scalar_remainder");
+
+ // Four-way classification:
+ // vector_body && scalar_remainder → epilogue vectorized remainder
+ // !vector_body && scalar_remainder → scalar remainder after vectorization
+ // vector_body && !scalar_remainder → main vectorized loop
+ // neither → plain loop
+ const char *LoopKind =
+ (IsVectorBody && IsScalarRemainder) ? "epilogue vectorized remainder loop"
+ : IsScalarRemainder ? "scalar remainder loop after vectorization"
+ : IsVectorBody ? "vectorized loop"
+ : "loop";
+
// Report the unrolling decision.
if (CompletelyUnroll) {
LLVM_DEBUG(dbgs() << "COMPLETELY UNROLLING loop %" << Header->getName()
<< " with trip count " << ULO.Count << "!\n");
if (ORE)
ORE->emit([&]() {
- return OptimizationRemark(DEBUG_TYPE, "FullyUnrolled", L->getStartLoc(),
- L->getHeader())
- << "completely unrolled loop with "
- << NV("UnrollCount", ULO.Count) << " iterations";
+ OptimizationRemark Diag(DEBUG_TYPE, "FullyUnrolled", L->getStartLoc(),
+ L->getHeader());
+ Diag << "completely unrolled " << LoopKind << " with ";
+ Diag << NV("UnrollCount", ULO.Count) << " iterations";
+ return Diag;
});
} else {
LLVM_DEBUG(dbgs() << "UNROLLING loop %" << Header->getName() << " by "
@@ -630,7 +650,8 @@ llvm::UnrollLoop(Loop *L, UnrollLoopOptions ULO, LoopInfo *LI,
ORE->emit([&]() {
OptimizationRemark Diag(DEBUG_TYPE, "PartialUnrolled", L->getStartLoc(),
L->getHeader());
- Diag << "unrolled loop by a factor of " << NV("UnrollCount", ULO.Count);
+ Diag << "unrolled " << LoopKind << " by a factor of ";
+ Diag << NV("UnrollCount", ULO.Count);
if (ULO.Runtime)
Diag << " with run-time trip count";
return Diag;
diff --git a/llvm/lib/Transforms/Vectorize/VPlan.cpp b/llvm/lib/Transforms/Vectorize/VPlan.cpp
index 919166099a4d1..8b2d1dbe42ed8 100644
--- a/llvm/lib/Transforms/Vectorize/VPlan.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlan.cpp
@@ -65,6 +65,19 @@ const char LLVMLoopVectorizeFollowupEpilogue[] =
"llvm.loop.vectorize.followup_epilogue";
/// @}
+/// Add a boolean metadata attribute to a loop's loop-ID node.
+static void addBooleanLoopAttribute(Loop *L, StringRef Name) {
+ LLVMContext &Context = L->getHeader()->getContext();
+ MDNode *AttrMD = MDNode::get(
+ Context,
+ {MDString::get(Context, Name),
+ ConstantAsMetadata::get(ConstantInt::get(Context, APInt(32, 1)))});
+ MDNode *LoopID = L->getLoopID();
+ MDNode *NewLoopID =
+ makePostTransformationMetadata(Context, LoopID, {}, {AttrMD});
+ L->setLoopID(NewLoopID);
+}
+
extern cl::opt<unsigned> ForceTargetInstructionCost;
extern cl::opt<unsigned> NumberOfStoresToPredicate;
@@ -1703,6 +1716,12 @@ void LoopVectorizationPlanner::updateLoopMetadataAndProfileInfo(
Hints.setAlreadyVectorized();
}
}
+ // Tag the scalar remainder so downstream passes (e.g. the unroller and
+ // WarnMissedTransforms) can produce more informative remarks. This is set
+ // unconditionally (including during epilogue vectorization) so that the
+ // final scalar tail is always identifiable.
+ if (Plan.getScalarPreheader()->hasPredecessors())
+ addBooleanLoopAttribute(OrigLoop, "llvm.loop.vectorize.scalar_remainder");
if (!VectorLoop)
return;
@@ -1722,6 +1741,12 @@ void LoopVectorizationPlanner::updateLoopMetadataAndProfileInfo(
Hints.setAlreadyVectorized();
}
}
+ // Tag the vector loop body so downstream passes can identify it. During
+ // epilogue vectorization the epilogue vector loop naturally inherits
+ // scalar_remainder from the pass-1 remainder and gets vector_body here,
+ // giving it both tags — which uniquely identifies it as an epilogue
+ // vectorized remainder.
+ addBooleanLoopAttribute(VectorLoop, "llvm.loop.vectorize.vector_body");
TargetTransformInfo::UnrollingPreferences UP;
TTI.getUnrollingPreferences(VectorLoop, *PSE.getSE(), UP, ORE);
if (!UP.UnrollVectorizedLoop || VectorizingEpilogue)
diff --git a/llvm/test/CodeGen/AArch64/aggressive-interleaving.ll b/llvm/test/CodeGen/AArch64/aggressive-interleaving.ll
index 57b4703073e81..948bd3d76cf52 100644
--- a/llvm/test/CodeGen/AArch64/aggressive-interleaving.ll
+++ b/llvm/test/CodeGen/AArch64/aggressive-interleaving.ll
@@ -90,7 +90,7 @@ define void @test_interleave_reduction(ptr %arg, ptr %arg1) {
; A320-NEXT: [[NEXT_I32]] = getelementptr inbounds i32, ptr [[PHI_PTR_I32]], i64 1
; A320-NEXT: [[NEXT_F64]] = getelementptr inbounds double, ptr [[PHI_PTR_F64]], i64 1
; A320-NEXT: [[DONE:%.*]] = icmp eq ptr [[NEXT_I32]], [[TPM29]]
-; A320-NEXT: br i1 [[DONE]], label %[[EXIT_INNER]], label %[[INNER]], !llvm.loop [[LOOP3:![0-9]+]]
+; A320-NEXT: br i1 [[DONE]], label %[[EXIT_INNER]], label %[[INNER]], !llvm.loop [[LOOP4:![0-9]+]]
; A320: [[EXIT_INNER]]:
; A320-NEXT: [[TPM50_LCSSA:%.*]] = phi double [ [[TPM50]], %[[INNER]] ], [ [[BIN_RDX]], %[[MIDDLE_BLOCK]] ]
; A320-NEXT: [[TPM35:%.*]] = getelementptr inbounds double, ptr [[TPM19]], i64 0
@@ -169,7 +169,7 @@ define double @sum_reduction(ptr nocapture readonly %a, i64 %n) {
; A320-NEXT: [[TMP4]] = fadd fast <2 x double> [[VEC_PHI1]], [[WIDE_LOAD2]]
; A320-NEXT: [[INDEX_NEXT]] = add nuw i64 [[TMP1]], 4
; A320-NEXT: [[TMP8:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
-; A320-NEXT: br i1 [[TMP8]], label %[[MIDDLE_BLOCK:.*]], label %[[VECTOR_BODY]], !llvm.loop [[LOOP4:![0-9]+]]
+; A320-NEXT: br i1 [[TMP8]], label %[[MIDDLE_BLOCK:.*]], label %[[VECTOR_BODY]], !llvm.loop [[LOOP6:![0-9]+]]
; A320: [[MIDDLE_BLOCK]]:
; A320-NEXT: [[BIN_RDX:%.*]] = fadd fast <2 x double> [[TMP4]], [[TMP2]]
; A320-NEXT: [[TMP5:%.*]] = call fast double @llvm.vector.reduce.fadd.v2f64(double 0.000000e+00, <2 x double> [[BIN_RDX]])
@@ -187,7 +187,7 @@ define double @sum_reduction(ptr nocapture readonly %a, i64 %n) {
; A320-NEXT: [[SUM_NEXT]] = fadd fast double [[SUM]], [[VAL]]
; A320-NEXT: [[IV_NEXT]] = add nuw nsw i64 [[IV]], 1
; A320-NEXT: [[COND:%.*]] = icmp ult i64 [[IV_NEXT]], [[N]]
-; A320-NEXT: br i1 [[COND]], label %[[LOOP]], label %[[EXIT_LOOPEXIT]], !llvm.loop [[LOOP5:![0-9]+]]
+; A320-NEXT: br i1 [[COND]], label %[[LOOP]], label %[[EXIT_LOOPEXIT]], !llvm.loop [[LOOP7:![0-9]+]]
; A320: [[EXIT_LOOPEXIT]]:
; A320-NEXT: [[SUM_NEXT_LCSSA:%.*]] = phi double [ [[SUM_NEXT]], %[[LOOP]] ], [ [[TMP5]], %[[MIDDLE_BLOCK]] ]
; A320-NEXT: br label %[[EXIT]]
@@ -256,7 +256,7 @@ define double @dot_product(ptr nocapture readonly %a, ptr nocapture readonly %b,
; A320-NEXT: [[TMP7]] = fadd fast <2 x double> [[VEC_PHI1]], [[TMP10]]
; A320-NEXT: [[INDEX_NEXT]] = add nuw i64 [[TMP1]], 4
; A320-NEXT: [[TMP14:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
-; A320-NEXT: br i1 [[TMP14]], label %[[MIDDLE_BLOCK:.*]], label %[[VECTOR_BODY]], !llvm.loop [[LOOP6:![0-9]+]]
+; A320-NEXT: br i1 [[TMP14]], label %[[MIDDLE_BLOCK:.*]], label %[[VECTOR_BODY]], !llvm.loop [[LOOP8:![0-9]+]]
; A320: [[MIDDLE_BLOCK]]:
; A320-NEXT: [[BIN_RDX:%.*]] = fadd fast <2 x double> [[TMP7]], [[TMP6]]
; A320-NEXT: [[TMP9:%.*]] = call fast double @llvm.vector.reduce.fadd.v2f64(double 0.000000e+00, <2 x double> [[BIN_RDX]])
@@ -277,7 +277,7 @@ define double @dot_product(ptr nocapture readonly %a, ptr nocapture readonly %b,
; A320-NEXT: [[ACC_NEXT]] = fadd fast double [[ACC]], [[PROD]]
; A320-NEXT: [[IV_NEXT]] = add nuw nsw i64 [[IV]], 1
; A320-NEXT: [[COND:%.*]] = icmp ult i64 [[IV_NEXT]], [[N]]
-; A320-NEXT: br i1 [[COND]], label %[[LOOP]], label %[[EXIT_LOOPEXIT]], !llvm.loop [[LOOP7:![0-9]+]]
+; A320-NEXT: br i1 [[COND]], label %[[LOOP]], label %[[EXIT_LOOPEXIT]], !llvm.loop [[LOOP9:![0-9]+]]
; A320: [[EXIT_LOOPEXIT]]:
; A320-NEXT: [[ACC_NEXT_LCSSA:%.*]] = phi double [ [[ACC_NEXT]], %[[LOOP]] ], [ [[TMP9]], %[[MIDDLE_BLOCK]] ]
; A320-NEXT: br label %[[EXIT]]
@@ -313,12 +313,14 @@ exit:
ret double %res
}
;.
-; A320: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]]}
+; A320: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]], [[META3:![0-9]+]]}
; A320: [[META1]] = !{!"llvm.loop.isvectorized", i32 1}
-; A320: [[META2]] = !{!"llvm.loop.unroll.runtime.disable"}
-; A320: [[LOOP3]] = distinct !{[[LOOP3]], [[META1]]}
-; A320: [[LOOP4]] = distinct !{[[LOOP4]], [[META1]], [[META2]]}
-; A320: [[LOOP5]] = distinct !{[[LOOP5]], [[META2]], [[META1]]}
-; A320: [[LOOP6]] = distinct !{[[LOOP6]], [[META1]], [[META2]]}
-; A320: [[LOOP7]] = distinct !{[[LOOP7]], [[META2]], [[META1]]}
+; A320: [[META2]] = !{!"llvm.loop.vectorize.vector_body", i32 1}
+; A320: [[META3]] = !{!"llvm.loop.unroll.runtime.disable"}
+; A320: [[LOOP4]] = distinct !{[[LOOP4]], [[META1]], [[META5:![0-9]+]]}
+; A320: [[META5]] = !{!"llvm.loop.vectorize.scalar_remainder", i32 1}
+; A320: [[LOOP6]] = distinct !{[[LOOP6]], [[META1]], [[META2]], [[META3]]}
+; A320: [[LOOP7]] = distinct !{[[LOOP7]], [[META3]], [[META1]], [[META5]]}
+; A320: [[LOOP8]] = distinct !{[[LOOP8]], [[META1]], [[META2]], [[META3]]}
+; A320: [[LOOP9]] = distinct !{[[LOOP9]], [[META3]], [[META1]], [[META5]]}
;.
diff --git a/llvm/test/CodeGen/AMDGPU/vectorize-unroll-metadata.ll b/llvm/test/CodeGen/AMDGPU/vectorize-unroll-metadata.ll
index eccaae6cc7de3..e589ef29d4725 100644
--- a/llvm/test/CodeGen/AMDGPU/vectorize-unroll-metadata.ll
+++ b/llvm/test/CodeGen/AMDGPU/vectorize-unroll-metadata.ll
@@ -1,20 +1,47 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
; RUN: opt -mtriple=amdgcn-- -mcpu=gfx90a -passes=loop-vectorize %s -S -o - | FileCheck %s
-; CHECK-LABEL: @test
-; CHECK-LABEL: vector.body:
-; CHECK: br i1 %{{[0-9]+}}, label %middle.block, label %vector.body, !llvm.loop !0
-; CHECK-LABEL: middle.block:
-; CHECK-LABEL: scalar.ph:
-; CHECK-LABEL: loop.body:
-; CHECK: br i1 %cond, label %exit, label %loop.body, !llvm.loop !2
-; CHECK: !0 = distinct !{!0, !1}
-; CHECK: !1 = !{!"llvm.loop.isvectorized", i32 1}
-; CHECK: !2 = distinct !{!2, !3, !1}
-; CHECK: !3 = !{!"llvm.loop.unroll.runtime.disable"}
define amdgpu_kernel void @test(ptr addrspace(1) %out, ptr addrspace(3) %lds, i32 %n) {
-
+; CHECK-LABEL: define amdgpu_kernel void @test(
+; CHECK-SAME: ptr addrspace(1) [[OUT:%.*]], ptr addrspace(3) [[LDS:%.*]], i32 [[N:%.*]]) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT: [[ENTRY:.*]]:
+; CHECK-NEXT: [[SMAX:%.*]] = call i32 @llvm.smax.i32(i32 [[N]], i32 0)
+; CHECK-NEXT: [[TMP0:%.*]] = add nuw i32 [[SMAX]], 1
+; CHECK-NEXT: [[MIN_ITERS_CHECK:%.*]] = icmp ult i32 [[TMP0]], 2
+; CHECK-NEXT: br i1 [[MIN_ITERS_CHECK]], label %[[SCALAR_PH:.*]], label %[[VECTOR_PH:.*]]
+; CHECK: [[VECTOR_PH]]:
+; CHECK-NEXT: [[N_MOD_VF:%.*]] = urem i32 [[TMP0]], 2
+; CHECK-NEXT: [[N_VEC:%.*]] = sub i32 [[TMP0]], [[N_MOD_VF]]
+; CHECK-NEXT: br label %[[VECTOR_BODY:.*]]
+; CHECK: [[VECTOR_BODY]]:
+; CHECK-NEXT: [[INDEX:%.*]] = phi i32 [ 0, %[[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], %[[VECTOR_BODY]] ]
+; CHECK-NEXT: [[TMP1:%.*]] = getelementptr i32, ptr addrspace(3) [[LDS]], i32 [[INDEX]]
+; CHECK-NEXT: [[WIDE_LOAD:%.*]] = load <2 x i32>, ptr addrspace(3) [[TMP1]], align 4
+; CHECK-NEXT: [[TMP2:%.*]] = getelementptr i32, ptr addrspace(1) [[OUT]], i32 [[INDEX]]
+; CHECK-NEXT: store <2 x i32> [[WIDE_LOAD]], ptr addrspace(1) [[TMP2]], align 4
+; CHECK-NEXT: [[INDEX_NEXT]] = add nuw i32 [[INDEX]], 2
+; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]
+; CHECK-NEXT: br i1 [[TMP3]], label %[[MIDDLE_BLOCK:.*]], label %[[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
+; CHECK: [[MIDDLE_BLOCK]]:
+; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i32 [[TMP0]], [[N_VEC]]
+; CHECK-NEXT: br i1 [[CMP_N]], label %[[EXIT:.*]], label %[[SCALAR_PH]]
+; CHECK: [[SCALAR_PH]]:
+; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i32 [ [[N_VEC]], %[[MIDDLE_BLOCK]] ], [ 0, %[[ENTRY]] ]
+; CHECK-NEXT: br label %[[LOOP_BODY:.*]]
+; CHECK: [[LOOP_BODY]]:
+; CHECK-NEXT: [[COUNTER:%.*]] = phi i32 [ [[BC_RESUME_VAL]], %[[SCALAR_PH]] ], [ [[INC:%.*]], %[[LOOP_BODY]] ]
+; CHECK-NEXT: [[PTR_LDS:%.*]] = getelementptr i32, ptr addrspace(3) [[LDS]], i32 [[COUNTER]]
+; CHECK-NEXT: [[VAL:%.*]] = load i32, ptr addrspace(3) [[PTR_LDS]], align 4
+; CHECK-NEXT: [[PTR_OUT:%.*]] = getelementptr i32, ptr addrspace(1) [[OUT]], i32 [[COUNTER]]
+; CHECK-NEXT: store i32 [[VAL]], ptr addrspace(1) [[PTR_OUT]], align 4
+; CHECK-NEXT: [[INC]] = add i32 [[COUNTER]], 1
+; CHECK-NEXT: [[COND:%.*]] = icmp sge i32 [[COUNTER]], [[N]]
+; CHECK-NEXT: br i1 [[COND]], label %[[EXIT]], label %[[LOOP_BODY]], !llvm.loop [[LOOP3:![0-9]+]]
+; CHECK: [[EXIT]]:
+; CHECK-NEXT: ret void
+;
entry:
br label %loop.body
@@ -32,3 +59,11 @@ exit:
ret void
}
+;.
+; CHECK: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]]}
+; CHECK: [[META1]] = !{!"llvm.loop.isvectorized", i32 1}
+; CHECK: [[META2]] = !{!"llvm.loop.vectorize.vector_body", i32 1}
+; CHECK: [[LOOP3]] = distinct !{[[LOOP3]], [[META4:![0-9]+]], [[META1]], [[META5:![0-9]+]]}
+; CHECK: [[META4]] = !{!"llvm.loop.unroll.runtime.disable"}
+; CHECK: [[META5]] = !{!"llvm.loop.vectorize.scalar_remainder", i32 1}
+;.
diff --git a/llvm/test/Transforms/LoopTransformWarning/amdgpu-vectorized-remainder-unroll-warning.ll b/llvm/test/Transforms/LoopTransformWarning/amdgpu-vectorized-remainder-unroll-warning.ll
new file mode 100644
index 0000000000000..2decef85692e7
--- /dev/null
+++ b/llvm/test/Transforms/LoopTransformWarning/amdgpu-vectorized-remainder-unroll-warning.ll
@@ -0,0 +1,59 @@
+; RUN: opt < %s -passes='loop-vectorize,loop-unroll,transform-warning' -force-vector-width=4 -force-vector-interleave=1 -disable-output -pass-remark...
[truncated]
``````````
</details>
https://github.com/llvm/llvm-project/pull/190258
More information about the llvm-commits
mailing list