[all-commits] [llvm/llvm-project] 65d9b8: [LoopVectorize] Support vectorization of compressi...

[Benjamin Maxwell via All-commits]

  Branch: refs/heads/users/MacDue/compressed.vec
  Home:   https://github.com/llvm/llvm-project
  Commit: 65d9b886b0a7d190e61c84a4bfb09bf931e502bd
      https://github.com/llvm/llvm-project/commit/65d9b886b0a7d190e61c84a4bfb09bf931e502bd
  Author: Sergey Kachkov <sergey.kachkov at syntacore.com>
  Date:   2026-09-23 (Wed, 23 Sep 2026)

  Changed paths:
    M llvm/include/llvm/Analysis/VectorUtils.h
    M llvm/include/llvm/Transforms/Utils/LoopUtils.h
    M llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h
    M llvm/lib/Analysis/VectorUtils.cpp
    M llvm/lib/IR/IntrinsicInst.cpp
    M llvm/lib/Transforms/Utils/LoopUtils.cpp
    M llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp
    M llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.cpp
    M llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
    M llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
    M llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h
    M llvm/lib/Transforms/Vectorize/VPlan.cpp
    M llvm/lib/Transforms/Vectorize/VPlan.h
    M llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
    M llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
    M llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
    M llvm/lib/Transforms/Vectorize/VPlanTransforms.h
    M llvm/lib/Transforms/Vectorize/VPlanUtils.cpp
    A llvm/test/Transforms/LoopVectorize/VPlan/compress-idioms.ll
    M llvm/test/Transforms/LoopVectorize/VPlan/vplan-print-before-after-all.ll
    M llvm/test/Transforms/LoopVectorize/compress-idioms-negative-tests.ll
    M llvm/unittests/Transforms/Vectorize/VPlanTestBase.h

  Log Message:
  -----------
  [LoopVectorize] Support vectorization of compressing patterns in VPlan

RFC link: https://discourse.llvm.org/t/rfc-loop-vectorization-of-compress-store-expand-load-patterns/86442

This adds loop vectorizer support for "compressing" patterns,
for example:

```
int dst_idx = 0;
for (int i = 0; i < n; i++) {
  if (cond[i])
    dst[dst_idx++] = src[i];
}
```

Can be vectorized with a `llvm.masked.compressstore` as:

```
int dst_idx = 0;
for (int i = 0; i < n; i++) {
  %cond = load(%cond) != 0
  %src = masked.load(%src[i], %cond)
  masked.compressstore(%src, %dst[dst_idx], %cond)
  dst_idx += num.active.lanes(%cond)
}
```

and:

```
int src_idx = 0;
for (int i = 0; i < n; i++) {
  if (cond[i])
    dst[i] = src[src_idx++];
}
```

Can be vectorized with a `llvm.masked.expandload` as:

```
int src_idx = 0;
for (int i = 0; i < n; i++) {
  %cond = load(%cond) != 0
  %src = masked.expandload(%src[src_idx], %cond)
  masked.store(%src, %dst[i], %cond)
  src_idx += num.active.lanes(%cond)
}
```

This uses the new `MonotonicDescriptor` to recognize
monotonic/compressing patterns. The phis are mapped to a new
`VPMonotonicPHIRecipe`, this will map to a scalar phi. We only
allow uniform uses of monotonic phis in the loop (e.g., as the pointer
to a compressed load/store).

Compressed loads/stores are recognized with
`LoopVectorizationLegality::isCompressedPtr`. Currently, we only allow
cases where:

- The (monotonic) pointer has a stride equal to the access size
- The memory operation is predicated with the same condition as the increment

This is a continuation Sergey Kachkov's patch (#140723).

There are a number of changes from the initial patch:

- Expandloads/compresstores directly use `VPWidenMemIntrinsic`
- ComputeMonotonicResult is replaced with existing VP instructions
- AArch64, VPlan, and target-agnostic tests have been added
- This style of vectorization if off by default
  - The switch can be flipped soon after this patch lands

Tests, fixes, and design rework

Add comment

Add out-of-loop use check

Fixups

Fixups

Fixups

Add test

Remove AArch64 flags

Rename MonotonicPHI (and related) to ConditionalInduction*

Add other target tests

Flip flag


  Commit: 13658ea51634f29656ffdf5972c79eefecaa34c6
      https://github.com/llvm/llvm-project/commit/13658ea51634f29656ffdf5972c79eefecaa34c6
  Author: Benjamin Maxwell <benjamin.maxwell at arm.com>
  Date:   2026-09-23 (Wed, 23 Sep 2026)

  Changed paths:
    M llvm/test/Transforms/LoopVectorize/AArch64/compress-idioms.ll
    M llvm/test/Transforms/LoopVectorize/X86/compress-idioms.ll
    M llvm/test/Transforms/LoopVectorize/compress-idioms.ll
    M llvm/test/Transforms/LoopVectorize/compress-store-vec-epilogue.ll

  Log Message:
  -----------
  Update checks


  Commit: 368f39f8d5a77bb8172093024ed09563f9a11e3a
      https://github.com/llvm/llvm-project/commit/368f39f8d5a77bb8172093024ed09563f9a11e3a
  Author: Benjamin Maxwell <benjamin.maxwell at arm.com>
  Date:   2026-09-23 (Wed, 23 Sep 2026)

  Changed paths:
    M llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.cpp
    M llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
    M llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

  Log Message:
  -----------
  Always pass a vector type to TTI.isLegalMaskedCompressStore/ExpandLoad


  Commit: 39313089ead854435198a7b4b41979fb52779118
      https://github.com/llvm/llvm-project/commit/39313089ead854435198a7b4b41979fb52779118
  Author: Benjamin Maxwell <benjamin.maxwell at arm.com>
  Date:   2026-09-23 (Wed, 23 Sep 2026)

  Changed paths:
    M llvm/lib/Transforms/Utils/LoopUtils.cpp

  Log Message:
  -----------
  Rebase fixups


Compare: https://github.com/llvm/llvm-project/compare/8927b902e498...39313089ead8

To unsubscribe from these emails, change your notification settings at https://github.com/llvm/llvm-project/settings/notifications