[all-commits] [llvm/llvm-project] 65eccc: [LoopVectorize] Support vectorization of compressi...

Fri Sep 25 08:13:22 PDT 2026

  Branch: refs/heads/users/MacDue/compressed.vec
  Home:   https://github.com/llvm/llvm-project
  Commit: 65ecccbc5569b43ef5cba656e6d858f51754aaf1
      https://github.com/llvm/llvm-project/commit/65ecccbc5569b43ef5cba656e6d858f51754aaf1
  Author: Sergey Kachkov <sergey.kachkov at syntacore.com>
  Date:   2026-09-25 (Fri, 25 Sep 2026)

  Changed paths:
    M llvm/include/llvm/Analysis/VectorUtils.h
    M llvm/include/llvm/Transforms/Utils/LoopUtils.h
    M llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h
    M llvm/lib/Analysis/VectorUtils.cpp
    M llvm/lib/IR/IntrinsicInst.cpp
    M llvm/lib/Transforms/Utils/LoopUtils.cpp
    M llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp
    M llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.cpp
    M llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
    M llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
    M llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h
    M llvm/lib/Transforms/Vectorize/VPlan.cpp
    M llvm/lib/Transforms/Vectorize/VPlan.h
    M llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
    M llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
    M llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
    M llvm/lib/Transforms/Vectorize/VPlanTransforms.h
    M llvm/lib/Transforms/Vectorize/VPlanUtils.cpp
    A llvm/test/Transforms/LoopVectorize/VPlan/compress-idioms.ll
    M llvm/test/Transforms/LoopVectorize/VPlan/vplan-print-before-after-all.ll
    M llvm/test/Transforms/LoopVectorize/compress-idioms-negative-tests.ll
    M llvm/unittests/Transforms/Vectorize/VPlanTestBase.h

  Log Message:
  -----------
  [LoopVectorize] Support vectorization of compressing patterns in VPlan

RFC link: https://discourse.llvm.org/t/rfc-loop-vectorization-of-compress-store-expand-load-patterns/86442

This adds loop vectorizer support for "compressing" patterns,
for example:

```
int dst_idx = 0;
for (int i = 0; i < n; i++) {
  if (cond[i])
    dst[dst_idx++] = src[i];
}
```

Can be vectorized with a `llvm.masked.compressstore` as:

```
int dst_idx = 0;
for (int i = 0; i < n; i++) {
  %cond = load(%cond) != 0
  %src = masked.load(%src[i], %cond)
  masked.compressstore(%src, %dst[dst_idx], %cond)
  dst_idx += num.active.lanes(%cond)
}
```

and:

```
int src_idx = 0;
for (int i = 0; i < n; i++) {
  if (cond[i])
    dst[i] = src[src_idx++];
}
```

Can be vectorized with a `llvm.masked.expandload` as:

```
int src_idx = 0;
for (int i = 0; i < n; i++) {
  %cond = load(%cond) != 0
  %src = masked.expandload(%src[src_idx], %cond)
  masked.store(%src, %dst[i], %cond)
  src_idx += num.active.lanes(%cond)
}
```

This uses the new `MonotonicDescriptor` to recognize
monotonic/compressing patterns. The phis are mapped to a new
`VPMonotonicPHIRecipe`, this will map to a scalar phi. We only
allow uniform uses of monotonic phis in the loop (e.g., as the pointer
to a compressed load/store).

Compressed loads/stores are recognized with
`LoopVectorizationLegality::isCompressedPtr`. Currently, we only allow
cases where:

- The (monotonic) pointer has a stride equal to the access size
- The memory operation is predicated with the same condition as the increment

This is a continuation Sergey Kachkov's patch (#140723).

There are a number of changes from the initial patch:

- Expandloads/compresstores directly use `VPWidenMemIntrinsic`
- ComputeMonotonicResult is replaced with existing VP instructions
- AArch64, VPlan, and target-agnostic tests have been added
- This style of vectorization if off by default
  - The switch can be flipped soon after this patch lands

Tests, fixes, and design rework

Add comment

Add out-of-loop use check

Fixups

Fixups

Fixups

Add test

Remove AArch64 flags

Rename MonotonicPHI (and related) to ConditionalInduction*

Add other target tests

Flip flag

  Commit: a79ad539bf659e1a8b9f6bcc732d5090ebad0397
      https://github.com/llvm/llvm-project/commit/a79ad539bf659e1a8b9f6bcc732d5090ebad0397
  Author: Benjamin Maxwell <benjamin.maxwell at arm.com>
  Date:   2026-09-25 (Fri, 25 Sep 2026)

  Changed paths:
    M llvm/test/Transforms/LoopVectorize/AArch64/compress-idioms.ll
    M llvm/test/Transforms/LoopVectorize/X86/compress-idioms.ll
    M llvm/test/Transforms/LoopVectorize/compress-idioms.ll
    M llvm/test/Transforms/LoopVectorize/compress-store-vec-epilogue.ll

  Log Message:
  -----------
  Update checks

  Commit: ed189ba949de84173e1d03969719701adfc229d3
      https://github.com/llvm/llvm-project/commit/ed189ba949de84173e1d03969719701adfc229d3
  Author: Benjamin Maxwell <benjamin.maxwell at arm.com>
  Date:   2026-09-25 (Fri, 25 Sep 2026)

  Changed paths:
    M llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.cpp
    M llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
    M llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

  Log Message:
  -----------
  Always pass a vector type to TTI.isLegalMaskedCompressStore/ExpandLoad

  Commit: f7f6d06ce3a2b2858011525f49226dc25c880d39
      https://github.com/llvm/llvm-project/commit/f7f6d06ce3a2b2858011525f49226dc25c880d39
  Author: Benjamin Maxwell <benjamin.maxwell at arm.com>
  Date:   2026-09-25 (Fri, 25 Sep 2026)

  Changed paths:
    M llvm/lib/Transforms/Utils/LoopUtils.cpp

  Log Message:
  -----------
  Rebase fixups

  Commit: ce28845d39002e1dde040837cd65900067f7bdde
      https://github.com/llvm/llvm-project/commit/ce28845d39002e1dde040837cd65900067f7bdde
  Author: Benjamin Maxwell <benjamin.maxwell at arm.com>
  Date:   2026-09-25 (Fri, 25 Sep 2026)

  Changed paths:
    M llvm/include/llvm/Transforms/Utils/LoopUtils.h

  Log Message:
  -----------
  Add LLVM_ABI marker

  Commit: 8440f58fd2b4d862c7b7f4c025a65ef81b0561e4
      https://github.com/llvm/llvm-project/commit/8440f58fd2b4d862c7b7f4c025a65ef81b0561e4
  Author: Benjamin Maxwell <benjamin.maxwell at arm.com>
  Date:   2026-09-25 (Fri, 25 Sep 2026)

  Changed paths:
    M llvm/test/Transforms/LoopVectorize/RISCV/compress-idioms.ll
    M llvm/test/Transforms/LoopVectorize/X86/compress-idioms.ll

  Log Message:
  -----------
  Rebase tests

Compare: https://github.com/llvm/llvm-project/compare/d8d6957668d1...8440f58fd2b4

To unsubscribe from these emails, change your notification settings at https://github.com/llvm/llvm-project/settings/notifications