[all-commits] [llvm/llvm-project] a10e0c: [LoopVectorize] Support vectorization of compressi...

Mon Sep 28 04:04:58 PDT 2026

  Branch: refs/heads/users/MacDue/compressed.vec
  Home:   https://github.com/llvm/llvm-project
  Commit: a10e0c03b705acabf00dd9d4d350974d623e3d3e
      https://github.com/llvm/llvm-project/commit/a10e0c03b705acabf00dd9d4d350974d623e3d3e
  Author: Sergey Kachkov <sergey.kachkov at syntacore.com>
  Date:   2026-09-28 (Mon, 28 Sep 2026)

  Changed paths:
    M llvm/include/llvm/Analysis/VectorUtils.h
    M llvm/include/llvm/Transforms/Utils/LoopUtils.h
    M llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h
    M llvm/lib/Analysis/VectorUtils.cpp
    M llvm/lib/IR/IntrinsicInst.cpp
    M llvm/lib/Transforms/Utils/LoopUtils.cpp
    M llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp
    M llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.cpp
    M llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
    M llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
    M llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h
    M llvm/lib/Transforms/Vectorize/VPlan.cpp
    M llvm/lib/Transforms/Vectorize/VPlan.h
    M llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
    M llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
    M llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
    M llvm/lib/Transforms/Vectorize/VPlanTransforms.h
    M llvm/lib/Transforms/Vectorize/VPlanUtils.cpp
    A llvm/test/Transforms/LoopVectorize/VPlan/compress-idioms.ll
    M llvm/test/Transforms/LoopVectorize/VPlan/vplan-print-before-after-all.ll
    M llvm/test/Transforms/LoopVectorize/compress-idioms-negative-tests.ll
    M llvm/unittests/Transforms/Vectorize/VPlanTestBase.h

  Log Message:
  -----------
  [LoopVectorize] Support vectorization of compressing patterns in VPlan

RFC link: https://discourse.llvm.org/t/rfc-loop-vectorization-of-compress-store-expand-load-patterns/86442

This adds loop vectorizer support for "compressing" patterns,
for example:

```
int dst_idx = 0;
for (int i = 0; i < n; i++) {
  if (cond[i])
    dst[dst_idx++] = src[i];
}
```

Can be vectorized with a `llvm.masked.compressstore` as:

```
int dst_idx = 0;
for (int i = 0; i < n; i++) {
  %cond = load(%cond) != 0
  %src = masked.load(%src[i], %cond)
  masked.compressstore(%src, %dst[dst_idx], %cond)
  dst_idx += num.active.lanes(%cond)
}
```

and:

```
int src_idx = 0;
for (int i = 0; i < n; i++) {
  if (cond[i])
    dst[i] = src[src_idx++];
}
```

Can be vectorized with a `llvm.masked.expandload` as:

```
int src_idx = 0;
for (int i = 0; i < n; i++) {
  %cond = load(%cond) != 0
  %src = masked.expandload(%src[src_idx], %cond)
  masked.store(%src, %dst[i], %cond)
  src_idx += num.active.lanes(%cond)
}
```

This uses the new `MonotonicDescriptor` to recognize
monotonic/compressing patterns. The phis are mapped to a new
`VPMonotonicPHIRecipe`, this will map to a scalar phi. We only
allow uniform uses of monotonic phis in the loop (e.g., as the pointer
to a compressed load/store).

Compressed loads/stores are recognized with
`LoopVectorizationLegality::isCompressedPtr`. Currently, we only allow
cases where:

- The (monotonic) pointer has a stride equal to the access size
- The memory operation is predicated with the same condition as the increment

This is a continuation Sergey Kachkov's patch (#140723).

There are a number of changes from the initial patch:

- Expandloads/compresstores directly use `VPWidenMemIntrinsic`
- ComputeMonotonicResult is replaced with existing VP instructions
- AArch64, VPlan, and target-agnostic tests have been added
- This style of vectorization if off by default
  - The switch can be flipped soon after this patch lands

Tests, fixes, and design rework

Add comment

Add out-of-loop use check

Fixups

Fixups

Fixups

Add test

Remove AArch64 flags

Rename MonotonicPHI (and related) to ConditionalInduction*

Add other target tests

Flip flag

  Commit: a6724eb4f7f993b3bf2bbaa9cd3efe3f982b25d8
      https://github.com/llvm/llvm-project/commit/a6724eb4f7f993b3bf2bbaa9cd3efe3f982b25d8
  Author: Benjamin Maxwell <benjamin.maxwell at arm.com>
  Date:   2026-09-28 (Mon, 28 Sep 2026)

  Changed paths:
    M llvm/test/Transforms/LoopVectorize/AArch64/compress-idioms.ll
    M llvm/test/Transforms/LoopVectorize/X86/compress-idioms.ll
    M llvm/test/Transforms/LoopVectorize/compress-idioms.ll
    M llvm/test/Transforms/LoopVectorize/compress-store-vec-epilogue.ll

  Log Message:
  -----------
  Update checks

  Commit: baf242d2ce6c765a19bfa145e3c6c24be2444a7b
      https://github.com/llvm/llvm-project/commit/baf242d2ce6c765a19bfa145e3c6c24be2444a7b
  Author: Benjamin Maxwell <benjamin.maxwell at arm.com>
  Date:   2026-09-28 (Mon, 28 Sep 2026)

  Changed paths:
    M llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.cpp
    M llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
    M llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

  Log Message:
  -----------
  Always pass a vector type to TTI.isLegalMaskedCompressStore/ExpandLoad

  Commit: 6c2f633d2b451ef4f7a1170fb9d9fd000c2dd27d
      https://github.com/llvm/llvm-project/commit/6c2f633d2b451ef4f7a1170fb9d9fd000c2dd27d
  Author: Benjamin Maxwell <benjamin.maxwell at arm.com>
  Date:   2026-09-28 (Mon, 28 Sep 2026)

  Changed paths:
    M llvm/lib/Transforms/Utils/LoopUtils.cpp

  Log Message:
  -----------
  Rebase fixups

  Commit: d96e0992609d29f349a0fa2d2de01bf06103355b
      https://github.com/llvm/llvm-project/commit/d96e0992609d29f349a0fa2d2de01bf06103355b
  Author: Benjamin Maxwell <benjamin.maxwell at arm.com>
  Date:   2026-09-28 (Mon, 28 Sep 2026)

  Changed paths:
    M llvm/include/llvm/Transforms/Utils/LoopUtils.h

  Log Message:
  -----------
  Add LLVM_ABI marker

  Commit: 9ccc84cdf6793a28000faa9ce0c9a13b8ff89b44
      https://github.com/llvm/llvm-project/commit/9ccc84cdf6793a28000faa9ce0c9a13b8ff89b44
  Author: Benjamin Maxwell <benjamin.maxwell at arm.com>
  Date:   2026-09-28 (Mon, 28 Sep 2026)

  Changed paths:
    M llvm/test/Transforms/LoopVectorize/RISCV/compress-idioms.ll
    M llvm/test/Transforms/LoopVectorize/X86/compress-idioms.ll

  Log Message:
  -----------
  Rebase tests

  Commit: 3945d893c75ff2429f4cd32cfb3380596b89c4c3
      https://github.com/llvm/llvm-project/commit/3945d893c75ff2429f4cd32cfb3380596b89c4c3
  Author: Benjamin Maxwell <benjamin.maxwell at arm.com>
  Date:   2026-09-28 (Mon, 28 Sep 2026)

  Changed paths:
    M llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

  Log Message:
  -----------
  Rebase fixups

  Commit: 281d23a05d9270b959834aad1d496f419337dfe0
      https://github.com/llvm/llvm-project/commit/281d23a05d9270b959834aad1d496f419337dfe0
  Author: Benjamin Maxwell <benjamin.maxwell at arm.com>
  Date:   2026-09-28 (Mon, 28 Sep 2026)

  Changed paths:
    M llvm/include/llvm/Transforms/Utils/LoopUtils.h
    M llvm/lib/Transforms/Utils/LoopUtils.cpp
    M llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp
    M llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
    M llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h
    M llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
    M llvm/test/Transforms/LoopVectorize/compress-idioms-negative-tests.ll

  Log Message:
  -----------
  Fixups

Compare: https://github.com/llvm/llvm-project/compare/8440f58fd2b4...281d23a05d92

To unsubscribe from these emails, change your notification settings at https://github.com/llvm/llvm-project/settings/notifications