[llvm] [LAA] Don't check same-address accesses against each other on retry (PR #226428)
Tim Besard via llvm-commits
llvm-commits at lists.llvm.org
Fri Sep 25 03:25:21 PDT 2026
https://github.com/maleadt created https://github.com/llvm/llvm-project/pull/226428
When LAA retries an unknown dependence with runtime checks, canCheckPtrAtRT removes the affected dependence classes, giving each pointer its own dependence set and checking group. This also separates safe dependences, including pointers accessing the same address each iteration, such as stores to A[i] in both branches of an if. Their identical bounds make the runtime check always fail, so the vector loop never runs.
Regroup accesses whose pointers have the same AddRec in the loop, whose loads and stores all have the same type, and whose stride is one element. They access the same bytes each iteration and disjoint bytes across iterations, so their only dependence has distance 0, preserved by vectorization. Consecutive accesses cannot join an interleave group that could reorder them, which matters because the retry discarded those dependences. Sharing a checking group also removes duplicate checks against other accesses.
Only the retry changes, after which recorded dependences are cleared. LoopLoadElimination and other passes relying on checks between classes while dependences are available are unaffected.
Motivation: Julia stores `Union{Float32, Nothing}` array values and type tags in one allocation, with tags at a runtime offset. Copy loops were vectorized with an always-failing check. In a reduced X86 test, memchecks drop from 7 comparisons (including `tags < tags_end` against itself) to 4, letting the vector loop run: 0.33 -> 0.07 ns/element on Zen 5 (n = 10000).
Assisted by: Codex 5.5, Astra 6
Part of a series of patches developed while migrating Julia to LLVM 23
>From d6f5223318cc838822e9791fd84c7b71f35b5dd6 Mon Sep 17 00:00:00 2001
From: Tim Besard <tim.besard at gmail.com>
Date: Thu, 24 Sep 2026 12:42:04 +0200
Subject: [PATCH 1/2] [LAA] Add tests for same-address runtime checks (NFC)
When LAA retries with runtime checks after finding an unknown
dependence, different pointers that access the same address in each
iteration are checked against each other, and that check always fails.
Also add a case where the pointers to the same address are also used by
a wider load, whose dependences are not recorded because of
-max-dependences.
---
.../retry-runtime-checks-same-address.ll | 371 ++++++++++++++++++
.../runtime-checks-max-dependences.ll | 122 ++++++
.../X86/runtime-checks-same-address.ll | 135 +++++++
3 files changed, 628 insertions(+)
create mode 100644 llvm/test/Analysis/LoopAccessAnalysis/retry-runtime-checks-same-address.ll
create mode 100644 llvm/test/Transforms/LoopVectorize/X86/runtime-checks-same-address.ll
diff --git a/llvm/test/Analysis/LoopAccessAnalysis/retry-runtime-checks-same-address.ll b/llvm/test/Analysis/LoopAccessAnalysis/retry-runtime-checks-same-address.ll
new file mode 100644
index 00000000000000..74cc187bc26698
--- /dev/null
+++ b/llvm/test/Analysis/LoopAccessAnalysis/retry-runtime-checks-same-address.ll
@@ -0,0 +1,371 @@
+; NOTE: Assertions have been autogenerated by utils/update_analyze_test_checks.py UTC_ARGS: --version 6
+; RUN: opt -passes='print<access-info>' -disable-output %s 2>&1 | FileCheck %s
+
+; The accesses to %A.sel and %A have a non-constant distance, so the
+; dependence analysis gives up and LAA retries with runtime checks between all
+; accesses to %A. Accesses through different pointers to the same address in
+; each iteration only depend on each other within an iteration. They must not
+; be checked against each other, as that check would always fail.
+
+define void @same_address_complementary_stores(ptr %A, i64 %off, ptr noalias %c, i64 %n) {
+; CHECK-LABEL: 'same_address_complementary_stores'
+; CHECK-NEXT: loop:
+; CHECK-NEXT: Memory dependences are safe with run-time checks
+; CHECK-NEXT: Dependences:
+; CHECK-NEXT: Run-time memory checks:
+; CHECK-NEXT: Check 0:
+; CHECK-NEXT: Comparing group GRP0:
+; CHECK-NEXT: %gep.sel.0 = getelementptr inbounds i8, ptr %A.sel, i64 %iv
+; CHECK-NEXT: Against group GRP1:
+; CHECK-NEXT: %gep.A = getelementptr inbounds i32, ptr %A, i64 %iv
+; CHECK-NEXT: Check 1:
+; CHECK-NEXT: Comparing group GRP0:
+; CHECK-NEXT: %gep.sel.0 = getelementptr inbounds i8, ptr %A.sel, i64 %iv
+; CHECK-NEXT: Against group GRP2:
+; CHECK-NEXT: %gep.sel.1 = getelementptr inbounds i8, ptr %A.sel, i64 %iv
+; CHECK-NEXT: Check 2:
+; CHECK-NEXT: Comparing group GRP1:
+; CHECK-NEXT: %gep.A = getelementptr inbounds i32, ptr %A, i64 %iv
+; CHECK-NEXT: Against group GRP2:
+; CHECK-NEXT: %gep.sel.1 = getelementptr inbounds i8, ptr %A.sel, i64 %iv
+; CHECK-NEXT: Grouped accesses:
+; CHECK-NEXT: Group GRP0:
+; CHECK-NEXT: (Low: (%off + %A) High: (%off + %n + %A))
+; CHECK-NEXT: Member: {(%off + %A),+,1}<nw><%loop>
+; CHECK-NEXT: Group GRP1:
+; CHECK-NEXT: (Low: %A High: ((4 * %n) + %A))
+; CHECK-NEXT: Member: {%A,+,4}<%loop>
+; CHECK-NEXT: Group GRP2:
+; CHECK-NEXT: (Low: (%off + %A) High: (%off + %n + %A))
+; CHECK-NEXT: Member: {(%off + %A),+,1}<nw><%loop>
+; CHECK-EMPTY:
+; CHECK-NEXT: Non vectorizable stores to invariant address were not found in loop.
+; CHECK-NEXT: SCEV assumptions:
+; CHECK-EMPTY:
+; CHECK-NEXT: Expressions re-written:
+;
+entry:
+ %A.sel = getelementptr i8, ptr %A, i64 %off
+ br label %loop
+
+loop:
+ %iv = phi i64 [ 0, %entry ], [ %iv.next, %latch ]
+ %gep.c = getelementptr inbounds i8, ptr %c, i64 %iv
+ %c.val = load i8, ptr %gep.c, align 1
+ %cond = icmp eq i8 %c.val, 0
+ br i1 %cond, label %then, label %else
+
+then:
+ %gep.A = getelementptr inbounds i32, ptr %A, i64 %iv
+ store i32 0, ptr %gep.A, align 4
+ %gep.sel.1 = getelementptr inbounds i8, ptr %A.sel, i64 %iv
+ store i8 1, ptr %gep.sel.1, align 1
+ br label %latch
+
+else:
+ %gep.sel.0 = getelementptr inbounds i8, ptr %A.sel, i64 %iv
+ store i8 0, ptr %gep.sel.0, align 1
+ br label %latch
+
+latch:
+ %iv.next = add nuw nsw i64 %iv, 1
+ %ec = icmp eq i64 %iv.next, %n
+ br i1 %ec, label %exit, label %loop
+
+exit:
+ ret void
+}
+
+define void @same_address_load_store(ptr %A, i64 %off, i64 %n) {
+; CHECK-LABEL: 'same_address_load_store'
+; CHECK-NEXT: loop:
+; CHECK-NEXT: Memory dependences are safe with run-time checks
+; CHECK-NEXT: Dependences:
+; CHECK-NEXT: Run-time memory checks:
+; CHECK-NEXT: Check 0:
+; CHECK-NEXT: Comparing group GRP0:
+; CHECK-NEXT: %gep.A = getelementptr inbounds i32, ptr %A, i64 %iv
+; CHECK-NEXT: Against group GRP1:
+; CHECK-NEXT: %gep.sel.1 = getelementptr inbounds i8, ptr %A.sel, i64 %iv
+; CHECK-NEXT: Check 1:
+; CHECK-NEXT: Comparing group GRP0:
+; CHECK-NEXT: %gep.A = getelementptr inbounds i32, ptr %A, i64 %iv
+; CHECK-NEXT: Against group GRP2:
+; CHECK-NEXT: %gep.sel.0 = getelementptr inbounds i8, ptr %A.sel, i64 %iv
+; CHECK-NEXT: Check 2:
+; CHECK-NEXT: Comparing group GRP1:
+; CHECK-NEXT: %gep.sel.1 = getelementptr inbounds i8, ptr %A.sel, i64 %iv
+; CHECK-NEXT: Against group GRP2:
+; CHECK-NEXT: %gep.sel.0 = getelementptr inbounds i8, ptr %A.sel, i64 %iv
+; CHECK-NEXT: Grouped accesses:
+; CHECK-NEXT: Group GRP0:
+; CHECK-NEXT: (Low: %A High: ((4 * %n) + %A))
+; CHECK-NEXT: Member: {%A,+,4}<nuw><%loop>
+; CHECK-NEXT: Group GRP1:
+; CHECK-NEXT: (Low: (%off + %A) High: (%off + %n + %A))
+; CHECK-NEXT: Member: {(%off + %A),+,1}<nw><%loop>
+; CHECK-NEXT: Group GRP2:
+; CHECK-NEXT: (Low: (%off + %A) High: (%off + %n + %A))
+; CHECK-NEXT: Member: {(%off + %A),+,1}<nw><%loop>
+; CHECK-EMPTY:
+; CHECK-NEXT: Non vectorizable stores to invariant address were not found in loop.
+; CHECK-NEXT: SCEV assumptions:
+; CHECK-EMPTY:
+; CHECK-NEXT: Expressions re-written:
+;
+entry:
+ %A.sel = getelementptr i8, ptr %A, i64 %off
+ br label %loop
+
+loop:
+ %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
+ %gep.A = getelementptr inbounds i32, ptr %A, i64 %iv
+ store i32 0, ptr %gep.A, align 4
+ %gep.sel.0 = getelementptr inbounds i8, ptr %A.sel, i64 %iv
+ %l = load i8, ptr %gep.sel.0, align 1
+ %add = add i8 %l, 1
+ %gep.sel.1 = getelementptr inbounds i8, ptr %A.sel, i64 %iv
+ store i8 %add, ptr %gep.sel.1, align 1
+ %iv.next = add nuw nsw i64 %iv, 1
+ %ec = icmp eq i64 %iv.next, %n
+ br i1 %ec, label %exit, label %loop
+
+exit:
+ ret void
+}
+
+define void @same_address_negative_stride(ptr %A, i64 %off, i64 %n) {
+; CHECK-LABEL: 'same_address_negative_stride'
+; CHECK-NEXT: loop:
+; CHECK-NEXT: Memory dependences are safe with run-time checks
+; CHECK-NEXT: Dependences:
+; CHECK-NEXT: Run-time memory checks:
+; CHECK-NEXT: Check 0:
+; CHECK-NEXT: Comparing group GRP0:
+; CHECK-NEXT: %gep.A = getelementptr inbounds i32, ptr %A, i64 %iv.next
+; CHECK-NEXT: Against group GRP1:
+; CHECK-NEXT: %gep.sel.0 = getelementptr inbounds i8, ptr %A.sel, i64 %iv.next
+; CHECK-NEXT: Check 1:
+; CHECK-NEXT: Comparing group GRP0:
+; CHECK-NEXT: %gep.A = getelementptr inbounds i32, ptr %A, i64 %iv.next
+; CHECK-NEXT: Against group GRP2:
+; CHECK-NEXT: %gep.sel.1 = getelementptr inbounds i8, ptr %A.sel, i64 %iv.next
+; CHECK-NEXT: Check 2:
+; CHECK-NEXT: Comparing group GRP1:
+; CHECK-NEXT: %gep.sel.0 = getelementptr inbounds i8, ptr %A.sel, i64 %iv.next
+; CHECK-NEXT: Against group GRP2:
+; CHECK-NEXT: %gep.sel.1 = getelementptr inbounds i8, ptr %A.sel, i64 %iv.next
+; CHECK-NEXT: Grouped accesses:
+; CHECK-NEXT: Group GRP0:
+; CHECK-NEXT: (Low: (-4 + (4 * (1 smin %n)) + %A) High: ((4 * %n) + %A))
+; CHECK-NEXT: Member: {(-4 + (4 * %n) + %A),+,-4}<%loop>
+; CHECK-NEXT: Group GRP1:
+; CHECK-NEXT: (Low: (-1 + (1 smin %n) + %off + %A) High: (%off + %n + %A))
+; CHECK-NEXT: Member: {(-1 + %off + %n + %A),+,-1}<nw><%loop>
+; CHECK-NEXT: Group GRP2:
+; CHECK-NEXT: (Low: (-1 + (1 smin %n) + %off + %A) High: (%off + %n + %A))
+; CHECK-NEXT: Member: {(-1 + %off + %n + %A),+,-1}<nw><%loop>
+; CHECK-EMPTY:
+; CHECK-NEXT: Non vectorizable stores to invariant address were not found in loop.
+; CHECK-NEXT: SCEV assumptions:
+; CHECK-EMPTY:
+; CHECK-NEXT: Expressions re-written:
+;
+entry:
+ %A.sel = getelementptr i8, ptr %A, i64 %off
+ br label %loop
+
+loop:
+ %iv = phi i64 [ %n, %entry ], [ %iv.next, %loop ]
+ %iv.next = add nsw i64 %iv, -1
+ %gep.A = getelementptr inbounds i32, ptr %A, i64 %iv.next
+ store i32 0, ptr %gep.A, align 4
+ %gep.sel.0 = getelementptr inbounds i8, ptr %A.sel, i64 %iv.next
+ store i8 0, ptr %gep.sel.0, align 1
+ %gep.sel.1 = getelementptr inbounds i8, ptr %A.sel, i64 %iv.next
+ store i8 1, ptr %gep.sel.1, align 1
+ %ec = icmp sgt i64 %iv, 1
+ br i1 %ec, label %loop, label %exit
+
+exit:
+ ret void
+}
+
+; The accesses have the same address in each iteration, but different sizes,
+; keep checking them.
+define void @same_address_different_sizes(ptr %A, i64 %off, i64 %n) {
+; CHECK-LABEL: 'same_address_different_sizes'
+; CHECK-NEXT: loop:
+; CHECK-NEXT: Memory dependences are safe with run-time checks
+; CHECK-NEXT: Dependences:
+; CHECK-NEXT: Run-time memory checks:
+; CHECK-NEXT: Check 0:
+; CHECK-NEXT: Comparing group GRP0:
+; CHECK-NEXT: %gep.A = getelementptr inbounds i32, ptr %A, i64 %iv
+; CHECK-NEXT: Against group GRP1:
+; CHECK-NEXT: %gep.sel.0 = getelementptr inbounds i16, ptr %A.sel, i64 %iv
+; CHECK-NEXT: Check 1:
+; CHECK-NEXT: Comparing group GRP0:
+; CHECK-NEXT: %gep.A = getelementptr inbounds i32, ptr %A, i64 %iv
+; CHECK-NEXT: Against group GRP2:
+; CHECK-NEXT: %gep.sel.1 = getelementptr inbounds i8, ptr %A.sel, i64 %iv.2
+; CHECK-NEXT: Check 2:
+; CHECK-NEXT: Comparing group GRP1:
+; CHECK-NEXT: %gep.sel.0 = getelementptr inbounds i16, ptr %A.sel, i64 %iv
+; CHECK-NEXT: Against group GRP2:
+; CHECK-NEXT: %gep.sel.1 = getelementptr inbounds i8, ptr %A.sel, i64 %iv.2
+; CHECK-NEXT: Grouped accesses:
+; CHECK-NEXT: Group GRP0:
+; CHECK-NEXT: (Low: %A High: ((4 * %n) + %A))
+; CHECK-NEXT: Member: {%A,+,4}<nuw><%loop>
+; CHECK-NEXT: Group GRP1:
+; CHECK-NEXT: (Low: (%off + %A) High: ((2 * %n) + %off + %A))
+; CHECK-NEXT: Member: {(%off + %A),+,2}<nw><%loop>
+; CHECK-NEXT: Group GRP2:
+; CHECK-NEXT: (Low: (%off + %A) High: (-1 + (2 * %n) + %off + %A))
+; CHECK-NEXT: Member: {(%off + %A),+,2}<nw><%loop>
+; CHECK-EMPTY:
+; CHECK-NEXT: Non vectorizable stores to invariant address were not found in loop.
+; CHECK-NEXT: SCEV assumptions:
+; CHECK-EMPTY:
+; CHECK-NEXT: Expressions re-written:
+;
+entry:
+ %A.sel = getelementptr i8, ptr %A, i64 %off
+ br label %loop
+
+loop:
+ %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
+ %gep.A = getelementptr inbounds i32, ptr %A, i64 %iv
+ store i32 0, ptr %gep.A, align 4
+ %gep.sel.0 = getelementptr inbounds i16, ptr %A.sel, i64 %iv
+ store i16 0, ptr %gep.sel.0, align 2
+ %iv.2 = shl nuw nsw i64 %iv, 1
+ %gep.sel.1 = getelementptr inbounds i8, ptr %A.sel, i64 %iv.2
+ store i8 1, ptr %gep.sel.1, align 1
+ %iv.next = add nuw nsw i64 %iv, 1
+ %ec = icmp eq i64 %iv.next, %n
+ br i1 %ec, label %exit, label %loop
+
+exit:
+ ret void
+}
+
+; The accesses have the same address in each iteration, but are strided and
+; could be reordered as part of interleave groups, keep checking them.
+define void @same_address_strided(ptr %A, i64 %off, i64 %n) {
+; CHECK-LABEL: 'same_address_strided'
+; CHECK-NEXT: loop:
+; CHECK-NEXT: Memory dependences are safe with run-time checks
+; CHECK-NEXT: Dependences:
+; CHECK-NEXT: Run-time memory checks:
+; CHECK-NEXT: Check 0:
+; CHECK-NEXT: Comparing group GRP0:
+; CHECK-NEXT: %gep.A = getelementptr inbounds i32, ptr %A, i64 %iv.2
+; CHECK-NEXT: Against group GRP1:
+; CHECK-NEXT: %gep.sel.0 = getelementptr inbounds i8, ptr %A.sel, i64 %iv.2
+; CHECK-NEXT: Check 1:
+; CHECK-NEXT: Comparing group GRP0:
+; CHECK-NEXT: %gep.A = getelementptr inbounds i32, ptr %A, i64 %iv.2
+; CHECK-NEXT: Against group GRP2:
+; CHECK-NEXT: %gep.sel.1 = getelementptr inbounds i8, ptr %A.sel, i64 %iv.2
+; CHECK-NEXT: Check 2:
+; CHECK-NEXT: Comparing group GRP1:
+; CHECK-NEXT: %gep.sel.0 = getelementptr inbounds i8, ptr %A.sel, i64 %iv.2
+; CHECK-NEXT: Against group GRP2:
+; CHECK-NEXT: %gep.sel.1 = getelementptr inbounds i8, ptr %A.sel, i64 %iv.2
+; CHECK-NEXT: Grouped accesses:
+; CHECK-NEXT: Group GRP0:
+; CHECK-NEXT: (Low: %A High: (-4 + (8 * %n) + %A))
+; CHECK-NEXT: Member: {%A,+,8}<nuw><%loop>
+; CHECK-NEXT: Group GRP1:
+; CHECK-NEXT: (Low: (%off + %A) High: (-1 + (2 * %n) + %off + %A))
+; CHECK-NEXT: Member: {(%off + %A),+,2}<nw><%loop>
+; CHECK-NEXT: Group GRP2:
+; CHECK-NEXT: (Low: (%off + %A) High: (-1 + (2 * %n) + %off + %A))
+; CHECK-NEXT: Member: {(%off + %A),+,2}<nw><%loop>
+; CHECK-EMPTY:
+; CHECK-NEXT: Non vectorizable stores to invariant address were not found in loop.
+; CHECK-NEXT: SCEV assumptions:
+; CHECK-EMPTY:
+; CHECK-NEXT: Expressions re-written:
+;
+entry:
+ %A.sel = getelementptr i8, ptr %A, i64 %off
+ br label %loop
+
+loop:
+ %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
+ %iv.2 = shl nuw nsw i64 %iv, 1
+ %gep.A = getelementptr inbounds i32, ptr %A, i64 %iv.2
+ store i32 0, ptr %gep.A, align 4
+ %gep.sel.0 = getelementptr inbounds i8, ptr %A.sel, i64 %iv.2
+ store i8 0, ptr %gep.sel.0, align 1
+ %gep.sel.1 = getelementptr inbounds i8, ptr %A.sel, i64 %iv.2
+ store i8 1, ptr %gep.sel.1, align 1
+ %iv.next = add nuw nsw i64 %iv, 1
+ %ec = icmp eq i64 %iv.next, %n
+ br i1 %ec, label %exit, label %loop
+
+exit:
+ ret void
+}
+
+; The accesses have the same address and size in each iteration, but different
+; types, keep checking them.
+define void @same_address_different_types(ptr %A, i64 %off, i64 %n) {
+; CHECK-LABEL: 'same_address_different_types'
+; CHECK-NEXT: loop:
+; CHECK-NEXT: Memory dependences are safe with run-time checks
+; CHECK-NEXT: Dependences:
+; CHECK-NEXT: Run-time memory checks:
+; CHECK-NEXT: Check 0:
+; CHECK-NEXT: Comparing group GRP0:
+; CHECK-NEXT: %gep.A = getelementptr inbounds i8, ptr %A, i64 %iv
+; CHECK-NEXT: Against group GRP1:
+; CHECK-NEXT: %gep.sel.0 = getelementptr inbounds i32, ptr %A.sel, i64 %iv
+; CHECK-NEXT: Check 1:
+; CHECK-NEXT: Comparing group GRP0:
+; CHECK-NEXT: %gep.A = getelementptr inbounds i8, ptr %A, i64 %iv
+; CHECK-NEXT: Against group GRP2:
+; CHECK-NEXT: %gep.sel.1 = getelementptr inbounds float, ptr %A.sel, i64 %iv
+; CHECK-NEXT: Check 2:
+; CHECK-NEXT: Comparing group GRP1:
+; CHECK-NEXT: %gep.sel.0 = getelementptr inbounds i32, ptr %A.sel, i64 %iv
+; CHECK-NEXT: Against group GRP2:
+; CHECK-NEXT: %gep.sel.1 = getelementptr inbounds float, ptr %A.sel, i64 %iv
+; CHECK-NEXT: Grouped accesses:
+; CHECK-NEXT: Group GRP0:
+; CHECK-NEXT: (Low: %A High: (%n + %A))
+; CHECK-NEXT: Member: {%A,+,1}<nuw><%loop>
+; CHECK-NEXT: Group GRP1:
+; CHECK-NEXT: (Low: (%off + %A) High: ((4 * %n) + %off + %A))
+; CHECK-NEXT: Member: {(%off + %A),+,4}<nw><%loop>
+; CHECK-NEXT: Group GRP2:
+; CHECK-NEXT: (Low: (%off + %A) High: ((4 * %n) + %off + %A))
+; CHECK-NEXT: Member: {(%off + %A),+,4}<nw><%loop>
+; CHECK-EMPTY:
+; CHECK-NEXT: Non vectorizable stores to invariant address were not found in loop.
+; CHECK-NEXT: SCEV assumptions:
+; CHECK-EMPTY:
+; CHECK-NEXT: Expressions re-written:
+;
+entry:
+ %A.sel = getelementptr i8, ptr %A, i64 %off
+ br label %loop
+
+loop:
+ %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
+ %gep.A = getelementptr inbounds i8, ptr %A, i64 %iv
+ store i8 0, ptr %gep.A, align 1
+ %gep.sel.0 = getelementptr inbounds i32, ptr %A.sel, i64 %iv
+ store i32 0, ptr %gep.sel.0, align 4
+ %gep.sel.1 = getelementptr inbounds float, ptr %A.sel, i64 %iv
+ store float 1.0, ptr %gep.sel.1, align 4
+ %iv.next = add nuw nsw i64 %iv, 1
+ %ec = icmp eq i64 %iv.next, %n
+ br i1 %ec, label %exit, label %loop
+
+exit:
+ ret void
+}
diff --git a/llvm/test/Analysis/LoopAccessAnalysis/runtime-checks-max-dependences.ll b/llvm/test/Analysis/LoopAccessAnalysis/runtime-checks-max-dependences.ll
index 5b17920bde85e8..0eecc5eb888e18 100644
--- a/llvm/test/Analysis/LoopAccessAnalysis/runtime-checks-max-dependences.ll
+++ b/llvm/test/Analysis/LoopAccessAnalysis/runtime-checks-max-dependences.ll
@@ -1,6 +1,7 @@
; NOTE: Assertions have been autogenerated by utils/update_analyze_test_checks.py UTC_ARGS: --version 5
; RUN: opt -passes='print<access-info>' -disable-output %s 2>&1 | FileCheck --check-prefix=DEFAULT %s
; RUN: opt -passes='print<access-info>' -disable-output -max-dependences=5 %s 2>&1 | FileCheck --check-prefix=MAXDEP5 %s
+; RUN: opt -passes='print<access-info>' -disable-output -max-dependences=1 %s 2>&1 | FileCheck --check-prefix=MAXDEP1 %s
define void @dependences_with_strides(ptr %dst, i64 %stride) {
; DEFAULT-LABEL: 'dependences_with_strides'
@@ -75,6 +76,17 @@ define void @dependences_with_strides(ptr %dst, i64 %stride) {
; MAXDEP5-EMPTY:
; MAXDEP5-NEXT: Expressions re-written:
;
+; MAXDEP1-LABEL: 'dependences_with_strides'
+; MAXDEP1-NEXT: loop:
+; MAXDEP1-NEXT: Too many dependences, not recorded
+; MAXDEP1-NEXT: Run-time memory checks:
+; MAXDEP1-NEXT: Grouped accesses:
+; MAXDEP1-EMPTY:
+; MAXDEP1-NEXT: Non vectorizable stores to invariant address were not found in loop.
+; MAXDEP1-NEXT: SCEV assumptions:
+; MAXDEP1-EMPTY:
+; MAXDEP1-NEXT: Expressions re-written:
+;
entry:
br label %loop
@@ -94,3 +106,113 @@ loop:
exit:
ret void
}
+
+; %p and %q have the same address in each iteration, but %p is also used by a
+; wider load, which overlaps the byte stored through %q in the next iteration.
+; The dependence between the load and the store through %q is not recorded with
+; -max-dependences=1, so %p and %q must still be checked against each other.
+define void @same_address_mixed_access_types(ptr %A, i64 %off, ptr noalias %out, i64 %n) {
+; DEFAULT-LABEL: 'same_address_mixed_access_types'
+; DEFAULT-NEXT: loop:
+; DEFAULT-NEXT: Report: unsafe dependent memory operations in loop. Use #pragma clang loop distribute(enable) to allow loop distribution to attempt to isolate the offending operations into a separate loop
+; DEFAULT-NEXT: Unknown data dependence.
+; DEFAULT-NEXT: Dependences:
+; DEFAULT-NEXT: Unknown:
+; DEFAULT-NEXT: store i8 2, ptr %q, align 1 ->
+; DEFAULT-NEXT: %l.r = load i8, ptr %r, align 1
+; DEFAULT-EMPTY:
+; DEFAULT-NEXT: Unknown:
+; DEFAULT-NEXT: store i8 1, ptr %p, align 1 ->
+; DEFAULT-NEXT: %l.r = load i8, ptr %r, align 1
+; DEFAULT-EMPTY:
+; DEFAULT-NEXT: IndirectUnsafe:
+; DEFAULT-NEXT: %l.p = load i16, ptr %p, align 1 ->
+; DEFAULT-NEXT: store i8 2, ptr %q, align 1
+; DEFAULT-EMPTY:
+; DEFAULT-NEXT: IndirectUnsafe:
+; DEFAULT-NEXT: store i8 1, ptr %p, align 1 ->
+; DEFAULT-NEXT: %l.p = load i16, ptr %p, align 1
+; DEFAULT-EMPTY:
+; DEFAULT-NEXT: Forward:
+; DEFAULT-NEXT: store i8 1, ptr %p, align 1 ->
+; DEFAULT-NEXT: store i8 2, ptr %q, align 1
+; DEFAULT-EMPTY:
+; DEFAULT-NEXT: Run-time memory checks:
+; DEFAULT-NEXT: Grouped accesses:
+; DEFAULT-EMPTY:
+; DEFAULT-NEXT: Non vectorizable stores to invariant address were not found in loop.
+; DEFAULT-NEXT: SCEV assumptions:
+; DEFAULT-EMPTY:
+; DEFAULT-NEXT: Expressions re-written:
+;
+; MAXDEP5-LABEL: 'same_address_mixed_access_types'
+; MAXDEP5-NEXT: loop:
+; MAXDEP5-NEXT: Too many dependences, not recorded
+; MAXDEP5-NEXT: Run-time memory checks:
+; MAXDEP5-NEXT: Grouped accesses:
+; MAXDEP5-EMPTY:
+; MAXDEP5-NEXT: Non vectorizable stores to invariant address were not found in loop.
+; MAXDEP5-NEXT: SCEV assumptions:
+; MAXDEP5-EMPTY:
+; MAXDEP5-NEXT: Expressions re-written:
+;
+; MAXDEP1-LABEL: 'same_address_mixed_access_types'
+; MAXDEP1-NEXT: loop:
+; MAXDEP1-NEXT: Memory dependences are safe with run-time checks
+; MAXDEP1-NEXT: Too many dependences, not recorded
+; MAXDEP1-NEXT: Run-time memory checks:
+; MAXDEP1-NEXT: Check 0:
+; MAXDEP1-NEXT: Comparing group GRP0:
+; MAXDEP1-NEXT: %p = getelementptr inbounds i8, ptr %A, i64 %iv
+; MAXDEP1-NEXT: Against group GRP1:
+; MAXDEP1-NEXT: %q = getelementptr inbounds i8, ptr %A, i64 %iv
+; MAXDEP1-NEXT: Check 1:
+; MAXDEP1-NEXT: Comparing group GRP0:
+; MAXDEP1-NEXT: %p = getelementptr inbounds i8, ptr %A, i64 %iv
+; MAXDEP1-NEXT: Against group GRP2:
+; MAXDEP1-NEXT: %r = getelementptr inbounds i8, ptr %A.off, i64 %iv
+; MAXDEP1-NEXT: Check 2:
+; MAXDEP1-NEXT: Comparing group GRP1:
+; MAXDEP1-NEXT: %q = getelementptr inbounds i8, ptr %A, i64 %iv
+; MAXDEP1-NEXT: Against group GRP2:
+; MAXDEP1-NEXT: %r = getelementptr inbounds i8, ptr %A.off, i64 %iv
+; MAXDEP1-NEXT: Grouped accesses:
+; MAXDEP1-NEXT: Group GRP0:
+; MAXDEP1-NEXT: (Low: %A High: (%n + %A))
+; MAXDEP1-NEXT: Member: {%A,+,1}<nuw><%loop>
+; MAXDEP1-NEXT: Group GRP1:
+; MAXDEP1-NEXT: (Low: %A High: (%n + %A))
+; MAXDEP1-NEXT: Member: {%A,+,1}<nuw><%loop>
+; MAXDEP1-NEXT: Group GRP2:
+; MAXDEP1-NEXT: (Low: (%off + %A) High: (%off + %n + %A))
+; MAXDEP1-NEXT: Member: {(%off + %A),+,1}<nw><%loop>
+; MAXDEP1-EMPTY:
+; MAXDEP1-NEXT: Non vectorizable stores to invariant address were not found in loop.
+; MAXDEP1-NEXT: SCEV assumptions:
+; MAXDEP1-EMPTY:
+; MAXDEP1-NEXT: Expressions re-written:
+;
+entry:
+ %A.off = getelementptr i8, ptr %A, i64 %off
+ br label %loop
+
+loop:
+ %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
+ %p = getelementptr inbounds i8, ptr %A, i64 %iv
+ store i8 1, ptr %p, align 1
+ %l.p = load i16, ptr %p, align 1
+ %q = getelementptr inbounds i8, ptr %A, i64 %iv
+ store i8 2, ptr %q, align 1
+ %r = getelementptr inbounds i8, ptr %A.off, i64 %iv
+ %l.r = load i8, ptr %r, align 1
+ %ext = zext i8 %l.r to i16
+ %res = add i16 %l.p, %ext
+ %gep.out = getelementptr inbounds i16, ptr %out, i64 %iv
+ store i16 %res, ptr %gep.out, align 2
+ %iv.next = add nuw nsw i64 %iv, 1
+ %ec = icmp eq i64 %iv.next, %n
+ br i1 %ec, label %exit, label %loop
+
+exit:
+ ret void
+}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/runtime-checks-same-address.ll b/llvm/test/Transforms/LoopVectorize/X86/runtime-checks-same-address.ll
new file mode 100644
index 00000000000000..2d7e8afbd4446b
--- /dev/null
+++ b/llvm/test/Transforms/LoopVectorize/X86/runtime-checks-same-address.ll
@@ -0,0 +1,135 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --check-globals none --filter-out-after "scalar.ph:" --version 6
+; RUN: opt -passes=loop-vectorize -mcpu=haswell -force-vector-width=4 -force-vector-interleave=1 -S %s | FileCheck %s
+
+target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+; Copy of an array of a union of i32 and a singleton type, stored as values
+; followed by type tags in a single allocation. The distance between the value
+; and the tag of an element is not a constant, so all accesses are checked at
+; runtime, but the two stores of the tag (to the same address) must not be
+; checked against each other.
+define void @union_array_copy(ptr %src, i64 %src.len, ptr %dst, i64 %dst.len, i64 %n) {
+; CHECK-LABEL: define void @union_array_copy(
+; CHECK-SAME: ptr [[SRC:%.*]], i64 [[SRC_LEN:%.*]], ptr [[DST:%.*]], i64 [[DST_LEN:%.*]], i64 [[N:%.*]]) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT: [[ENTRY:.*:]]
+; CHECK-NEXT: [[SRC_TAGS:%.*]] = getelementptr inbounds nuw [4 x i8], ptr [[SRC]], i64 [[SRC_LEN]]
+; CHECK-NEXT: [[DST_TAGS:%.*]] = getelementptr [4 x i8], ptr [[DST]], i64 [[DST_LEN]]
+; CHECK-NEXT: [[MIN_ITERS_CHECK:%.*]] = icmp ult i64 [[N]], 4
+; CHECK-NEXT: br i1 [[MIN_ITERS_CHECK]], label %[[SCALAR_PH:.*]], label %[[VECTOR_MEMCHECK:.*]]
+; CHECK: [[VECTOR_MEMCHECK]]:
+; CHECK-NEXT: [[TMP0:%.*]] = shl i64 [[DST_LEN]], 2
+; CHECK-NEXT: [[TMP1:%.*]] = add i64 [[N]], [[TMP0]]
+; CHECK-NEXT: [[SCEVGEP:%.*]] = getelementptr i8, ptr [[DST]], i64 [[TMP1]]
+; CHECK-NEXT: [[TMP2:%.*]] = shl i64 [[N]], 2
+; CHECK-NEXT: [[SCEVGEP1:%.*]] = getelementptr i8, ptr [[DST]], i64 [[TMP2]]
+; CHECK-NEXT: [[SCEVGEP2:%.*]] = getelementptr i8, ptr [[SRC]], i64 [[TMP2]]
+; CHECK-NEXT: [[TMP3:%.*]] = shl nuw nsw i64 [[SRC_LEN]], 2
+; CHECK-NEXT: [[TMP4:%.*]] = add i64 [[N]], [[TMP3]]
+; CHECK-NEXT: [[SCEVGEP3:%.*]] = getelementptr i8, ptr [[SRC]], i64 [[TMP4]]
+; CHECK-NEXT: [[BOUND2:%.*]] = icmp ult ptr [[DST_TAGS]], [[SCEVGEP]]
+; CHECK-NEXT: [[BOUND3:%.*]] = icmp ult ptr [[DST_TAGS]], [[SCEVGEP]]
+; CHECK-NEXT: [[BOUND0:%.*]] = icmp ult ptr [[DST_TAGS]], [[SCEVGEP1]]
+; CHECK-NEXT: [[BOUND1:%.*]] = icmp ult ptr [[DST]], [[SCEVGEP]]
+; CHECK-NEXT: [[FOUND_CONFLICT:%.*]] = and i1 [[BOUND0]], [[BOUND1]]
+; CHECK-NEXT: [[CONFLICT_RDX1:%.*]] = or i1 [[BOUND2]], [[FOUND_CONFLICT]]
+; CHECK-NEXT: [[BOUND04:%.*]] = icmp ult ptr [[DST_TAGS]], [[SCEVGEP2]]
+; CHECK-NEXT: [[BOUND15:%.*]] = icmp ult ptr [[SRC]], [[SCEVGEP]]
+; CHECK-NEXT: [[FOUND_CONFLICT6:%.*]] = and i1 [[BOUND04]], [[BOUND15]]
+; CHECK-NEXT: [[CONFLICT_RDX:%.*]] = or i1 [[CONFLICT_RDX1]], [[FOUND_CONFLICT6]]
+; CHECK-NEXT: [[BOUND07:%.*]] = icmp ult ptr [[DST_TAGS]], [[SCEVGEP3]]
+; CHECK-NEXT: [[BOUND18:%.*]] = icmp ult ptr [[SRC_TAGS]], [[SCEVGEP]]
+; CHECK-NEXT: [[FOUND_CONFLICT9:%.*]] = and i1 [[BOUND07]], [[BOUND18]]
+; CHECK-NEXT: [[CONFLICT_RDX10:%.*]] = or i1 [[CONFLICT_RDX]], [[FOUND_CONFLICT9]]
+; CHECK-NEXT: [[BOUND011:%.*]] = icmp ult ptr [[DST_TAGS]], [[SCEVGEP1]]
+; CHECK-NEXT: [[BOUND112:%.*]] = icmp ult ptr [[DST]], [[SCEVGEP]]
+; CHECK-NEXT: [[FOUND_CONFLICT13:%.*]] = and i1 [[BOUND011]], [[BOUND112]]
+; CHECK-NEXT: [[CONFLICT_RDX14:%.*]] = or i1 [[CONFLICT_RDX10]], [[FOUND_CONFLICT13]]
+; CHECK-NEXT: [[BOUND015:%.*]] = icmp ult ptr [[DST_TAGS]], [[SCEVGEP2]]
+; CHECK-NEXT: [[BOUND116:%.*]] = icmp ult ptr [[SRC]], [[SCEVGEP]]
+; CHECK-NEXT: [[FOUND_CONFLICT17:%.*]] = and i1 [[BOUND015]], [[BOUND116]]
+; CHECK-NEXT: [[CONFLICT_RDX18:%.*]] = or i1 [[CONFLICT_RDX14]], [[FOUND_CONFLICT17]]
+; CHECK-NEXT: [[BOUND019:%.*]] = icmp ult ptr [[DST_TAGS]], [[SCEVGEP3]]
+; CHECK-NEXT: [[BOUND120:%.*]] = icmp ult ptr [[SRC_TAGS]], [[SCEVGEP]]
+; CHECK-NEXT: [[FOUND_CONFLICT21:%.*]] = and i1 [[BOUND019]], [[BOUND120]]
+; CHECK-NEXT: [[CONFLICT_RDX22:%.*]] = or i1 [[CONFLICT_RDX18]], [[FOUND_CONFLICT21]]
+; CHECK-NEXT: [[BOUND023:%.*]] = icmp ult ptr [[DST]], [[SCEVGEP2]]
+; CHECK-NEXT: [[BOUND124:%.*]] = icmp ult ptr [[SRC]], [[SCEVGEP1]]
+; CHECK-NEXT: [[FOUND_CONFLICT25:%.*]] = and i1 [[BOUND023]], [[BOUND124]]
+; CHECK-NEXT: [[CONFLICT_RDX26:%.*]] = or i1 [[CONFLICT_RDX22]], [[FOUND_CONFLICT25]]
+; CHECK-NEXT: [[BOUND027:%.*]] = icmp ult ptr [[DST]], [[SCEVGEP3]]
+; CHECK-NEXT: [[BOUND128:%.*]] = icmp ult ptr [[SRC_TAGS]], [[SCEVGEP1]]
+; CHECK-NEXT: [[FOUND_CONFLICT29:%.*]] = and i1 [[BOUND027]], [[BOUND128]]
+; CHECK-NEXT: [[CONFLICT_RDX30:%.*]] = or i1 [[CONFLICT_RDX26]], [[FOUND_CONFLICT29]]
+; CHECK-NEXT: br i1 [[CONFLICT_RDX30]], label %[[SCALAR_PH]], label %[[VECTOR_PH:.*]]
+; CHECK: [[VECTOR_PH]]:
+; CHECK-NEXT: [[TMP5:%.*]] = and i64 [[N]], 3
+; CHECK-NEXT: [[N_VEC:%.*]] = sub i64 [[N]], [[TMP5]]
+; CHECK-NEXT: br label %[[VECTOR_BODY:.*]]
+; CHECK: [[VECTOR_BODY]]:
+; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, %[[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], %[[VECTOR_BODY]] ]
+; CHECK-NEXT: [[TMP13:%.*]] = add i64 [[INDEX]], 1
+; CHECK-NEXT: [[TMP17:%.*]] = add i64 [[INDEX]], 2
+; CHECK-NEXT: [[TMP22:%.*]] = add i64 [[INDEX]], 3
+; CHECK-NEXT: [[TMP6:%.*]] = getelementptr inbounds i8, ptr [[SRC_TAGS]], i64 [[INDEX]]
+; CHECK-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i8>, ptr [[TMP6]], align 1, !alias.scope [[META0:![0-9]+]]
+; CHECK-NEXT: [[TMP7:%.*]] = icmp ne <4 x i8> [[WIDE_LOAD]], zeroinitializer
+; CHECK-NEXT: [[TMP8:%.*]] = getelementptr i32, ptr [[SRC]], i64 [[INDEX]]
+; CHECK-NEXT: [[WIDE_MASKED_LOAD:%.*]] = call <4 x i32> @llvm.masked.load.v4i32.p0(ptr align 4 [[TMP8]], <4 x i1> [[TMP7]], <4 x i32> poison), !alias.scope [[META3:![0-9]+]]
+; CHECK-NEXT: [[TMP9:%.*]] = getelementptr i32, ptr [[DST]], i64 [[INDEX]]
+; CHECK-NEXT: [[TMP10:%.*]] = getelementptr inbounds i8, ptr [[DST_TAGS]], i64 [[INDEX]]
+; CHECK-NEXT: [[TMP14:%.*]] = getelementptr inbounds i8, ptr [[DST_TAGS]], i64 [[TMP13]]
+; CHECK-NEXT: [[TMP15:%.*]] = getelementptr inbounds i8, ptr [[DST_TAGS]], i64 [[TMP17]]
+; CHECK-NEXT: [[TMP16:%.*]] = getelementptr inbounds i8, ptr [[DST_TAGS]], i64 [[TMP22]]
+; CHECK-NEXT: [[TMP11:%.*]] = select <4 x i1> [[TMP7]], <4 x i8> splat (i8 1), <4 x i8> zeroinitializer
+; CHECK-NEXT: [[TMP18:%.*]] = extractelement <4 x i8> [[TMP11]], i64 0
+; CHECK-NEXT: store i8 [[TMP18]], ptr [[TMP10]], align 1
+; CHECK-NEXT: [[TMP19:%.*]] = extractelement <4 x i8> [[TMP11]], i64 1
+; CHECK-NEXT: store i8 [[TMP19]], ptr [[TMP14]], align 1
+; CHECK-NEXT: [[TMP20:%.*]] = extractelement <4 x i8> [[TMP11]], i64 2
+; CHECK-NEXT: store i8 [[TMP20]], ptr [[TMP15]], align 1
+; CHECK-NEXT: [[TMP21:%.*]] = extractelement <4 x i8> [[TMP11]], i64 3
+; CHECK-NEXT: store i8 [[TMP21]], ptr [[TMP16]], align 1
+; CHECK-NEXT: call void @llvm.masked.store.v4i32.p0(<4 x i32> [[WIDE_MASKED_LOAD]], ptr align 4 [[TMP9]], <4 x i1> [[TMP7]]), !alias.scope [[META5:![0-9]+]], !noalias [[META7:![0-9]+]]
+; CHECK-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
+; CHECK-NEXT: [[TMP12:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
+; CHECK-NEXT: br i1 [[TMP12]], label %[[MIDDLE_BLOCK:.*]], label %[[VECTOR_BODY]], !llvm.loop [[LOOP8:![0-9]+]]
+; CHECK: [[MIDDLE_BLOCK]]:
+; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i64 [[N]], [[N_VEC]]
+; CHECK-NEXT: br i1 [[CMP_N]], [[EXIT:label %.*]], label %[[SCALAR_PH]]
+; CHECK: [[SCALAR_PH]]:
+;
+entry:
+ %src.tags = getelementptr inbounds nuw [4 x i8], ptr %src, i64 %src.len
+ %dst.tags = getelementptr inbounds nuw [4 x i8], ptr %dst, i64 %dst.len
+ br label %loop
+
+loop:
+ %iv = phi i64 [ 0, %entry ], [ %iv.next, %latch ]
+ %src.tag.gep = getelementptr inbounds i8, ptr %src.tags, i64 %iv
+ %tag = load i8, ptr %src.tag.gep, align 1
+ %is.value = icmp ne i8 %tag, 0
+ br i1 %is.value, label %value, label %singleton
+
+value:
+ %src.gep = getelementptr inbounds i32, ptr %src, i64 %iv
+ %v = load i32, ptr %src.gep, align 4
+ %dst.gep = getelementptr inbounds i32, ptr %dst, i64 %iv
+ %dst.tag.gep.1 = getelementptr inbounds i8, ptr %dst.tags, i64 %iv
+ store i8 1, ptr %dst.tag.gep.1, align 1
+ store i32 %v, ptr %dst.gep, align 4
+ br label %latch
+
+singleton:
+ %dst.tag.gep.0 = getelementptr inbounds i8, ptr %dst.tags, i64 %iv
+ store i8 0, ptr %dst.tag.gep.0, align 1
+ br label %latch
+
+latch:
+ %iv.next = add nuw nsw i64 %iv, 1
+ %ec = icmp eq i64 %iv.next, %n
+ br i1 %ec, label %exit, label %loop
+
+exit:
+ ret void
+}
>From 6a535e907aaf2aeff39913579f923a7566aa8756 Mon Sep 17 00:00:00 2001
From: Tim Besard <tim.besard at gmail.com>
Date: Thu, 24 Sep 2026 12:49:34 +0200
Subject: [PATCH 2/2] [LAA] Don't check same-address accesses against each
other on retry
When the dependence checker finds an unknown dependence and LAA retries
with runtime checks, canCheckPtrAtRT removes the dependence classes of
the accesses involved, so that each of their pointers gets its own
dependence set and checking group. This also separates accesses whose
dependences were safe, including different pointers that access the
same address in each iteration, such as stores to A[i] in both branches
of an if. Their bounds are identical, so the runtime check between them
always fails and the vector loop never runs.
Put such accesses back into a common dependence class when their
pointers have the same AddRec in the loop, all loads and stores through
them have the same type, and the stride is one element. They then
access the same bytes in each iteration, and bytes no other iteration
accesses, so the only dependence between them has a distance of 0,
which vectorization preserves. Being consecutive, they cannot be part
of an interleave group, which could reorder them; the dependences that
would prevent that are gone after the retry. As they share a class,
they also share a checking group, which removes their duplicate checks
against the other accesses.
This only changes the retry, after which the recorded dependences are
cleared, so passes like LoopLoadElimination, which rely on the checks
between classes when dependences are available, are not affected.
This comes up in Julia, which stores the values and the type tags of an
array of a union like Union{Float32, Nothing} in a single allocation:
the loops copying such arrays were vectorized with a runtime check that
always failed.
Assisted-by: Claude Code, Codex
---
llvm/lib/Analysis/LoopAccessAnalysis.cpp | 36 ++++++++++++++++++
.../retry-runtime-checks-same-address.ll | 37 +------------------
.../X86/runtime-checks-same-address.ll | 29 ++++-----------
3 files changed, 45 insertions(+), 57 deletions(-)
diff --git a/llvm/lib/Analysis/LoopAccessAnalysis.cpp b/llvm/lib/Analysis/LoopAccessAnalysis.cpp
index 409d9ceb5b8124..08952a2a7e277b 100644
--- a/llvm/lib/Analysis/LoopAccessAnalysis.cpp
+++ b/llvm/lib/Analysis/LoopAccessAnalysis.cpp
@@ -1526,6 +1526,42 @@ bool AccessAnalysis::canCheckPtrAtRT(RuntimePointerChecking &RtCheck,
continue;
}
+ // Removing dependence classes above also separates different pointers that
+ // access the same address in each iteration, like stores to A[i] in both
+ // branches of an if, and a runtime check between them would always fail.
+ // Put such accesses back into a common class if all loads and stores
+ // through them have the same type and the stride is one element: they then
+ // access the same bytes in each iteration, and bytes no other iteration
+ // accesses, so they only depend on each other within an iteration, which
+ // vectorization preserves. Being consecutive, they cannot be part of an
+ // interleave group, which could reorder them.
+ SmallDenseMap<std::pair<const SCEV *, Type *>, MemAccessInfo, 4>
+ SameAddressAccesses;
+ for (MemAccessInfo Access : AccessInfos) {
+ if (DepCands.contains(Access))
+ continue;
+ Value *Ptr = Access.getPointer();
+ SmallSetVector<Type *, 1> AccessTys;
+ for (bool IsWrite : {false, true})
+ if (auto It = Accesses.find(MemAccessInfo(Ptr, IsWrite));
+ It != Accesses.end())
+ AccessTys.insert_range(It->second);
+ if (AccessTys.size() != 1)
+ continue;
+ Type *AccessTy = AccessTys.front();
+ const auto *AR = dyn_cast<SCEVAddRecExpr>(PSE.getSCEV(Ptr));
+ if (!AR)
+ continue;
+ std::optional<int64_t> Stride =
+ getStrideFromAddRec(AR, TheLoop, AccessTy, Ptr, PSE);
+ if (Stride != 1 && Stride != -1)
+ continue;
+ auto [It, Inserted] =
+ SameAddressAccesses.try_emplace({AR, AccessTy}, Access);
+ if (!Inserted)
+ DepCands.unionSets(It->second, Access);
+ }
+
for (auto &Access : AccessInfos) {
for (const auto &AccessTy : Accesses[Access]) {
if (!createCheckForAccess(RtCheck, Access, AccessTy, StridesMap,
diff --git a/llvm/test/Analysis/LoopAccessAnalysis/retry-runtime-checks-same-address.ll b/llvm/test/Analysis/LoopAccessAnalysis/retry-runtime-checks-same-address.ll
index 74cc187bc26698..5455cde87150d5 100644
--- a/llvm/test/Analysis/LoopAccessAnalysis/retry-runtime-checks-same-address.ll
+++ b/llvm/test/Analysis/LoopAccessAnalysis/retry-runtime-checks-same-address.ll
@@ -16,28 +16,17 @@ define void @same_address_complementary_stores(ptr %A, i64 %off, ptr noalias %c,
; CHECK-NEXT: Check 0:
; CHECK-NEXT: Comparing group GRP0:
; CHECK-NEXT: %gep.sel.0 = getelementptr inbounds i8, ptr %A.sel, i64 %iv
-; CHECK-NEXT: Against group GRP1:
-; CHECK-NEXT: %gep.A = getelementptr inbounds i32, ptr %A, i64 %iv
-; CHECK-NEXT: Check 1:
-; CHECK-NEXT: Comparing group GRP0:
-; CHECK-NEXT: %gep.sel.0 = getelementptr inbounds i8, ptr %A.sel, i64 %iv
-; CHECK-NEXT: Against group GRP2:
; CHECK-NEXT: %gep.sel.1 = getelementptr inbounds i8, ptr %A.sel, i64 %iv
-; CHECK-NEXT: Check 2:
-; CHECK-NEXT: Comparing group GRP1:
+; CHECK-NEXT: Against group GRP1:
; CHECK-NEXT: %gep.A = getelementptr inbounds i32, ptr %A, i64 %iv
-; CHECK-NEXT: Against group GRP2:
-; CHECK-NEXT: %gep.sel.1 = getelementptr inbounds i8, ptr %A.sel, i64 %iv
; CHECK-NEXT: Grouped accesses:
; CHECK-NEXT: Group GRP0:
; CHECK-NEXT: (Low: (%off + %A) High: (%off + %n + %A))
; CHECK-NEXT: Member: {(%off + %A),+,1}<nw><%loop>
+; CHECK-NEXT: Member: {(%off + %A),+,1}<nw><%loop>
; CHECK-NEXT: Group GRP1:
; CHECK-NEXT: (Low: %A High: ((4 * %n) + %A))
; CHECK-NEXT: Member: {%A,+,4}<%loop>
-; CHECK-NEXT: Group GRP2:
-; CHECK-NEXT: (Low: (%off + %A) High: (%off + %n + %A))
-; CHECK-NEXT: Member: {(%off + %A),+,1}<nw><%loop>
; CHECK-EMPTY:
; CHECK-NEXT: Non vectorizable stores to invariant address were not found in loop.
; CHECK-NEXT: SCEV assumptions:
@@ -87,15 +76,6 @@ define void @same_address_load_store(ptr %A, i64 %off, i64 %n) {
; CHECK-NEXT: %gep.A = getelementptr inbounds i32, ptr %A, i64 %iv
; CHECK-NEXT: Against group GRP1:
; CHECK-NEXT: %gep.sel.1 = getelementptr inbounds i8, ptr %A.sel, i64 %iv
-; CHECK-NEXT: Check 1:
-; CHECK-NEXT: Comparing group GRP0:
-; CHECK-NEXT: %gep.A = getelementptr inbounds i32, ptr %A, i64 %iv
-; CHECK-NEXT: Against group GRP2:
-; CHECK-NEXT: %gep.sel.0 = getelementptr inbounds i8, ptr %A.sel, i64 %iv
-; CHECK-NEXT: Check 2:
-; CHECK-NEXT: Comparing group GRP1:
-; CHECK-NEXT: %gep.sel.1 = getelementptr inbounds i8, ptr %A.sel, i64 %iv
-; CHECK-NEXT: Against group GRP2:
; CHECK-NEXT: %gep.sel.0 = getelementptr inbounds i8, ptr %A.sel, i64 %iv
; CHECK-NEXT: Grouped accesses:
; CHECK-NEXT: Group GRP0:
@@ -104,8 +84,6 @@ define void @same_address_load_store(ptr %A, i64 %off, i64 %n) {
; CHECK-NEXT: Group GRP1:
; CHECK-NEXT: (Low: (%off + %A) High: (%off + %n + %A))
; CHECK-NEXT: Member: {(%off + %A),+,1}<nw><%loop>
-; CHECK-NEXT: Group GRP2:
-; CHECK-NEXT: (Low: (%off + %A) High: (%off + %n + %A))
; CHECK-NEXT: Member: {(%off + %A),+,1}<nw><%loop>
; CHECK-EMPTY:
; CHECK-NEXT: Non vectorizable stores to invariant address were not found in loop.
@@ -145,15 +123,6 @@ define void @same_address_negative_stride(ptr %A, i64 %off, i64 %n) {
; CHECK-NEXT: %gep.A = getelementptr inbounds i32, ptr %A, i64 %iv.next
; CHECK-NEXT: Against group GRP1:
; CHECK-NEXT: %gep.sel.0 = getelementptr inbounds i8, ptr %A.sel, i64 %iv.next
-; CHECK-NEXT: Check 1:
-; CHECK-NEXT: Comparing group GRP0:
-; CHECK-NEXT: %gep.A = getelementptr inbounds i32, ptr %A, i64 %iv.next
-; CHECK-NEXT: Against group GRP2:
-; CHECK-NEXT: %gep.sel.1 = getelementptr inbounds i8, ptr %A.sel, i64 %iv.next
-; CHECK-NEXT: Check 2:
-; CHECK-NEXT: Comparing group GRP1:
-; CHECK-NEXT: %gep.sel.0 = getelementptr inbounds i8, ptr %A.sel, i64 %iv.next
-; CHECK-NEXT: Against group GRP2:
; CHECK-NEXT: %gep.sel.1 = getelementptr inbounds i8, ptr %A.sel, i64 %iv.next
; CHECK-NEXT: Grouped accesses:
; CHECK-NEXT: Group GRP0:
@@ -162,8 +131,6 @@ define void @same_address_negative_stride(ptr %A, i64 %off, i64 %n) {
; CHECK-NEXT: Group GRP1:
; CHECK-NEXT: (Low: (-1 + (1 smin %n) + %off + %A) High: (%off + %n + %A))
; CHECK-NEXT: Member: {(-1 + %off + %n + %A),+,-1}<nw><%loop>
-; CHECK-NEXT: Group GRP2:
-; CHECK-NEXT: (Low: (-1 + (1 smin %n) + %off + %A) High: (%off + %n + %A))
; CHECK-NEXT: Member: {(-1 + %off + %n + %A),+,-1}<nw><%loop>
; CHECK-EMPTY:
; CHECK-NEXT: Non vectorizable stores to invariant address were not found in loop.
diff --git a/llvm/test/Transforms/LoopVectorize/X86/runtime-checks-same-address.ll b/llvm/test/Transforms/LoopVectorize/X86/runtime-checks-same-address.ll
index 2d7e8afbd4446b..bfbfa79535ba60 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/runtime-checks-same-address.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/runtime-checks-same-address.ll
@@ -27,28 +27,13 @@ define void @union_array_copy(ptr %src, i64 %src.len, ptr %dst, i64 %dst.len, i6
; CHECK-NEXT: [[TMP3:%.*]] = shl nuw nsw i64 [[SRC_LEN]], 2
; CHECK-NEXT: [[TMP4:%.*]] = add i64 [[N]], [[TMP3]]
; CHECK-NEXT: [[SCEVGEP3:%.*]] = getelementptr i8, ptr [[SRC]], i64 [[TMP4]]
-; CHECK-NEXT: [[BOUND2:%.*]] = icmp ult ptr [[DST_TAGS]], [[SCEVGEP]]
-; CHECK-NEXT: [[BOUND3:%.*]] = icmp ult ptr [[DST_TAGS]], [[SCEVGEP]]
-; CHECK-NEXT: [[BOUND0:%.*]] = icmp ult ptr [[DST_TAGS]], [[SCEVGEP1]]
-; CHECK-NEXT: [[BOUND1:%.*]] = icmp ult ptr [[DST]], [[SCEVGEP]]
-; CHECK-NEXT: [[FOUND_CONFLICT:%.*]] = and i1 [[BOUND0]], [[BOUND1]]
-; CHECK-NEXT: [[CONFLICT_RDX1:%.*]] = or i1 [[BOUND2]], [[FOUND_CONFLICT]]
-; CHECK-NEXT: [[BOUND04:%.*]] = icmp ult ptr [[DST_TAGS]], [[SCEVGEP2]]
-; CHECK-NEXT: [[BOUND15:%.*]] = icmp ult ptr [[SRC]], [[SCEVGEP]]
-; CHECK-NEXT: [[FOUND_CONFLICT6:%.*]] = and i1 [[BOUND04]], [[BOUND15]]
-; CHECK-NEXT: [[CONFLICT_RDX:%.*]] = or i1 [[CONFLICT_RDX1]], [[FOUND_CONFLICT6]]
-; CHECK-NEXT: [[BOUND07:%.*]] = icmp ult ptr [[DST_TAGS]], [[SCEVGEP3]]
-; CHECK-NEXT: [[BOUND18:%.*]] = icmp ult ptr [[SRC_TAGS]], [[SCEVGEP]]
-; CHECK-NEXT: [[FOUND_CONFLICT9:%.*]] = and i1 [[BOUND07]], [[BOUND18]]
-; CHECK-NEXT: [[CONFLICT_RDX10:%.*]] = or i1 [[CONFLICT_RDX]], [[FOUND_CONFLICT9]]
; CHECK-NEXT: [[BOUND011:%.*]] = icmp ult ptr [[DST_TAGS]], [[SCEVGEP1]]
; CHECK-NEXT: [[BOUND112:%.*]] = icmp ult ptr [[DST]], [[SCEVGEP]]
; CHECK-NEXT: [[FOUND_CONFLICT13:%.*]] = and i1 [[BOUND011]], [[BOUND112]]
-; CHECK-NEXT: [[CONFLICT_RDX14:%.*]] = or i1 [[CONFLICT_RDX10]], [[FOUND_CONFLICT13]]
; CHECK-NEXT: [[BOUND015:%.*]] = icmp ult ptr [[DST_TAGS]], [[SCEVGEP2]]
; CHECK-NEXT: [[BOUND116:%.*]] = icmp ult ptr [[SRC]], [[SCEVGEP]]
; CHECK-NEXT: [[FOUND_CONFLICT17:%.*]] = and i1 [[BOUND015]], [[BOUND116]]
-; CHECK-NEXT: [[CONFLICT_RDX18:%.*]] = or i1 [[CONFLICT_RDX14]], [[FOUND_CONFLICT17]]
+; CHECK-NEXT: [[CONFLICT_RDX18:%.*]] = or i1 [[FOUND_CONFLICT13]], [[FOUND_CONFLICT17]]
; CHECK-NEXT: [[BOUND019:%.*]] = icmp ult ptr [[DST_TAGS]], [[SCEVGEP3]]
; CHECK-NEXT: [[BOUND120:%.*]] = icmp ult ptr [[SRC_TAGS]], [[SCEVGEP]]
; CHECK-NEXT: [[FOUND_CONFLICT21:%.*]] = and i1 [[BOUND019]], [[BOUND120]]
@@ -83,17 +68,17 @@ define void @union_array_copy(ptr %src, i64 %src.len, ptr %dst, i64 %dst.len, i6
; CHECK-NEXT: [[TMP16:%.*]] = getelementptr inbounds i8, ptr [[DST_TAGS]], i64 [[TMP22]]
; CHECK-NEXT: [[TMP11:%.*]] = select <4 x i1> [[TMP7]], <4 x i8> splat (i8 1), <4 x i8> zeroinitializer
; CHECK-NEXT: [[TMP18:%.*]] = extractelement <4 x i8> [[TMP11]], i64 0
-; CHECK-NEXT: store i8 [[TMP18]], ptr [[TMP10]], align 1
+; CHECK-NEXT: store i8 [[TMP18]], ptr [[TMP10]], align 1, !alias.scope [[META5:![0-9]+]], !noalias [[META7:![0-9]+]]
; CHECK-NEXT: [[TMP19:%.*]] = extractelement <4 x i8> [[TMP11]], i64 1
-; CHECK-NEXT: store i8 [[TMP19]], ptr [[TMP14]], align 1
+; CHECK-NEXT: store i8 [[TMP19]], ptr [[TMP14]], align 1, !alias.scope [[META5]], !noalias [[META7]]
; CHECK-NEXT: [[TMP20:%.*]] = extractelement <4 x i8> [[TMP11]], i64 2
-; CHECK-NEXT: store i8 [[TMP20]], ptr [[TMP15]], align 1
+; CHECK-NEXT: store i8 [[TMP20]], ptr [[TMP15]], align 1, !alias.scope [[META5]], !noalias [[META7]]
; CHECK-NEXT: [[TMP21:%.*]] = extractelement <4 x i8> [[TMP11]], i64 3
-; CHECK-NEXT: store i8 [[TMP21]], ptr [[TMP16]], align 1
-; CHECK-NEXT: call void @llvm.masked.store.v4i32.p0(<4 x i32> [[WIDE_MASKED_LOAD]], ptr align 4 [[TMP9]], <4 x i1> [[TMP7]]), !alias.scope [[META5:![0-9]+]], !noalias [[META7:![0-9]+]]
+; CHECK-NEXT: store i8 [[TMP21]], ptr [[TMP16]], align 1, !alias.scope [[META5]], !noalias [[META7]]
+; CHECK-NEXT: call void @llvm.masked.store.v4i32.p0(<4 x i32> [[WIDE_MASKED_LOAD]], ptr align 4 [[TMP9]], <4 x i1> [[TMP7]]), !alias.scope [[META9:![0-9]+]], !noalias [[META10:![0-9]+]]
; CHECK-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
; CHECK-NEXT: [[TMP12:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
-; CHECK-NEXT: br i1 [[TMP12]], label %[[MIDDLE_BLOCK:.*]], label %[[VECTOR_BODY]], !llvm.loop [[LOOP8:![0-9]+]]
+; CHECK-NEXT: br i1 [[TMP12]], label %[[MIDDLE_BLOCK:.*]], label %[[VECTOR_BODY]], !llvm.loop [[LOOP11:![0-9]+]]
; CHECK: [[MIDDLE_BLOCK]]:
; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i64 [[N]], [[N_VEC]]
; CHECK-NEXT: br i1 [[CMP_N]], [[EXIT:label %.*]], label %[[SCALAR_PH]]
More information about the llvm-commits
mailing list