[Mlir-commits] [mlir] [mlir][VectorToLLVM] emit `inbounds|nuw` GEP flags when lowering `vector.load/store` (PR #202118)
Federico Bruzzone
llvmlistbot at llvm.org
Thu Jun 11 06:38:02 PDT 2026
FedericoBruzzone wrote:
@banach-space I managed to get away sooner than expected. Let me explain here.
> NOTE: `affine-super-vectorize` now emits `in_bounds = [true]` on `vector.transfer_read/write` when statically provable (#201180, and the refactoring #202766) . `convert-vector-to-llvm` sees an in-bounds transfer and emits a plain `llvm.load/llvm.store` instead of `llvm.intr.masked.load`. The masked-intrinsic overhead (~3x on AArch64/NEON) is gone.
As an example, I'll use a single high-level `linalg.matmul` (no vectors):
```mlir
// tiny_matmul.mlir
func.func @matmul(%A: memref<16x16xf32>,
%B: memref<16x16xf32>,
%C: memref<16x16xf32>) {
linalg.matmul ins(%A, %B : memref<16x16xf32>, memref<16x16xf32>)
outs(%C : memref<16x16xf32>)
return
}
```
Now consider two pipelines. The only difference: Path 1 inserts `--affine-super-vectorize="virtual-vector-size=4"` (and `--convert-ub-to-llvm` for the UB ops it introduces) between tiling and `--lower-affine`.
## 1. tiled, LLVM auto-vectorization
```
mlir-opt tiny_matmul.mlir \
--convert-linalg-to-affine-loops \
--affine-loop-tile="tile-sizes=4,4,4" \
--lower-affine \
--convert-scf-to-cf \
--convert-cf-to-llvm \
--convert-vector-to-llvm \
--convert-arith-to-llvm \
--convert-math-to-llvm \
--convert-func-to-llvm \
--finalize-memref-to-llvm \
--convert-index-to-llvm \
--reconcile-unrealized-casts \
| mlir-translate --mlir-to-llvmir -o path1.ll
```
; Generated LLVM IR (index arithmetic and loads)
```llvm
%71 = mul nuw nsw i64 %57, 16 ; <----- nsw+nuw: LLVM (I think SCEV) knows no overflow is possible
%72 = add nuw nsw i64 %71, %67
%73 = getelementptr inbounds nuw float, ptr %70, i64 %72
; ^^^^^^^^^^^^
; this pointer stays within the allocation bounds
```
## 2. explicit affine-super-vectorize
Generated LLVM IR (same index arithmetic, flags gone):
```llvm
%71 = mul i64 %57, 16 ; <----- no nsw/nuw: LLVM (I think SCEV) cannot prove monotonicity
%72 = add i64 %71, %67
%73 = getelementptr float, ptr %70, i64 %72 ; <----- no inbounds: BasicAA falls back to conservative
```
Yet the IR already contains explicit vectors:
```llvm
%74 = load <1 x float>, ptr %73, align 4
%98 = fmul <4 x float> %77, %87 ; <----- vector type is there
%99 = fadd <4 x float> %97, %98
```
## **Why would this matter if MLIR already generates vectorized code?**
Because "vectorized LLVM IR" is not necessary equal to "vectorized assembly".
**Path 2** has `fmul <4 x float>` in the IR (the vector type is correct) but the data arrives via `ld1.s` lane-by-lane. Without `inbounds nuw` on the GEPs, LLVM (I think BasicAA) cannot prove that the 4 accesses are contiguous and non-overlapping, so it cannot lower `load <4 x float>` to a single `ldr q` (128-bit in "one cycle"). Instead, each lane of the vector is loaded individually before the multiply:
`clang -O3 -march=native -S path2.ll -o path2.s`
```asm
ldr s5, [x7] ; lane 0 — scalar load
add x24, x7, #4
ld1.s { v5 }[1], [x24] ; lane 1
add x24, x7, #8
ld1.s { v5 }[2], [x24] ; lane 2
add x24, x7, #12
ld1.s { v5 }[3], [x24] ; lane 3
fmul.4s v4, v5, v4[0] ; <----- v5 feeds here: 4 separate loads (1 vector multiply)
fadd.4s v3, v3, v4
```
**Path 1** (tiled, no explicit vectorization) has no vector ops in the IR at all, but because `nuw nsw` and `inbounds nuw` are present, LLVM (again I think SCEV and BasicAA) have the guarantees it needs. After `-O3 -march=native`, LLVM is free to choose the optimal strategy: for this 16x16 case it fully unrolls with clean scalar loads, for larger matrices it auto-vectorizes with contiguous wide loads (from my experiments with 512x512 and Tile=16 LLVM generates 68 `ldr q` + 64 `fmul.4s` with zero` ld1.s`).
`clang -O3 -march=native -S path1.ll -o path1.s`
```
ldr s1, [x15, x2] ; <----- direct scalar load, no lane-by-lane
fmul s0, s0, s1
ldr s3, [x15, x3]
fmul s1, s1, s3
ldr s1, [x7]
ldr s3, [x15, x4]
fmul s1, s1, s3
```
This is the concrete mechanism behind the ~2.4x slowdown I measured in [my silly and nonsensical study](https://federicobruzzone.github.io/posts/mlir-study.html) (the RQ5).
Below is the AI-generated script based on this comment of mine. It does everything and displays what I'm reporting.
I'm using a machine with M4 Pro Aarch64.
<details>
<summary>check_matmul_pipelines.sh</summary>
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
MLIR_FILE="${SCRIPT_DIR}/tiny_matmul.mlir"
LL_PATH1="${SCRIPT_DIR}/path1.ll"
LL_PATH2="${SCRIPT_DIR}/path2.ll"
ASM_PATH1="${SCRIPT_DIR}/path1.s"
ASM_PATH2="${SCRIPT_DIR}/path2.s"
PASS=0
FAIL=0
pass() { PASS=$((PASS+1)); echo " ✓ $1"; }
fail() { FAIL=$((FAIL+1)); echo " ✗ $1"; }
cleanup() {
rm -f "$MLIR_FILE" "$LL_PATH1" "$LL_PATH2" "$ASM_PATH1" "$ASM_PATH2"
}
trap cleanup EXIT
# --- 1. Write the MLIR test file ---
cat > "$MLIR_FILE" << 'MLIR'
func.func @matmul(%A: memref<16x16xf32>,
%B: memref<16x16xf32>,
%C: memref<16x16xf32>) {
linalg.matmul ins(%A, %B : memref<16x16xf32>, memref<16x16xf32>)
outs(%C : memref<16x16xf32>)
return
}
MLIR
echo ">>> Created $MLIR_FILE"
# --- 2. PATH 1: tiled, LLVM auto-vectorization ---
echo ""
echo "=== Path 1: tiled (affine-loop-tile), no explicit vectorization ==="
mlir-opt "$MLIR_FILE" \
--convert-linalg-to-affine-loops \
--affine-loop-tile="tile-sizes=4,4,4" \
--lower-affine \
--convert-scf-to-cf \
--convert-cf-to-llvm \
--convert-vector-to-llvm \
--convert-arith-to-llvm \
--convert-math-to-llvm \
--convert-func-to-llvm \
--finalize-memref-to-llvm \
--convert-index-to-llvm \
--reconcile-unrealized-casts \
| mlir-translate --mlir-to-llvmir -o "$LL_PATH1"
echo ">>> LLVM IR written to $LL_PATH1"
# Check 1a: mul nuw nsw i64 (snippet: mul nuw nsw i64 %57, 16)
if grep -qE 'mul nuw nsw i64 %[0-9]+, 16' "$LL_PATH1" 2>/dev/null; then
pass "Path 1 LLVM [1/3]: 'mul nuw nsw i64 %.., 16' found"
else
fail "Path 1 LLVM [1/3]: missing 'mul nuw nsw i64'"
fi
# Check 1b: add nuw nsw i64 (snippet: add nuw nsw i64 %71, %67)
if grep -qE 'add nuw nsw i64 %[0-9]+, %[0-9]+' "$LL_PATH1" 2>/dev/null; then
pass "Path 1 LLVM [2/3]: 'add nuw nsw i64 %.., %..' found"
else
fail "Path 1 LLVM [2/3]: missing 'add nuw nsw i64'"
fi
# Check 1c: getelementptr inbounds nuw float (snippet: getelementptr inbounds nuw float, ptr %70, i64 %72)
if grep -qE 'getelementptr inbounds nuw float, ptr %[0-9]+, i64 %[0-9]+' "$LL_PATH1" 2>/dev/null; then
pass "Path 1 LLVM [3/3]: 'getelementptr inbounds nuw float' found"
else
fail "Path 1 LLVM [3/3]: missing 'getelementptr inbounds nuw float'"
fi
# --- 3. PATH 2: explicit affine-super-vectorize ---
echo ""
echo "=== Path 2: with affine-super-vectorize ==="
mlir-opt "$MLIR_FILE" \
--convert-linalg-to-affine-loops \
--affine-loop-tile="tile-sizes=4,4,4" \
--affine-super-vectorize="virtual-vector-size=4" \
--convert-ub-to-llvm \
--lower-affine \
--convert-scf-to-cf \
--convert-cf-to-llvm \
--convert-vector-to-llvm \
--convert-arith-to-llvm \
--convert-math-to-llvm \
--convert-func-to-llvm \
--finalize-memref-to-llvm \
--convert-index-to-llvm \
--reconcile-unrealized-casts \
| mlir-translate --mlir-to-llvmir -o "$LL_PATH2"
echo ">>> LLVM IR written to $LL_PATH2"
# Check 2a: mul WITHOUT nuw/nsw (snippet: mul i64 %57, 16 — no flags)
if grep -qE 'mul i64 %[0-9]+, 16' "$LL_PATH2" 2>/dev/null; then
if grep -qE 'mul nuw nsw i64 %[0-9]+, 16' "$LL_PATH2" 2>/dev/null; then
fail "Path 2 LLVM [1/5]: expected plain 'mul i64' but found 'mul nuw nsw'"
else
pass "Path 2 LLVM [1/5]: 'mul i64 %.., 16' (no nuw/nsw)"
fi
else
fail "Path 2 LLVM [1/5]: missing 'mul i64'"
fi
# Check 2b: add WITHOUT nuw/nsw (snippet: add i64 %71, %67 — no flags)
if grep -qE 'add i64 %[0-9]+, %[0-9]+' "$LL_PATH2" 2>/dev/null; then
if grep -qE 'add nuw nsw i64 %[0-9]+, %[0-9]+' "$LL_PATH2" 2>/dev/null; then
fail "Path 2 LLVM [2/5]: expected plain 'add i64' but found 'add nuw nsw'"
else
pass "Path 2 LLVM [2/5]: 'add i64 %.., %..' (no nuw/nsw)"
fi
else
fail "Path 2 LLVM [2/5]: missing 'add i64'"
fi
# Check 2c: getelementptr WITHOUT inbounds (snippet: getelementptr float, ptr %70, i64 %72)
if grep -qE 'getelementptr float, ptr %[0-9]+, i64 %[0-9]+' "$LL_PATH2" 2>/dev/null; then
if grep -qE 'getelementptr inbounds nuw float' "$LL_PATH2" 2>/dev/null; then
fail "Path 2 LLVM [3/5]: expected plain GEP but found 'inbounds nuw'"
else
pass "Path 2 LLVM [3/5]: 'getelementptr float' (no inbounds)"
fi
else
if grep -qE 'getelementptr float, ptr %[0-9]+' "$LL_PATH2" 2>/dev/null; then
if grep -qE 'getelementptr inbounds nuw float' "$LL_PATH2" 2>/dev/null; then
fail "Path 2 LLVM [3/5]: expected plain GEP but found 'inbounds nuw'"
else
pass "Path 2 LLVM [3/5]: 'getelementptr float' (no inbounds)"
fi
else
fail "Path 2 LLVM [3/5]: missing 'getelementptr float'"
fi
fi
# Check 2d: load <1 x float> (snippet: load <1 x float>, ptr %73, align 4)
if grep -qE 'load <1 x float>' "$LL_PATH2" 2>/dev/null; then
pass "Path 2 LLVM [4/5]: 'load <1 x float>' found"
else
fail "Path 2 LLVM [4/5]: missing 'load <1 x float>'"
fi
# Check 2e: fmul <4 x float> + fadd <4 x float> (snippet: fmul <4 x float> %77, %87 / fadd <4 x float> %97, %98)
VEC_COUNT=0
grep -qE 'fmul <4 x float>' "$LL_PATH2" 2>/dev/null && VEC_COUNT=$((VEC_COUNT+1))
grep -qE 'fadd <4 x float>' "$LL_PATH2" 2>/dev/null && VEC_COUNT=$((VEC_COUNT+1))
if [ "$VEC_COUNT" -eq 2 ]; then
pass "Path 2 LLVM [5/5]: 'fmul <4 x float>' + 'fadd <4 x float>' found"
else
fail "Path 2 LLVM [5/5]: missing vector fmul/fadd (found $VEC_COUNT/2)"
fi
# --- 4. Compile both to assembly with clang ---
echo ""
echo "=== Compiling to assembly (clang -O3 -march=native) ==="
clang -O3 -march=native -S "$LL_PATH1" -o "$ASM_PATH1" 2>/dev/null
echo ">>> Assembly written to $ASM_PATH1"
clang -O3 -march=native -S "$LL_PATH2" -o "$ASM_PATH2" 2>/dev/null
echo ">>> Assembly written to $ASM_PATH2"
# Check 3a: Path 2 ASM — ld1.s lane-by-lane loads (snippet: 3x ld1.s { v5 }[1..3], [x24])
LD1_COUNT=0
if grep -q 'ld1.s' "$ASM_PATH2" 2>/dev/null; then
LD1_COUNT=$(grep -c 'ld1.s' "$ASM_PATH2" 2>/dev/null || echo 0)
fi
if [ "$LD1_COUNT" -ge 3 ]; then
pass "Path 2 ASM [1/3]: $LD1_COUNT ld1.s (lane-by-lane pattern)"
else
fail "Path 2 ASM [1/3]: expected >=3 ld1.s, found $LD1_COUNT"
fi
# Check 3b: Path 2 ASM — fmul.4s (snippet: fmul.4s v4, v5, v4[0])
if grep -qE 'fmul\.4s' "$ASM_PATH2" 2>/dev/null; then
pass "Path 2 ASM [2/3]: 'fmul.4s' found"
else
fail "Path 2 ASM [2/3]: missing 'fmul.4s'"
fi
# Check 3c: Path 2 ASM — fadd.4s (snippet: fadd.4s v3, v3, v4)
if grep -qE 'fadd\.4s' "$ASM_PATH2" 2>/dev/null; then
pass "Path 2 ASM [3/3]: 'fadd.4s' found"
else
fail "Path 2 ASM [3/3]: missing 'fadd.4s'"
fi
# Check 3d: Path 1 ASM — NO ld1.s (contiguous/scalar, not lane-by-lane)
if grep -q 'ld1.s' "$ASM_PATH1" 2>/dev/null; then
fail "Path 1 ASM [1/2]: unexpectedly contains ld1.s"
else
pass "Path 1 ASM [1/2]: no ld1.s (scalar instead)"
fi
# Check 3e: Path 1 ASM — scalar ldr / fmul (snippet: ldr s1, [x15, x2] / fmul s0, s0, s1 / ...)
SCALAR_FMUL=$(grep -cE 'fmul\s+s[0-9]+' "$ASM_PATH1" 2>/dev/null || echo 0)
if [ "$SCALAR_FMUL" -ge 1 ]; then
pass "Path 1 ASM [2/2]: scalar 'fmul s' found ($SCALAR_FMUL)"
else
fail "Path 1 ASM [2/2]: no scalar 'fmul s'"
fi
# --- Summary ---
echo ""
echo "============================================"
echo " Results: $PASS passed, $FAIL failed"
echo "============================================"
if [ "$FAIL" -ne 0 ]; then
echo " Intermediate files left for inspection:"
echo " $LL_PATH1"
echo " $LL_PATH2"
echo " $ASM_PATH1"
echo " $ASM_PATH2"
echo ""
echo " Re-run manually with reduced pipeline to debug."
exit 1
else
echo " All checks passed."
exit 0
fi
</details>
https://github.com/llvm/llvm-project/pull/202118
More information about the Mlir-commits
mailing list