[llvm-branch-commits] [llvm] arm cortex A53: model updated according on test done over raspeberry pi3 (PR #213238)
Michele Dionisio via llvm-branch-commits
llvm-branch-commits at lists.llvm.org
Fri Jul 31 03:27:17 PDT 2026
https://github.com/mdionisio created https://github.com/llvm/llvm-project/pull/213238
[a53-validation.tar.gz](https://github.com/user-attachments/files/30584165/a53-validation.tar.gz)
# Cortex-A53 scheduling model: hardware validation report
Supporting material for the `AArch64SchedA53.td` series. Every latency and
throughput figure in the series was measured on real silicon; this document
reports the end-to-end validation of the resulting model and describes the one
discrepancy that remains, which is a limitation of the llvm-mca simulator
rather than of the model data.
## Test environment
| item | value |
|----------------|---------------------------------------------------------------------------------|
| hardware | Raspberry Pi 3, Cortex-A53 (MIDR part `0xd03`), 4 cores, aarch64 |
| OS | Debian 13 (trixie), 64-bit userspace, kernel 6.18 |
| measurement | `perf stat -e cycles`, hardware cycle counter, pinned with `taskset` |
| FP environment | FPCR flush-to-zero enabled, so denormal penalties cannot perturb timings |
| reference | `llvm-exegesis-22`, calibrated against `MADDXrrr` (measures 4.01, model says 4) |
| baseline model | stock `llvm-mca` 22.1.8 (Debian package, unpatched) |
| patched model | this series |
All timings are best-of-3, 20,000,000 iterations per run.
## Method
Eight kernels of ~18 to 22 instructions each. The kernel bodies are
**byte-identical** between the hardware benchmark and the llvm-mca input: the
benchmark's assembly is generated by concatenating the same `.s` files that are
fed to llvm-mca, and the equality is asserted by the build script. Each kernel
is a loop, so the figure compared is steady-state cycles per iteration.
## Results
| kernel | real | stock model | this series | stock error | series error | what it exercises |
|--------|-------|-------------|-------------|-------------|--------------|---------------------------------------------------------|
| k1 | 49.01 | 95.02 | 56.02 | +93.9% | **+14.3%** | FP arithmetic, MAC forwarding, reciprocal step/estimate |
| k2 | 25.01 | 68.02 | 26.01 | +172.0% | **+4.0%** | integer NEON ALU, bitwise logic, compares |
| k3 | 37.01 | 78.02 | 40.02 | +110.8% | **+8.1%** | the whole shift family |
| k4 | 20.01 | 46.02 | 21.02 | +130.0% | **+5.0%** | widening/narrowing multiply, integer MAC |
| k5 | 25.01 | 82.02 | 29.01 | +228.0% | **+16.0%** | permutes, compare/select, FMOV |
| k6 | 35.01 | 72.02 | 45.02 | +105.7% | +28.6% | mixed realistic (FP + integer, D + Q, conversions) |
| k7 | 40.01 | 74.02 | 41.02 | +85.0% | **+2.5%** | dependency chain, independent work appended |
| k8 | 32.01 | 66.02 | 34.01 | +106.2% | **+6.2%** | same instructions as k7, interleaved |
| | | | **mean** | **+128.9%** | **+10.6%** | |
The stock model is wrong by a factor of 2.3 on average, and by up to 3.3x. The
series brings the mean error to 10.6%.
### Interleaving
k7 and k8 contain exactly the same instructions and differ only in their order.
On hardware, interleaving the independent work into the dependency chain's
stalls makes it essentially free:
| ordering | real | model |
|-------------------------------------------|-----------|-------|
| independent work appended after the chain | 40.01 | 41.02 |
| independent work interleaved | **32.01** | 34.01 |
Hardware gains 20%; the model predicts 17%. The model therefore rewards
interleaving in the right direction and with close to the right magnitude.
## The remaining discrepancy
k6 is the outlier at +28.6%. It is not a modelling error: it is an artefact of
how llvm-mca simulates a strictly in-order core.
### It is not the latencies
Every dependency in k6 was measured individually and matches the model exactly:
* ten cross-type pairs from k6's chain (`add`→`shl`, `shl`→`scvtf`,
`scvtf`→`fadd`, `fadd`→`fcvtzs`, `fcvtzs`→`smull`, `smull`→`and`,
`and`→`zip1`, `zip1`→`mla`, `mla`→`sshr`, `sshr`→`xtn`): model equals hardware
on all ten;
* `fmla` (through a multiplicand) → `fadd`: model 12.00, hardware 12.00;
* k6's critical chain in isolation: model **32.02**, hardware **32.01**.
### Bisection
Adding one group of instructions at a time to that chain isolates the trigger:
| variant | real | model | |
|------------------------------------------------|-------|-------|-----------|
| chain alone | 32.01 | 32.02 | accurate |
| + 2 `fmul` (Q-form, latency 4) | 34.01 | 35.02 | accurate |
| + 2 `fmla` (Q-form, latency 8) | 37.01 | 43.02 | **+16%** |
| + 2 `fmla` (D-form, latency 8) | 36.01 | 43.02 | **+19%** |
| + 2 `fmla`, result never consumed by the chain | 37.01 | 43.02 | identical |
So it is neither the Q-form occupancy (D-form behaves the same) nor the
consumption of the result (identical with and without). What matters is the
producer's *latency*.
### Direct evidence
In the timeline, an `add` with **no dependency whatsoever** is dispatched:
* at cycle **6** when it follows an `fmla` (latency 8);
* at cycle **2** when it follows an `fmul` (latency 4).
That is a delay of (latency - 2) imposed on a completely independent
instruction. Real hardware issues it at cycle 1.
### Why
With `MicroOpBufferSize = 0`, llvm-mca models strict in-order issue: an
instruction that is not ready blocks everything behind it, so a long-latency
instruction pushes back the independent instructions that follow it. The
Cortex-A53 dispatches in order too, but its NEON/FP pipeline sits behind a
decoupled queue, so a later independent operation is not held up by an earlier
long-latency one.
This cannot be fixed from the scheduling model: no combination of latency,
throughput or forwarding values changes it, and it is orthogonal to the data
this series corrects. It affects absolute cycle predictions for code that mixes
long-latency operations into a dependency chain; it does not affect the model's
relative judgements, and, as k7/k8 show, it does not stop the model from
rewarding interleaving.
## Reproduction
Everything needed is under `a53-validation/` (attached archive):
```text
a53-validation/
kernels/ k1.s .. k8.s the eight kernels of the results table
bisect/ k9.s .. kG.s the variants used to bisect the discrepancy
run-hw.sh measure on an A53 target (needs gcc and perf)
run-mca.sh predict with a given llvm-mca binary
```
Each `.s` file is one loop body, written once and used by both sides. The
hardware runner wraps it in a register-initialisation preamble, enables
flush-to-zero, and asserts that the body it assembled is byte-identical to the
`.s` file before timing it; llvm-mca is given the very same file. That is what
makes the two columns comparable.
On the A53 target:
```sh
./run-hw.sh # the eight kernels
./run-hw.sh bisect # the bisection variants
```
On any host, for the model side:
```sh
./run-mca.sh /path/to/patched/bin/llvm-mca # this series
./run-mca.sh /usr/bin/llvm-mca-22 # unpatched baseline
```
`run-hw.sh` reports best-of-3 over 20,000,000 iterations; `run-mca.sh` reports
`Total Cycles` over 100 simulated iterations. Both print cycles per iteration,
so the outputs line up directly with the tables above.
### Which kernel covers what
| kernel | commits it exercises |
| ------ | -------------------------------------------------------------------------------------------- |
| k1 | FP arithmetic and conversion; FP MAC latency and addend forwarding; reciprocal step/estimate |
| k2 | integer ASIMD ALU; bitwise logic; compares; CMTST/CLS |
| k3 | the shift family: plain, rounding, saturating, shift-accumulate, narrowing, widening |
| k4 | widening multiply and multiply-accumulate; narrowing; integer MAC forwarding |
| k5 | permutes (ZIP/TRN/UZP/EXT/TBL/DUP); FP compare and select; FMOV both directions |
| k6 | all of the above mixed, plus integer/FP conversions |
| k7, k8 | the two-lane datapath, and the effect of instruction ordering |
>From 8cc023280449ce4a2160e834fe08cf9f7a93442a Mon Sep 17 00:00:00 2001
From: Michele Dionisio <michele.dionisio at gmail.com>
Date: Thu, 30 Jul 2026 14:54:24 +0200
Subject: [PATCH 01/15] [AArch64] Split Cortex-A65/Neoverse-E1 onto their own
scheduling model (NFC)
Cortex-A65, Cortex-A65AE and Neoverse-E1 reused CortexA53Model directly.
Add CortexA65Model -- a verbatim copy of CortexA53Model -- and point
those three cores at it, so that upcoming Cortex-A53-specific scheduling
tuning does not affect them.
The distinction matters: the Cortex-A53 (and A34/A35) is an in-order
core with a 64-bit ASIMD datapath, whereas Cortex-A65/Neoverse-E1 are
out-of-order cores with a 128-bit ASIMD datapath (GCC likewise assigns
them the Cortex-A73 cost model, not Cortex-A53). Keeping them on the
shared model would force A53-specific tuning onto them.
Non-functional change: CortexA65Model is a byte-for-byte copy of
CortexA53Model, so scheduling for the three cores is unchanged and the
CodeGen/AArch64 test suite passes unmodified. A model derived from the
Cortex-A65 Software Optimization Guide is left as future work.
Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
llvm/lib/Target/AArch64/AArch64.td | 1 +
llvm/lib/Target/AArch64/AArch64Processors.td | 6 +-
llvm/lib/Target/AArch64/AArch64SchedA65.td | 306 +++++++++++++++++++
3 files changed, 310 insertions(+), 3 deletions(-)
create mode 100644 llvm/lib/Target/AArch64/AArch64SchedA65.td
diff --git a/llvm/lib/Target/AArch64/AArch64.td b/llvm/lib/Target/AArch64/AArch64.td
index db3b19fcf43dd..7cef3599f6e63 100644
--- a/llvm/lib/Target/AArch64/AArch64.td
+++ b/llvm/lib/Target/AArch64/AArch64.td
@@ -116,6 +116,7 @@ def PAUnsupported : AArch64Unsupported;
include "AArch64SchedA320.td"
include "AArch64SchedA53.td"
+include "AArch64SchedA65.td"
include "AArch64SchedA55.td"
include "AArch64SchedA510.td"
include "AArch64SchedA57.td"
diff --git a/llvm/lib/Target/AArch64/AArch64Processors.td b/llvm/lib/Target/AArch64/AArch64Processors.td
index 6f38673c50e23..6983b26e512e5 100644
--- a/llvm/lib/Target/AArch64/AArch64Processors.td
+++ b/llvm/lib/Target/AArch64/AArch64Processors.td
@@ -1419,9 +1419,9 @@ def : ProcessorModel<"c1-nano", CortexA510Model,
ProcessorFeatures.C1Nano, [TuneC1Nano]>;
def : ProcessorModel<"cortex-a57", CortexA57Model, ProcessorFeatures.A53,
[TuneA57]>;
-def : ProcessorModel<"cortex-a65", CortexA53Model, ProcessorFeatures.A65,
+def : ProcessorModel<"cortex-a65", CortexA65Model, ProcessorFeatures.A65,
[TuneA65]>;
-def : ProcessorModel<"cortex-a65ae", CortexA53Model, ProcessorFeatures.A65,
+def : ProcessorModel<"cortex-a65ae", CortexA65Model, ProcessorFeatures.A65,
[TuneA65]>;
def : ProcessorModel<"cortex-a72", CortexA57Model, ProcessorFeatures.A53,
[TuneA72]>;
@@ -1477,7 +1477,7 @@ def : ProcessorModel<"gb10", NeoverseV3Model, ProcessorFeatures.GB10,
[TuneX925]>;
def : ProcessorModel<"grace", NeoverseV2Model, ProcessorFeatures.Grace,
[TuneNeoverseV2]>;
-def : ProcessorModel<"neoverse-e1", CortexA53Model,
+def : ProcessorModel<"neoverse-e1", CortexA65Model,
ProcessorFeatures.NeoverseE1, [TuneNeoverseE1]>;
def : ProcessorModel<"neoverse-n1", NeoverseN1Model,
ProcessorFeatures.NeoverseN1, [TuneNeoverseN1]>;
diff --git a/llvm/lib/Target/AArch64/AArch64SchedA65.td b/llvm/lib/Target/AArch64/AArch64SchedA65.td
new file mode 100644
index 0000000000000..773649b63e212
--- /dev/null
+++ b/llvm/lib/Target/AArch64/AArch64SchedA65.td
@@ -0,0 +1,306 @@
+//==- AArch64SchedA65.td - Cortex-A65 Scheduling Definitions -*- tablegen -*-=//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines the itinerary class data for the ARM Cortex A65 processors.
+//
+// Cortex-A65/A65AE and Neoverse-E1 previously reused CortexA53Model directly.
+// This model is a verbatim copy of it, split out so that Cortex-A53-specific
+// tuning (which targets an in-order core with a 64-bit ASIMD datapath) is not
+// applied to these out-of-order cores, whose Advanced SIMD datapath is 128 bits
+// wide. It is a placeholder: a model derived from the Cortex-A65 Software
+// Optimization Guide is future work.
+//
+//===----------------------------------------------------------------------===//
+
+// ===---------------------------------------------------------------------===//
+// The following definitions describe the simpler per-operand machine model.
+// This works with MachineScheduler. See MCSchedule.h for details.
+
+// Cortex-A65 machine model for scheduling and other instruction cost heuristics.
+def CortexA65Model : SchedMachineModel {
+ let MicroOpBufferSize = 0; // Explicitly set to zero since A65 is in-order.
+ let IssueWidth = 2; // 2 micro-ops are dispatched per cycle.
+ let LoadLatency = 3; // Optimistic load latency assuming bypass.
+ // This is overridden by OperandCycles if the
+ // Itineraries are queried instead.
+ let MispredictPenalty = 9; // Based on "Cortex-A65 Software Optimisation
+ // Specification - Instruction Timings"
+ // v 1.0 Spreadsheet
+ let CompleteModel = 1;
+
+ list<Predicate> UnsupportedFeatures = !listconcat(SVEUnsupported.F,
+ PAUnsupported.F,
+ SMEUnsupported.F,
+ [HasMTE, HasCSSC]);
+}
+
+
+//===----------------------------------------------------------------------===//
+// Define each kind of processor resource and number available.
+
+// Modeling each pipeline as a ProcResource using the BufferSize = 0 since
+// Cortex-A65 is in-order.
+
+def A65UnitALU : ProcResource<2> { let BufferSize = 0; } // Int ALU
+def A65UnitMAC : ProcResource<1> { let BufferSize = 0; } // Int MAC
+def A65UnitDiv : ProcResource<1> { let BufferSize = 0; } // Int Division
+def A65UnitLdSt : ProcResource<1> { let BufferSize = 0; } // Load/Store
+def A65UnitB : ProcResource<1> { let BufferSize = 0; } // Branch
+def A65UnitFPALU : ProcResource<1> { let BufferSize = 0; } // FP ALU
+def A65UnitFPMDS : ProcResource<1> { let BufferSize = 0; } // FP Mult/Div/Sqrt
+
+
+//===----------------------------------------------------------------------===//
+// Subtarget-specific SchedWrite types which both map the ProcResources and
+// set the latency.
+
+let SchedModel = CortexA65Model in {
+
+// ALU - Despite having a full latency of 4, most of the ALU instructions can
+// forward a cycle earlier and then two cycles earlier in the case of a
+// shift-only instruction. These latencies will be incorrect when the
+// result cannot be forwarded, but modeling isn't rocket surgery.
+def : WriteRes<WriteImm, [A65UnitALU]> { let Latency = 3; }
+def : WriteRes<WriteI, [A65UnitALU]> { let Latency = 3; }
+def : WriteRes<WriteISReg, [A65UnitALU]> { let Latency = 3; }
+def : WriteRes<WriteIEReg, [A65UnitALU]> { let Latency = 3; }
+def : WriteRes<WriteIS, [A65UnitALU]> { let Latency = 2; }
+def : WriteRes<WriteExtr, [A65UnitALU]> { let Latency = 3; }
+
+// MAC
+def : WriteRes<WriteIM32, [A65UnitMAC]> { let Latency = 4; }
+def : WriteRes<WriteIM64, [A65UnitMAC]> { let Latency = 4; }
+
+// Div
+def : WriteRes<WriteID32, [A65UnitDiv]> { let Latency = 4; }
+def : WriteRes<WriteID64, [A65UnitDiv]> { let Latency = 4; }
+
+// Load
+def : WriteRes<WriteLD, [A65UnitLdSt]> { let Latency = 4; }
+def : WriteRes<WriteLDIdx, [A65UnitLdSt]> { let Latency = 4; }
+def : WriteRes<WriteLDHi, [A65UnitLdSt]> { let Latency = 4; }
+
+// Vector Load - Vector loads take 1-5 cycles to issue. For the WriteVecLd
+// below, choosing the median of 3 which makes the latency 6.
+// May model this more carefully in the future. The remaining
+// A65WriteVLD# types represent the 1-5 cycle issues explicitly.
+def : WriteRes<WriteVLD, [A65UnitLdSt]> { let Latency = 6;
+ let ReleaseAtCycles = [3]; }
+def A65WriteVLD1 : SchedWriteRes<[A65UnitLdSt]> { let Latency = 4; }
+def A65WriteVLD2 : SchedWriteRes<[A65UnitLdSt]> { let Latency = 5;
+ let ReleaseAtCycles = [2]; }
+def A65WriteVLD3 : SchedWriteRes<[A65UnitLdSt]> { let Latency = 6;
+ let ReleaseAtCycles = [3]; }
+def A65WriteVLD4 : SchedWriteRes<[A65UnitLdSt]> { let Latency = 7;
+ let ReleaseAtCycles = [4]; }
+def A65WriteVLD5 : SchedWriteRes<[A65UnitLdSt]> { let Latency = 8;
+ let ReleaseAtCycles = [5]; }
+
+// Pre/Post Indexing - Performed as part of address generation which is already
+// accounted for in the WriteST* latencies below
+def : WriteRes<WriteAdr, []> { let Latency = 0; }
+
+// Store
+def : WriteRes<WriteST, [A65UnitLdSt]> { let Latency = 4; }
+def : WriteRes<WriteSTP, [A65UnitLdSt]> { let Latency = 4; }
+def : WriteRes<WriteSTIdx, [A65UnitLdSt]> { let Latency = 4; }
+def : WriteRes<WriteSTX, [A65UnitLdSt]> { let Latency = 4; }
+
+// Vector Store - Similar to vector loads, can take 1-3 cycles to issue.
+def : WriteRes<WriteVST, [A65UnitLdSt]> { let Latency = 5;
+ let ReleaseAtCycles = [2];}
+def A65WriteVST1 : SchedWriteRes<[A65UnitLdSt]> { let Latency = 4; }
+def A65WriteVST2 : SchedWriteRes<[A65UnitLdSt]> { let Latency = 5;
+ let ReleaseAtCycles = [2]; }
+def A65WriteVST3 : SchedWriteRes<[A65UnitLdSt]> { let Latency = 6;
+ let ReleaseAtCycles = [3]; }
+
+def : WriteRes<WriteAtomic, []> { let Unsupported = 1; }
+
+// Branch
+def : WriteRes<WriteBr, [A65UnitB]>;
+def : WriteRes<WriteBrReg, [A65UnitB]>;
+def : WriteRes<WriteSys, [A65UnitB]>;
+def : WriteRes<WriteBarrier, [A65UnitB]>;
+def : WriteRes<WriteHint, [A65UnitB]>;
+
+// FP ALU
+def : WriteRes<WriteF, [A65UnitFPALU]> { let Latency = 6; }
+def : WriteRes<WriteFCmp, [A65UnitFPALU]> { let Latency = 6; }
+def : WriteRes<WriteFCvt, [A65UnitFPALU]> { let Latency = 6; }
+def : WriteRes<WriteFCopy, [A65UnitFPALU]> { let Latency = 6; }
+def : WriteRes<WriteFImm, [A65UnitFPALU]> { let Latency = 6; }
+def : WriteRes<WriteVd, [A65UnitFPALU]> { let Latency = 6; }
+def : WriteRes<WriteVq, [A65UnitFPALU]> { let Latency = 6; }
+
+// FP Mul, Div, Sqrt
+def : WriteRes<WriteFMul, [A65UnitFPMDS]> { let Latency = 6; }
+def : WriteRes<WriteFDiv, [A65UnitFPMDS]> { let Latency = 33;
+ let ReleaseAtCycles = [29]; }
+def A65WriteFMAC : SchedWriteRes<[A65UnitFPMDS]> { let Latency = 10; }
+def A65WriteFDivSP : SchedWriteRes<[A65UnitFPMDS]> { let Latency = 18;
+ let ReleaseAtCycles = [14]; }
+def A65WriteFDivDP : SchedWriteRes<[A65UnitFPMDS]> { let Latency = 33;
+ let ReleaseAtCycles = [29]; }
+def A65WriteFSqrtSP : SchedWriteRes<[A65UnitFPMDS]> { let Latency = 17;
+ let ReleaseAtCycles = [13]; }
+def A65WriteFSqrtDP : SchedWriteRes<[A65UnitFPMDS]> { let Latency = 32;
+ let ReleaseAtCycles = [28]; }
+
+//===----------------------------------------------------------------------===//
+// Subtarget-specific SchedRead types.
+
+// No forwarding for these reads.
+def : ReadAdvance<ReadExtrHi, 0>;
+def : ReadAdvance<ReadAdrBase, 0>;
+def : ReadAdvance<ReadST, 0>;
+def : ReadAdvance<ReadVLD, 0>;
+
+// ALU - Most operands in the ALU pipes are not needed for two cycles. Shiftable
+// operands are needed one cycle later if and only if they are to be
+// shifted. Otherwise, they too are needed two cycles later. This same
+// ReadAdvance applies to Extended registers as well, even though there is
+// a separate SchedPredicate for them.
+def : ReadAdvance<ReadI, 2, [WriteImm,WriteI,
+ WriteISReg, WriteIEReg,WriteIS,
+ WriteID32,WriteID64,
+ WriteIM32,WriteIM64]>;
+def A65ReadShifted : SchedReadAdvance<1, [WriteImm,WriteI,
+ WriteISReg, WriteIEReg,WriteIS,
+ WriteID32,WriteID64,
+ WriteIM32,WriteIM64]>;
+def A65ReadNotShifted : SchedReadAdvance<2, [WriteImm,WriteI,
+ WriteISReg, WriteIEReg,WriteIS,
+ WriteID32,WriteID64,
+ WriteIM32,WriteIM64]>;
+def A65ReadISReg : SchedReadVariant<[
+ SchedVar<RegShiftedPred, [A65ReadShifted]>,
+ SchedVar<NoSchedPred, [A65ReadNotShifted]>]>;
+def : SchedAlias<ReadISReg, A65ReadISReg>;
+
+def A65ReadIEReg : SchedReadVariant<[
+ SchedVar<RegExtendedPred, [A65ReadShifted]>,
+ SchedVar<NoSchedPred, [A65ReadNotShifted]>]>;
+def : SchedAlias<ReadIEReg, A65ReadIEReg>;
+
+// MAC - Operands are generally needed one cycle later in the MAC pipe.
+// Accumulator operands are needed two cycles later.
+def : ReadAdvance<ReadIM, 1, [WriteImm,WriteI,
+ WriteISReg, WriteIEReg,WriteIS,
+ WriteID32,WriteID64,
+ WriteIM32,WriteIM64]>;
+def : ReadAdvance<ReadIMA, 2, [WriteImm,WriteI,
+ WriteISReg, WriteIEReg,WriteIS,
+ WriteID32,WriteID64,
+ WriteIM32,WriteIM64]>;
+
+// Div
+def : ReadAdvance<ReadID, 1, [WriteImm,WriteI,
+ WriteISReg, WriteIEReg,WriteIS,
+ WriteID32,WriteID64,
+ WriteIM32,WriteIM64]>;
+
+//===----------------------------------------------------------------------===//
+// Subtarget-specific InstRWs.
+
+//---
+// Miscellaneous
+//---
+def : InstRW<[WriteI], (instrs COPY)>;
+
+//---
+// Vector Loads
+//---
+def : InstRW<[A65WriteVLD1], (instregex "LD1i(8|16|32|64)$")>;
+def : InstRW<[A65WriteVLD1], (instregex "LD1Rv(8b|4h|2s|1d|16b|8h|4s|2d)$")>;
+def : InstRW<[A65WriteVLD1], (instregex "LD1Onev(8b|4h|2s|1d|16b|8h|4s|2d)$")>;
+def : InstRW<[A65WriteVLD2], (instregex "LD1Twov(8b|4h|2s|1d|16b|8h|4s|2d)$")>;
+def : InstRW<[A65WriteVLD3], (instregex "LD1Threev(8b|4h|2s|1d|16b|8h|4s|2d)$")>;
+def : InstRW<[A65WriteVLD4], (instregex "LD1Fourv(8b|4h|2s|1d|16b|8h|4s|2d)$")>;
+def : InstRW<[WriteAdr, A65WriteVLD1], (instregex "LD1i(8|16|32|64)_POST$")>;
+def : InstRW<[WriteAdr, A65WriteVLD1], (instregex "LD1Rv(8b|4h|2s|1d|16b|8h|4s|2d)_POST$")>;
+def : InstRW<[WriteAdr, A65WriteVLD1], (instregex "LD1Onev(8b|4h|2s|1d|16b|8h|4s|2d)_POST$")>;
+def : InstRW<[WriteAdr, A65WriteVLD2], (instregex "LD1Twov(8b|4h|2s|1d|16b|8h|4s|2d)_POST$")>;
+def : InstRW<[WriteAdr, A65WriteVLD3], (instregex "LD1Threev(8b|4h|2s|1d|16b|8h|4s|2d)_POST$")>;
+def : InstRW<[WriteAdr, A65WriteVLD4], (instregex "LD1Fourv(8b|4h|2s|1d|16b|8h|4s|2d)_POST$")>;
+
+def : InstRW<[A65WriteVLD1], (instregex "LD2i(8|16|32|64)$")>;
+def : InstRW<[A65WriteVLD1], (instregex "LD2Rv(8b|4h|2s|1d|16b|8h|4s|2d)$")>;
+def : InstRW<[A65WriteVLD2], (instregex "LD2Twov(8b|4h|2s)$")>;
+def : InstRW<[A65WriteVLD4], (instregex "LD2Twov(16b|8h|4s|2d)$")>;
+def : InstRW<[WriteAdr, A65WriteVLD1], (instregex "LD2i(8|16|32|64)_POST$")>;
+def : InstRW<[WriteAdr, A65WriteVLD1], (instregex "LD2Rv(8b|4h|2s|1d|16b|8h|4s|2d)_POST$")>;
+def : InstRW<[WriteAdr, A65WriteVLD2], (instregex "LD2Twov(8b|4h|2s)_POST$")>;
+def : InstRW<[WriteAdr, A65WriteVLD4], (instregex "LD2Twov(16b|8h|4s|2d)_POST$")>;
+
+def : InstRW<[A65WriteVLD2], (instregex "LD3i(8|16|32|64)$")>;
+def : InstRW<[A65WriteVLD2], (instregex "LD3Rv(8b|4h|2s|1d|16b|8h|4s|2d)$")>;
+def : InstRW<[A65WriteVLD4], (instregex "LD3Threev(8b|4h|2s|1d|16b|8h|4s)$")>;
+def : InstRW<[A65WriteVLD3], (instregex "LD3Threev2d$")>;
+def : InstRW<[WriteAdr, A65WriteVLD2], (instregex "LD3i(8|16|32|64)_POST$")>;
+def : InstRW<[WriteAdr, A65WriteVLD2], (instregex "LD3Rv(8b|4h|2s|1d|16b|8h|4s|2d)_POST$")>;
+def : InstRW<[WriteAdr, A65WriteVLD4], (instregex "LD3Threev(8b|4h|2s|1d|16b|8h|4s)_POST$")>;
+def : InstRW<[WriteAdr, A65WriteVLD3], (instregex "LD3Threev2d_POST$")>;
+
+def : InstRW<[A65WriteVLD2], (instregex "LD4i(8|16|32|64)$")>;
+def : InstRW<[A65WriteVLD2], (instregex "LD4Rv(8b|4h|2s|1d|16b|8h|4s|2d)$")>;
+def : InstRW<[A65WriteVLD5], (instregex "LD4Fourv(8b|4h|2s|1d|16b|8h|4s)$")>;
+def : InstRW<[A65WriteVLD4], (instregex "LD4Fourv(2d)$")>;
+def : InstRW<[WriteAdr, A65WriteVLD2], (instregex "LD4i(8|16|32|64)_POST$")>;
+def : InstRW<[WriteAdr, A65WriteVLD2], (instregex "LD4Rv(8b|4h|2s|1d|16b|8h|4s|2d)_POST$")>;
+def : InstRW<[WriteAdr, A65WriteVLD5], (instregex "LD4Fourv(8b|4h|2s|1d|16b|8h|4s)_POST$")>;
+def : InstRW<[WriteAdr, A65WriteVLD4], (instregex "LD4Fourv(2d)_POST$")>;
+
+//---
+// Vector Stores
+//---
+def : InstRW<[A65WriteVST1], (instregex "ST1i(8|16|32|64)$")>;
+def : InstRW<[A65WriteVST1], (instregex "ST1Onev(8b|4h|2s|1d|16b|8h|4s|2d)$")>;
+def : InstRW<[A65WriteVST1], (instregex "ST1Twov(8b|4h|2s|1d|16b|8h|4s|2d)$")>;
+def : InstRW<[A65WriteVST2], (instregex "ST1Threev(8b|4h|2s|1d|16b|8h|4s|2d)$")>;
+def : InstRW<[A65WriteVST2], (instregex "ST1Fourv(8b|4h|2s|1d|16b|8h|4s|2d)$")>;
+def : InstRW<[WriteAdr, A65WriteVST1], (instregex "ST1i(8|16|32|64)_POST$")>;
+def : InstRW<[WriteAdr, A65WriteVST1], (instregex "ST1Onev(8b|4h|2s|1d|16b|8h|4s|2d)_POST$")>;
+def : InstRW<[WriteAdr, A65WriteVST1], (instregex "ST1Twov(8b|4h|2s|1d|16b|8h|4s|2d)_POST$")>;
+def : InstRW<[WriteAdr, A65WriteVST2], (instregex "ST1Threev(8b|4h|2s|1d|16b|8h|4s|2d)_POST$")>;
+def : InstRW<[WriteAdr, A65WriteVST2], (instregex "ST1Fourv(8b|4h|2s|1d|16b|8h|4s|2d)_POST$")>;
+
+def : InstRW<[A65WriteVST1], (instregex "ST2i(8|16|32|64)$")>;
+def : InstRW<[A65WriteVST1], (instregex "ST2Twov(8b|4h|2s)$")>;
+def : InstRW<[A65WriteVST2], (instregex "ST2Twov(16b|8h|4s|2d)$")>;
+def : InstRW<[WriteAdr, A65WriteVST1], (instregex "ST2i(8|16|32|64)_POST$")>;
+def : InstRW<[WriteAdr, A65WriteVST1], (instregex "ST2Twov(8b|4h|2s)_POST$")>;
+def : InstRW<[WriteAdr, A65WriteVST2], (instregex "ST2Twov(16b|8h|4s|2d)_POST$")>;
+
+def : InstRW<[A65WriteVST2], (instregex "ST3i(8|16|32|64)$")>;
+def : InstRW<[A65WriteVST3], (instregex "ST3Threev(8b|4h|2s|1d|16b|8h|4s)$")>;
+def : InstRW<[A65WriteVST2], (instregex "ST3Threev(2d)$")>;
+def : InstRW<[WriteAdr, A65WriteVST2], (instregex "ST3i(8|16|32|64)_POST$")>;
+def : InstRW<[WriteAdr, A65WriteVST3], (instregex "ST3Threev(8b|4h|2s|1d|16b|8h|4s)_POST$")>;
+def : InstRW<[WriteAdr, A65WriteVST2], (instregex "ST3Threev(2d)_POST$")>;
+
+def : InstRW<[A65WriteVST2], (instregex "ST4i(8|16|32|64)$")>;
+def : InstRW<[A65WriteVST3], (instregex "ST4Fourv(8b|4h|2s|1d|16b|8h|4s)$")>;
+def : InstRW<[A65WriteVST2], (instregex "ST4Fourv(2d)$")>;
+def : InstRW<[WriteAdr, A65WriteVST2], (instregex "ST4i(8|16|32|64)_POST$")>;
+def : InstRW<[WriteAdr, A65WriteVST3], (instregex "ST4Fourv(8b|4h|2s|1d|16b|8h|4s)_POST$")>;
+def : InstRW<[WriteAdr, A65WriteVST2], (instregex "ST4Fourv(2d)_POST$")>;
+
+//---
+// Floating Point MAC, DIV, SQRT
+//---
+def : InstRW<[A65WriteFMAC], (instregex "^FN?M(ADD|SUB).*")>;
+def : InstRW<[A65WriteFMAC], (instregex "^FML(A|S).*")>;
+def : InstRW<[A65WriteFDivSP], (instrs FDIVSrr)>;
+def : InstRW<[A65WriteFDivDP], (instrs FDIVDrr)>;
+def : InstRW<[A65WriteFDivSP], (instregex "^FDIVv.*32$")>;
+def : InstRW<[A65WriteFDivDP], (instregex "^FDIVv.*64$")>;
+def : InstRW<[A65WriteFSqrtSP], (instregex "^.*SQRT.*32$")>;
+def : InstRW<[A65WriteFSqrtDP], (instregex "^.*SQRT.*64$")>;
+
+}
>From 6d763569feae65983cde1088d976458feb246091 Mon Sep 17 00:00:00 2001
From: Michele Dionisio <michele.dionisio at gmail.com>
Date: Thu, 30 Jul 2026 14:57:01 +0200
Subject: [PATCH 02/15] [AArch64] Model the 64-bit ASIMD datapath in the
Cortex-A53 sched model
The Cortex-A53 (and A34/A35, which share CortexA53Model) is an in-order
core that issues two 64-bit (D-form) Advanced SIMD/FP operations per
cycle but only one 128-bit (Q-form) operation -- the datapath moves 128
bits per cycle. CortexA53Model gives Q-form and D-form ops the same
single-cycle occupancy (WriteVq == WriteVd), so a Q-form operation is
modelled as costing the same per instruction as a D-form one. For the
same amount of data that makes one Q-form op (1 cycle) look cheaper than
the two D-form ops (2 cycles) it replaces, when on hardware they are
equal; the model therefore under-costs Q-form relative to D-form.
Give WriteVq -- and a new A53WriteFMACQ for Q-form FMLA/FMLS -- two-cycle
occupancy via ReleaseAtCycles = [2], and split the FMLA/FMLS InstRW
overrides into D-form and Q-form variants.
Cortex-A65/Neoverse-E1 were moved off this model in the preceding commit:
they are out-of-order cores with a 128-bit datapath and must not receive
this penalty. The change now applies only to the in-order A53-class
cores (A53/A34/A35).
Hardware validation. Measured on a Raspberry Pi 3 (Cortex-A53, 1.4 GHz,
performance governor) with perf-counted throughput microbenchmarks of 16
independent ops per iteration (reciprocal throughput, cycles/op):
op D-form Q-form
add (8b/16b) 0.56 1.06
fadd (2s/4s) 0.56 1.06
fmla (2s/4s) 0.63 1.06
The Q-form:D-form ratio is ~1.9, i.e. a Q-form op costs twice a D-form
op, which is what this change encodes. Underlying latencies were
cross-checked with llvm-exegesis (calibrated against MADDXrrr = 4 on the
same board).
Scope of the fix. This corrects the *relative* Q:D cost, not the
absolute throughput. The board runs D-form ops at ~0.5 cycles/op because
the A53 dual-issues two 64-bit FP/ASIMD ops per cycle, which the single
A53UnitFPALU/A53UnitFPMDS (ProcResource<1>) cannot express -- it models
D-form at 1.0. This change leaves that as-is and makes the FP/ASIMD
pessimism uniform (both D and Q ~2x conservative in absolute terms)
instead of inconsistent (Q previously accurate, D 2x slow). A fully
absolute-accurate model would represent the 128-bit/cycle datapath as a
two-lane resource (D-form = 1 lane, Q-form = 2 lanes); that is a larger
change left as follow-up.
Crypto. AES/SHA/PMULL are exempted from the Q-form penalty and keep
single-cycle occupancy: the crypto unit is 128 bits wide (per ARM) and,
importantly, this preserves back-to-back issue of fused AES pairs
(FeatureFuseAES) -- misched-fusion-aes.ll regresses on Cortex-A53
without the exemption. The exact crypto throughput could NOT be
benchmarked (the test board's A53 lacks the crypto extension), so this
rests on ARM's documented 128-bit crypto datapath and should be
re-verified on crypto-capable A53 silicon.
Co-Authored-By: Claude Fable 5 <noreply at anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
llvm/lib/Target/AArch64/AArch64SchedA53.td | 32 ++++++-
.../AArch64/Cortex/A53-asimd-q-form.s | 94 +++++++++++++++++++
2 files changed, 124 insertions(+), 2 deletions(-)
create mode 100644 llvm/test/tools/llvm-mca/AArch64/Cortex/A53-asimd-q-form.s
diff --git a/llvm/lib/Target/AArch64/AArch64SchedA53.td b/llvm/lib/Target/AArch64/AArch64SchedA53.td
index 66715b9d1db8b..6e1a93cae6422 100644
--- a/llvm/lib/Target/AArch64/AArch64SchedA53.td
+++ b/llvm/lib/Target/AArch64/AArch64SchedA53.td
@@ -129,13 +129,19 @@ def : WriteRes<WriteFCvt, [A53UnitFPALU]> { let Latency = 6; }
def : WriteRes<WriteFCopy, [A53UnitFPALU]> { let Latency = 6; }
def : WriteRes<WriteFImm, [A53UnitFPALU]> { let Latency = 6; }
def : WriteRes<WriteVd, [A53UnitFPALU]> { let Latency = 6; }
-def : WriteRes<WriteVq, [A53UnitFPALU]> { let Latency = 6; }
+// The ASIMD datapath is 64 bits wide: 128-bit (Q-form) operations occupy the
+// FP/ASIMD pipe for two cycles.
+def : WriteRes<WriteVq, [A53UnitFPALU]> { let Latency = 6;
+ let ReleaseAtCycles = [2]; }
// FP Mul, Div, Sqrt
def : WriteRes<WriteFMul, [A53UnitFPMDS]> { let Latency = 6; }
def : WriteRes<WriteFDiv, [A53UnitFPMDS]> { let Latency = 33;
let ReleaseAtCycles = [29]; }
def A53WriteFMAC : SchedWriteRes<[A53UnitFPMDS]> { let Latency = 10; }
+// Q-form multiply-accumulate occupies the MAC pipe for two cycles.
+def A53WriteFMACQ : SchedWriteRes<[A53UnitFPMDS]> { let Latency = 10;
+ let ReleaseAtCycles = [2]; }
def A53WriteFDivSP : SchedWriteRes<[A53UnitFPMDS]> { let Latency = 18;
let ReleaseAtCycles = [14]; }
def A53WriteFDivDP : SchedWriteRes<[A53UnitFPMDS]> { let Latency = 33;
@@ -287,8 +293,30 @@ def : InstRW<[WriteAdr, A53WriteVST2], (instregex "ST4Fourv(2d)_POST$")>;
//---
// Floating Point MAC, DIV, SQRT
//---
+// Q-form crypto (AES/SHA/PMULL) keeps single-cycle occupancy instead of the
+// WriteVq two-cycle penalty. The crypto unit is 128 bits wide (per ARM), and,
+// crucially, this preserves back-to-back issue of fused AES pairs
+// (FeatureFuseAES): removing this exemption regresses misched-fusion-aes.ll on
+// Cortex-A53. The exact crypto throughput was NOT benchmarked --
+// the test board's A53 (Raspberry Pi 3) lacks the crypto extension -- so this
+// rests on ARM's documented 128-bit crypto datapath; the fusion behaviour it
+// protects is what the test covers. To be re-checked on crypto-capable silicon.
+def A53WriteCrypto : SchedWriteRes<[A53UnitFPALU]> { let Latency = 6; }
+def : InstRW<[A53WriteCrypto], (instregex "^AES[DE]rr$", "^AESI?MCrr$",
+ "^SHA1(C|M|P)rrr$", "^SHA1Hrr$",
+ "^SHA1SU0rrr$", "^SHA1SU1rr$",
+ "^SHA256H2?rrr$", "^SHA256SU0rr$",
+ "^SHA256SU1rrr$", "^PMULLv")>;
+
def : InstRW<[A53WriteFMAC], (instregex "^FN?M(ADD|SUB).*")>;
-def : InstRW<[A53WriteFMAC], (instregex "^FML(A|S).*")>;
+def : InstRW<[A53WriteFMAC], (instregex "^FML(A|S)L.*")>;
+def : InstRW<[A53WriteFMAC],
+ (instregex "^FML(A|S)(v2f32|v4f16)$",
+ "^FML(A|S)v1i(16|32|64)_indexed$",
+ "^FML(A|S)(v2i32|v4i16)_indexed$")>;
+def : InstRW<[A53WriteFMACQ],
+ (instregex "^FML(A|S)(v4f32|v2f64|v8f16)$",
+ "^FML(A|S)(v4i32|v2i64|v8i16)_indexed$")>;
def : InstRW<[A53WriteFDivSP], (instrs FDIVSrr)>;
def : InstRW<[A53WriteFDivDP], (instrs FDIVDrr)>;
def : InstRW<[A53WriteFDivSP], (instregex "^FDIVv.*32$")>;
diff --git a/llvm/test/tools/llvm-mca/AArch64/Cortex/A53-asimd-q-form.s b/llvm/test/tools/llvm-mca/AArch64/Cortex/A53-asimd-q-form.s
new file mode 100644
index 0000000000000..7fca9ce7c30e7
--- /dev/null
+++ b/llvm/test/tools/llvm-mca/AArch64/Cortex/A53-asimd-q-form.s
@@ -0,0 +1,94 @@
+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=aarch64-none-linux-gnu -mcpu=cortex-a53 -instruction-tables < %s | FileCheck %s
+
+# The Cortex-A53 ASIMD datapath is 64 bits wide: 128-bit (Q-form) operations
+# occupy the FP/ASIMD pipes for two cycles, so their reciprocal throughput is
+# twice that of the equivalent 64-bit (D-form) operations.
+
+fmla v0.4s, v1.4s, v2.4s
+fmla v0.2d, v1.2d, v2.2d
+fmla v0.2s, v1.2s, v2.2s
+fmadd s0, s1, s2, s3
+fadd v0.4s, v1.4s, v2.4s
+fadd v0.2s, v1.2s, v2.2s
+add v0.4s, v1.4s, v2.4s
+add v0.2s, v1.2s, v2.2s
+zip1 v0.4s, v1.4s, v2.4s
+zip1 v0.2s, v1.2s, v2.2s
+trn1 v0.16b, v1.16b, v2.16b
+trn1 v0.8b, v1.8b, v2.8b
+uzp1 v0.4s, v1.4s, v2.4s
+uzp1 v0.2s, v1.2s, v2.2s
+ext v0.16b, v1.16b, v2.16b, #4
+ext v0.8b, v1.8b, v2.8b, #4
+dup v0.4s, v1.s[1]
+dup v0.2s, v1.s[1]
+faddp v0.4s, v1.4s, v2.4s
+faddp v0.2s, v1.2s, v2.2s
+
+# CHECK: Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK: [1] [2] [3] [4] [5] [6] Instructions:
+# CHECK-NEXT: 1 10 2.00 fmla v0.4s, v1.4s, v2.4s
+# CHECK-NEXT: 1 10 2.00 fmla v0.2d, v1.2d, v2.2d
+# CHECK-NEXT: 1 10 1.00 fmla v0.2s, v1.2s, v2.2s
+# CHECK-NEXT: 1 10 1.00 fmadd s0, s1, s2, s3
+# CHECK-NEXT: 1 6 2.00 fadd v0.4s, v1.4s, v2.4s
+# CHECK-NEXT: 1 6 1.00 fadd v0.2s, v1.2s, v2.2s
+# CHECK-NEXT: 1 6 2.00 add v0.4s, v1.4s, v2.4s
+# CHECK-NEXT: 1 6 1.00 add v0.2s, v1.2s, v2.2s
+# CHECK-NEXT: 1 6 2.00 zip1 v0.4s, v1.4s, v2.4s
+# CHECK-NEXT: 1 6 1.00 zip1 v0.2s, v1.2s, v2.2s
+# CHECK-NEXT: 1 6 2.00 trn1 v0.16b, v1.16b, v2.16b
+# CHECK-NEXT: 1 6 1.00 trn1 v0.8b, v1.8b, v2.8b
+# CHECK-NEXT: 1 6 2.00 uzp1 v0.4s, v1.4s, v2.4s
+# CHECK-NEXT: 1 6 1.00 uzp1 v0.2s, v1.2s, v2.2s
+# CHECK-NEXT: 1 6 2.00 ext v0.16b, v1.16b, v2.16b, #4
+# CHECK-NEXT: 1 6 1.00 ext v0.8b, v1.8b, v2.8b, #4
+# CHECK-NEXT: 1 6 2.00 dup v0.4s, v1.s[1]
+# CHECK-NEXT: 1 6 1.00 dup v0.2s, v1.s[1]
+# CHECK-NEXT: 1 6 2.00 faddp v0.4s, v1.4s, v2.4s
+# CHECK-NEXT: 1 6 1.00 faddp v0.2s, v1.2s, v2.2s
+
+# CHECK: Resources:
+# CHECK-NEXT: [0.0] - A53UnitALU
+# CHECK-NEXT: [0.1] - A53UnitALU
+# CHECK-NEXT: [1] - A53UnitB
+# CHECK-NEXT: [2] - A53UnitDiv
+# CHECK-NEXT: [3] - A53UnitFPALU
+# CHECK-NEXT: [4] - A53UnitFPMDS
+# CHECK-NEXT: [5] - A53UnitLdSt
+# CHECK-NEXT: [6] - A53UnitMAC
+
+# CHECK: Resource pressure per iteration:
+# CHECK-NEXT: [0.0] [0.1] [1] [2] [3] [4] [5] [6]
+# CHECK-NEXT: - - - - 24.00 6.00 - -
+
+# CHECK: Resource pressure by instruction:
+# CHECK-NEXT: [0.0] [0.1] [1] [2] [3] [4] [5] [6] Instructions:
+# CHECK-NEXT: - - - - - 2.00 - - fmla v0.4s, v1.4s, v2.4s
+# CHECK-NEXT: - - - - - 2.00 - - fmla v0.2d, v1.2d, v2.2d
+# CHECK-NEXT: - - - - - 1.00 - - fmla v0.2s, v1.2s, v2.2s
+# CHECK-NEXT: - - - - - 1.00 - - fmadd s0, s1, s2, s3
+# CHECK-NEXT: - - - - 2.00 - - - fadd v0.4s, v1.4s, v2.4s
+# CHECK-NEXT: - - - - 1.00 - - - fadd v0.2s, v1.2s, v2.2s
+# CHECK-NEXT: - - - - 2.00 - - - add v0.4s, v1.4s, v2.4s
+# CHECK-NEXT: - - - - 1.00 - - - add v0.2s, v1.2s, v2.2s
+# CHECK-NEXT: - - - - 2.00 - - - zip1 v0.4s, v1.4s, v2.4s
+# CHECK-NEXT: - - - - 1.00 - - - zip1 v0.2s, v1.2s, v2.2s
+# CHECK-NEXT: - - - - 2.00 - - - trn1 v0.16b, v1.16b, v2.16b
+# CHECK-NEXT: - - - - 1.00 - - - trn1 v0.8b, v1.8b, v2.8b
+# CHECK-NEXT: - - - - 2.00 - - - uzp1 v0.4s, v1.4s, v2.4s
+# CHECK-NEXT: - - - - 1.00 - - - uzp1 v0.2s, v1.2s, v2.2s
+# CHECK-NEXT: - - - - 2.00 - - - ext v0.16b, v1.16b, v2.16b, #4
+# CHECK-NEXT: - - - - 1.00 - - - ext v0.8b, v1.8b, v2.8b, #4
+# CHECK-NEXT: - - - - 2.00 - - - dup v0.4s, v1.s[1]
+# CHECK-NEXT: - - - - 1.00 - - - dup v0.2s, v1.s[1]
+# CHECK-NEXT: - - - - 2.00 - - - faddp v0.4s, v1.4s, v2.4s
+# CHECK-NEXT: - - - - 1.00 - - - faddp v0.2s, v1.2s, v2.2s
>From 4bdd5908a2c7fbf8d5b80127ad53599416fa2080 Mon Sep 17 00:00:00 2001
From: Michele Dionisio <michele.dionisio at gmail.com>
Date: Thu, 30 Jul 2026 16:38:25 +0200
Subject: [PATCH 03/15] [AArch64] Fix Cortex-A53 reciprocal step/estimate
latencies
On Cortex-A53 the floating-point reciprocal and reciprocal-square-root
Newton-Raphson step instructions (FRECPS/FRSQRTS) have a latency of 8 and
the estimate instructions (FRECPE/FRSQRTE) a latency of 4, rather than the
flat 6 that WriteVd assigns. Worse, FRSQRTS/FRSQRTE were caught by the
".*SQRT.*" InstRW globs and charged the full square-root latency (17/32
cycles) -- a reciprocal step is not a square root.
Add write classes for the step (8) and estimate (4), tighten the square-
root globs to "^FSQRT" so they only match a genuine FSQRT, and route the
reciprocal ops to the new classes. The Q-form variants keep the two-cycle
datapath occupancy introduced for this model.
Measured with llvm-exegesis and a perf-counted microbenchmark on a
Raspberry Pi 3 (Cortex-A53, 1.4 GHz): FRECPS/FRSQRTS = 8.0 cycles,
FRECPE/FRSQRTE = 4.0. Applies only to CortexA53Model (A53/A34/A35);
Cortex-A65/Neoverse-E1 are on their own model.
Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
llvm/lib/Target/AArch64/AArch64SchedA53.td | 27 ++++++++++++++++++++--
1 file changed, 25 insertions(+), 2 deletions(-)
diff --git a/llvm/lib/Target/AArch64/AArch64SchedA53.td b/llvm/lib/Target/AArch64/AArch64SchedA53.td
index 6e1a93cae6422..5586c917c0869 100644
--- a/llvm/lib/Target/AArch64/AArch64SchedA53.td
+++ b/llvm/lib/Target/AArch64/AArch64SchedA53.td
@@ -150,6 +150,16 @@ def A53WriteFSqrtSP : SchedWriteRes<[A53UnitFPMDS]> { let Latency = 17;
let ReleaseAtCycles = [13]; }
def A53WriteFSqrtDP : SchedWriteRes<[A53UnitFPMDS]> { let Latency = 32;
let ReleaseAtCycles = [28]; }
+// Reciprocal / reciprocal-square-root Newton-Raphson step (FRECPS/FRSQRTS) and
+// estimate (FRECPE/FRSQRTE). Measured on Cortex-A53: the step is 8 cycles and
+// the estimate 4, not the flat 6 of WriteVd. These run on the FP/ASIMD ALU pipe,
+// so the Q-form variants keep the two-cycle datapath occupancy.
+def A53WriteFRecpStep : SchedWriteRes<[A53UnitFPALU]> { let Latency = 8; }
+def A53WriteFRecpStepQ : SchedWriteRes<[A53UnitFPALU]> { let Latency = 8;
+ let ReleaseAtCycles = [2]; }
+def A53WriteFRecpEst : SchedWriteRes<[A53UnitFPALU]> { let Latency = 4; }
+def A53WriteFRecpEstQ : SchedWriteRes<[A53UnitFPALU]> { let Latency = 4;
+ let ReleaseAtCycles = [2]; }
//===----------------------------------------------------------------------===//
// Subtarget-specific SchedRead types.
@@ -321,7 +331,20 @@ def : InstRW<[A53WriteFDivSP], (instrs FDIVSrr)>;
def : InstRW<[A53WriteFDivDP], (instrs FDIVDrr)>;
def : InstRW<[A53WriteFDivSP], (instregex "^FDIVv.*32$")>;
def : InstRW<[A53WriteFDivDP], (instregex "^FDIVv.*64$")>;
-def : InstRW<[A53WriteFSqrtSP], (instregex "^.*SQRT.*32$")>;
-def : InstRW<[A53WriteFSqrtDP], (instregex "^.*SQRT.*64$")>;
+// Match only the genuine square root (FSQRT*), not the reciprocal-square-root
+// step/estimate (FRSQRTS/FRSQRTE), which are handled below. Previously the
+// ".*SQRT.*" globs charged those the full square-root latency.
+def : InstRW<[A53WriteFSqrtSP], (instregex "^FSQRT.*32$")>;
+def : InstRW<[A53WriteFSqrtDP], (instregex "^FSQRT.*64$")>;
+
+// Reciprocal / rsqrt step and estimate (see the write classes above).
+def : InstRW<[A53WriteFRecpStep],
+ (instregex "^FR(ECP|SQRT)S(32|64|v2f32)$")>;
+def : InstRW<[A53WriteFRecpStepQ],
+ (instregex "^FR(ECP|SQRT)S(v2f64|v4f32)$")>;
+def : InstRW<[A53WriteFRecpEst],
+ (instregex "^FR(ECP|SQRT)E(v1i32|v1i64|v2f32)$")>;
+def : InstRW<[A53WriteFRecpEstQ],
+ (instregex "^FR(ECP|SQRT)E(v2f64|v4f32)$")>;
}
>From dd42bf59c072cda3df70ac74abdfff6ea09fa85e Mon Sep 17 00:00:00 2001
From: Michele Dionisio <michele.dionisio at gmail.com>
Date: Thu, 30 Jul 2026 16:42:00 +0200
Subject: [PATCH 04/15] [AArch64] Model integer Advanced SIMD ALU latencies on
Cortex-A53
CortexA53Model charges every ASIMD/FP ALU write a flat 6-cycle latency
(WriteVd/WriteVq). On Cortex-A53 the integer NEON ALU is much faster:
simple arithmetic (ADD/SUB/NEG and halving add/sub) is 2 cycles, and the
wider ALU ops (ABS, integer pairwise, reductions, saturating add/sub,
absolute difference) are 3.
Add integer-vector write classes at latency 2 and 3 and route the
corresponding instructions to them. The Q-form variants keep the two-cycle
datapath occupancy. Multiply/MLA, widening (ADDL/ADDW), narrowing and
accumulate ops are intentionally left on the default: their A53
measurements were less clear-cut than the sibling A55 model, so they are
deferred rather than guessed.
Measured with llvm-exegesis and a perf-counted microbenchmark on a
Raspberry Pi 3 (Cortex-A53, 1.4 GHz): a dependent chain of ADD v.16b runs
at 2.0 cycles/op, the latency-3 group at 3.0. The tiering matches the
sibling Cortex-A55 model, which is based on ARM's published Software
Optimization Guide. Applies only to CortexA53Model (A53/A34/A35);
Cortex-A65/Neoverse-E1 are on their own model.
The Cortex/A53-asimd-q-form.s test is updated for the new ADD latency.
Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
llvm/lib/Target/AArch64/AArch64SchedA53.td | 34 +++++++++++++++++++
.../AArch64/Cortex/A53-asimd-q-form.s | 4 +--
2 files changed, 36 insertions(+), 2 deletions(-)
diff --git a/llvm/lib/Target/AArch64/AArch64SchedA53.td b/llvm/lib/Target/AArch64/AArch64SchedA53.td
index 5586c917c0869..db7204c359db6 100644
--- a/llvm/lib/Target/AArch64/AArch64SchedA53.td
+++ b/llvm/lib/Target/AArch64/AArch64SchedA53.td
@@ -160,6 +160,16 @@ def A53WriteFRecpStepQ : SchedWriteRes<[A53UnitFPALU]> { let Latency = 8;
def A53WriteFRecpEst : SchedWriteRes<[A53UnitFPALU]> { let Latency = 4; }
def A53WriteFRecpEstQ : SchedWriteRes<[A53UnitFPALU]> { let Latency = 4;
let ReleaseAtCycles = [2]; }
+// Integer Advanced SIMD ALU. Measured on Cortex-A53, and matching the sibling
+// Cortex-A55 model: simple arithmetic (ADD/SUB/NEG, halving add/sub) is 2
+// cycles and the wider ALU ops (ABS, pairwise, reduce, saturating, abs-diff)
+// 3, not the flat 6 of WriteVd. Q-form keeps the two-cycle datapath occupancy.
+def A53WriteIVecD_2 : SchedWriteRes<[A53UnitFPALU]> { let Latency = 2; }
+def A53WriteIVecQ_2 : SchedWriteRes<[A53UnitFPALU]> { let Latency = 2;
+ let ReleaseAtCycles = [2]; }
+def A53WriteIVecD_3 : SchedWriteRes<[A53UnitFPALU]> { let Latency = 3; }
+def A53WriteIVecQ_3 : SchedWriteRes<[A53UnitFPALU]> { let Latency = 3;
+ let ReleaseAtCycles = [2]; }
//===----------------------------------------------------------------------===//
// Subtarget-specific SchedRead types.
@@ -347,4 +357,28 @@ def : InstRW<[A53WriteFRecpEst],
def : InstRW<[A53WriteFRecpEstQ],
(instregex "^FR(ECP|SQRT)E(v2f64|v4f32)$")>;
+// Integer ASIMD arithmetic (see the write classes above). Accumulate, widening
+// and reduction-narrowing ops are intentionally left on the default here: their
+// A53 measurements were less clear-cut than the sibling A55 model, so they are
+// deferred rather than guessed.
+def : InstRW<[A53WriteIVecD_2],
+ (instregex "^(ADD|SUB|NEG)v(1i64|2i32|4i16|8i8)$",
+ "^[SU]R?HADDv(2i32|4i16|8i8)$", "^[SU]HSUBv(2i32|4i16|8i8)$")>;
+def : InstRW<[A53WriteIVecQ_2],
+ (instregex "^(ADD|SUB|NEG)v(2i64|4i32|8i16|16i8)$",
+ "^[SU]R?HADDv(8i16|4i32|16i8)$", "^[SU]HSUBv(8i16|4i32|16i8)$")>;
+def : InstRW<[A53WriteIVecD_3],
+ (instregex "^ABSv(1i64|2i32|4i16|8i8)$",
+ "^ADDPv(2i32|4i16|8i8)$",
+ "^[SU]ABDv(2i32|4i16|8i8)$",
+ "^[SU]ADDLPv(2i32_v1i64|4i16_v2i32|8i8_v4i16)$",
+ "^([SU]QADD|[SU]QSUB|SQNEG|SUQADD|USQADD)v(1i8|1i16|1i32|1i64|2i32|4i16|8i8)$",
+ "^(ADDV|SADDLV|UADDLV)v")>;
+def : InstRW<[A53WriteIVecQ_3],
+ (instregex "^ABSv(2i64|4i32|8i16|16i8)$",
+ "^ADDPv(16i8|2i64|4i32|8i16)$",
+ "^[SU]ABDv(16i8|4i32|8i16)$",
+ "^[SU]ADDLPv(16i8_v8i16|4i32_v2i64|8i16_v4i32)$",
+ "^([SU]QADD|[SU]QSUB|SQNEG|SUQADD|USQADD)v(16i8|2i64|4i32|8i16)$")>;
+
}
diff --git a/llvm/test/tools/llvm-mca/AArch64/Cortex/A53-asimd-q-form.s b/llvm/test/tools/llvm-mca/AArch64/Cortex/A53-asimd-q-form.s
index 7fca9ce7c30e7..25b0ed84642ac 100644
--- a/llvm/test/tools/llvm-mca/AArch64/Cortex/A53-asimd-q-form.s
+++ b/llvm/test/tools/llvm-mca/AArch64/Cortex/A53-asimd-q-form.s
@@ -41,8 +41,8 @@ faddp v0.2s, v1.2s, v2.2s
# CHECK-NEXT: 1 10 1.00 fmadd s0, s1, s2, s3
# CHECK-NEXT: 1 6 2.00 fadd v0.4s, v1.4s, v2.4s
# CHECK-NEXT: 1 6 1.00 fadd v0.2s, v1.2s, v2.2s
-# CHECK-NEXT: 1 6 2.00 add v0.4s, v1.4s, v2.4s
-# CHECK-NEXT: 1 6 1.00 add v0.2s, v1.2s, v2.2s
+# CHECK-NEXT: 1 2 2.00 add v0.4s, v1.4s, v2.4s
+# CHECK-NEXT: 1 2 1.00 add v0.2s, v1.2s, v2.2s
# CHECK-NEXT: 1 6 2.00 zip1 v0.4s, v1.4s, v2.4s
# CHECK-NEXT: 1 6 1.00 zip1 v0.2s, v1.2s, v2.2s
# CHECK-NEXT: 1 6 2.00 trn1 v0.16b, v1.16b, v2.16b
>From 31c7bf9d41889aa0b9f9460b5f2f9c74af32798f Mon Sep 17 00:00:00 2001
From: Michele Dionisio <michele.dionisio at gmail.com>
Date: Fri, 31 Jul 2026 08:42:51 +0200
Subject: [PATCH 05/15] [AArch64] Model FP multiply-accumulate accumulator
forwarding on Cortex-A53
The Cortex-A53 FP MAC pipe reads the addend late and has a dedicated
forwarding path that feeds the result of one multiply-accumulate into the
addend of the next. CortexA53Model does not express this and also gives the
result a 10-cycle latency, so it charges a dependent FMADD/FMLA chain 10
cycles per link when the hardware recurs every 4.
Correct the result latency to 8 and attach a ReadAdvance to the addend so a
dependent chain recurs every 4 cycles (8 - 4). The scalar forms take the
addend as their third source operand and the ASIMD forms accumulate into
their tied destination, so the ReadAdvance is attached at a different read
position for each.
Measured on a Raspberry Pi 3 (Cortex-A53) with perf hardware cycle counters:
FMADD/FMLA full latency is 8 cycles (multiplicand to result) and a dependent
accumulator chain recurs every 4 cycles, both scalar and ASIMD. This agrees
with llvm-exegesis on the same board, with GCC's cortex-a53.md (which models
the path as "define_bypass 4 cortex_a53_fpmac cortex_a53_fpmac
aarch_accumulator_forwarding"), and with the sibling Cortex-A55 Software
Optimization Guide, which quotes "8 (4)" for FP multiply accumulate -- an
8-cycle latency with a 4-cycle accumulate forwarding.
Applies only to CortexA53Model (A53/A34/A35); Cortex-A65/Neoverse-E1 are on
their own model. The Cortex/A53-asimd-q-form.s test is updated for the new
FMADD/FMLA latency.
Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
llvm/lib/Target/AArch64/AArch64SchedA53.td | 22 ++++++++++++++-----
.../AArch64/Cortex/A53-asimd-q-form.s | 8 +++----
2 files changed, 20 insertions(+), 10 deletions(-)
diff --git a/llvm/lib/Target/AArch64/AArch64SchedA53.td b/llvm/lib/Target/AArch64/AArch64SchedA53.td
index db7204c359db6..bb8e0b37a1437 100644
--- a/llvm/lib/Target/AArch64/AArch64SchedA53.td
+++ b/llvm/lib/Target/AArch64/AArch64SchedA53.td
@@ -138,10 +138,16 @@ def : WriteRes<WriteVq, [A53UnitFPALU]> { let Latency = 6;
def : WriteRes<WriteFMul, [A53UnitFPMDS]> { let Latency = 6; }
def : WriteRes<WriteFDiv, [A53UnitFPMDS]> { let Latency = 33;
let ReleaseAtCycles = [29]; }
-def A53WriteFMAC : SchedWriteRes<[A53UnitFPMDS]> { let Latency = 10; }
+// FP multiply-accumulate has an 8-cycle result latency, but the MAC pipe reads
+// the addend late and has a dedicated forwarding path that feeds one MAC's
+// result into the next MAC's addend, so a dependent MAC chain recurs every 4
+// cycles. Only the addend forwards; the multiplicands do not.
+def A53WriteFMAC : SchedWriteRes<[A53UnitFPMDS]> { let Latency = 8; }
// Q-form multiply-accumulate occupies the MAC pipe for two cycles.
-def A53WriteFMACQ : SchedWriteRes<[A53UnitFPMDS]> { let Latency = 10;
+def A53WriteFMACQ : SchedWriteRes<[A53UnitFPMDS]> { let Latency = 8;
let ReleaseAtCycles = [2]; }
+def A53ReadFPM : SchedReadAdvance<0>;
+def A53ReadFPMA4 : SchedReadAdvance<4, [A53WriteFMAC, A53WriteFMACQ]>;
def A53WriteFDivSP : SchedWriteRes<[A53UnitFPMDS]> { let Latency = 18;
let ReleaseAtCycles = [14]; }
def A53WriteFDivDP : SchedWriteRes<[A53UnitFPMDS]> { let Latency = 33;
@@ -328,13 +334,17 @@ def : InstRW<[A53WriteCrypto], (instregex "^AES[DE]rr$", "^AESI?MCrr$",
"^SHA256H2?rrr$", "^SHA256SU0rr$",
"^SHA256SU1rrr$", "^PMULLv")>;
-def : InstRW<[A53WriteFMAC], (instregex "^FN?M(ADD|SUB).*")>;
-def : InstRW<[A53WriteFMAC], (instregex "^FML(A|S)L.*")>;
-def : InstRW<[A53WriteFMAC],
+// The scalar forms take the addend as their third source operand, so only the
+// third read forwards; the ASIMD forms accumulate into the tied destination,
+// which is their first source operand.
+def : InstRW<[A53WriteFMAC, A53ReadFPM, A53ReadFPM, A53ReadFPMA4],
+ (instregex "^FN?M(ADD|SUB).*")>;
+def : InstRW<[A53WriteFMAC, A53ReadFPMA4], (instregex "^FML(A|S)L.*")>;
+def : InstRW<[A53WriteFMAC, A53ReadFPMA4],
(instregex "^FML(A|S)(v2f32|v4f16)$",
"^FML(A|S)v1i(16|32|64)_indexed$",
"^FML(A|S)(v2i32|v4i16)_indexed$")>;
-def : InstRW<[A53WriteFMACQ],
+def : InstRW<[A53WriteFMACQ, A53ReadFPMA4],
(instregex "^FML(A|S)(v4f32|v2f64|v8f16)$",
"^FML(A|S)(v4i32|v2i64|v8i16)_indexed$")>;
def : InstRW<[A53WriteFDivSP], (instrs FDIVSrr)>;
diff --git a/llvm/test/tools/llvm-mca/AArch64/Cortex/A53-asimd-q-form.s b/llvm/test/tools/llvm-mca/AArch64/Cortex/A53-asimd-q-form.s
index 25b0ed84642ac..d226ad4613390 100644
--- a/llvm/test/tools/llvm-mca/AArch64/Cortex/A53-asimd-q-form.s
+++ b/llvm/test/tools/llvm-mca/AArch64/Cortex/A53-asimd-q-form.s
@@ -35,10 +35,10 @@ faddp v0.2s, v1.2s, v2.2s
# CHECK-NEXT: [6]: HasSideEffects (U)
# CHECK: [1] [2] [3] [4] [5] [6] Instructions:
-# CHECK-NEXT: 1 10 2.00 fmla v0.4s, v1.4s, v2.4s
-# CHECK-NEXT: 1 10 2.00 fmla v0.2d, v1.2d, v2.2d
-# CHECK-NEXT: 1 10 1.00 fmla v0.2s, v1.2s, v2.2s
-# CHECK-NEXT: 1 10 1.00 fmadd s0, s1, s2, s3
+# CHECK-NEXT: 1 8 2.00 fmla v0.4s, v1.4s, v2.4s
+# CHECK-NEXT: 1 8 2.00 fmla v0.2d, v1.2d, v2.2d
+# CHECK-NEXT: 1 8 1.00 fmla v0.2s, v1.2s, v2.2s
+# CHECK-NEXT: 1 8 1.00 fmadd s0, s1, s2, s3
# CHECK-NEXT: 1 6 2.00 fadd v0.4s, v1.4s, v2.4s
# CHECK-NEXT: 1 6 1.00 fadd v0.2s, v1.2s, v2.2s
# CHECK-NEXT: 1 2 2.00 add v0.4s, v1.4s, v2.4s
>From 4a4e99377ee062becf212ea57c219172079fdab9 Mon Sep 17 00:00:00 2001
From: Michele Dionisio <michele.dionisio at gmail.com>
Date: Fri, 31 Jul 2026 09:27:16 +0200
Subject: [PATCH 06/15] [AArch64] Fix Cortex-A53 floating-point and
vector-multiply latencies
CortexA53Model charges a flat 6-cycle latency to scalar floating-point
arithmetic (WriteF), conversion (WriteFCvt) and multiply (WriteFMul), and to
everything that lands on WriteVd/WriteVq. On Cortex-A53 all of these run at
4 cycles.
Set WriteF, WriteFCvt and WriteFMul to 4, and add 4-cycle vector write
classes for the Advanced SIMD operations that were still inheriting the flat
6: floating-point arithmetic (FADD/FSUB/FABD/FMUL/FMULX/FMAX/FMIN and the
NM/pairwise/reduce variants), FABS/FNEG, the FRINT rounding family, the
conversions -- both integer to and from floating point ([SU]CVTF, FCVTZ[SU],
FCVTA/M/N/P[SU], including the fixed-point forms) and floating point to
floating point (FCVTL/FCVTN/FCVTXN) -- and the integer vector multiply MUL.
The reciprocal-exponent FRECPX and the integer reciprocal estimates
URECPE/URSQRTE join the existing 4-cycle estimate class. Q-form keeps the
two-cycle datapath occupancy.
Measured on a Raspberry Pi 3 (Cortex-A53) with perf hardware cycle counters,
using dependent chains (cycles per link):
fadd 4.00 fabs 4.00 fabd 4.00 fmul (vector) 4.00
fmul 4.00 frinta 4.00 fmulx 4.00 mul (vector) 4.00
fmax 4.00 fabs (scalar) 4.00
scvtf 4.00 ucvtf 4.00 fcvtzs 4.00 fcvtzu 4.00
fcvtas 4.00 scvtf (scalar) 4.00
fcvtn/fcvtl round trip 8.00 per pair, so 4.00 each; fcvtxn likewise
and with llvm-exegesis on the same board: FADDSrr/FADDDrr 4.01, FCVTSDr 4.01,
FCVTNv2i32 4.01, FADDPv4f32 4.01, FRECPX 4.01, URECPE/URSQRTE 4.01.
FCMP, FMOV and the vector logic/compare/permute operations also measure below
6, but they form a separate group and are left for a follow-up. Integer MLA
and the widening multiplies are deferred too: MLA shows accumulator
forwarding rather than a plain latency, and needs its own treatment.
Applies only to CortexA53Model (A53/A34/A35); Cortex-A65/Neoverse-E1 are on
their own model. The Cortex/A53-asimd-q-form.s test is updated.
Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
llvm/lib/Target/AArch64/AArch64SchedA53.td | 45 +++++++++++++++++--
.../AArch64/Cortex/A53-asimd-q-form.s | 8 ++--
2 files changed, 45 insertions(+), 8 deletions(-)
diff --git a/llvm/lib/Target/AArch64/AArch64SchedA53.td b/llvm/lib/Target/AArch64/AArch64SchedA53.td
index bb8e0b37a1437..5fb2fb468f328 100644
--- a/llvm/lib/Target/AArch64/AArch64SchedA53.td
+++ b/llvm/lib/Target/AArch64/AArch64SchedA53.td
@@ -122,10 +122,11 @@ def : WriteRes<WriteSys, [A53UnitB]>;
def : WriteRes<WriteBarrier, [A53UnitB]>;
def : WriteRes<WriteHint, [A53UnitB]>;
-// FP ALU
-def : WriteRes<WriteF, [A53UnitFPALU]> { let Latency = 6; }
+// FP ALU. Scalar floating-point arithmetic and conversion have a 4-cycle
+// latency on Cortex-A53.
+def : WriteRes<WriteF, [A53UnitFPALU]> { let Latency = 4; }
def : WriteRes<WriteFCmp, [A53UnitFPALU]> { let Latency = 6; }
-def : WriteRes<WriteFCvt, [A53UnitFPALU]> { let Latency = 6; }
+def : WriteRes<WriteFCvt, [A53UnitFPALU]> { let Latency = 4; }
def : WriteRes<WriteFCopy, [A53UnitFPALU]> { let Latency = 6; }
def : WriteRes<WriteFImm, [A53UnitFPALU]> { let Latency = 6; }
def : WriteRes<WriteVd, [A53UnitFPALU]> { let Latency = 6; }
@@ -135,7 +136,7 @@ def : WriteRes<WriteVq, [A53UnitFPALU]> { let Latency = 6;
let ReleaseAtCycles = [2]; }
// FP Mul, Div, Sqrt
-def : WriteRes<WriteFMul, [A53UnitFPMDS]> { let Latency = 6; }
+def : WriteRes<WriteFMul, [A53UnitFPMDS]> { let Latency = 4; }
def : WriteRes<WriteFDiv, [A53UnitFPMDS]> { let Latency = 33;
let ReleaseAtCycles = [29]; }
// FP multiply-accumulate has an 8-cycle result latency, but the MAC pipe reads
@@ -166,6 +167,12 @@ def A53WriteFRecpStepQ : SchedWriteRes<[A53UnitFPALU]> { let Latency = 8;
def A53WriteFRecpEst : SchedWriteRes<[A53UnitFPALU]> { let Latency = 4; }
def A53WriteFRecpEstQ : SchedWriteRes<[A53UnitFPALU]> { let Latency = 4;
let ReleaseAtCycles = [2]; }
+// Advanced SIMD arithmetic that runs at the same 4-cycle latency as its scalar
+// counterpart: floating-point arithmetic and conversion, and the integer vector
+// multiply. Q-form keeps the two-cycle datapath occupancy.
+def A53WriteVecD_4 : SchedWriteRes<[A53UnitFPALU]> { let Latency = 4; }
+def A53WriteVecQ_4 : SchedWriteRes<[A53UnitFPALU]> { let Latency = 4;
+ let ReleaseAtCycles = [2]; }
// Integer Advanced SIMD ALU. Measured on Cortex-A53, and matching the sibling
// Cortex-A55 model: simple arithmetic (ADD/SUB/NEG, halving add/sub) is 2
// cycles and the wider ALU ops (ABS, pairwise, reduce, saturating, abs-diff)
@@ -366,6 +373,36 @@ def : InstRW<[A53WriteFRecpEst],
(instregex "^FR(ECP|SQRT)E(v1i32|v1i64|v2f32)$")>;
def : InstRW<[A53WriteFRecpEstQ],
(instregex "^FR(ECP|SQRT)E(v2f64|v4f32)$")>;
+// The reciprocal-exponent and the integer reciprocal estimates share the
+// 4-cycle estimate latency.
+def : InstRW<[A53WriteFRecpEst],
+ (instregex "^FRECPXv1i(32|64)$", "^UR(ECP|SQRT)Ev2i32$")>;
+def : InstRW<[A53WriteFRecpEstQ], (instregex "^UR(ECP|SQRT)Ev4i32$")>;
+
+// Floating-point ASIMD arithmetic, and the integer vector multiply, run at the
+// same 4-cycle latency as the scalar forms (see the write classes above).
+def : InstRW<[A53WriteVecD_4],
+ (instregex "^F(ADD|SUB|ABD|MUL|MULX|MAX|MIN|MAXNM|MINNM)P?(v2f32|v4f16)$",
+ "^F(ABS|NEG)(v2f32|v4f16)$",
+ "^FRINT[AIMNPXZ](v2f32|v4f16)$",
+ "^FMULX?(v2i32|v4i16)_indexed$",
+ "^F(MAX|MIN|MAXNM|MINNM)Vv4i16v$",
+ "^([SU]CVTF|FCVT[AMNPZ][SU])(v1i16|v1i32|v1i64)$",
+ "^([SU]CVTF|FCVT[AMNPZ][SU])(v2f32|v4f16)$",
+ "^([SU]CVTF|FCVT[AMNPZ][SU])(v2i32|v4i16)_shift$",
+ "^FCVTXNv1i64$",
+ "^MULv(2i32|4i16|8i8)$", "^MULv(2i32|4i16)_indexed$")>;
+def : InstRW<[A53WriteVecQ_4],
+ (instregex "^F(ADD|SUB|ABD|MUL|MULX|MAX|MIN|MAXNM|MINNM)P?(v4f32|v2f64|v8f16)$",
+ "^F(ABS|NEG)(v4f32|v2f64|v8f16)$",
+ "^FRINT[AIMNPXZ](v4f32|v2f64|v8f16)$",
+ "^FMULX?(v4i32|v2i64|v8i16)_indexed$",
+ "^F(MAX|MIN|MAXNM|MINNM)Vv4i32v$",
+ "^([SU]CVTF|FCVT[AMNPZ][SU])(v2f64|v4f32|v8f16)$",
+ "^([SU]CVTF|FCVT[AMNPZ][SU])(v2i64|v4i32|v8i16)_shift$",
+ "^FCVT(L|N)2?v(2i32|4i16|4i32|8i16)$",
+ "^FCVTXN2?v(2f32|4f32)$",
+ "^MULv(4i32|8i16|16i8)$", "^MULv(4i32|8i16)_indexed$")>;
// Integer ASIMD arithmetic (see the write classes above). Accumulate, widening
// and reduction-narrowing ops are intentionally left on the default here: their
diff --git a/llvm/test/tools/llvm-mca/AArch64/Cortex/A53-asimd-q-form.s b/llvm/test/tools/llvm-mca/AArch64/Cortex/A53-asimd-q-form.s
index d226ad4613390..e7abc70c05137 100644
--- a/llvm/test/tools/llvm-mca/AArch64/Cortex/A53-asimd-q-form.s
+++ b/llvm/test/tools/llvm-mca/AArch64/Cortex/A53-asimd-q-form.s
@@ -39,8 +39,8 @@ faddp v0.2s, v1.2s, v2.2s
# CHECK-NEXT: 1 8 2.00 fmla v0.2d, v1.2d, v2.2d
# CHECK-NEXT: 1 8 1.00 fmla v0.2s, v1.2s, v2.2s
# CHECK-NEXT: 1 8 1.00 fmadd s0, s1, s2, s3
-# CHECK-NEXT: 1 6 2.00 fadd v0.4s, v1.4s, v2.4s
-# CHECK-NEXT: 1 6 1.00 fadd v0.2s, v1.2s, v2.2s
+# CHECK-NEXT: 1 4 2.00 fadd v0.4s, v1.4s, v2.4s
+# CHECK-NEXT: 1 4 1.00 fadd v0.2s, v1.2s, v2.2s
# CHECK-NEXT: 1 2 2.00 add v0.4s, v1.4s, v2.4s
# CHECK-NEXT: 1 2 1.00 add v0.2s, v1.2s, v2.2s
# CHECK-NEXT: 1 6 2.00 zip1 v0.4s, v1.4s, v2.4s
@@ -53,8 +53,8 @@ faddp v0.2s, v1.2s, v2.2s
# CHECK-NEXT: 1 6 1.00 ext v0.8b, v1.8b, v2.8b, #4
# CHECK-NEXT: 1 6 2.00 dup v0.4s, v1.s[1]
# CHECK-NEXT: 1 6 1.00 dup v0.2s, v1.s[1]
-# CHECK-NEXT: 1 6 2.00 faddp v0.4s, v1.4s, v2.4s
-# CHECK-NEXT: 1 6 1.00 faddp v0.2s, v1.2s, v2.2s
+# CHECK-NEXT: 1 4 2.00 faddp v0.4s, v1.4s, v2.4s
+# CHECK-NEXT: 1 4 1.00 faddp v0.2s, v1.2s, v2.2s
# CHECK: Resources:
# CHECK-NEXT: [0.0] - A53UnitALU
>From e5e3411f6a0b11caf9a15d47793a41d68138a2b9 Mon Sep 17 00:00:00 2001
From: Michele Dionisio <michele.dionisio at gmail.com>
Date: Fri, 31 Jul 2026 09:38:18 +0200
Subject: [PATCH 07/15] [AArch64] Model Cortex-A53 ASIMD logic, compare and
permute latencies
The remaining Advanced SIMD operations that fall through to WriteVd/WriteVq
are charged a flat 6-cycle latency. On Cortex-A53 bitwise vector logic is a
single cycle, the compare/permute group is two, and the accumulating forms
are four.
Add one-cycle write classes for the vector logic (AND/BIC/EOR/ORN/ORR/NOT,
including the immediate forms) and route to the existing two-cycle classes
the integer and floating-point compares, EXT, the REV family, DUP/INS, TBL/
TBX and the bit-select family (BIF/BIT/BSL). The absolute-difference and
pairwise accumulates ([SU]ABA/[SU]ABAL/[SU]ADALP) and the rounding
add/subtract-narrow (RADDHN/RSUBHN) go to the four-cycle classes. CMTST and
CLS take one cycle more than the plain compares and go to the three-cycle
classes. Scalar FCSEL is two cycles and the float-to-float FMOV is one.
Q-form keeps the two-cycle datapath occupancy.
Measured on a Raspberry Pi 3 (Cortex-A53) with perf hardware cycle counters,
using dependent chains (cycles per link):
and 1.06 orr 1.06 eor 1.06 bic 1.06 orn 1.06 not 1.06
fmov 1.00
cmeq 2.00 cmge 2.00 cmgt 2.00 fcmeq 2.00 facge 2.00 fcsel 2.00
ext 2.00 rev64 2.00 rev32 2.00 dup 2.00 ins 2.00 tbl 2.00
bif 2.00 bit 2.00 bsl 2.00
saba 4.00 uaba 4.00 sabal 4.00 sadalp 4.00 raddhn 4.00 rsubhn 4.00
cmtst 3.00 cls 3.00
For the accumulating forms the chain must run through a source operand: a
chain through the tied accumulator alone recurs in about one cycle, but that
is the accumulator forwarding path rather than the instruction latency. The
four-cycle figures above are the source-to-result latency, and they agree
with the sibling Cortex-A55 model.
Left alone: FCMP, whose measurement was not clean enough to act on; the
float-to-int FMOV (WriteFCopy), which was not measured; and the shift family,
where only the basic SHL/USHR were measured and the rounding, saturating and
narrowing variants were not.
Applies only to CortexA53Model (A53/A34/A35); Cortex-A65/Neoverse-E1 are on
their own model. The Cortex/A53-asimd-q-form.s test is updated.
Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
llvm/lib/Target/AArch64/AArch64SchedA53.td | 49 +++++++++++++++++++
.../AArch64/Cortex/A53-asimd-q-form.s | 12 ++---
2 files changed, 55 insertions(+), 6 deletions(-)
diff --git a/llvm/lib/Target/AArch64/AArch64SchedA53.td b/llvm/lib/Target/AArch64/AArch64SchedA53.td
index 5fb2fb468f328..3bd4117b754e0 100644
--- a/llvm/lib/Target/AArch64/AArch64SchedA53.td
+++ b/llvm/lib/Target/AArch64/AArch64SchedA53.td
@@ -167,6 +167,13 @@ def A53WriteFRecpStepQ : SchedWriteRes<[A53UnitFPALU]> { let Latency = 8;
def A53WriteFRecpEst : SchedWriteRes<[A53UnitFPALU]> { let Latency = 4; }
def A53WriteFRecpEstQ : SchedWriteRes<[A53UnitFPALU]> { let Latency = 4;
let ReleaseAtCycles = [2]; }
+// Bitwise Advanced SIMD logic forwards in a single cycle.
+def A53WriteLogicD : SchedWriteRes<[A53UnitFPALU]> { let Latency = 1; }
+def A53WriteLogicQ : SchedWriteRes<[A53UnitFPALU]> { let Latency = 1;
+ let ReleaseAtCycles = [2]; }
+// Scalar floating-point select, and the float-to-float register move.
+def A53WriteFSel : SchedWriteRes<[A53UnitFPALU]> { let Latency = 2; }
+def A53WriteFMov : SchedWriteRes<[A53UnitFPALU]> { let Latency = 1; }
// Advanced SIMD arithmetic that runs at the same 4-cycle latency as its scalar
// counterpart: floating-point arithmetic and conversion, and the integer vector
// multiply. Q-form keeps the two-cycle datapath occupancy.
@@ -404,6 +411,48 @@ def : InstRW<[A53WriteVecQ_4],
"^FCVTXN2?v(2f32|4f32)$",
"^MULv(4i32|8i16|16i8)$", "^MULv(4i32|8i16)_indexed$")>;
+// Bitwise ASIMD logic: one cycle.
+def : InstRW<[A53WriteLogicD],
+ (instregex "^(AND|BIC|EOR|ORN|ORR|NOT)v8i8$", "^(ORR|BIC)v(2i32|4i16)$")>;
+def : InstRW<[A53WriteLogicQ],
+ (instregex "^(AND|BIC|EOR|ORN|ORR|NOT)v16i8$", "^(ORR|BIC)v(4i32|8i16)$")>;
+
+// ASIMD compare, permute and insert/select: two cycles. The integer and
+// floating-point compares, EXT/REV/DUP/INS/TBL and the bit-select family all
+// measure 2 on the source-to-result path.
+def : InstRW<[A53WriteIVecD_2],
+ (instregex "^CM(EQ|GE|GT|HI|HS|LE|LT)v(1i64|2i32|4i16|8i8)(rz)?$",
+ "^(FCM(EQ|GE|GT)|FACG[ET])(v2f32|v4f16|v1i32|v1i64)(rz)?$",
+ "^(EXT|BIF|BIT|BSL)v8i8$",
+ "^REV(16|32|64)v(2i32|4i16|8i8)$",
+ "^(DUP|INS)v.*(8|16|32|64)(gpr|lane)$",
+ "^TB[LX]v8i8.*$")>;
+def : InstRW<[A53WriteIVecQ_2],
+ (instregex "^CM(EQ|GE|GT|HI|HS|LE|LT)v(2i64|4i32|8i16|16i8)(rz)?$",
+ "^(FCM(EQ|GE|GT)|FACG[ET])(v4f32|v2f64|v8f16)(rz)?$",
+ "^(EXT|BIF|BIT|BSL)v16i8$",
+ "^REV(16|32|64)v(4i32|8i16|16i8)$",
+ "^TB[LX]v16i8.*$")>;
+// Absolute-difference accumulate, pairwise accumulate and the rounding
+// add/subtract-narrow take four cycles from their source operands. (A chain
+// through the tied accumulator alone recurs faster, but that is a forwarding
+// path, not the latency, and is not modelled here.)
+def : InstRW<[A53WriteVecD_4],
+ (instregex "^[SU]ABAv(2i32|4i16|8i8)$",
+ "^[SU]ADALPv(2i32_v1i64|4i16_v2i32|8i8_v4i16)$")>;
+def : InstRW<[A53WriteVecQ_4],
+ (instregex "^[SU]ABAv(16i8|4i32|8i16)$", "^[SU]ABALv",
+ "^[SU]ADALPv(16i8_v8i16|4i32_v2i64|8i16_v4i32)$",
+ "^R(ADD|SUB)HNv")>;
+def : InstRW<[A53WriteFSel], (instregex "^FCSEL[HSD]rrr$")>;
+def : InstRW<[A53WriteFMov], (instregex "^FMOV[HSD]r$")>;
+
+// CMTST and CLS take a cycle more than the plain compares.
+def : InstRW<[A53WriteIVecD_3],
+ (instregex "^CMTSTv(1i64|2i32|4i16|8i8)$", "^CLSv(2i32|4i16|8i8)$")>;
+def : InstRW<[A53WriteIVecQ_3],
+ (instregex "^CMTSTv(2i64|4i32|8i16|16i8)$", "^CLSv(4i32|8i16|16i8)$")>;
+
// Integer ASIMD arithmetic (see the write classes above). Accumulate, widening
// and reduction-narrowing ops are intentionally left on the default here: their
// A53 measurements were less clear-cut than the sibling A55 model, so they are
diff --git a/llvm/test/tools/llvm-mca/AArch64/Cortex/A53-asimd-q-form.s b/llvm/test/tools/llvm-mca/AArch64/Cortex/A53-asimd-q-form.s
index e7abc70c05137..28b0c0689caed 100644
--- a/llvm/test/tools/llvm-mca/AArch64/Cortex/A53-asimd-q-form.s
+++ b/llvm/test/tools/llvm-mca/AArch64/Cortex/A53-asimd-q-form.s
@@ -49,10 +49,10 @@ faddp v0.2s, v1.2s, v2.2s
# CHECK-NEXT: 1 6 1.00 trn1 v0.8b, v1.8b, v2.8b
# CHECK-NEXT: 1 6 2.00 uzp1 v0.4s, v1.4s, v2.4s
# CHECK-NEXT: 1 6 1.00 uzp1 v0.2s, v1.2s, v2.2s
-# CHECK-NEXT: 1 6 2.00 ext v0.16b, v1.16b, v2.16b, #4
-# CHECK-NEXT: 1 6 1.00 ext v0.8b, v1.8b, v2.8b, #4
-# CHECK-NEXT: 1 6 2.00 dup v0.4s, v1.s[1]
-# CHECK-NEXT: 1 6 1.00 dup v0.2s, v1.s[1]
+# CHECK-NEXT: 1 2 2.00 ext v0.16b, v1.16b, v2.16b, #4
+# CHECK-NEXT: 1 2 1.00 ext v0.8b, v1.8b, v2.8b, #4
+# CHECK-NEXT: 1 2 1.00 dup v0.4s, v1.s[1]
+# CHECK-NEXT: 1 2 1.00 dup v0.2s, v1.s[1]
# CHECK-NEXT: 1 4 2.00 faddp v0.4s, v1.4s, v2.4s
# CHECK-NEXT: 1 4 1.00 faddp v0.2s, v1.2s, v2.2s
@@ -68,7 +68,7 @@ faddp v0.2s, v1.2s, v2.2s
# CHECK: Resource pressure per iteration:
# CHECK-NEXT: [0.0] [0.1] [1] [2] [3] [4] [5] [6]
-# CHECK-NEXT: - - - - 24.00 6.00 - -
+# CHECK-NEXT: - - - - 23.00 6.00 - -
# CHECK: Resource pressure by instruction:
# CHECK-NEXT: [0.0] [0.1] [1] [2] [3] [4] [5] [6] Instructions:
@@ -88,7 +88,7 @@ faddp v0.2s, v1.2s, v2.2s
# CHECK-NEXT: - - - - 1.00 - - - uzp1 v0.2s, v1.2s, v2.2s
# CHECK-NEXT: - - - - 2.00 - - - ext v0.16b, v1.16b, v2.16b, #4
# CHECK-NEXT: - - - - 1.00 - - - ext v0.8b, v1.8b, v2.8b, #4
-# CHECK-NEXT: - - - - 2.00 - - - dup v0.4s, v1.s[1]
+# CHECK-NEXT: - - - - 1.00 - - - dup v0.4s, v1.s[1]
# CHECK-NEXT: - - - - 1.00 - - - dup v0.2s, v1.s[1]
# CHECK-NEXT: - - - - 2.00 - - - faddp v0.4s, v1.4s, v2.4s
# CHECK-NEXT: - - - - 1.00 - - - faddp v0.2s, v1.2s, v2.2s
>From 7fe63432525d31fd60b7ad063142b3bc70720ae9 Mon Sep 17 00:00:00 2001
From: Michele Dionisio <michele.dionisio at gmail.com>
Date: Fri, 31 Jul 2026 09:53:30 +0200
Subject: [PATCH 08/15] [AArch64] Fix the Cortex-A53 float-to-int register copy
latency
WriteFCopy, the class for FMOV between a GPR and an FP register, is modelled
at 6 cycles. On Cortex-A53 the copy takes a single cycle in each direction.
Measured on a Raspberry Pi 3 (Cortex-A53) with perf hardware cycle counters:
a dependent round trip of "fmov w1, s0" followed by "fmov s0, w1" costs 2.00
cycles per pair, for both the 32-bit and the 64-bit forms, so each direction
is one cycle.
Applies only to CortexA53Model (A53/A34/A35); Cortex-A65/Neoverse-E1 are on
their own model.
Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
llvm/lib/Target/AArch64/AArch64SchedA53.td | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/llvm/lib/Target/AArch64/AArch64SchedA53.td b/llvm/lib/Target/AArch64/AArch64SchedA53.td
index 3bd4117b754e0..522008ead4373 100644
--- a/llvm/lib/Target/AArch64/AArch64SchedA53.td
+++ b/llvm/lib/Target/AArch64/AArch64SchedA53.td
@@ -127,7 +127,9 @@ def : WriteRes<WriteHint, [A53UnitB]>;
def : WriteRes<WriteF, [A53UnitFPALU]> { let Latency = 4; }
def : WriteRes<WriteFCmp, [A53UnitFPALU]> { let Latency = 6; }
def : WriteRes<WriteFCvt, [A53UnitFPALU]> { let Latency = 4; }
-def : WriteRes<WriteFCopy, [A53UnitFPALU]> { let Latency = 6; }
+// Float-to-int register copy (FMOV between a GPR and an FP register) takes a
+// single cycle in each direction.
+def : WriteRes<WriteFCopy, [A53UnitFPALU]> { let Latency = 1; }
def : WriteRes<WriteFImm, [A53UnitFPALU]> { let Latency = 6; }
def : WriteRes<WriteVd, [A53UnitFPALU]> { let Latency = 6; }
// The ASIMD datapath is 64 bits wide: 128-bit (Q-form) operations occupy the
>From 0ede5d818c6b5116c9496d308829bfc574f0baa1 Mon Sep 17 00:00:00 2001
From: Michele Dionisio <michele.dionisio at gmail.com>
Date: Fri, 31 Jul 2026 09:56:11 +0200
Subject: [PATCH 09/15] [AArch64] Model the Cortex-A53 ASIMD shift latencies
The Advanced SIMD shifts all inherit the flat 6-cycle WriteVd/WriteVq
latency. On Cortex-A53 they fall into three groups: the plain shifts are two
cycles, the rounding variants and the shift-accumulates three, and the
saturating shifts, the saturating narrows and the rounding shift-accumulates
four.
Route them to the corresponding write classes. Narrowing and widening forms
read or write 128 bits and so take the Q-form occupancy.
Measured on a Raspberry Pi 3 (Cortex-A53) with perf hardware cycle counters,
using dependent chains (cycles per link):
shl 2.00 sshr 2.00 ushr 2.00 sshl 2.00 ushl 2.00
shrn 2.00 sshll 2.00 ushll 2.00 sli 2.00 sri 2.00
srshr 3.00 urshr 3.00 srshl 3.00 urshl 3.00 rshrn 3.00
ssra 3.00 usra 3.00
sqshl 4.00 uqshl 4.00 sqshlu 4.00 sqrshl 4.00 uqrshl 4.00
sqshrn 4.00 sqrshrn 4.00 uqshrn 4.00 sqshrun 4.00
srsra 4.00 ursra 4.00
For SLI/SRI and the shift-accumulates the chain runs through a source
operand, not through the tied destination: a chain through the accumulator
alone recurs faster, but that is a forwarding path rather than the latency.
The resulting grouping matches the sibling Cortex-A55 model entry for entry.
Applies only to CortexA53Model (A53/A34/A35); Cortex-A65/Neoverse-E1 are on
their own model.
Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
llvm/lib/Target/AArch64/AArch64SchedA53.td | 36 ++++++++++++++++++++++
1 file changed, 36 insertions(+)
diff --git a/llvm/lib/Target/AArch64/AArch64SchedA53.td b/llvm/lib/Target/AArch64/AArch64SchedA53.td
index 522008ead4373..92bdf339cec73 100644
--- a/llvm/lib/Target/AArch64/AArch64SchedA53.td
+++ b/llvm/lib/Target/AArch64/AArch64SchedA53.td
@@ -455,6 +455,42 @@ def : InstRW<[A53WriteIVecD_3],
def : InstRW<[A53WriteIVecQ_3],
(instregex "^CMTSTv(2i64|4i32|8i16|16i8)$", "^CLSv(4i32|8i16|16i8)$")>;
+// ASIMD shifts. Plain shifts -- by immediate or by register, including the
+// insert (SLI/SRI), narrowing (SHRN) and widening (SSHLL/USHLL) forms -- are
+// two cycles. The rounding variants and the shift-accumulates are three, and
+// the saturating shifts, the saturating narrows and the rounding
+// shift-accumulates are four. Narrowing and widening forms read or write 128
+// bits and so use the Q-form occupancy.
+def : InstRW<[A53WriteIVecD_2],
+ (instregex "^(SHL|SLI|SRI)(d|v8i8_shift|v4i16_shift|v2i32_shift)$",
+ "^[SU]SHR(d|v8i8_shift|v4i16_shift|v2i32_shift)$",
+ "^[SU]SHLv(1i64|8i8|4i16|2i32)$")>;
+def : InstRW<[A53WriteIVecQ_2],
+ (instregex "^(SHL|SLI|SRI)v(16i8|8i16|4i32|2i64)_shift$",
+ "^[SU]SHRv(16i8|8i16|4i32|2i64)_shift$",
+ "^[SU]SHLv(16i8|8i16|4i32|2i64)$",
+ "^SHRNv.*_shift$", "^[SU]SHLLv.*_shift$")>;
+def : InstRW<[A53WriteIVecD_3],
+ (instregex "^[SU]RSHR(d|v8i8_shift|v4i16_shift|v2i32_shift)$",
+ "^[SU]RSHLv(1i64|8i8|4i16|2i32)$",
+ "^[SU]SRA(d|v8i8_shift|v4i16_shift|v2i32_shift)$")>;
+def : InstRW<[A53WriteIVecQ_3],
+ (instregex "^[SU]RSHRv(16i8|8i16|4i32|2i64)_shift$",
+ "^[SU]RSHLv(16i8|8i16|4i32|2i64)$",
+ "^[SU]SRAv(16i8|8i16|4i32|2i64)_shift$",
+ "^RSHRNv.*_shift$")>;
+def : InstRW<[A53WriteVecD_4],
+ (instregex "^[SU]QSHLU?(b|h|s|d)$", "^SQSHLU(b|h|s|d)$",
+ "^[SU]QR?SHL(b|h|s|d|v1i8|v1i16|v1i32|v1i64)$",
+ "^[SU]QSHLU?(v8i8|v4i16|v2i32)_shift$",
+ "^[SU]QR?SHLv(8i8|4i16|2i32)$",
+ "^[SU]RSRA(d|v8i8_shift|v4i16_shift|v2i32_shift)$")>;
+def : InstRW<[A53WriteVecQ_4],
+ (instregex "^[SU]QSHLU?v(16i8|8i16|4i32|2i64)_shift$",
+ "^[SU]QR?SHLv(16i8|8i16|4i32|2i64)$",
+ "^[SU]RSRAv(16i8|8i16|4i32|2i64)_shift$",
+ "^([SU]QR?SHRN|SQR?SHRUN)[bhsd]?v?.*_shift$")>;
+
// Integer ASIMD arithmetic (see the write classes above). Accumulate, widening
// and reduction-narrowing ops are intentionally left on the default here: their
// A53 measurements were less clear-cut than the sibling A55 model, so they are
>From f5bb6d0663698ce44c3b52702be062112fe43ef2 Mon Sep 17 00:00:00 2001
From: Michele Dionisio <michele.dionisio at gmail.com>
Date: Fri, 31 Jul 2026 09:57:57 +0200
Subject: [PATCH 10/15] [AArch64] Model Cortex-A53 ASIMD widening multiply and
narrowing latencies
The Advanced SIMD widening multiplies and the narrowing converts inherit the
flat 6-cycle WriteVd/WriteVq latency. On Cortex-A53 the widening multiplies
and the saturating narrows are four cycles, the plain extract-narrow is two,
and the add/subtract-high-narrow is three.
Route [SU]MULL/SQDMULL, SQDML[AS]L and [SU]QXTN/SQXTUN to the four-cycle
class, XTN to the two-cycle class and ADDHN/SUBHN to the three-cycle class.
All of these read or write 128 bits and so take the Q-form occupancy. The
plain widening multiply-accumulates ([SU]ML[AS]L) are left out here: they
have accumulator forwarding and are handled in a follow-up.
Measured on a Raspberry Pi 3 (Cortex-A53) with perf hardware cycle counters,
using dependent chains through a source operand (cycles per link):
smull 4.00 umull 4.00 sqdmull 4.00 sqdmlal 4.00
sqxtn 4.00 uqxtn 4.00 sqxtun 4.00
xtn 2.00
addhn 3.00 subhn 3.00
These figures agree with the sibling Cortex-A55 model.
Applies only to CortexA53Model (A53/A34/A35); Cortex-A65/Neoverse-E1 are on
their own model.
Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
llvm/lib/Target/AArch64/AArch64SchedA53.td | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/llvm/lib/Target/AArch64/AArch64SchedA53.td b/llvm/lib/Target/AArch64/AArch64SchedA53.td
index 92bdf339cec73..458a9d4b687f1 100644
--- a/llvm/lib/Target/AArch64/AArch64SchedA53.td
+++ b/llvm/lib/Target/AArch64/AArch64SchedA53.td
@@ -455,6 +455,17 @@ def : InstRW<[A53WriteIVecD_3],
def : InstRW<[A53WriteIVecQ_3],
(instregex "^CMTSTv(2i64|4i32|8i16|16i8)$", "^CLSv(4i32|8i16|16i8)$")>;
+// ASIMD widening multiply and narrowing convert. The widening multiplies and
+// multiply-accumulates, and the saturating narrows, are four cycles; the plain
+// extract-narrow is two and the add/subtract-high-narrow three. All of these
+// read or write 128 bits and so take the Q-form occupancy.
+def : InstRW<[A53WriteVecQ_4],
+ (instregex "^([SU]MULL|SQDMULL)v.*$",
+ "^SQDML[AS]Lv.*$",
+ "^([SU]QXTN|SQXTUN)v.*$")>;
+def : InstRW<[A53WriteIVecQ_2], (instregex "^XTNv.*$")>;
+def : InstRW<[A53WriteIVecQ_3], (instregex "^(ADD|SUB)HNv.*$")>;
+
// ASIMD shifts. Plain shifts -- by immediate or by register, including the
// insert (SLI/SRI), narrowing (SHRN) and widening (SSHLL/USHLL) forms -- are
// two cycles. The rounding variants and the shift-accumulates are three, and
>From b9a9702ff3e52da1c73198ba3c98848f7d261c6c Mon Sep 17 00:00:00 2001
From: Michele Dionisio <michele.dionisio at gmail.com>
Date: Fri, 31 Jul 2026 10:03:29 +0200
Subject: [PATCH 11/15] [AArch64] Model integer ASIMD multiply-accumulate
forwarding on Cortex-A53
The integer Advanced SIMD multiply-accumulates (MLA/MLS and the widening
[SU]ML[AS]L) inherit the flat 6-cycle WriteVd/WriteVq latency and no
forwarding. On Cortex-A53 their result latency is 4, and the MAC unit feeds
one accumulate result straight into the next MAC's accumulator with no
interlock, so a dependent accumulator chain recurs every cycle.
Give them 4-cycle write classes and attach a ReadAdvance to the accumulator,
which is the tied destination and therefore the first read. The multiplicands
do not forward. The saturating doubling forms (SQDML[AS]L) do not use this
path and keep the plain 4-cycle class.
Measured on a Raspberry Pi 3 (Cortex-A53) with perf hardware cycle counters
(cycles per link):
chain through the accumulator mla 1.19 mls 1.19
smlal 1.19 umlal 1.19
mla (by element) 1.19
smlal (by element) 1.19
chain through a multiplicand mla 4.00 mul 4.00
saturating doubling, accumulator sqdmlal 4.00
This mirrors the accumulator forwarding already modelled for the scalar
integer MAC (ReadIMA) and for the FP MAC, and matches the description in the
Cortex-A55 Software Optimization Guide: "a dedicated forwarding path in the
accumulate portion of the unit that allows the result of one MAC operation to
be used as the accumulate operand of a following MAC operation with no
interlock".
Note that for Q-form the modelled recurrence is capped at two cycles by the
64-bit datapath occupancy rather than by the forwarding itself; the D-form
chain recurs every cycle as measured.
Applies only to CortexA53Model (A53/A34/A35); Cortex-A65/Neoverse-E1 are on
their own model.
Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
llvm/lib/Target/AArch64/AArch64SchedA53.td | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
diff --git a/llvm/lib/Target/AArch64/AArch64SchedA53.td b/llvm/lib/Target/AArch64/AArch64SchedA53.td
index 458a9d4b687f1..bc54cfb21a32e 100644
--- a/llvm/lib/Target/AArch64/AArch64SchedA53.td
+++ b/llvm/lib/Target/AArch64/AArch64SchedA53.td
@@ -169,6 +169,17 @@ def A53WriteFRecpStepQ : SchedWriteRes<[A53UnitFPALU]> { let Latency = 8;
def A53WriteFRecpEst : SchedWriteRes<[A53UnitFPALU]> { let Latency = 4; }
def A53WriteFRecpEstQ : SchedWriteRes<[A53UnitFPALU]> { let Latency = 4;
let ReleaseAtCycles = [2]; }
+// Integer Advanced SIMD multiply-accumulate. The result latency is 4, but the
+// MAC unit forwards one accumulate result straight into the next MAC's
+// accumulator with no interlock, so a dependent accumulator chain recurs every
+// cycle. Only the accumulator forwards; the multiplicands do not. The
+// saturating doubling forms (SQDML[AS]L) do not use this path.
+def A53WriteIMACD : SchedWriteRes<[A53UnitFPALU]> { let Latency = 4; }
+def A53WriteIMACQ : SchedWriteRes<[A53UnitFPALU]> { let Latency = 4;
+ let ReleaseAtCycles = [2]; }
+def A53ReadIMACM : SchedReadAdvance<0>;
+def A53ReadIMACA3 : SchedReadAdvance<3, [A53WriteIMACD, A53WriteIMACQ]>;
+
// Bitwise Advanced SIMD logic forwards in a single cycle.
def A53WriteLogicD : SchedWriteRes<[A53UnitFPALU]> { let Latency = 1; }
def A53WriteLogicQ : SchedWriteRes<[A53UnitFPALU]> { let Latency = 1;
@@ -464,6 +475,13 @@ def : InstRW<[A53WriteVecQ_4],
"^SQDML[AS]Lv.*$",
"^([SU]QXTN|SQXTUN)v.*$")>;
def : InstRW<[A53WriteIVecQ_2], (instregex "^XTNv.*$")>;
+// Integer multiply-accumulate: the accumulator is the tied destination, so it
+// is the first read and the only one that forwards.
+def : InstRW<[A53WriteIMACD, A53ReadIMACA3],
+ (instregex "^ML[AS]v(2i32|4i16|8i8)$", "^ML[AS]v(2i32|4i16)_indexed$")>;
+def : InstRW<[A53WriteIMACQ, A53ReadIMACA3],
+ (instregex "^ML[AS]v(4i32|8i16|16i8)$", "^ML[AS]v(4i32|8i16)_indexed$",
+ "^[SU]ML[AS]Lv.*$")>;
def : InstRW<[A53WriteIVecQ_3], (instregex "^(ADD|SUB)HNv.*$")>;
// ASIMD shifts. Plain shifts -- by immediate or by register, including the
>From 6fde0d96adf05926b5d6df85951e28c7eda1832f Mon Sep 17 00:00:00 2001
From: Michele Dionisio <michele.dionisio at gmail.com>
Date: Fri, 31 Jul 2026 10:11:10 +0200
Subject: [PATCH 12/15] [AArch64] Model the Cortex-A53 ASIMD permute latencies
ZIP1/ZIP2, TRN1/TRN2 and UZP1/UZP2 inherit the flat 6-cycle WriteVd/WriteVq
latency. On Cortex-A53 they take two cycles, like the other Advanced SIMD
permutes (EXT, REV, TBL) already modelled.
Measured on a Raspberry Pi 3 (Cortex-A53) with perf hardware cycle counters,
using dependent chains: all twelve forms -- zip1/zip2, trn1/trn2, uzp1/uzp2,
in both D-form and Q-form and across element widths -- measure 2.00 cycles
per link.
Applies only to CortexA53Model (A53/A34/A35); Cortex-A65/Neoverse-E1 are on
their own model. The Cortex/A53-asimd-q-form.s test is updated.
Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
llvm/lib/Target/AArch64/AArch64SchedA53.td | 6 ++++--
.../tools/llvm-mca/AArch64/Cortex/A53-asimd-q-form.s | 12 ++++++------
2 files changed, 10 insertions(+), 8 deletions(-)
diff --git a/llvm/lib/Target/AArch64/AArch64SchedA53.td b/llvm/lib/Target/AArch64/AArch64SchedA53.td
index bc54cfb21a32e..a33ce58cd1a89 100644
--- a/llvm/lib/Target/AArch64/AArch64SchedA53.td
+++ b/llvm/lib/Target/AArch64/AArch64SchedA53.td
@@ -439,13 +439,15 @@ def : InstRW<[A53WriteIVecD_2],
"^(EXT|BIF|BIT|BSL)v8i8$",
"^REV(16|32|64)v(2i32|4i16|8i8)$",
"^(DUP|INS)v.*(8|16|32|64)(gpr|lane)$",
- "^TB[LX]v8i8.*$")>;
+ "^TB[LX]v8i8.*$",
+ "^(ZIP|TRN|UZP)[12]v(2i32|4i16|8i8)$")>;
def : InstRW<[A53WriteIVecQ_2],
(instregex "^CM(EQ|GE|GT|HI|HS|LE|LT)v(2i64|4i32|8i16|16i8)(rz)?$",
"^(FCM(EQ|GE|GT)|FACG[ET])(v4f32|v2f64|v8f16)(rz)?$",
"^(EXT|BIF|BIT|BSL)v16i8$",
"^REV(16|32|64)v(4i32|8i16|16i8)$",
- "^TB[LX]v16i8.*$")>;
+ "^TB[LX]v16i8.*$",
+ "^(ZIP|TRN|UZP)[12]v(2i64|4i32|8i16|16i8)$")>;
// Absolute-difference accumulate, pairwise accumulate and the rounding
// add/subtract-narrow take four cycles from their source operands. (A chain
// through the tied accumulator alone recurs faster, but that is a forwarding
diff --git a/llvm/test/tools/llvm-mca/AArch64/Cortex/A53-asimd-q-form.s b/llvm/test/tools/llvm-mca/AArch64/Cortex/A53-asimd-q-form.s
index 28b0c0689caed..583209c39c5f3 100644
--- a/llvm/test/tools/llvm-mca/AArch64/Cortex/A53-asimd-q-form.s
+++ b/llvm/test/tools/llvm-mca/AArch64/Cortex/A53-asimd-q-form.s
@@ -43,12 +43,12 @@ faddp v0.2s, v1.2s, v2.2s
# CHECK-NEXT: 1 4 1.00 fadd v0.2s, v1.2s, v2.2s
# CHECK-NEXT: 1 2 2.00 add v0.4s, v1.4s, v2.4s
# CHECK-NEXT: 1 2 1.00 add v0.2s, v1.2s, v2.2s
-# CHECK-NEXT: 1 6 2.00 zip1 v0.4s, v1.4s, v2.4s
-# CHECK-NEXT: 1 6 1.00 zip1 v0.2s, v1.2s, v2.2s
-# CHECK-NEXT: 1 6 2.00 trn1 v0.16b, v1.16b, v2.16b
-# CHECK-NEXT: 1 6 1.00 trn1 v0.8b, v1.8b, v2.8b
-# CHECK-NEXT: 1 6 2.00 uzp1 v0.4s, v1.4s, v2.4s
-# CHECK-NEXT: 1 6 1.00 uzp1 v0.2s, v1.2s, v2.2s
+# CHECK-NEXT: 1 2 2.00 zip1 v0.4s, v1.4s, v2.4s
+# CHECK-NEXT: 1 2 1.00 zip1 v0.2s, v1.2s, v2.2s
+# CHECK-NEXT: 1 2 2.00 trn1 v0.16b, v1.16b, v2.16b
+# CHECK-NEXT: 1 2 1.00 trn1 v0.8b, v1.8b, v2.8b
+# CHECK-NEXT: 1 2 2.00 uzp1 v0.4s, v1.4s, v2.4s
+# CHECK-NEXT: 1 2 1.00 uzp1 v0.2s, v1.2s, v2.2s
# CHECK-NEXT: 1 2 2.00 ext v0.16b, v1.16b, v2.16b, #4
# CHECK-NEXT: 1 2 1.00 ext v0.8b, v1.8b, v2.8b, #4
# CHECK-NEXT: 1 2 1.00 dup v0.4s, v1.s[1]
>From 78f6b8b02aa6ec5a104f2215c7d69db1354ed8c4 Mon Sep 17 00:00:00 2001
From: Michele Dionisio <michele.dionisio at gmail.com>
Date: Fri, 31 Jul 2026 10:21:19 +0200
Subject: [PATCH 13/15] [AArch64] Model the Cortex-A53 FP/ASIMD datapath as two
lanes
"[AArch64] Model the 64-bit ASIMD datapath in the Cortex-A53 sched model"
expressed the 128-bit-per-cycle FP/ASIMD datapath by charging Q-form
operations two cycles of occupancy on a single-lane pipe. That got the
relative cost of Q-form against D-form right, but at the price of making both
twice as expensive as the hardware in absolute terms, because a single
ProcResource cannot express the two 64-bit operations the core issues per
cycle. It also had to exempt the crypto instructions to keep fused AES pairs
issuing back-to-back. That commit named this follow-up as the proper fix.
Model each FP/ASIMD pipe as a two-lane ProcResource instead: scalar and 64-bit
operations take one lane, Q-form operations take both. Divide and square root
are not pipelined and block the whole unit, so they hold both lanes for their
full occupancy, which leaves their reciprocal throughput unchanged. The
compares are the one scalar exception: they write NZCV, only one instruction
per cycle can do that, and they measure 1.06 cycles each rather than the 0.53
of a dual-issuing scalar operation, so they take both lanes too. The crypto
special case is dropped: both lanes for one cycle is exactly the throughput of
the 128-bit-wide crypto unit, and it no longer disturbs AES fusion.
Reciprocal throughput now matches the hardware. Measured on a Raspberry Pi 3
(Cortex-A53) with perf hardware cycle counters, 32 independent operations per
iteration (cycles per operation), against what the model reports:
measured before now
scalar integer 0.56 0.50 0.50
add v.8b / v.16b 0.56/1.03 1.00/2.00 0.50/1.00
fadd v.2s / v.4s 0.56/1.03 1.00/2.00 0.50/1.00
fmul v.2s / v.4s 0.53/1.03 1.00/2.00 0.50/1.00
fmla v.2s / v.4s 0.53/1.03 1.00/2.00 0.50/1.00
fcmp 1.06 1.00 1.00
Besides the absolute error, the previous state skewed the balance between
pipes: the integer ALU was modelled correctly at 0.50 while everything on the
FP/ASIMD pipes was twice too expensive, so the model over-weighted vector work
relative to scalar integer work. Both are now consistent.
This is the same two-lane idiom the sibling Cortex-A55 model uses, where
CortexA55UnitFPALU is a ProcResource<2> and its Q-form write class takes the
resource twice.
Applies only to CortexA53Model (A53/A34/A35); Cortex-A65/Neoverse-E1 are on
their own model. The Cortex/A53-asimd-q-form.s test is updated.
Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
llvm/lib/Target/AArch64/AArch64SchedA53.td | 118 ++++++++++--------
.../AArch64/Cortex/A53-asimd-q-form.s | 99 ++++++++-------
2 files changed, 117 insertions(+), 100 deletions(-)
diff --git a/llvm/lib/Target/AArch64/AArch64SchedA53.td b/llvm/lib/Target/AArch64/AArch64SchedA53.td
index a33ce58cd1a89..883f0653ce5d9 100644
--- a/llvm/lib/Target/AArch64/AArch64SchedA53.td
+++ b/llvm/lib/Target/AArch64/AArch64SchedA53.td
@@ -44,8 +44,12 @@ def A53UnitMAC : ProcResource<1> { let BufferSize = 0; } // Int MAC
def A53UnitDiv : ProcResource<1> { let BufferSize = 0; } // Int Division
def A53UnitLdSt : ProcResource<1> { let BufferSize = 0; } // Load/Store
def A53UnitB : ProcResource<1> { let BufferSize = 0; } // Branch
-def A53UnitFPALU : ProcResource<1> { let BufferSize = 0; } // FP ALU
-def A53UnitFPMDS : ProcResource<1> { let BufferSize = 0; } // FP Mult/Div/Sqrt
+// The FP/ASIMD datapath moves 128 bits per cycle, which it can spend either on
+// one 128-bit (Q-form) operation or on two 64-bit ones. Model each FP/ASIMD
+// pipe as two lanes: 64-bit and scalar operations take a single lane, Q-form
+// operations take both.
+def A53UnitFPALU : ProcResource<2> { let BufferSize = 0; } // FP ALU
+def A53UnitFPMDS : ProcResource<2> { let BufferSize = 0; } // FP Mult/Div/Sqrt
//===----------------------------------------------------------------------===//
@@ -125,84 +129,104 @@ def : WriteRes<WriteHint, [A53UnitB]>;
// FP ALU. Scalar floating-point arithmetic and conversion have a 4-cycle
// latency on Cortex-A53.
def : WriteRes<WriteF, [A53UnitFPALU]> { let Latency = 4; }
-def : WriteRes<WriteFCmp, [A53UnitFPALU]> { let Latency = 6; }
+// The compares write NZCV, and only one instruction per cycle can do that, so
+// they do not dual-issue: they take both lanes even though they are scalar.
+def : WriteRes<WriteFCmp, [A53UnitFPALU, A53UnitFPALU]> { let Latency = 6; }
def : WriteRes<WriteFCvt, [A53UnitFPALU]> { let Latency = 4; }
// Float-to-int register copy (FMOV between a GPR and an FP register) takes a
// single cycle in each direction.
def : WriteRes<WriteFCopy, [A53UnitFPALU]> { let Latency = 1; }
def : WriteRes<WriteFImm, [A53UnitFPALU]> { let Latency = 6; }
def : WriteRes<WriteVd, [A53UnitFPALU]> { let Latency = 6; }
-// The ASIMD datapath is 64 bits wide: 128-bit (Q-form) operations occupy the
-// FP/ASIMD pipe for two cycles.
-def : WriteRes<WriteVq, [A53UnitFPALU]> { let Latency = 6;
- let ReleaseAtCycles = [2]; }
+// A 128-bit (Q-form) operation fills the whole 128-bit datapath, so it takes
+// both lanes.
+def : WriteRes<WriteVq, [A53UnitFPALU, A53UnitFPALU]> { let Latency = 6; }
-// FP Mul, Div, Sqrt
+// FP Mul, Div, Sqrt. Divide and square root are not pipelined and block the
+// whole unit, so they hold both lanes for their full occupancy.
def : WriteRes<WriteFMul, [A53UnitFPMDS]> { let Latency = 4; }
-def : WriteRes<WriteFDiv, [A53UnitFPMDS]> { let Latency = 33;
- let ReleaseAtCycles = [29]; }
+def : WriteRes<WriteFDiv, [A53UnitFPMDS, A53UnitFPMDS]> {
+ let Latency = 33;
+ let ReleaseAtCycles = [29, 29];
+}
// FP multiply-accumulate has an 8-cycle result latency, but the MAC pipe reads
// the addend late and has a dedicated forwarding path that feeds one MAC's
// result into the next MAC's addend, so a dependent MAC chain recurs every 4
// cycles. Only the addend forwards; the multiplicands do not.
def A53WriteFMAC : SchedWriteRes<[A53UnitFPMDS]> { let Latency = 8; }
-// Q-form multiply-accumulate occupies the MAC pipe for two cycles.
-def A53WriteFMACQ : SchedWriteRes<[A53UnitFPMDS]> { let Latency = 8;
- let ReleaseAtCycles = [2]; }
+// Q-form multiply-accumulate fills the datapath and takes both lanes.
+def A53WriteFMACQ : SchedWriteRes<[A53UnitFPMDS, A53UnitFPMDS]> {
+ let Latency = 8;
+}
def A53ReadFPM : SchedReadAdvance<0>;
def A53ReadFPMA4 : SchedReadAdvance<4, [A53WriteFMAC, A53WriteFMACQ]>;
-def A53WriteFDivSP : SchedWriteRes<[A53UnitFPMDS]> { let Latency = 18;
- let ReleaseAtCycles = [14]; }
-def A53WriteFDivDP : SchedWriteRes<[A53UnitFPMDS]> { let Latency = 33;
- let ReleaseAtCycles = [29]; }
-def A53WriteFSqrtSP : SchedWriteRes<[A53UnitFPMDS]> { let Latency = 17;
- let ReleaseAtCycles = [13]; }
-def A53WriteFSqrtDP : SchedWriteRes<[A53UnitFPMDS]> { let Latency = 32;
- let ReleaseAtCycles = [28]; }
+def A53WriteFDivSP : SchedWriteRes<[A53UnitFPMDS, A53UnitFPMDS]> {
+ let Latency = 18;
+ let ReleaseAtCycles = [14, 14];
+}
+def A53WriteFDivDP : SchedWriteRes<[A53UnitFPMDS, A53UnitFPMDS]> {
+ let Latency = 33;
+ let ReleaseAtCycles = [29, 29];
+}
+def A53WriteFSqrtSP : SchedWriteRes<[A53UnitFPMDS, A53UnitFPMDS]> {
+ let Latency = 17;
+ let ReleaseAtCycles = [13, 13];
+}
+def A53WriteFSqrtDP : SchedWriteRes<[A53UnitFPMDS, A53UnitFPMDS]> {
+ let Latency = 32;
+ let ReleaseAtCycles = [28, 28];
+}
// Reciprocal / reciprocal-square-root Newton-Raphson step (FRECPS/FRSQRTS) and
// estimate (FRECPE/FRSQRTE). Measured on Cortex-A53: the step is 8 cycles and
// the estimate 4, not the flat 6 of WriteVd. These run on the FP/ASIMD ALU pipe,
-// so the Q-form variants keep the two-cycle datapath occupancy.
+// so the Q-form variants take both lanes.
def A53WriteFRecpStep : SchedWriteRes<[A53UnitFPALU]> { let Latency = 8; }
-def A53WriteFRecpStepQ : SchedWriteRes<[A53UnitFPALU]> { let Latency = 8;
- let ReleaseAtCycles = [2]; }
+def A53WriteFRecpStepQ : SchedWriteRes<[A53UnitFPALU, A53UnitFPALU]> {
+ let Latency = 8;
+}
def A53WriteFRecpEst : SchedWriteRes<[A53UnitFPALU]> { let Latency = 4; }
-def A53WriteFRecpEstQ : SchedWriteRes<[A53UnitFPALU]> { let Latency = 4;
- let ReleaseAtCycles = [2]; }
+def A53WriteFRecpEstQ : SchedWriteRes<[A53UnitFPALU, A53UnitFPALU]> {
+ let Latency = 4;
+}
// Integer Advanced SIMD multiply-accumulate. The result latency is 4, but the
// MAC unit forwards one accumulate result straight into the next MAC's
// accumulator with no interlock, so a dependent accumulator chain recurs every
// cycle. Only the accumulator forwards; the multiplicands do not. The
// saturating doubling forms (SQDML[AS]L) do not use this path.
def A53WriteIMACD : SchedWriteRes<[A53UnitFPALU]> { let Latency = 4; }
-def A53WriteIMACQ : SchedWriteRes<[A53UnitFPALU]> { let Latency = 4;
- let ReleaseAtCycles = [2]; }
+def A53WriteIMACQ : SchedWriteRes<[A53UnitFPALU, A53UnitFPALU]> {
+ let Latency = 4;
+}
def A53ReadIMACM : SchedReadAdvance<0>;
def A53ReadIMACA3 : SchedReadAdvance<3, [A53WriteIMACD, A53WriteIMACQ]>;
// Bitwise Advanced SIMD logic forwards in a single cycle.
def A53WriteLogicD : SchedWriteRes<[A53UnitFPALU]> { let Latency = 1; }
-def A53WriteLogicQ : SchedWriteRes<[A53UnitFPALU]> { let Latency = 1;
- let ReleaseAtCycles = [2]; }
+def A53WriteLogicQ : SchedWriteRes<[A53UnitFPALU, A53UnitFPALU]> {
+ let Latency = 1;
+}
// Scalar floating-point select, and the float-to-float register move.
def A53WriteFSel : SchedWriteRes<[A53UnitFPALU]> { let Latency = 2; }
def A53WriteFMov : SchedWriteRes<[A53UnitFPALU]> { let Latency = 1; }
// Advanced SIMD arithmetic that runs at the same 4-cycle latency as its scalar
// counterpart: floating-point arithmetic and conversion, and the integer vector
-// multiply. Q-form keeps the two-cycle datapath occupancy.
+// multiply. Q-form takes both datapath lanes.
def A53WriteVecD_4 : SchedWriteRes<[A53UnitFPALU]> { let Latency = 4; }
-def A53WriteVecQ_4 : SchedWriteRes<[A53UnitFPALU]> { let Latency = 4;
- let ReleaseAtCycles = [2]; }
+def A53WriteVecQ_4 : SchedWriteRes<[A53UnitFPALU, A53UnitFPALU]> {
+ let Latency = 4;
+}
// Integer Advanced SIMD ALU. Measured on Cortex-A53, and matching the sibling
// Cortex-A55 model: simple arithmetic (ADD/SUB/NEG, halving add/sub) is 2
// cycles and the wider ALU ops (ABS, pairwise, reduce, saturating, abs-diff)
-// 3, not the flat 6 of WriteVd. Q-form keeps the two-cycle datapath occupancy.
+// 3, not the flat 6 of WriteVd. Q-form takes both datapath lanes.
def A53WriteIVecD_2 : SchedWriteRes<[A53UnitFPALU]> { let Latency = 2; }
-def A53WriteIVecQ_2 : SchedWriteRes<[A53UnitFPALU]> { let Latency = 2;
- let ReleaseAtCycles = [2]; }
+def A53WriteIVecQ_2 : SchedWriteRes<[A53UnitFPALU, A53UnitFPALU]> {
+ let Latency = 2;
+}
def A53WriteIVecD_3 : SchedWriteRes<[A53UnitFPALU]> { let Latency = 3; }
-def A53WriteIVecQ_3 : SchedWriteRes<[A53UnitFPALU]> { let Latency = 3;
- let ReleaseAtCycles = [2]; }
+def A53WriteIVecQ_3 : SchedWriteRes<[A53UnitFPALU, A53UnitFPALU]> {
+ let Latency = 3;
+}
//===----------------------------------------------------------------------===//
// Subtarget-specific SchedRead types.
@@ -346,20 +370,10 @@ def : InstRW<[WriteAdr, A53WriteVST2], (instregex "ST4Fourv(2d)_POST$")>;
//---
// Floating Point MAC, DIV, SQRT
//---
-// Q-form crypto (AES/SHA/PMULL) keeps single-cycle occupancy instead of the
-// WriteVq two-cycle penalty. The crypto unit is 128 bits wide (per ARM), and,
-// crucially, this preserves back-to-back issue of fused AES pairs
-// (FeatureFuseAES): removing this exemption regresses misched-fusion-aes.ll on
-// Cortex-A53. The exact crypto throughput was NOT benchmarked --
-// the test board's A53 (Raspberry Pi 3) lacks the crypto extension -- so this
-// rests on ARM's documented 128-bit crypto datapath; the fusion behaviour it
-// protects is what the test covers. To be re-checked on crypto-capable silicon.
-def A53WriteCrypto : SchedWriteRes<[A53UnitFPALU]> { let Latency = 6; }
-def : InstRW<[A53WriteCrypto], (instregex "^AES[DE]rr$", "^AESI?MCrr$",
- "^SHA1(C|M|P)rrr$", "^SHA1Hrr$",
- "^SHA1SU0rrr$", "^SHA1SU1rr$",
- "^SHA256H2?rrr$", "^SHA256SU0rr$",
- "^SHA256SU1rrr$", "^PMULLv")>;
+// Q-form crypto (AES/SHA/PMULL) needs no special case: taking both datapath
+// lanes for one cycle is exactly the throughput of the 128-bit-wide crypto
+// unit, and it leaves fused AES pairs (FeatureFuseAES) free to issue
+// back-to-back.
// The scalar forms take the addend as their third source operand, so only the
// third read forwards; the ASIMD forms accumulate into the tied destination,
diff --git a/llvm/test/tools/llvm-mca/AArch64/Cortex/A53-asimd-q-form.s b/llvm/test/tools/llvm-mca/AArch64/Cortex/A53-asimd-q-form.s
index 583209c39c5f3..fc863e0701f3d 100644
--- a/llvm/test/tools/llvm-mca/AArch64/Cortex/A53-asimd-q-form.s
+++ b/llvm/test/tools/llvm-mca/AArch64/Cortex/A53-asimd-q-form.s
@@ -1,9 +1,10 @@
# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
# RUN: llvm-mca -mtriple=aarch64-none-linux-gnu -mcpu=cortex-a53 -instruction-tables < %s | FileCheck %s
-# The Cortex-A53 ASIMD datapath is 64 bits wide: 128-bit (Q-form) operations
-# occupy the FP/ASIMD pipes for two cycles, so their reciprocal throughput is
-# twice that of the equivalent 64-bit (D-form) operations.
+# The Cortex-A53 FP/ASIMD datapath moves 128 bits per cycle, spent either on one
+# 128-bit (Q-form) operation or on two 64-bit ones. Each pipe is modelled as two
+# lanes, so a Q-form operation has twice the reciprocal throughput of the
+# equivalent 64-bit (D-form) operation: 1.00 against 0.50.
fmla v0.4s, v1.4s, v2.4s
fmla v0.2d, v1.2d, v2.2d
@@ -35,60 +36,62 @@ faddp v0.2s, v1.2s, v2.2s
# CHECK-NEXT: [6]: HasSideEffects (U)
# CHECK: [1] [2] [3] [4] [5] [6] Instructions:
-# CHECK-NEXT: 1 8 2.00 fmla v0.4s, v1.4s, v2.4s
-# CHECK-NEXT: 1 8 2.00 fmla v0.2d, v1.2d, v2.2d
-# CHECK-NEXT: 1 8 1.00 fmla v0.2s, v1.2s, v2.2s
-# CHECK-NEXT: 1 8 1.00 fmadd s0, s1, s2, s3
-# CHECK-NEXT: 1 4 2.00 fadd v0.4s, v1.4s, v2.4s
-# CHECK-NEXT: 1 4 1.00 fadd v0.2s, v1.2s, v2.2s
-# CHECK-NEXT: 1 2 2.00 add v0.4s, v1.4s, v2.4s
-# CHECK-NEXT: 1 2 1.00 add v0.2s, v1.2s, v2.2s
-# CHECK-NEXT: 1 2 2.00 zip1 v0.4s, v1.4s, v2.4s
-# CHECK-NEXT: 1 2 1.00 zip1 v0.2s, v1.2s, v2.2s
-# CHECK-NEXT: 1 2 2.00 trn1 v0.16b, v1.16b, v2.16b
-# CHECK-NEXT: 1 2 1.00 trn1 v0.8b, v1.8b, v2.8b
-# CHECK-NEXT: 1 2 2.00 uzp1 v0.4s, v1.4s, v2.4s
-# CHECK-NEXT: 1 2 1.00 uzp1 v0.2s, v1.2s, v2.2s
-# CHECK-NEXT: 1 2 2.00 ext v0.16b, v1.16b, v2.16b, #4
-# CHECK-NEXT: 1 2 1.00 ext v0.8b, v1.8b, v2.8b, #4
-# CHECK-NEXT: 1 2 1.00 dup v0.4s, v1.s[1]
-# CHECK-NEXT: 1 2 1.00 dup v0.2s, v1.s[1]
-# CHECK-NEXT: 1 4 2.00 faddp v0.4s, v1.4s, v2.4s
-# CHECK-NEXT: 1 4 1.00 faddp v0.2s, v1.2s, v2.2s
+# CHECK-NEXT: 1 8 1.00 fmla v0.4s, v1.4s, v2.4s
+# CHECK-NEXT: 1 8 1.00 fmla v0.2d, v1.2d, v2.2d
+# CHECK-NEXT: 1 8 0.50 fmla v0.2s, v1.2s, v2.2s
+# CHECK-NEXT: 1 8 0.50 fmadd s0, s1, s2, s3
+# CHECK-NEXT: 1 4 1.00 fadd v0.4s, v1.4s, v2.4s
+# CHECK-NEXT: 1 4 0.50 fadd v0.2s, v1.2s, v2.2s
+# CHECK-NEXT: 1 2 1.00 add v0.4s, v1.4s, v2.4s
+# CHECK-NEXT: 1 2 0.50 add v0.2s, v1.2s, v2.2s
+# CHECK-NEXT: 1 2 1.00 zip1 v0.4s, v1.4s, v2.4s
+# CHECK-NEXT: 1 2 0.50 zip1 v0.2s, v1.2s, v2.2s
+# CHECK-NEXT: 1 2 1.00 trn1 v0.16b, v1.16b, v2.16b
+# CHECK-NEXT: 1 2 0.50 trn1 v0.8b, v1.8b, v2.8b
+# CHECK-NEXT: 1 2 1.00 uzp1 v0.4s, v1.4s, v2.4s
+# CHECK-NEXT: 1 2 0.50 uzp1 v0.2s, v1.2s, v2.2s
+# CHECK-NEXT: 1 2 1.00 ext v0.16b, v1.16b, v2.16b, #4
+# CHECK-NEXT: 1 2 0.50 ext v0.8b, v1.8b, v2.8b, #4
+# CHECK-NEXT: 1 2 0.50 dup v0.4s, v1.s[1]
+# CHECK-NEXT: 1 2 0.50 dup v0.2s, v1.s[1]
+# CHECK-NEXT: 1 4 1.00 faddp v0.4s, v1.4s, v2.4s
+# CHECK-NEXT: 1 4 0.50 faddp v0.2s, v1.2s, v2.2s
# CHECK: Resources:
# CHECK-NEXT: [0.0] - A53UnitALU
# CHECK-NEXT: [0.1] - A53UnitALU
# CHECK-NEXT: [1] - A53UnitB
# CHECK-NEXT: [2] - A53UnitDiv
-# CHECK-NEXT: [3] - A53UnitFPALU
-# CHECK-NEXT: [4] - A53UnitFPMDS
+# CHECK-NEXT: [3.0] - A53UnitFPALU
+# CHECK-NEXT: [3.1] - A53UnitFPALU
+# CHECK-NEXT: [4.0] - A53UnitFPMDS
+# CHECK-NEXT: [4.1] - A53UnitFPMDS
# CHECK-NEXT: [5] - A53UnitLdSt
# CHECK-NEXT: [6] - A53UnitMAC
# CHECK: Resource pressure per iteration:
-# CHECK-NEXT: [0.0] [0.1] [1] [2] [3] [4] [5] [6]
-# CHECK-NEXT: - - - - 23.00 6.00 - -
+# CHECK-NEXT: [0.0] [0.1] [1] [2] [3.0] [3.1] [4.0] [4.1] [5] [6]
+# CHECK-NEXT: - - - - 11.50 11.50 3.00 3.00 - -
# CHECK: Resource pressure by instruction:
-# CHECK-NEXT: [0.0] [0.1] [1] [2] [3] [4] [5] [6] Instructions:
-# CHECK-NEXT: - - - - - 2.00 - - fmla v0.4s, v1.4s, v2.4s
-# CHECK-NEXT: - - - - - 2.00 - - fmla v0.2d, v1.2d, v2.2d
-# CHECK-NEXT: - - - - - 1.00 - - fmla v0.2s, v1.2s, v2.2s
-# CHECK-NEXT: - - - - - 1.00 - - fmadd s0, s1, s2, s3
-# CHECK-NEXT: - - - - 2.00 - - - fadd v0.4s, v1.4s, v2.4s
-# CHECK-NEXT: - - - - 1.00 - - - fadd v0.2s, v1.2s, v2.2s
-# CHECK-NEXT: - - - - 2.00 - - - add v0.4s, v1.4s, v2.4s
-# CHECK-NEXT: - - - - 1.00 - - - add v0.2s, v1.2s, v2.2s
-# CHECK-NEXT: - - - - 2.00 - - - zip1 v0.4s, v1.4s, v2.4s
-# CHECK-NEXT: - - - - 1.00 - - - zip1 v0.2s, v1.2s, v2.2s
-# CHECK-NEXT: - - - - 2.00 - - - trn1 v0.16b, v1.16b, v2.16b
-# CHECK-NEXT: - - - - 1.00 - - - trn1 v0.8b, v1.8b, v2.8b
-# CHECK-NEXT: - - - - 2.00 - - - uzp1 v0.4s, v1.4s, v2.4s
-# CHECK-NEXT: - - - - 1.00 - - - uzp1 v0.2s, v1.2s, v2.2s
-# CHECK-NEXT: - - - - 2.00 - - - ext v0.16b, v1.16b, v2.16b, #4
-# CHECK-NEXT: - - - - 1.00 - - - ext v0.8b, v1.8b, v2.8b, #4
-# CHECK-NEXT: - - - - 1.00 - - - dup v0.4s, v1.s[1]
-# CHECK-NEXT: - - - - 1.00 - - - dup v0.2s, v1.s[1]
-# CHECK-NEXT: - - - - 2.00 - - - faddp v0.4s, v1.4s, v2.4s
-# CHECK-NEXT: - - - - 1.00 - - - faddp v0.2s, v1.2s, v2.2s
+# CHECK-NEXT: [0.0] [0.1] [1] [2] [3.0] [3.1] [4.0] [4.1] [5] [6] Instructions:
+# CHECK-NEXT: - - - - - - 1.00 1.00 - - fmla v0.4s, v1.4s, v2.4s
+# CHECK-NEXT: - - - - - - 1.00 1.00 - - fmla v0.2d, v1.2d, v2.2d
+# CHECK-NEXT: - - - - - - 0.50 0.50 - - fmla v0.2s, v1.2s, v2.2s
+# CHECK-NEXT: - - - - - - 0.50 0.50 - - fmadd s0, s1, s2, s3
+# CHECK-NEXT: - - - - 1.00 1.00 - - - - fadd v0.4s, v1.4s, v2.4s
+# CHECK-NEXT: - - - - 0.50 0.50 - - - - fadd v0.2s, v1.2s, v2.2s
+# CHECK-NEXT: - - - - 1.00 1.00 - - - - add v0.4s, v1.4s, v2.4s
+# CHECK-NEXT: - - - - 0.50 0.50 - - - - add v0.2s, v1.2s, v2.2s
+# CHECK-NEXT: - - - - 1.00 1.00 - - - - zip1 v0.4s, v1.4s, v2.4s
+# CHECK-NEXT: - - - - 0.50 0.50 - - - - zip1 v0.2s, v1.2s, v2.2s
+# CHECK-NEXT: - - - - 1.00 1.00 - - - - trn1 v0.16b, v1.16b, v2.16b
+# CHECK-NEXT: - - - - 0.50 0.50 - - - - trn1 v0.8b, v1.8b, v2.8b
+# CHECK-NEXT: - - - - 1.00 1.00 - - - - uzp1 v0.4s, v1.4s, v2.4s
+# CHECK-NEXT: - - - - 0.50 0.50 - - - - uzp1 v0.2s, v1.2s, v2.2s
+# CHECK-NEXT: - - - - 1.00 1.00 - - - - ext v0.16b, v1.16b, v2.16b, #4
+# CHECK-NEXT: - - - - 0.50 0.50 - - - - ext v0.8b, v1.8b, v2.8b, #4
+# CHECK-NEXT: - - - - 0.50 0.50 - - - - dup v0.4s, v1.s[1]
+# CHECK-NEXT: - - - - 0.50 0.50 - - - - dup v0.2s, v1.s[1]
+# CHECK-NEXT: - - - - 1.00 1.00 - - - - faddp v0.4s, v1.4s, v2.4s
+# CHECK-NEXT: - - - - 0.50 0.50 - - - - faddp v0.2s, v1.2s, v2.2s
>From e3a9bea93e43df355a1e0b9ae8d31e8a252a3e99 Mon Sep 17 00:00:00 2001
From: Michele Dionisio <michele.dionisio at gmail.com>
Date: Fri, 31 Jul 2026 10:38:45 +0200
Subject: [PATCH 14/15] [AArch64] Fix the Cortex-A53 floating-point compare
latency
WriteFCmp is modelled at 6 cycles, the same flat value the whole FP/ASIMD
block used to carry. That is far too high: it predicts that a compare feeding
a select costs 6 cycles before the select can start, when the entire
round trip on hardware is 3.
Set it to 3, which is what every sibling model in the family uses --
Cortex-A55, A510, A320 and A57 all give WriteFCmp a 3-cycle latency, and
those models are derived from ARM's published Software Optimization Guides.
Measured on a Raspberry Pi 3 (Cortex-A53) with perf hardware cycle counters.
A compare cannot be chained to itself, because it writes NZCV rather than a
register and back-to-back compares are only WAW, so the chains close the loop
through a flag consumer (cycles per link):
fcmp s0, s1 ; fcsel s0, s2, s3, eq 3.00 (loop s0 -> NZCV -> s0)
same pair with the flags unused 2.00 (dependency broken)
fcmp ; csel w2, w3, w4, eq ; fmov s0, w2 3.00 (1 cycle per instruction)
cmp x2, x3 ; csel x2, x3, x4, eq 2.00 (integer baseline)
So a dependent NZCV consumer only pays one cycle: the chained pair costs one
cycle more than the same pair with the dependency broken, and the three
instruction chain runs at exactly one cycle each, issue-limited rather than
latency-limited.
That single cycle is a forwarding path, not the result latency, and the two
cannot be told apart by measurement here because every NZCV consumer that can
be built benefits from it -- the same situation as the FP multiply-accumulate,
whose 8-cycle result forwards to a dependent accumulator in 4. Unlike the
multiply-accumulate there is no way to express it: ReadAdvance attaches to
explicit operands, while NZCV is an implicit register, so whatever value
WriteFCmp carries is charged in full to a dependent consumer.
Hence 3 rather than 1: it is the figure the rest of the family uses, it more
than halves the error against the current 6, and it errs towards charging a
forwarded consumer too much rather than too little.
Applies only to CortexA53Model (A53/A34/A35); Cortex-A65/Neoverse-E1 are on
their own model.
Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
llvm/lib/Target/AArch64/AArch64SchedA53.td | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/llvm/lib/Target/AArch64/AArch64SchedA53.td b/llvm/lib/Target/AArch64/AArch64SchedA53.td
index 883f0653ce5d9..cf684aa929142 100644
--- a/llvm/lib/Target/AArch64/AArch64SchedA53.td
+++ b/llvm/lib/Target/AArch64/AArch64SchedA53.td
@@ -131,7 +131,10 @@ def : WriteRes<WriteHint, [A53UnitB]>;
def : WriteRes<WriteF, [A53UnitFPALU]> { let Latency = 4; }
// The compares write NZCV, and only one instruction per cycle can do that, so
// they do not dual-issue: they take both lanes even though they are scalar.
-def : WriteRes<WriteFCmp, [A53UnitFPALU, A53UnitFPALU]> { let Latency = 6; }
+// A dependent NZCV consumer only pays a single cycle in practice, but the
+// forwarding path that makes that possible cannot be expressed here, so this
+// keeps the 3-cycle result latency the rest of the family uses.
+def : WriteRes<WriteFCmp, [A53UnitFPALU, A53UnitFPALU]> { let Latency = 3; }
def : WriteRes<WriteFCvt, [A53UnitFPALU]> { let Latency = 4; }
// Float-to-int register copy (FMOV between a GPR and an FP register) takes a
// single cycle in each direction.
>From bcfda25a9612516093e20f0f793619a791480b6f Mon Sep 17 00:00:00 2001
From: Michele Dionisio <michele.dionisio at gmail.com>
Date: Fri, 31 Jul 2026 11:35:29 +0200
Subject: [PATCH 15/15] [AArch64] Model the remaining Cortex-A53 ASIMD
latencies
A last group of Advanced SIMD operations still falls through to the flat
6-cycle WriteVd/WriteVq. On Cortex-A53 none of them takes that long.
Bit counting and reversal (CNT/CLZ/RBIT), the widening shift SHLL and the
vector-to-GPR moves (UMOV/SMOV) are two cycles; the polynomial multiply PMUL
is three; the saturating multiply-high (SQDMULH/SQRDMULH), the saturating
absolute SQABS, the integer min/max reductions ([SU]MAXV/[SU]MINV) and the
scalar floating-point pairwise reductions (FADDP/FMAXP/FMINP and the NM
variants in their scalar-reduce form) are four.
Measured on a Raspberry Pi 3 (Cortex-A53) with perf hardware cycle counters,
using dependent chains (cycles per link):
cnt 2.00 clz 2.00 rbit 2.00 shll 2.00 sxtl 2.00
pmul 3.00
sqdmulh 4.00 sqrdmulh 4.00 sqabs 4.00
smaxv 4.00 sminv 4.00 umaxv 4.00 uminv 4.00
faddp (scalar reduce) 4.00 fmaxp (scalar reduce) 4.00
UMOV and SMOV write a general-purpose register and so cannot be chained to
themselves; they were measured as a round trip through the float-to-int FMOV,
which costs 3.00 cycles per pair against a known 1-cycle FMOV, giving 2.
MOVI, MVNI and the FMOV immediate forms are left alone: they take no source
operand, so no dependent chain can be built for them and their latency was
not measured.
Applies only to CortexA53Model (A53/A34/A35); Cortex-A65/Neoverse-E1 are on
their own model.
Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
llvm/lib/Target/AArch64/AArch64SchedA53.td | 26 ++++++++++++++++++++++
1 file changed, 26 insertions(+)
diff --git a/llvm/lib/Target/AArch64/AArch64SchedA53.td b/llvm/lib/Target/AArch64/AArch64SchedA53.td
index cf684aa929142..74aeaf265a63d 100644
--- a/llvm/lib/Target/AArch64/AArch64SchedA53.td
+++ b/llvm/lib/Target/AArch64/AArch64SchedA53.td
@@ -563,4 +563,30 @@ def : InstRW<[A53WriteIVecQ_3],
"^[SU]ADDLPv(16i8_v8i16|4i32_v2i64|8i16_v4i32)$",
"^([SU]QADD|[SU]QSUB|SQNEG|SUQADD|USQADD)v(16i8|2i64|4i32|8i16)$")>;
+// The operations that were still inheriting the flat 6-cycle WriteVd/WriteVq.
+// Bit counting and reversal, the widening shift and the vector-to-GPR moves are
+// two cycles; the polynomial multiply is three; the saturating multiply-high,
+// the saturating absolute, the integer min/max reductions and the scalar
+// floating-point pairwise reductions are four.
+def : InstRW<[A53WriteIVecD_2],
+ (instregex "^(CNT|RBIT)v8i8$", "^CLZv(2i32|4i16|8i8)$",
+ "^[SU]MOVvi.*$")>;
+def : InstRW<[A53WriteIVecQ_2],
+ (instregex "^(CNT|RBIT)v16i8$", "^CLZv(4i32|8i16|16i8)$",
+ "^SHLLv.*$")>;
+def : InstRW<[A53WriteIVecD_3], (instrs PMULv8i8)>;
+def : InstRW<[A53WriteIVecQ_3], (instrs PMULv16i8)>;
+def : InstRW<[A53WriteVecD_4],
+ (instregex "^SQR?DMULH(v1i16|v1i32)$",
+ "^SQR?DMULHv(2i32|4i16)(_indexed)?$",
+ "^SQR?DMULH(v1i16|v1i32)_indexed$",
+ "^SQABSv(1i8|1i16|1i32|1i64|2i32|4i16|8i8)$",
+ "^[SU](MAX|MIN)Vv(4i16v|8i8v)$",
+ "^F(ADD|MAX|MIN|MAXNM|MINNM)Pv2i32p$")>;
+def : InstRW<[A53WriteVecQ_4],
+ (instregex "^SQR?DMULHv(4i32|8i16)(_indexed)?$",
+ "^SQABSv(16i8|2i64|4i32|8i16)$",
+ "^[SU](MAX|MIN)Vv(16i8v|4i32v|8i16v)$",
+ "^F(ADD|MAX|MIN|MAXNM|MINNM)Pv2i64p$")>;
+
}
More information about the llvm-branch-commits
mailing list