[llvm] [X86][CostModel] Add per-shape gather/scatter cost tables for AMD znver4+ (PR #199488)
Sumukh J Bharadwaj via llvm-commits
llvm-commits at lists.llvm.org
Thu Sep 17 03:51:25 PDT 2026
================
@@ -6588,11 +6590,191 @@ InstructionCost X86TTIImpl::getCFInstrCost(unsigned Opcode,
return TTI::TCC_Free;
}
+// Pick a representative masked gather/scatter opcode for a data and index
+// shape, used to read the body cost from the schedule model. The integer and FP
+// variants of a shape share a scheduling class, so the integer form is
+// returned. Returns 0 for unsupported shapes.
+static unsigned getAVX512GSRepresentativeOpcode(bool IsLoad, MVT VT,
+ unsigned IndexBits) {
+ if (!VT.isVector())
+ return 0;
+ unsigned NumElts = VT.getVectorNumElements();
+ unsigned EltBits = VT.getScalarSizeInBits();
+ if (IndexBits != 32 && IndexBits != 64)
+ return 0;
+
+ if (IsLoad) {
+ if (EltBits == 32) {
+ if (IndexBits == 32)
+ return NumElts == 4 ? X86::VPGATHERDDZ128rm
+ : NumElts == 8 ? X86::VPGATHERDDZ256rm
+ : NumElts == 16 ? X86::VPGATHERDDZrm
+ : 0;
+ return NumElts == 4 ? X86::VPGATHERQDZ256rm
+ : NumElts == 8 ? X86::VPGATHERQDZrm
+ : 0;
+ }
+ if (EltBits == 64) {
+ if (IndexBits == 32)
+ return NumElts == 4 ? X86::VPGATHERDQZ256rm
+ : NumElts == 8 ? X86::VPGATHERDQZrm
+ : 0;
+ return NumElts == 4 ? X86::VPGATHERQQZ256rm
+ : NumElts == 8 ? X86::VPGATHERQQZrm
+ : 0;
+ }
+ return 0;
+ }
+ if (EltBits == 32) {
+ if (IndexBits == 32)
+ return NumElts == 4 ? X86::VPSCATTERDDZ128mr
+ : NumElts == 8 ? X86::VPSCATTERDDZ256mr
+ : NumElts == 16 ? X86::VPSCATTERDDZmr
+ : 0;
+ return NumElts == 4 ? X86::VPSCATTERQDZ256mr
+ : NumElts == 8 ? X86::VPSCATTERQDZmr
+ : 0;
+ }
+ if (EltBits == 64) {
+ if (IndexBits == 32)
+ return NumElts == 4 ? X86::VPSCATTERDQZ256mr
+ : NumElts == 8 ? X86::VPSCATTERDQZmr
+ : 0;
+ return NumElts == 4 ? X86::VPSCATTERQQZ256mr
+ : NumElts == 8 ? X86::VPSCATTERQQZmr
+ : 0;
+ }
+ return 0;
+}
+
+// Read an opcode's reciprocal-throughput body cost from this subtarget's
+// schedule model, or nullopt (caller falls back) when there is no
+// per-instruction model, the class is invalid or a variant (variants need a
+// real MachineInstr), or there is no real per-shape override.
+static std::optional<unsigned> getSchedModelGSBody(unsigned Opc,
+ const X86Subtarget *ST) {
+ const MCSchedModel &SM = ST->getSchedModel();
+ if (!SM.hasInstrSchedModel())
+ return std::nullopt;
+
+ const auto *TII = ST->getInstrInfo();
+ auto ReciprocalThroughputOf = [&](unsigned Opcode) -> std::optional<double> {
+ unsigned SClassID = TII->get(Opcode).getSchedClass();
+ const MCSchedClassDesc *SCDesc = SM.getSchedClassDesc(SClassID);
+ if (!SCDesc || !SCDesc->isValid() || SCDesc->isVariant())
+ return std::nullopt;
+ return MCSchedModel::getReciprocalThroughput(*ST, *SCDesc);
+ };
+
+ std::optional<double> RThru = ReciprocalThroughputOf(Opc);
+ if (!RThru)
+ return std::nullopt;
+
+ // A valid class alone is not enough: an unmodelled op also has one, and
+ // would round to 0. The cheapest a real gather/scatter can be is a plain
+ // vector load, so anything at or below the load's throughput is the generic
+ // default in disguise. Reading the baseline from the model avoids a magic
+ // floor.
+ std::optional<double> LoadRThru = ReciprocalThroughputOf(X86::VMOVUPSZrm);
+ if (!LoadRThru || *RThru <= *LoadRThru)
+ return std::nullopt;
+
+ return static_cast<unsigned>(std::lround(*RThru));
+}
+
+// Hardware body cost of a single native-width masked gather/scatter, read live
+// from the schedule model; getZenGSOverhead adds the profitability premium.
+std::optional<unsigned>
+X86TTIImpl::getModeledGSInstrCost(bool IsLoad, Type *SrcVTy,
+ unsigned IndexSize,
+ TTI::TargetCostKind CostKind) const {
+ if (CostKind != TTI::TCK_RecipThroughput || !ST->hasAVX512() ||
+ !ST->hasPreferGSCostTable() || !SrcVTy)
+ return std::nullopt;
+ EVT VT = TLI->getValueType(DL, SrcVTy);
+ if (!VT.isSimple())
+ return std::nullopt;
+ // An encoding whose data and index operands are both sub-512-bit requires
+ // AVX512VL. A v8i32 operation with i64 indices still uses a full zmm index
+ // and therefore does not.
+ unsigned IndexVectorBits = IndexSize * VT.getVectorNumElements();
+ if (VT.getSizeInBits() < 512 && IndexVectorBits < 512 && !ST->hasVLX())
+ return std::nullopt;
+ unsigned Opc =
+ getAVX512GSRepresentativeOpcode(IsLoad, VT.getSimpleVT(), IndexSize);
+ if (!Opc)
+ return std::nullopt;
+ return getSchedModelGSBody(Opc, ST);
+}
+
+// Per-shape vectorize-vs-scalarize OVERHEAD for AMD znver4+ gather/scatter
+// (TuningPreferGSCostTable): the calibrated break-even total, at which the
----------------
amd-subharad wrote:
Measured on both generations. Two of the four were a schedule-model defect; the other two are a real generational split, and I would rather put the numbers up than pick the tie-break unilaterally.
Sixteen lanes was a defect. Znver4 runs the two vpscatterqd that codegen emits slightly slower than the single vpscatterdd, 21.66 cycles against 21.05. Zn4WriteVPSCATTERQDZ's fitted residual had the pair 11% faster instead, which is what pushed the qword totals under the dword ones. Refitting that entry against its counterpart corrects the order, and v16i32 and v16f32 now total 47 under either index width. #212997 carries it. The eight-lane rows move 25 to 26 in the same direction, which is also the measured order (10.93 against 9.26).
Four lanes is not a defect. The two encodings genuinely differ by generation:
four-lane scatter, cycles/instr dword idx qword idx ratio
Zen4 8645HS 5.38 5.38 1.000
Zen5 9950X 5.53 4.61 0.833
Znver4Model 8.25 6.50 0.788
Zen4 is indifferent to index width at four lanes; Zen5 runs the qword form about 17% faster. Both readings are stable to three decimals across runs, and the emitted encodings were checked (vpscatterdd with an xmm index against vpscatterqd with a ymm index).
Note where Zn4WriteVPSCATTERQDZ256's residual of 26 sits: close to Zen5's ratio and far from Zen4's. It is effectively carrying Zen5 behavior in a model named Znver4. Since znver4, znver5 and znver6 all select Znver4Model and share one tuning bit, one number has to serve all three, so the two readings cannot both be honored.
That leaves a tie-break rather than a fix. Keeping 26 keeps v4i32 and v4f32 at 18, which is Zen5's truth and about 6% optimistic on Zen4. Refitting to 33 moves them to 19, matching the calibrated row exactly and resolving all four cases you raised, at the cost of a roughly 20% pessimistic body on that one instruction for znver5 and znver6.
The asymmetry argues for the pessimistic reading, since underpricing emits a slow scatter while overpricing only forgoes an upside. But the whole disagreement is one cost unit, and it is really an argument for a Znver5 schedule model rather than for either value. What should be the choice made here?
https://github.com/llvm/llvm-project/pull/199488
More information about the llvm-commits
mailing list