[llvm] [TableGen] Intern custom operand match sequences (PR #202620)
David Zbarsky via llvm-commits
llvm-commits at lists.llvm.org
Tue Jun 9 06:59:46 PDT 2026
https://github.com/dzbarsky created https://github.com/llvm/llvm-project/pull/202620
Custom assembly operand matching emits a flat record for every mnemonic and operand class combination. AMDGPU repeats many complete ordered sequences, producing 291,413 rows and a 51 MB generated matcher source.
Intern each distinct ordered sequence of operand mask, class, and required features. Emit one compact mnemonic descriptor containing the pooled offset and length, while retaining the empty-mnemonic full-scan behavior for matchers without a mnemonic-first grammar. Matching order and semantics remain unchanged.
For LLVM 22 AMDGPU this reduces generated matcher source from 51,036,672 to 31,302,339 bytes and pooled storage to 140,734 rows plus 2,678 descriptors.
The stripped Darwin arm64 all-tools multicall binary decreases from 153,046,248 to 150,552,984 bytes, saving 2,493,264 bytes (1.63%). A standalone llvm-mc decreases from 34,823,760 to 32,335,312 bytes, saving 2,488,448 bytes (7.15%).
Validation:
- Regenerated the complete AMDGPU matcher twice; output was byte-identical.
- Reconstructed all 291,413 canonical ordered rows from the pooled representation with exact semantic equality.
- Added and passed a focused TableGen test for mnemonic-first and non-mnemonic-first matchers.
- Existing gfx11_asm_vopd.s positive modes and wave64 diagnostics passed FileCheck; baseline and patched output were byte-identical.
- Full multicall build: https://app.buildbuddy.io/invocation/e85ce1e4-bece-4fff-9963-56002dde63e5
LLVM has no dedicated assembly-parser benchmark target. Across 80 runs of the existing 15,036-line gfx11_asm_vopd.s corpus, baseline mean/median were 44.526/41.984 ms and pooled mean/median were 42.695/41.157 ms; variance exceeds the difference and shows no observable regression.
Work towards #202616
>From f163207e34ab9b64afa7fc5e07fb216238a79d73 Mon Sep 17 00:00:00 2001
From: David Zbarsky <dzbarsky at gmail.com>
Date: Mon, 8 Jun 2026 14:14:42 -0400
Subject: [PATCH] [TableGen] Intern custom operand match sequences
Custom assembly operand matching emits a flat record for every mnemonic and operand class combination. AMDGPU repeats many complete ordered sequences, producing 291,413 rows and a 51 MB generated matcher source.
Intern each distinct ordered sequence of operand mask, class, and required features. Emit one compact mnemonic descriptor containing the pooled offset and length, while retaining the empty-mnemonic full-scan behavior for matchers without a mnemonic-first grammar. Matching order and semantics remain unchanged.
For LLVM 22 AMDGPU this reduces generated matcher source from 51,036,672 to 31,302,339 bytes and pooled storage to 140,734 rows plus 2,678 descriptors.
The stripped Darwin arm64 all-tools multicall binary decreases from 153,046,248 to 150,552,984 bytes, saving 2,493,264 bytes (1.63%). A standalone llvm-mc decreases from 34,823,760 to 32,335,312 bytes, saving 2,488,448 bytes (7.15%).
Validation:
- Regenerated the complete AMDGPU matcher twice; output was byte-identical.
- Reconstructed all 291,413 canonical ordered rows from the pooled representation with exact semantic equality.
- Added and passed a focused TableGen test for mnemonic-first and non-mnemonic-first matchers.
- Existing gfx11_asm_vopd.s positive modes and wave64 diagnostics passed FileCheck; baseline and patched output were byte-identical.
- Full multicall build: https://app.buildbuddy.io/invocation/e85ce1e4-bece-4fff-9963-56002dde63e5
LLVM has no dedicated assembly-parser benchmark target. Across 80 runs of the existing 15,036-line gfx11_asm_vopd.s corpus, baseline mean/median were 44.526/41.984 ms and pooled mean/median were 42.695/41.157 ms; variance exceeds the difference and shows no observable regression.
---
.../AsmMatcherCustomOperandInterning.td | 100 +++++++++
llvm/utils/TableGen/AsmMatcherEmitter.cpp | 208 ++++++++++++------
2 files changed, 239 insertions(+), 69 deletions(-)
create mode 100644 llvm/test/TableGen/AsmMatcherCustomOperandInterning.td
diff --git a/llvm/test/TableGen/AsmMatcherCustomOperandInterning.td b/llvm/test/TableGen/AsmMatcherCustomOperandInterning.td
new file mode 100644
index 0000000000000..94200c9ac3d26
--- /dev/null
+++ b/llvm/test/TableGen/AsmMatcherCustomOperandInterning.td
@@ -0,0 +1,100 @@
+// RUN: llvm-tblgen -gen-asm-matcher -I %p/../../include %s \
+// RUN: | FileCheck %s --check-prefixes=CHECK,MNEMONIC-FIRST
+// RUN: llvm-tblgen -gen-asm-matcher -DNO_MNEMONIC_FIRST \
+// RUN: -I %p/../../include %s \
+// RUN: | FileCheck %s --check-prefixes=CHECK,NO-MNEMONIC-FIRST
+
+include "llvm/Target/Target.td"
+
+def ArchInstrInfo : InstrInfo;
+
+#ifdef NO_MNEMONIC_FIRST
+def ArchAsmParser : AsmParser {
+ let HasMnemonicFirst = 0;
+}
+#else
+def ArchAsmParser : AsmParser;
+#endif
+
+def Arch : Target {
+ let InstructionSet = ArchInstrInfo;
+ let AssemblyParsers = [ArchAsmParser];
+}
+
+def Reg : Register<"reg">;
+def RegClass : RegisterClass<"Arch", [i32], 32, (add Reg)>;
+
+def FeatureA : SubtargetFeature<"feature-a", "FeatureA", "true",
+ "Enable feature A">;
+def HasFeatureA : Predicate<"Subtarget->hasFeatureA()">,
+ AssemblerPredicate<(all_of FeatureA)>;
+
+def AClass : AsmOperandClass {
+ let Name = "A";
+ let ParserMethod = "parseA";
+}
+
+def BClass : AsmOperandClass {
+ let Name = "B";
+ let ParserMethod = "parseB";
+}
+
+def AOperand : Operand<i32> {
+ let ParserMatchClass = AClass;
+}
+
+def BOperand : Operand<i32> {
+ let ParserMatchClass = BClass;
+}
+
+class PairInst<string Mnemonic, list<Predicate> RequiredPredicates>
+ : Instruction {
+ let Size = 4;
+ let OutOperandList = (outs);
+ let InOperandList = (ins AOperand:$a, BOperand:$b);
+ let AsmString = Mnemonic # " $a, $b";
+ let Predicates = RequiredPredicates;
+}
+
+class SingleInst<string Mnemonic> : Instruction {
+ let Size = 4;
+ let OutOperandList = (outs);
+ let InOperandList = (ins AOperand:$a);
+ let AsmString = Mnemonic # " $a";
+}
+
+def BarFeature : PairInst<"bar", [HasFeatureA]>;
+def BarBase : PairInst<"bar", []>;
+def Baz : SingleInst<"baz">;
+def FooFeature : PairInst<"foo", [HasFeatureA]>;
+def FooBase : PairInst<"foo", []>;
+
+// Bar and foo have identical ordered match sequences. Baz has a distinct
+// one-record sequence.
+//
+// CHECK: static const OperandMatchEntry OperandMatchSequenceTable[5] = {
+// CHECK-NEXT: /* Operand Mask, Operand Class, Features */
+// CHECK-NEXT: /* 0 */ { {{[0-9]+}} /* {{[0-9]+}} */, MCK_A, AMFBS_HasFeatureA },
+// CHECK-NEXT: /* 1 */ { {{[0-9]+}} /* {{[0-9]+}} */, MCK_B, AMFBS_HasFeatureA },
+// CHECK-NEXT: /* 2 */ { {{[0-9]+}} /* {{[0-9]+}} */, MCK_A, AMFBS_None },
+// CHECK-NEXT: /* 3 */ { {{[0-9]+}} /* {{[0-9]+}} */, MCK_B, AMFBS_None },
+// CHECK-NEXT: /* 4 */ { {{[0-9]+}} /* {{[0-9]+}} */, MCK_A, AMFBS_None },
+// CHECK: static const OperandMatchMnemonicEntry OperandMatchMnemonicTable[3] = {
+// CHECK-NEXT: /* Match Sequence Offset, Mnemonic, Match Sequence Length */
+// CHECK-NEXT: { 0, {{[0-9]+}} /* bar */, 4 },
+// CHECK-NEXT: { 4, {{[0-9]+}} /* baz */, 1 },
+// CHECK-NEXT: { 0, {{[0-9]+}} /* foo */, 4 },
+
+// The generated parser walks each mnemonic's interned sequence in order.
+// MNEMONIC-FIRST: auto MnemonicIt =
+// MNEMONIC-FIRST: for (unsigned I = MnemonicIt->MatchSequenceOffset;
+// MNEMONIC-FIRST-NEXT: I != MatchSequenceEnd; ++I) {
+// NO-MNEMONIC-FIRST: auto MnemonicBegin = std::begin(OperandMatchMnemonicTable);
+// NO-MNEMONIC-FIRST-NEXT: auto MnemonicEnd = std::end(OperandMatchMnemonicTable);
+// NO-MNEMONIC-FIRST-NEXT: if (!Mnemonic.empty()) {
+// NO-MNEMONIC-FIRST: for (const OperandMatchMnemonicEntry *MnemonicIt =
+// NO-MNEMONIC-FIRST-SAME: MnemonicBegin;
+// NO-MNEMONIC-FIRST-NEXT: MnemonicIt != MnemonicEnd; ++MnemonicIt) {
+// NO-MNEMONIC-FIRST: for (unsigned I = MnemonicIt->MatchSequenceOffset;
+// NO-MNEMONIC-FIRST-NEXT: I != MatchSequenceEnd; ++I) {
+// CHECK-NEXT: const OperandMatchEntry &Entry = OperandMatchSequenceTable[I];
diff --git a/llvm/utils/TableGen/AsmMatcherEmitter.cpp b/llvm/utils/TableGen/AsmMatcherEmitter.cpp
index 4c61d64ec215a..be31585ca59c1 100644
--- a/llvm/utils/TableGen/AsmMatcherEmitter.cpp
+++ b/llvm/utils/TableGen/AsmMatcherEmitter.cpp
@@ -80,9 +80,9 @@
// instructions. The target specific custom operand parsing works in the
// following way:
//
-// 1. A operand match table is built, each entry contains a mnemonic, an
-// operand class, a mask for all operand positions for that same
-// class/mnemonic and target features to be checked while trying to match.
+// 1. Each mnemonic references an ordered sequence of operand match records
+// containing an operand class, operand mask, and required target features.
+// Identical sequences are interned.
//
// 2. The operand matcher will try every possible entry with the same
// mnemonic and will check if the target feature for this mnemonic also
@@ -121,6 +121,8 @@
#include <forward_list>
#include <map>
#include <set>
+#include <tuple>
+#include <vector>
using namespace llvm;
@@ -2991,15 +2993,64 @@ emitCustomOperandParsing(raw_ostream &OS, CodeGenTarget &Target,
const StringToOffsetTable &StringTable,
unsigned MaxMnemonicIndex, unsigned MaxFeaturesIndex,
bool HasMnemonicFirst, const Record &AsmParser) {
+ using OperandMatchDataTy = std::tuple<unsigned, StringRef, std::string>;
+ using OperandMatchSequenceTy = std::vector<OperandMatchDataTy>;
+ struct OperandMatchMnemonicInfo {
+ unsigned MnemonicOffset;
+ std::string Mnemonic;
+ unsigned MatchSequenceOffset;
+ unsigned MatchSequenceLength;
+ };
+
unsigned MaxMask = 0;
for (const OperandMatchEntry &OMI : Info.OperandMatchInfo) {
MaxMask |= OMI.OperandMask;
}
+ auto getRequiredFeaturesName = [](const MatchableInfo &MI) {
+ std::string Name = "AMFBS";
+ if (MI.RequiredFeatures.empty())
+ return Name + "_None";
+ for (const auto &F : MI.RequiredFeatures)
+ Name += "_" + F->TheDef->getName().str();
+ return Name;
+ };
+
+ std::map<OperandMatchSequenceTy, unsigned> MatchSequenceOffsets;
+ OperandMatchSequenceTy MatchSequenceTable;
+ std::vector<OperandMatchMnemonicInfo> MnemonicTable;
+ unsigned MaxMatchSequenceLength = 0;
+
+ for (auto I = Info.OperandMatchInfo.begin(), E = Info.OperandMatchInfo.end();
+ I != E;) {
+ const MatchableInfo &FirstMI = *I->MI;
+ std::string Mnemonic = FirstMI.Mnemonic.lower();
+ std::string LenMnemonic = char(Mnemonic.size()) + Mnemonic;
+ unsigned MnemonicOffset = *StringTable.GetStringOffset(LenMnemonic);
+ OperandMatchSequenceTy MatchSequence;
+
+ do {
+ const MatchableInfo &MI = *I->MI;
+ MatchSequence.emplace_back(I->OperandMask, I->CI->Name,
+ getRequiredFeaturesName(MI));
+ ++I;
+ } while (I != E && I->MI->Mnemonic.equals_insensitive(Mnemonic));
+
+ auto [SequenceIt, Inserted] = MatchSequenceOffsets.try_emplace(
+ MatchSequence, MatchSequenceTable.size());
+ if (Inserted)
+ llvm::append_range(MatchSequenceTable, MatchSequence);
+
+ MaxMatchSequenceLength = std::max(
+ MaxMatchSequenceLength, static_cast<unsigned>(MatchSequence.size()));
+ MnemonicTable.push_back({MnemonicOffset, std::move(Mnemonic),
+ SequenceIt->second,
+ static_cast<unsigned>(MatchSequence.size())});
+ }
+
// Emit the static custom operand parsing table;
OS << "namespace {\n";
OS << " struct OperandMatchEntry {\n";
- OS << " " << getMinimalTypeForRange(MaxMnemonicIndex) << " Mnemonic;\n";
OS << " " << getMinimalTypeForRange(MaxMask) << " OperandMask;\n";
OS << " "
<< getMinimalTypeForRange(
@@ -3007,7 +3058,15 @@ emitCustomOperandParsing(raw_ostream &OS, CodeGenTarget &Target,
2 /* Include 'InvalidMatchClass' and 'OptionalMatchClass' */)
<< " Class;\n";
OS << " " << getMinimalTypeForRange(MaxFeaturesIndex)
- << " RequiredFeaturesIdx;\n\n";
+ << " RequiredFeaturesIdx;\n";
+ OS << " };\n\n";
+
+ OS << " struct OperandMatchMnemonicEntry {\n";
+ OS << " " << getMinimalTypeForRange(MatchSequenceTable.size() - 1)
+ << " MatchSequenceOffset;\n";
+ OS << " " << getMinimalTypeForRange(MaxMnemonicIndex) << " Mnemonic;\n";
+ OS << " " << getMinimalTypeForRange(MaxMatchSequenceLength)
+ << " MatchSequenceLength;\n\n";
OS << " StringRef getMnemonic() const {\n";
OS << " return StringRef(MnemonicTable + Mnemonic + 1,\n";
OS << " MnemonicTable[Mnemonic]);\n";
@@ -3016,56 +3075,45 @@ emitCustomOperandParsing(raw_ostream &OS, CodeGenTarget &Target,
OS << " // Predicate for searching for an opcode.\n";
OS << " struct LessOpcodeOperand {\n";
- OS << " bool operator()(const OperandMatchEntry &LHS, StringRef RHS) {\n";
+ OS << " bool operator()(const OperandMatchMnemonicEntry &LHS, "
+ "StringRef RHS) {\n";
OS << " return LHS.getMnemonic() < RHS;\n";
OS << " }\n";
- OS << " bool operator()(StringRef LHS, const OperandMatchEntry &RHS) {\n";
+ OS << " bool operator()(StringRef LHS, const "
+ "OperandMatchMnemonicEntry &RHS) {\n";
OS << " return LHS < RHS.getMnemonic();\n";
OS << " }\n";
- OS << " bool operator()(const OperandMatchEntry &LHS,";
- OS << " const OperandMatchEntry &RHS) {\n";
- OS << " return LHS.getMnemonic() < RHS.getMnemonic();\n";
- OS << " }\n";
OS << " };\n";
OS << "} // end anonymous namespace\n\n";
- OS << "static const OperandMatchEntry OperandMatchTable["
- << Info.OperandMatchInfo.size() << "] = {\n";
-
- OS << " /* Operand List Mnemonic, Mask, Operand Class, Features */\n";
- for (const OperandMatchEntry &OMI : Info.OperandMatchInfo) {
- const MatchableInfo &II = *OMI.MI;
-
- OS << " { ";
+ OS << "static const OperandMatchEntry OperandMatchSequenceTable["
+ << MatchSequenceTable.size() << "] = {\n";
- // Store a pascal-style length byte in the mnemonic.
- std::string LenMnemonic = char(II.Mnemonic.size()) + II.Mnemonic.lower();
- OS << *StringTable.GetStringOffset(LenMnemonic) << " /* " << II.Mnemonic
- << " */, ";
-
- OS << OMI.OperandMask;
+ OS << " /* Operand Mask, Operand Class, Features */\n";
+ for (const auto &[Index, Data] : enumerate(MatchSequenceTable)) {
+ const auto &[OperandMask, Class, RequiredFeatures] = Data;
+ OS << " /* " << Index << " */ { ";
+ OS << OperandMask;
OS << " /* ";
ListSeparator LS;
for (int i = 0, e = 31; i != e; ++i)
- if (OMI.OperandMask & (1 << i))
+ if (OperandMask & (1 << i))
OS << LS << i;
OS << " */, ";
-
- OS << OMI.CI->Name;
-
- // Write the required features mask.
- OS << ", AMFBS";
- if (II.RequiredFeatures.empty())
- OS << "_None";
- else
- for (const auto &F : II.RequiredFeatures)
- OS << '_' << F->TheDef->getName();
-
- OS << " },\n";
+ OS << Class << ", " << RequiredFeatures << " },\n";
}
OS << "};\n\n";
+ OS << "static const OperandMatchMnemonicEntry OperandMatchMnemonicTable["
+ << MnemonicTable.size() << "] = {\n";
+ OS << " /* Match Sequence Offset, Mnemonic, Match Sequence Length */\n";
+ for (const OperandMatchMnemonicInfo &Entry : MnemonicTable)
+ OS << " { " << Entry.MatchSequenceOffset << ", " << Entry.MnemonicOffset
+ << " /* " << Entry.Mnemonic << " */, " << Entry.MatchSequenceLength
+ << " },\n";
+ OS << "};\n\n";
+
// Emit the operand class switch to call the correct custom parser for
// the found operand class.
OS << "ParseStatus " << Target.getName() << ClassName << "::\n"
@@ -3105,51 +3153,73 @@ emitCustomOperandParsing(raw_ostream &OS, CodeGenTarget &Target,
// Emit code to search the table.
OS << " // Search the table.\n";
+ std::string Indent;
if (HasMnemonicFirst) {
- OS << " auto MnemonicRange =\n";
- OS << " std::equal_range(std::begin(OperandMatchTable), "
- "std::end(OperandMatchTable),\n";
+ OS << " auto MnemonicIt =\n";
+ OS << " std::lower_bound(std::begin(OperandMatchMnemonicTable), "
+ "std::end(OperandMatchMnemonicTable),\n";
OS << " Mnemonic, LessOpcodeOperand());\n\n";
+ OS << " if (MnemonicIt == std::end(OperandMatchMnemonicTable) ||\n";
+ OS << " MnemonicIt->getMnemonic() != Mnemonic)\n";
+ OS << " return ParseStatus::NoMatch;\n\n";
+ OS << " const unsigned MatchSequenceEnd =\n";
+ OS << " MnemonicIt->MatchSequenceOffset + "
+ "MnemonicIt->MatchSequenceLength;\n";
+ OS << " for (unsigned I = MnemonicIt->MatchSequenceOffset;\n";
+ OS << " I != MatchSequenceEnd; ++I) {\n";
+ Indent = " ";
} else {
- OS << " auto MnemonicRange = std::pair(std::begin(OperandMatchTable),"
- " std::end(OperandMatchTable));\n";
- OS << " if (!Mnemonic.empty())\n";
- OS << " MnemonicRange =\n";
- OS << " std::equal_range(std::begin(OperandMatchTable), "
- "std::end(OperandMatchTable),\n";
- OS << " Mnemonic, LessOpcodeOperand());\n\n";
+ OS << " auto MnemonicBegin = std::begin(OperandMatchMnemonicTable);\n";
+ OS << " auto MnemonicEnd = std::end(OperandMatchMnemonicTable);\n";
+ OS << " if (!Mnemonic.empty()) {\n";
+ OS << " MnemonicBegin =\n";
+ OS << " std::lower_bound(MnemonicBegin, MnemonicEnd,\n";
+ OS << " Mnemonic, LessOpcodeOperand());\n";
+ OS << " if (MnemonicBegin == MnemonicEnd ||\n";
+ OS << " MnemonicBegin->getMnemonic() != Mnemonic)\n";
+ OS << " return ParseStatus::NoMatch;\n";
+ OS << " MnemonicEnd = MnemonicBegin + 1;\n";
+ OS << " }\n";
+ OS << '\n';
+ OS << " for (const OperandMatchMnemonicEntry *MnemonicIt = "
+ "MnemonicBegin;\n";
+ OS << " MnemonicIt != MnemonicEnd; ++MnemonicIt) {\n";
+ OS << " const unsigned MatchSequenceEnd =\n";
+ OS << " MnemonicIt->MatchSequenceOffset + "
+ "MnemonicIt->MatchSequenceLength;\n";
+ OS << " for (unsigned I = MnemonicIt->MatchSequenceOffset;\n";
+ OS << " I != MatchSequenceEnd; ++I) {\n";
+ Indent = " ";
}
-
- OS << " if (MnemonicRange.first == MnemonicRange.second)\n";
- OS << " return ParseStatus::NoMatch;\n\n";
-
- OS << " for (const OperandMatchEntry *it = MnemonicRange.first,\n"
- << " *ie = MnemonicRange.second; it != ie; ++it) {\n";
-
- OS << " // equal_range guarantees that instruction mnemonic matches.\n";
- OS << " assert(Mnemonic == it->getMnemonic());\n\n";
+ OS << Indent
+ << "const OperandMatchEntry &Entry = OperandMatchSequenceTable[I];\n";
// Emit check that the required features are available.
- OS << " // check if the available features match\n";
- OS << " const FeatureBitset &RequiredFeatures = "
- "FeatureBitsets[it->RequiredFeaturesIdx];\n";
- OS << " if (!ParseForAllFeatures && (AvailableFeatures & "
+ OS << Indent << "// check if the available features match\n";
+ OS << Indent
+ << "const FeatureBitset &RequiredFeatures = "
+ "FeatureBitsets[Entry.RequiredFeaturesIdx];\n";
+ OS << Indent
+ << "if (!ParseForAllFeatures && (AvailableFeatures & "
"RequiredFeatures) != RequiredFeatures)\n";
- OS << " continue;\n\n";
+ OS << Indent << " continue;\n\n";
// Emit check to ensure the operand number matches.
- OS << " // check if the operand in question has a custom parser.\n";
- OS << " if (!(it->OperandMask & (1 << NextOpNum)))\n";
- OS << " continue;\n\n";
+ OS << Indent << "// check if the operand in question has a custom parser.\n";
+ OS << Indent << "if (!(Entry.OperandMask & (1 << NextOpNum)))\n";
+ OS << Indent << " continue;\n\n";
// Emit call to the custom parser method
StringRef ParserName = AsmParser.getValueAsString("OperandParserMethod");
if (ParserName.empty())
ParserName = "tryCustomParseOperand";
- OS << " // call custom parse method to handle the operand\n";
- OS << " ParseStatus Result = " << ParserName << "(Operands, it->Class);\n";
- OS << " if (!Result.isNoMatch())\n";
- OS << " return Result;\n";
+ OS << Indent << "// call custom parse method to handle the operand\n";
+ OS << Indent << "ParseStatus Result = " << ParserName
+ << "(Operands, Entry.Class);\n";
+ OS << Indent << "if (!Result.isNoMatch())\n";
+ OS << Indent << " return Result;\n";
+ if (!HasMnemonicFirst)
+ OS << " }\n";
OS << " }\n\n";
OS << " // Okay, we had no match.\n";
More information about the llvm-commits
mailing list