[llvm] [TableGen][IR] Store intrinsic name lengths in offset entries (PR #202645)
David Zbarsky via llvm-commits
llvm-commits at lists.llvm.org
Tue Jun 9 07:01:03 PDT 2026
https://github.com/dzbarsky created https://github.com/llvm/llvm-project/pull/202645
IntrinsicImpl.inc currently terminates every intrinsic name with NUL even
though every lookup already goes through the parallel 32-bit offset table.
That stores 16,379 terminators and requires lookup to recover lengths by
scanning for NUL.
Concatenate the names without per-entry terminators and use the high eight bits
of each offset entry for the name length, leaving 24 bits for the byte offset.
Construct StringRefs directly from the packed offset and length. Generated
static assertions verify the offset width, maximum name length, storage size,
and offset-table count. Add a round-trip test for every intrinsic name.
In an arm64 Release build:
* Intrinsic name storage decreases from 476,658 to 460,278 bytes (-16,380).
* Intrinsics.cpp.o decreases by 16,256 bytes.
* opt decreases from 56,520,560 to 56,504,048 bytes (-16,512).
* stripped opt also decreases by 16,512 bytes.
* Text increases by 172 bytes.
A direct lookup benchmark measured 29.47% faster lookupIntrinsicID calls
(95% bootstrap CI 20.09% to 36.86%) and 91.29% faster getBaseName calls
(CI 89.86% to 92.47%). A process-level opt startup benchmark measured +0.81%
(CI -2.43% to +4.15%); its control measured +0.19% (CI -3.04% to +3.54%).
All 16,379 generated names round-tripped with identical checksums. Validation
also passed 871 IR unit tests, five focused intrinsic tests, seven lit tests,
and a compile of the MSVC-compatible character-array emission path.
Work towards #202616
>From 65747bbfceed3233737c6f27de7fcd8c02b6c508 Mon Sep 17 00:00:00 2001
From: David Zbarsky <dzbarsky at gmail.com>
Date: Tue, 9 Jun 2026 06:59:28 -0400
Subject: [PATCH] [TableGen][IR] Store intrinsic name lengths in offset entries
IntrinsicImpl.inc currently terminates every intrinsic name with NUL even
though every lookup already goes through the parallel 32-bit offset table.
That stores 16,379 terminators and requires lookup to recover lengths by
scanning for NUL.
Concatenate the names without per-entry terminators and use the high eight bits
of each offset entry for the name length, leaving 24 bits for the byte offset.
Construct StringRefs directly from the packed offset and length. Generated
static assertions verify the offset width, maximum name length, storage size,
and offset-table count. Add a round-trip test for every intrinsic name.
In an arm64 Release build:
* Intrinsic name storage decreases from 476,658 to 460,278 bytes (-16,380).
* Intrinsics.cpp.o decreases by 16,256 bytes.
* opt decreases from 56,520,560 to 56,504,048 bytes (-16,512).
* stripped opt also decreases by 16,512 bytes.
* Text increases by 172 bytes.
A direct lookup benchmark measured 29.47% faster lookupIntrinsicID calls
(95% bootstrap CI 20.09% to 36.86%) and 91.29% faster getBaseName calls
(CI 89.86% to 92.47%). A process-level opt startup benchmark measured +0.81%
(CI -2.43% to +4.15%); its control measured +0.19% (CI -3.04% to +3.54%).
All 16,379 generated names round-tripped with identical checksums. Validation
also passed 871 IR unit tests, five focused intrinsic tests, seven lit tests,
and a compile of the MSVC-compatible character-array emission path.
---
llvm/lib/IR/Intrinsics.cpp | 34 ++++++---
llvm/unittests/IR/IntrinsicsTest.cpp | 7 ++
.../utils/TableGen/Basic/IntrinsicEmitter.cpp | 76 ++++++++++++++++---
3 files changed, 95 insertions(+), 22 deletions(-)
diff --git a/llvm/lib/IR/Intrinsics.cpp b/llvm/lib/IR/Intrinsics.cpp
index b0bad878f0c6f..0d880b8e03e19 100644
--- a/llvm/lib/IR/Intrinsics.cpp
+++ b/llvm/lib/IR/Intrinsics.cpp
@@ -36,6 +36,7 @@
#include "llvm/IR/Type.h"
#include "llvm/Support/FormatVariadic.h"
#include "llvm/Support/MathExtras.h"
+#include <iterator>
using namespace llvm;
@@ -50,9 +51,17 @@ static bool isSignatureValid(FunctionType *FTy,
#define GET_INTRINSIC_NAME_TABLE
#include "llvm/IR/IntrinsicImpl.inc"
+static_assert(std::size(IntrinsicNameOffsetTable) == Intrinsic::num_intrinsics);
+
+static StringRef getIntrinsicName(unsigned TableEntry) {
+ unsigned Offset = TableEntry & IntrinsicNameOffsetMask;
+ unsigned Length = TableEntry >> IntrinsicNameLengthShift;
+ return StringRef(IntrinsicNameTableStorage + Offset, Length);
+}
+
StringRef Intrinsic::getBaseName(ID id) {
assert(id < num_intrinsics && "Invalid intrinsic ID!");
- return IntrinsicNameTable[IntrinsicNameOffsetTable[id]];
+ return getIntrinsicName(IntrinsicNameOffsetTable[id]);
}
StringRef Intrinsic::getName(ID id) {
@@ -684,33 +693,34 @@ static int lookupLLVMIntrinsicByName(ArrayRef<unsigned> NameOffsetTable,
CmpEnd = Name.find('.', CmpStart + 1);
CmpEnd = CmpEnd == StringRef::npos ? Name.size() : CmpEnd;
auto Cmp = [CmpStart, CmpEnd](auto LHS, auto RHS) {
- // `equal_range` requires the comparison to work with either side being an
- // offset or the value. Detect which kind each side is to set up the
+ // `equal_range` requires the comparison to work with either side being a
+ // table entry or the value. Detect which kind each side is to set up the
// compared strings.
- const char *LHSStr;
+ StringRef LHSStr;
if constexpr (std::is_integral_v<decltype(LHS)>)
- LHSStr = IntrinsicNameTable.getCString(LHS);
+ LHSStr = getIntrinsicName(LHS);
else
LHSStr = LHS;
- const char *RHSStr;
+ StringRef RHSStr;
if constexpr (std::is_integral_v<decltype(RHS)>)
- RHSStr = IntrinsicNameTable.getCString(RHS);
+ RHSStr = getIntrinsicName(RHS);
else
RHSStr = RHS;
- return strncmp(LHSStr + CmpStart, RHSStr + CmpStart, CmpEnd - CmpStart) <
- 0;
+ size_t CmpLength = CmpEnd - CmpStart;
+ return LHSStr.substr(CmpStart, CmpLength) <
+ RHSStr.substr(CmpStart, CmpLength);
};
LastLow = Low;
- std::tie(Low, High) = std::equal_range(Low, High, Name.data(), Cmp);
+ std::tie(Low, High) = std::equal_range(Low, High, Name, Cmp);
}
if (High - Low > 0)
LastLow = Low;
if (LastLow == NameOffsetTable.end())
return -1;
- StringRef NameFound = IntrinsicNameTable[*LastLow];
+ StringRef NameFound = getIntrinsicName(*LastLow);
if (Name == NameFound ||
(Name.starts_with(NameFound) && Name[NameFound.size()] == '.'))
return LastLow - NameOffsetTable.begin();
@@ -754,7 +764,7 @@ Intrinsic::ID Intrinsic::lookupIntrinsicID(StringRef Name) {
// If the intrinsic is not overloaded, require an exact match. If it is
// overloaded, require either exact or prefix match.
- const auto MatchSize = IntrinsicNameTable[NameOffsetTable[Idx]].size();
+ const auto MatchSize = getIntrinsicName(NameOffsetTable[Idx]).size();
assert(Name.size() >= MatchSize && "Expected either exact or prefix match");
bool IsExactMatch = Name.size() == MatchSize;
return IsExactMatch || Intrinsic::isOverloaded(ID) ? ID
diff --git a/llvm/unittests/IR/IntrinsicsTest.cpp b/llvm/unittests/IR/IntrinsicsTest.cpp
index 87d922d22eaac..cd0ce7ed26fed 100644
--- a/llvm/unittests/IR/IntrinsicsTest.cpp
+++ b/llvm/unittests/IR/IntrinsicsTest.cpp
@@ -80,6 +80,13 @@ TEST(IntrinsicNameLookup, Basic) {
EXPECT_EQ(memcpy_inline, lookupIntrinsicID("llvm.memcpy.inline.p0.p0.i1024"));
}
+TEST(IntrinsicNameLookup, AllNamesRoundTrip) {
+ for (unsigned ID = 1; ID != Intrinsic::num_intrinsics; ++ID) {
+ StringRef Name = Intrinsic::getBaseName(ID);
+ EXPECT_EQ(ID, Intrinsic::lookupIntrinsicID(Name)) << Name;
+ }
+}
+
TEST(IntrinsicNameLookup, NonNullterminatedStringRef) {
using namespace Intrinsic;
// This reproduces an issue where lookupIntrinsicID() can access memory beyond
diff --git a/llvm/utils/TableGen/Basic/IntrinsicEmitter.cpp b/llvm/utils/TableGen/Basic/IntrinsicEmitter.cpp
index 02b2cd9850997..295428409bbd3 100644
--- a/llvm/utils/TableGen/Basic/IntrinsicEmitter.cpp
+++ b/llvm/utils/TableGen/Basic/IntrinsicEmitter.cpp
@@ -24,6 +24,7 @@
#include "llvm/Support/raw_ostream.h"
#include "llvm/TableGen/CodeGenHelpers.h"
#include "llvm/TableGen/Error.h"
+#include "llvm/TableGen/Main.h"
#include "llvm/TableGen/Record.h"
#include "llvm/TableGen/StringToOffsetTable.h"
#include "llvm/TableGen/TableGenBackend.h"
@@ -279,31 +280,86 @@ void IntrinsicEmitter::EmitIntrinsicBitTable(
void IntrinsicEmitter::EmitIntrinsicToNameTable(
const CodeGenIntrinsicTable &Ints, raw_ostream &OS) {
- // Built up a table of the intrinsic names.
constexpr StringLiteral NotIntrinsic = "not_intrinsic";
- StringToOffsetTable Table;
- Table.GetOrAddStringOffset(NotIntrinsic);
+ SmallVector<StringRef> Names;
+ Names.push_back(NotIntrinsic);
for (const auto &Int : Ints)
- Table.GetOrAddStringOffset(Int.Name);
+ Names.push_back(Int.Name);
IfDefEmitter IfDef(OS, "GET_INTRINSIC_NAME_TABLE");
OS << R"(// Intrinsic ID to name table.
// Note that entry #0 is the invalid intrinsic!
+#ifdef __GNUC__
+#pragma GCC diagnostic push
+#pragma GCC diagnostic ignored "-Woverlength-strings"
+#endif
)";
- Table.EmitStringTableDef(OS, "IntrinsicNameTable");
+ unsigned TableSize = 0;
+ unsigned MaxNameLength = 0;
+ for (StringRef Name : Names) {
+ TableSize += Name.size();
+ MaxNameLength = std::max<unsigned>(MaxNameLength, Name.size());
+ }
+ bool UseChars = !EmitLongStrLiterals && TableSize > 64 * 1024;
+ OS << "static constexpr char IntrinsicNameTableStorage[] ="
+ << (UseChars ? "{\n" : "\n");
+ ListSeparator LineSep(UseChars ? ",\n" : "\n");
+ for (StringRef Name : Names) {
+ OS << LineSep << " ";
+ if (!UseChars) {
+ OS << "\"";
+ OS.write_escaped(Name);
+ OS << "\"";
+ continue;
+ }
+ ListSeparator CharSep(", ");
+ for (char C : Name) {
+ OS << CharSep << "'";
+ OS.write_escaped(StringRef(&C, 1));
+ OS << "'";
+ }
+ }
+ OS << LineSep << (UseChars ? "};" : ";");
+ OS << R"(
+#ifdef __GNUC__
+#pragma GCC diagnostic pop
+#endif
+)";
OS << R"(
static constexpr unsigned IntrinsicNameOffsetTable[] = {
)";
- OS << formatv(" {}, // {}\n", Table.GetStringOffset(NotIntrinsic),
- NotIntrinsic);
- for (const auto &Int : Ints)
- OS << formatv(" {}, // {}\n", Table.GetStringOffset(Int.Name), Int.Name);
+ constexpr unsigned OffsetBits = 24;
+ constexpr unsigned OffsetMask = (1u << OffsetBits) - 1;
+ constexpr unsigned LengthShift = OffsetBits;
+ unsigned Offset = 0;
+ for (StringRef Name : Names) {
+ if (Offset > OffsetMask)
+ PrintFatalError("intrinsic name table is too large to encode");
+ if (Name.size() > 0xffu)
+ PrintFatalError("intrinsic name is too long to encode");
+ unsigned Entry = Offset | (Name.size() << LengthShift);
+ OS << formatv(" 0x{0:X-8}, // {1}\n", Entry, Name);
+ Offset += Name.size();
+ }
- OS << "\n}; // IntrinsicNameOffsetTable\n";
+ OS << "\n}; // IntrinsicNameOffsetTable\n\n";
+ OS << formatv(
+ "static constexpr unsigned IntrinsicNameOffsetMask = 0x{0:X-8};\n"
+ "static constexpr unsigned IntrinsicNameLengthShift = {1};\n"
+ "static constexpr unsigned IntrinsicNameTableSize = {2};\n"
+ "static constexpr unsigned IntrinsicNameMaxLength = {3};\n"
+ "static_assert(IntrinsicNameTableSize <= "
+ "IntrinsicNameOffsetMask + 1);\n"
+ "static_assert(IntrinsicNameMaxLength <= 0xffu);\n"
+ "static_assert(sizeof(IntrinsicNameTableStorage) == "
+ "IntrinsicNameTableSize ||\n"
+ " sizeof(IntrinsicNameTableStorage) == "
+ "IntrinsicNameTableSize + 1);\n",
+ OffsetMask, LengthShift, TableSize, MaxNameLength);
}
void IntrinsicEmitter::EmitIntrinsicToOverloadTable(
More information about the llvm-commits
mailing list