[llvm] [llvm-exegesis] Skip benchmark entries with unknown opcodes in analysis mode (PR #201162)
Lawson Darrow via llvm-commits
llvm-commits at lists.llvm.org
Mon Jun 8 07:02:08 PDT 2026
https://github.com/Lawson-Darrow updated https://github.com/llvm/llvm-project/pull/201162
>From c30a96c4561e9e8216a9221e33eb8f43726b663c Mon Sep 17 00:00:00 2001
From: Lawson Darrow <lawson.darrow at gmail.com>
Date: Tue, 2 Jun 2026 13:22:49 -0400
Subject: [PATCH 1/4] [llvm-exegesis] Skip benchmark entries with unknown
opcodes in analysis mode
llvm-exegesis -mode=analysis currently aborts the entire run if any benchmark
entry references an unknown opcode (e.g. an old sample whose instruction has
since been renamed), discarding the results for every valid entry.
This adds an opt-in SkipInvalidEntries parameter to Benchmark::readYamls. When
set, an entry that fails to deserialize is reported via a warning, dropped, and
parsing continues with the remaining valid entries. The default remains false,
preserving the existing strict behavior for all other callers; analysis mode
passes true.
Fixes #144403
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply at anthropic.com>
---
.../X86/analysis-skip-unknown-opcode.test | 74 +++++++++++++++++++
.../llvm-exegesis/lib/BenchmarkResult.cpp | 44 +++++++++--
.../tools/llvm-exegesis/lib/BenchmarkResult.h | 6 +-
llvm/tools/llvm-exegesis/llvm-exegesis.cpp | 5 +-
.../llvm-exegesis/X86/BenchmarkResultTest.cpp | 55 ++++++++++++++
5 files changed, 177 insertions(+), 7 deletions(-)
create mode 100644 llvm/test/tools/llvm-exegesis/X86/analysis-skip-unknown-opcode.test
diff --git a/llvm/test/tools/llvm-exegesis/X86/analysis-skip-unknown-opcode.test b/llvm/test/tools/llvm-exegesis/X86/analysis-skip-unknown-opcode.test
new file mode 100644
index 0000000000000..3373cd01843e6
--- /dev/null
+++ b/llvm/test/tools/llvm-exegesis/X86/analysis-skip-unknown-opcode.test
@@ -0,0 +1,74 @@
+# Verify that in analysis mode an entry referencing an unknown opcode (e.g. a
+# sample that has bitrotted due to instruction renaming) is skipped with a
+# warning instead of aborting, and that the remaining valid entries are still
+# analyzed.
+
+# RUN: llvm-exegesis -mode=analysis -benchmarks-file=%s -analysis-clusters-output-file=- -analysis-clustering-epsilon=0.1 -analysis-inconsistency-epsilon=0.1 -analysis-numpoints=1 -analysis-clustering=naive 2> %t.err | FileCheck -check-prefix=CHECK-CLUSTERS %s
+# RUN: FileCheck -check-prefix=CHECK-WARN %s < %t.err
+
+# CHECK-WARN: warning: skipping benchmark entry: No opcode with name 'NOT_A_REAL_OPCODE'
+
+# CHECK-CLUSTERS: {{^}}cluster_id,opcode_name,config,sched_class,
+# CHECK-CLUSTERS-NEXT: {{^}}0,
+
+---
+mode: uops
+key:
+ instructions:
+ - 'NOT_A_REAL_OPCODE RAX'
+ config: ''
+ register_initial_values: []
+cpu_name: znver3
+llvm_triple: x86_64-pc-linux-gnu
+num_repetitions: 10000
+measurements: []
+error: ''
+...
+---
+mode: uops
+key:
+ instructions:
+ - 'CMOV16rm AX AX RDI i_0x1 %noreg i_0x0 %noreg i_0x0'
+ - 'CMOV16rm BP BP RDI i_0x1 %noreg i_0x40 %noreg i_0x0'
+ - 'CMOV16rm BX BX RDI i_0x1 %noreg i_0x80 %noreg i_0x0'
+ - 'CMOV16rm CX CX RDI i_0x1 %noreg i_0xc0 %noreg i_0x0'
+ - 'CMOV16rm DX DX RDI i_0x1 %noreg i_0x100 %noreg i_0x0'
+ - 'CMOV16rm SI SI RDI i_0x1 %noreg i_0x140 %noreg i_0x0'
+ - 'CMOV16rm R8W R8W RDI i_0x1 %noreg i_0x180 %noreg i_0x0'
+ - 'CMOV16rm R9W R9W RDI i_0x1 %noreg i_0x1c0 %noreg i_0x0'
+ - 'CMOV16rm R10W R10W RDI i_0x1 %noreg i_0x200 %noreg i_0x0'
+ - 'CMOV16rm R11W R11W RDI i_0x1 %noreg i_0x240 %noreg i_0x0'
+ - 'CMOV16rm R12W R12W RDI i_0x1 %noreg i_0x280 %noreg i_0x0'
+ - 'CMOV16rm R13W R13W RDI i_0x1 %noreg i_0x2c0 %noreg i_0x0'
+ - 'CMOV16rm R14W R14W RDI i_0x1 %noreg i_0x300 %noreg i_0x0'
+ - 'CMOV16rm R15W R15W RDI i_0x1 %noreg i_0x340 %noreg i_0x0'
+ config: ''
+ register_initial_values:
+ - 'AX=0x0'
+ - 'EFLAGS=0x0'
+ - 'BP=0x0'
+ - 'BX=0x0'
+ - 'CX=0x0'
+ - 'DX=0x0'
+ - 'SI=0x0'
+ - 'R8W=0x0'
+ - 'R9W=0x0'
+ - 'R10W=0x0'
+ - 'R11W=0x0'
+ - 'R12W=0x0'
+ - 'R13W=0x0'
+ - 'R14W=0x0'
+ - 'R15W=0x0'
+cpu_name: znver3
+llvm_triple: x86_64-pc-linux-gnu
+num_repetitions: 10000
+measurements:
+ - { key: Zn3Int, value: 1.0161, per_snippet_value: 14.2254 }
+ - { key: Zn3FPU, value: 0, per_snippet_value: 0 }
+ - { key: Zn3Load, value: 1.003, per_snippet_value: 14.042 }
+ - { key: Zn3Store, value: 0.0023, per_snippet_value: 0.0322 }
+ - { key: Zn3Divider, value: 0, per_snippet_value: 0 }
+ - { key: NumMicroOps, value: 1.0146, per_snippet_value: 14.2044 }
+error: ''
+info: instruction has tied variables, using static renaming.
+...
diff --git a/llvm/tools/llvm-exegesis/lib/BenchmarkResult.cpp b/llvm/tools/llvm-exegesis/lib/BenchmarkResult.cpp
index a597f413ee699..2df13ff7d7537 100644
--- a/llvm/tools/llvm-exegesis/lib/BenchmarkResult.cpp
+++ b/llvm/tools/llvm-exegesis/lib/BenchmarkResult.cpp
@@ -19,6 +19,7 @@
#include "llvm/Support/FileOutputBuffer.h"
#include "llvm/Support/FileSystem.h"
#include "llvm/Support/Format.h"
+#include "llvm/Support/WithColor.h"
#include "llvm/Support/raw_ostream.h"
static constexpr char kIntegerPrefix[] = "i_0x";
@@ -63,8 +64,22 @@ struct YamlContext {
std::string &getLastError() { return ErrorStream.str(); }
+ // Returns the recorded error to report to the YAML parser from a
+ // ScalarTraits::input. When ContinueOnError is set, recoverable per-entry
+ // errors (e.g. an unknown opcode in a bitrotted sample) are left recorded for
+ // the caller to inspect but are not reported to the parser, which would
+ // otherwise abort parsing of every remaining document.
+ StringRef getInputError() {
+ return ContinueOnError ? StringRef() : StringRef(getLastError());
+ }
+
raw_string_ostream &getErrorStream() { return ErrorStream; }
+ // When set, ScalarTraits::input swallows recoverable deserialization errors
+ // (see getInputError) so that readYamls can skip the offending entry instead
+ // of aborting the whole file.
+ bool ContinueOnError = false;
+
StringRef getRegName(MCRegister Reg) {
// Special case: Reg may be invalid. We have to deal with it explicitly.
if (!Reg.isValid())
@@ -177,7 +192,7 @@ template <> struct ScalarTraits<MCInst> {
static StringRef input(StringRef Scalar, void *Ctx, MCInst &Value) {
YamlContext &Context = getTypedContext(Ctx);
Context.deserializeMCInst(Scalar, Value);
- return Context.getLastError();
+ return Context.getInputError();
}
// By default strings are quoted only when necessary.
@@ -270,7 +285,7 @@ template <> struct ScalarTraits<exegesis::RegisterValue> {
Context.getErrorStream()
<< "Unknown initial register value: '" << String << "'";
}
- return Context.getLastError();
+ return Context.getInputError();
}
static QuotingType mustQuote(StringRef) { return QuotingType::Single; }
@@ -376,19 +391,38 @@ Expected<Benchmark> Benchmark::readYaml(const LLVMState &State,
}
Expected<std::vector<Benchmark>> Benchmark::readYamls(const LLVMState &State,
- MemoryBufferRef Buffer) {
+ MemoryBufferRef Buffer,
+ bool SkipInvalidEntries) {
yaml::Input Yin(Buffer);
YamlContext Context(State);
+ // In skip mode, recoverable per-entry errors (e.g. unknown opcodes) are
+ // recorded rather than reported to the parser, so a single bad entry can be
+ // dropped without aborting the read of the whole file.
+ Context.ContinueOnError = SkipInvalidEntries;
std::vector<Benchmark> Benchmarks;
+ unsigned NumSkippedEntries = 0;
while (Yin.setCurrentDocument()) {
Benchmarks.emplace_back();
yamlize(Yin, Benchmarks.back(), /*unused*/ true, Context);
if (Yin.error())
return errorCodeToError(Yin.error());
- if (!Context.getLastError().empty())
- return make_error<Failure>(Context.getLastError());
+ if (!Context.getLastError().empty()) {
+ if (!SkipInvalidEntries)
+ return make_error<Failure>(Context.getLastError());
+ // In skip mode, warn about the unparsable entry (e.g. an unknown opcode
+ // from a bitrotted sample), discard it, and continue so that a single bad
+ // entry doesn't abort the read of the whole file.
+ WithColor::warning() << "skipping benchmark entry: "
+ << Context.getLastError();
+ Context.getLastError().clear();
+ Benchmarks.pop_back();
+ ++NumSkippedEntries;
+ }
Yin.nextDocument();
}
+ if (NumSkippedEntries)
+ WithColor::warning() << "skipped " << NumSkippedEntries
+ << " benchmark entries that could not be parsed\n";
return std::move(Benchmarks);
}
diff --git a/llvm/tools/llvm-exegesis/lib/BenchmarkResult.h b/llvm/tools/llvm-exegesis/lib/BenchmarkResult.h
index 7984c8805cadc..c751a601fb765 100644
--- a/llvm/tools/llvm-exegesis/lib/BenchmarkResult.h
+++ b/llvm/tools/llvm-exegesis/lib/BenchmarkResult.h
@@ -136,8 +136,12 @@ struct Benchmark {
static Expected<Benchmark> readYaml(const LLVMState &State,
MemoryBufferRef Buffer);
+ // Deserializes benchmarks from the given Buffer. If SkipInvalidEntries is
+ // true, entries that fail to parse (e.g. referencing an unknown opcode) are
+ // warned about and skipped instead of aborting the read of the whole file.
static Expected<std::vector<Benchmark>>
- readYamls(const LLVMState &State, MemoryBufferRef Buffer);
+ readYamls(const LLVMState &State, MemoryBufferRef Buffer,
+ bool SkipInvalidEntries = false);
// Given a set of serialized instruction benchmarks, returns the set of
// triples and CPUs that appear in the list of benchmarks.
diff --git a/llvm/tools/llvm-exegesis/llvm-exegesis.cpp b/llvm/tools/llvm-exegesis/llvm-exegesis.cpp
index b5820f8669f1d..88e5b081eca11 100644
--- a/llvm/tools/llvm-exegesis/llvm-exegesis.cpp
+++ b/llvm/tools/llvm-exegesis/llvm-exegesis.cpp
@@ -668,8 +668,11 @@ static void analysisMain() {
// Read benchmarks.
const LLVMState State = ExitOnErr(
LLVMState::Create(TripleAndCpu.LLVMTriple, TripleAndCpu.CpuName));
+ // Skip entries that fail to parse (e.g. unknown opcodes from bitrotted
+ // samples) with a warning rather than aborting the whole analysis.
std::vector<Benchmark> Points = ExitOnFileError(
- BenchmarkFile, Benchmark::readYamls(State, *MemoryBuffer));
+ BenchmarkFile, Benchmark::readYamls(State, *MemoryBuffer,
+ /*SkipInvalidEntries=*/true));
outs() << "Parsed " << Points.size() << " benchmark points\n";
if (Points.empty()) {
diff --git a/llvm/unittests/tools/llvm-exegesis/X86/BenchmarkResultTest.cpp b/llvm/unittests/tools/llvm-exegesis/X86/BenchmarkResultTest.cpp
index 4636029f35017..f5ea870e10097 100644
--- a/llvm/unittests/tools/llvm-exegesis/X86/BenchmarkResultTest.cpp
+++ b/llvm/unittests/tools/llvm-exegesis/X86/BenchmarkResultTest.cpp
@@ -170,6 +170,61 @@ TEST(BenchmarkResultTest, PerInstructionStats) {
EXPECT_EQ(Stats.max(), 1.5);
EXPECT_EQ(Stats.avg(), 0.25); // (0.5+1.5-1.0+0.0) / 4
}
+
+TEST(BenchmarkResultTest, ReadYamlsSkipsInvalidEntries) {
+ LLVMInitializeX86TargetInfo();
+ LLVMInitializeX86Target();
+ LLVMInitializeX86TargetMC();
+ InitializeX86ExegesisTarget();
+
+ const LLVMState State =
+ cantFail(LLVMState::Create("x86_64-unknown-linux", "haswell"));
+ ExitOnError ExitOnErr;
+
+ // Serialize a single valid benchmark to YAML.
+ Benchmark Valid;
+ Valid.Key.Instructions.push_back(MCInstBuilder(X86::XOR32rr)
+ .addReg(X86::AL)
+ .addReg(X86::AH)
+ .addImm(123)
+ .addDFPImm(bit_cast<uint64_t>(0.5)));
+ Valid.Mode = Benchmark::Latency;
+ Valid.CpuName = "cpu_name";
+ Valid.LLVMTriple = "llvm_triple";
+ Valid.MinInstructions = 1;
+
+ std::string ValidYaml;
+ raw_string_ostream OS(ValidYaml);
+ ExitOnErr(Valid.writeYamlTo(State, OS));
+ OS.flush();
+
+ // Build a corrupted copy whose opcode name is unknown, then a two-document
+ // buffer with the bad entry first and the valid entry second.
+ const StringRef Opcode = "XOR32rr";
+ std::string InvalidYaml = ValidYaml;
+ const size_t Pos = InvalidYaml.find(Opcode);
+ ASSERT_NE(Pos, std::string::npos);
+ InvalidYaml.replace(Pos, Opcode.size(), "NOT_A_REAL_OPCODE");
+ const std::string Combined = InvalidYaml + ValidYaml;
+
+ // Without skipping, the unknown opcode aborts the read of the whole file.
+ {
+ auto Result =
+ Benchmark::readYamls(State, MemoryBufferRef(Combined, "data"));
+ EXPECT_FALSE(static_cast<bool>(Result));
+ consumeError(Result.takeError());
+ }
+
+ // With skipping, the bad entry is dropped and the valid one survives.
+ {
+ const auto Benchmarks = ExitOnErr(Benchmark::readYamls(
+ State, MemoryBufferRef(Combined, "data"), /*SkipInvalidEntries=*/true));
+ ASSERT_EQ(Benchmarks.size(), size_t{1});
+ EXPECT_EQ(Benchmarks[0].CpuName, "cpu_name");
+ EXPECT_THAT(Benchmarks[0].Key.Instructions,
+ Pointwise(EqMCInst(), Valid.Key.Instructions));
+ }
+}
} // namespace
} // namespace exegesis
} // namespace llvm
>From 7220a489ca91251976b94746feef8dbd514fece9 Mon Sep 17 00:00:00 2001
From: Lawson Darrow <lawson.darrow at gmail.com>
Date: Tue, 2 Jun 2026 14:40:51 -0400
Subject: [PATCH 2/4] [llvm-exegesis] Address review: skip invalid entries by
default
Skip invalid benchmark entries by default with a warning instead of
parameterizing readYamls. Drop the redundant unit test, add a check for
the skipped-entry count, and trim the analysis test's second section.
---
.../X86/analysis-skip-unknown-opcode.test | 39 +------------
.../llvm-exegesis/lib/BenchmarkResult.cpp | 19 +++----
.../tools/llvm-exegesis/lib/BenchmarkResult.h | 9 ++-
llvm/tools/llvm-exegesis/llvm-exegesis.cpp | 3 +-
.../llvm-exegesis/X86/BenchmarkResultTest.cpp | 55 -------------------
5 files changed, 16 insertions(+), 109 deletions(-)
diff --git a/llvm/test/tools/llvm-exegesis/X86/analysis-skip-unknown-opcode.test b/llvm/test/tools/llvm-exegesis/X86/analysis-skip-unknown-opcode.test
index 3373cd01843e6..1e4986ef93161 100644
--- a/llvm/test/tools/llvm-exegesis/X86/analysis-skip-unknown-opcode.test
+++ b/llvm/test/tools/llvm-exegesis/X86/analysis-skip-unknown-opcode.test
@@ -3,10 +3,11 @@
# warning instead of aborting, and that the remaining valid entries are still
# analyzed.
-# RUN: llvm-exegesis -mode=analysis -benchmarks-file=%s -analysis-clusters-output-file=- -analysis-clustering-epsilon=0.1 -analysis-inconsistency-epsilon=0.1 -analysis-numpoints=1 -analysis-clustering=naive 2> %t.err | FileCheck -check-prefix=CHECK-CLUSTERS %s
+# RUN: llvm-exegesis -mode=analysis -benchmarks-file=%s -analysis-clusters-output-file=- -analysis-clustering-epsilon=0.1 -analysis-inconsistency-epsilon=0.1 -analysis-numpoints=1 2> %t.err | FileCheck -check-prefix=CHECK-CLUSTERS %s
# RUN: FileCheck -check-prefix=CHECK-WARN %s < %t.err
# CHECK-WARN: warning: skipping benchmark entry: No opcode with name 'NOT_A_REAL_OPCODE'
+# CHECK-WARN: warning: skipped 1 benchmark entries that could not be parsed
# CHECK-CLUSTERS: {{^}}cluster_id,opcode_name,config,sched_class,
# CHECK-CLUSTERS-NEXT: {{^}}0,
@@ -29,46 +30,12 @@ mode: uops
key:
instructions:
- 'CMOV16rm AX AX RDI i_0x1 %noreg i_0x0 %noreg i_0x0'
- - 'CMOV16rm BP BP RDI i_0x1 %noreg i_0x40 %noreg i_0x0'
- - 'CMOV16rm BX BX RDI i_0x1 %noreg i_0x80 %noreg i_0x0'
- - 'CMOV16rm CX CX RDI i_0x1 %noreg i_0xc0 %noreg i_0x0'
- - 'CMOV16rm DX DX RDI i_0x1 %noreg i_0x100 %noreg i_0x0'
- - 'CMOV16rm SI SI RDI i_0x1 %noreg i_0x140 %noreg i_0x0'
- - 'CMOV16rm R8W R8W RDI i_0x1 %noreg i_0x180 %noreg i_0x0'
- - 'CMOV16rm R9W R9W RDI i_0x1 %noreg i_0x1c0 %noreg i_0x0'
- - 'CMOV16rm R10W R10W RDI i_0x1 %noreg i_0x200 %noreg i_0x0'
- - 'CMOV16rm R11W R11W RDI i_0x1 %noreg i_0x240 %noreg i_0x0'
- - 'CMOV16rm R12W R12W RDI i_0x1 %noreg i_0x280 %noreg i_0x0'
- - 'CMOV16rm R13W R13W RDI i_0x1 %noreg i_0x2c0 %noreg i_0x0'
- - 'CMOV16rm R14W R14W RDI i_0x1 %noreg i_0x300 %noreg i_0x0'
- - 'CMOV16rm R15W R15W RDI i_0x1 %noreg i_0x340 %noreg i_0x0'
config: ''
- register_initial_values:
- - 'AX=0x0'
- - 'EFLAGS=0x0'
- - 'BP=0x0'
- - 'BX=0x0'
- - 'CX=0x0'
- - 'DX=0x0'
- - 'SI=0x0'
- - 'R8W=0x0'
- - 'R9W=0x0'
- - 'R10W=0x0'
- - 'R11W=0x0'
- - 'R12W=0x0'
- - 'R13W=0x0'
- - 'R14W=0x0'
- - 'R15W=0x0'
+ register_initial_values: []
cpu_name: znver3
llvm_triple: x86_64-pc-linux-gnu
num_repetitions: 10000
measurements:
- - { key: Zn3Int, value: 1.0161, per_snippet_value: 14.2254 }
- - { key: Zn3FPU, value: 0, per_snippet_value: 0 }
- - { key: Zn3Load, value: 1.003, per_snippet_value: 14.042 }
- - { key: Zn3Store, value: 0.0023, per_snippet_value: 0.0322 }
- - { key: Zn3Divider, value: 0, per_snippet_value: 0 }
- { key: NumMicroOps, value: 1.0146, per_snippet_value: 14.2044 }
error: ''
-info: instruction has tied variables, using static renaming.
...
diff --git a/llvm/tools/llvm-exegesis/lib/BenchmarkResult.cpp b/llvm/tools/llvm-exegesis/lib/BenchmarkResult.cpp
index 2df13ff7d7537..1d1a376d3505b 100644
--- a/llvm/tools/llvm-exegesis/lib/BenchmarkResult.cpp
+++ b/llvm/tools/llvm-exegesis/lib/BenchmarkResult.cpp
@@ -391,14 +391,13 @@ Expected<Benchmark> Benchmark::readYaml(const LLVMState &State,
}
Expected<std::vector<Benchmark>> Benchmark::readYamls(const LLVMState &State,
- MemoryBufferRef Buffer,
- bool SkipInvalidEntries) {
+ MemoryBufferRef Buffer) {
yaml::Input Yin(Buffer);
YamlContext Context(State);
- // In skip mode, recoverable per-entry errors (e.g. unknown opcodes) are
- // recorded rather than reported to the parser, so a single bad entry can be
- // dropped without aborting the read of the whole file.
- Context.ContinueOnError = SkipInvalidEntries;
+ // Recoverable per-entry errors (e.g. unknown opcodes) are recorded rather
+ // than reported to the parser, so a single bad entry can be dropped without
+ // aborting the read of the whole file.
+ Context.ContinueOnError = true;
std::vector<Benchmark> Benchmarks;
unsigned NumSkippedEntries = 0;
while (Yin.setCurrentDocument()) {
@@ -407,11 +406,9 @@ Expected<std::vector<Benchmark>> Benchmark::readYamls(const LLVMState &State,
if (Yin.error())
return errorCodeToError(Yin.error());
if (!Context.getLastError().empty()) {
- if (!SkipInvalidEntries)
- return make_error<Failure>(Context.getLastError());
- // In skip mode, warn about the unparsable entry (e.g. an unknown opcode
- // from a bitrotted sample), discard it, and continue so that a single bad
- // entry doesn't abort the read of the whole file.
+ // Warn about the unparsable entry (e.g. an unknown opcode from a
+ // bitrotted sample), discard it, and continue so that a single bad entry
+ // doesn't abort the read of the whole file.
WithColor::warning() << "skipping benchmark entry: "
<< Context.getLastError();
Context.getLastError().clear();
diff --git a/llvm/tools/llvm-exegesis/lib/BenchmarkResult.h b/llvm/tools/llvm-exegesis/lib/BenchmarkResult.h
index c751a601fb765..6f3bbdfc7e17f 100644
--- a/llvm/tools/llvm-exegesis/lib/BenchmarkResult.h
+++ b/llvm/tools/llvm-exegesis/lib/BenchmarkResult.h
@@ -136,12 +136,11 @@ struct Benchmark {
static Expected<Benchmark> readYaml(const LLVMState &State,
MemoryBufferRef Buffer);
- // Deserializes benchmarks from the given Buffer. If SkipInvalidEntries is
- // true, entries that fail to parse (e.g. referencing an unknown opcode) are
- // warned about and skipped instead of aborting the read of the whole file.
+ // Deserializes benchmarks from the given Buffer. Entries that fail to parse
+ // (e.g. referencing an unknown opcode from a bitrotted sample) are warned
+ // about and skipped instead of aborting the read of the whole file.
static Expected<std::vector<Benchmark>>
- readYamls(const LLVMState &State, MemoryBufferRef Buffer,
- bool SkipInvalidEntries = false);
+ readYamls(const LLVMState &State, MemoryBufferRef Buffer);
// Given a set of serialized instruction benchmarks, returns the set of
// triples and CPUs that appear in the list of benchmarks.
diff --git a/llvm/tools/llvm-exegesis/llvm-exegesis.cpp b/llvm/tools/llvm-exegesis/llvm-exegesis.cpp
index 88e5b081eca11..3572abca8e162 100644
--- a/llvm/tools/llvm-exegesis/llvm-exegesis.cpp
+++ b/llvm/tools/llvm-exegesis/llvm-exegesis.cpp
@@ -671,8 +671,7 @@ static void analysisMain() {
// Skip entries that fail to parse (e.g. unknown opcodes from bitrotted
// samples) with a warning rather than aborting the whole analysis.
std::vector<Benchmark> Points = ExitOnFileError(
- BenchmarkFile, Benchmark::readYamls(State, *MemoryBuffer,
- /*SkipInvalidEntries=*/true));
+ BenchmarkFile, Benchmark::readYamls(State, *MemoryBuffer));
outs() << "Parsed " << Points.size() << " benchmark points\n";
if (Points.empty()) {
diff --git a/llvm/unittests/tools/llvm-exegesis/X86/BenchmarkResultTest.cpp b/llvm/unittests/tools/llvm-exegesis/X86/BenchmarkResultTest.cpp
index f5ea870e10097..4636029f35017 100644
--- a/llvm/unittests/tools/llvm-exegesis/X86/BenchmarkResultTest.cpp
+++ b/llvm/unittests/tools/llvm-exegesis/X86/BenchmarkResultTest.cpp
@@ -170,61 +170,6 @@ TEST(BenchmarkResultTest, PerInstructionStats) {
EXPECT_EQ(Stats.max(), 1.5);
EXPECT_EQ(Stats.avg(), 0.25); // (0.5+1.5-1.0+0.0) / 4
}
-
-TEST(BenchmarkResultTest, ReadYamlsSkipsInvalidEntries) {
- LLVMInitializeX86TargetInfo();
- LLVMInitializeX86Target();
- LLVMInitializeX86TargetMC();
- InitializeX86ExegesisTarget();
-
- const LLVMState State =
- cantFail(LLVMState::Create("x86_64-unknown-linux", "haswell"));
- ExitOnError ExitOnErr;
-
- // Serialize a single valid benchmark to YAML.
- Benchmark Valid;
- Valid.Key.Instructions.push_back(MCInstBuilder(X86::XOR32rr)
- .addReg(X86::AL)
- .addReg(X86::AH)
- .addImm(123)
- .addDFPImm(bit_cast<uint64_t>(0.5)));
- Valid.Mode = Benchmark::Latency;
- Valid.CpuName = "cpu_name";
- Valid.LLVMTriple = "llvm_triple";
- Valid.MinInstructions = 1;
-
- std::string ValidYaml;
- raw_string_ostream OS(ValidYaml);
- ExitOnErr(Valid.writeYamlTo(State, OS));
- OS.flush();
-
- // Build a corrupted copy whose opcode name is unknown, then a two-document
- // buffer with the bad entry first and the valid entry second.
- const StringRef Opcode = "XOR32rr";
- std::string InvalidYaml = ValidYaml;
- const size_t Pos = InvalidYaml.find(Opcode);
- ASSERT_NE(Pos, std::string::npos);
- InvalidYaml.replace(Pos, Opcode.size(), "NOT_A_REAL_OPCODE");
- const std::string Combined = InvalidYaml + ValidYaml;
-
- // Without skipping, the unknown opcode aborts the read of the whole file.
- {
- auto Result =
- Benchmark::readYamls(State, MemoryBufferRef(Combined, "data"));
- EXPECT_FALSE(static_cast<bool>(Result));
- consumeError(Result.takeError());
- }
-
- // With skipping, the bad entry is dropped and the valid one survives.
- {
- const auto Benchmarks = ExitOnErr(Benchmark::readYamls(
- State, MemoryBufferRef(Combined, "data"), /*SkipInvalidEntries=*/true));
- ASSERT_EQ(Benchmarks.size(), size_t{1});
- EXPECT_EQ(Benchmarks[0].CpuName, "cpu_name");
- EXPECT_THAT(Benchmarks[0].Key.Instructions,
- Pointwise(EqMCInst(), Valid.Key.Instructions));
- }
-}
} // namespace
} // namespace exegesis
} // namespace llvm
>From 1b965580aead8379ee75a256cd1c36265d58b91f Mon Sep 17 00:00:00 2001
From: Lawson Darrow <lawson.darrow at gmail.com>
Date: Fri, 5 Jun 2026 14:14:24 -0400
Subject: [PATCH 3/4] [llvm-exegesis] Add a test that malformed YAML in
analysis mode errors instead of being skipped
The skip path only swallows instruction/register scalar parse errors (e.g. an
unknown opcode from a bitrotted sample), so genuinely ill-formed YAML such as an
unknown key is still reported as a parse error rather than silently dropped. Add
a lit test for the unknown-key case that checks it is not treated as a skipped
benchmark entry.
---
.../X86/analysis-malformed-yaml-error.test | 25 +++++++++++++++++++
1 file changed, 25 insertions(+)
create mode 100644 llvm/test/tools/llvm-exegesis/X86/analysis-malformed-yaml-error.test
diff --git a/llvm/test/tools/llvm-exegesis/X86/analysis-malformed-yaml-error.test b/llvm/test/tools/llvm-exegesis/X86/analysis-malformed-yaml-error.test
new file mode 100644
index 0000000000000..25bccd277f259
--- /dev/null
+++ b/llvm/test/tools/llvm-exegesis/X86/analysis-malformed-yaml-error.test
@@ -0,0 +1,25 @@
+# Verify that in analysis mode genuinely ill-formed YAML (here an unknown key,
+# as opposed to an unknown opcode) is reported as a parse error rather than
+# being silently skipped the way a bitrotted/unknown-opcode entry is.
+
+# RUN: not llvm-exegesis -mode=analysis -benchmarks-file=%s -analysis-clusters-output-file=- -analysis-clustering=naive -analysis-numpoints=1 2> %t.err
+# RUN: FileCheck %s < %t.err
+
+# CHECK-NOT: skipping benchmark entry
+# CHECK: error: unknown key 'bar'
+
+---
+mode: uops
+key:
+ instructions:
+ - 'CMOV16rm AX AX RDI i_0x1 %noreg i_0x0 %noreg i_0x0'
+ config: ''
+ register_initial_values: []
+ foo: ''
+ bar: []
+cpu_name: znver3
+llvm_triple: x86_64-pc-linux-gnu
+num_repetitions: 10000
+measurements: []
+error: ''
+...
>From f274a209ac03a113316323c5683ae1c325b0e41d Mon Sep 17 00:00:00 2001
From: Lawson Darrow <lawson.darrow at gmail.com>
Date: Mon, 8 Jun 2026 10:01:37 -0400
Subject: [PATCH 4/4] [llvm-exegesis] Remove redundant comment at readYamls
call site
---
llvm/tools/llvm-exegesis/llvm-exegesis.cpp | 2 --
1 file changed, 2 deletions(-)
diff --git a/llvm/tools/llvm-exegesis/llvm-exegesis.cpp b/llvm/tools/llvm-exegesis/llvm-exegesis.cpp
index 3572abca8e162..b5820f8669f1d 100644
--- a/llvm/tools/llvm-exegesis/llvm-exegesis.cpp
+++ b/llvm/tools/llvm-exegesis/llvm-exegesis.cpp
@@ -668,8 +668,6 @@ static void analysisMain() {
// Read benchmarks.
const LLVMState State = ExitOnErr(
LLVMState::Create(TripleAndCpu.LLVMTriple, TripleAndCpu.CpuName));
- // Skip entries that fail to parse (e.g. unknown opcodes from bitrotted
- // samples) with a warning rather than aborting the whole analysis.
std::vector<Benchmark> Points = ExitOnFileError(
BenchmarkFile, Benchmark::readYamls(State, *MemoryBuffer));
More information about the llvm-commits
mailing list