[llvm] 7ffcbc2 - [llvm-exegesis] Skip benchmark entries with unknown opcodes in analysis mode (#201162)
via llvm-commits
llvm-commits at lists.llvm.org
Thu Jun 11 09:18:43 PDT 2026
Author: Lawson Darrow
Date: 2026-06-11T09:18:38-07:00
New Revision: 7ffcbc27ea719531e8bda54fa1b0aca5c8a7e095
URL: https://github.com/llvm/llvm-project/commit/7ffcbc27ea719531e8bda54fa1b0aca5c8a7e095
DIFF: https://github.com/llvm/llvm-project/commit/7ffcbc27ea719531e8bda54fa1b0aca5c8a7e095.diff
LOG: [llvm-exegesis] Skip benchmark entries with unknown opcodes in analysis mode (#201162)
Fixes #144403
`llvm-exegesis -mode=analysis` currently aborts the whole run if any
entry references an unknown opcode (e.g. an old sample whose instruction
was since renamed). A single bitrotted entry discards the results for
every valid one:
```
uops-test.yaml:5:7: error: No opcode with name 'VADDPDYrrr'
llvm-exegesis: 'uops-test.yaml': invalid argument
```
This adds an opt-in `SkipInvalidEntries` parameter to
`Benchmark::readYamls`. When set, the `YamlContext` records recoverable
per-entry deserialization errors (e.g. an unknown opcode or register)
instead of returning them from `ScalarTraits::input` to the YAML parser,
which would otherwise poison the stream and stop parsing. `readYamls`
then inspects the recorded error after each document, warns about the
offending entry, drops it, and continues with the rest. The default
stays `false`, preserving the existing strict behavior for all other
callers; analysis mode passes `true`.
For example, a file with one bitrotted entry and one valid entry now
produces:
```
warning: skipping benchmark entry: No opcode with name 'NOT_A_REAL_OPCODE'
warning: skipped 1 benchmark entries that could not be parsed
Parsed 1 benchmark points
```
Tests:
- Unit test in `BenchmarkResultTest.cpp` covering both strict (errors)
and skip (drops the bad entry, keeps the valid one) behavior.
- A lit test exercising analysis mode end-to-end: a file with one
unknown-opcode entry plus one valid entry still produces clusters output
and emits the warning.
The "did you mean '<nearest>'?" opcode suggestion mentioned in the issue
(via `OptTable::findNearest`) is left as a follow-up.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply at anthropic.com>
Added:
llvm/test/tools/llvm-exegesis/X86/analysis-malformed-yaml-error.test
llvm/test/tools/llvm-exegesis/X86/analysis-skip-unknown-opcode.test
Modified:
llvm/tools/llvm-exegesis/lib/BenchmarkResult.cpp
llvm/tools/llvm-exegesis/lib/BenchmarkResult.h
Removed:
################################################################################
diff --git a/llvm/test/tools/llvm-exegesis/X86/analysis-malformed-yaml-error.test b/llvm/test/tools/llvm-exegesis/X86/analysis-malformed-yaml-error.test
new file mode 100644
index 0000000000000..25bccd277f259
--- /dev/null
+++ b/llvm/test/tools/llvm-exegesis/X86/analysis-malformed-yaml-error.test
@@ -0,0 +1,25 @@
+# Verify that in analysis mode genuinely ill-formed YAML (here an unknown key,
+# as opposed to an unknown opcode) is reported as a parse error rather than
+# being silently skipped the way a bitrotted/unknown-opcode entry is.
+
+# RUN: not llvm-exegesis -mode=analysis -benchmarks-file=%s -analysis-clusters-output-file=- -analysis-clustering=naive -analysis-numpoints=1 2> %t.err
+# RUN: FileCheck %s < %t.err
+
+# CHECK-NOT: skipping benchmark entry
+# CHECK: error: unknown key 'bar'
+
+---
+mode: uops
+key:
+ instructions:
+ - 'CMOV16rm AX AX RDI i_0x1 %noreg i_0x0 %noreg i_0x0'
+ config: ''
+ register_initial_values: []
+ foo: ''
+ bar: []
+cpu_name: znver3
+llvm_triple: x86_64-pc-linux-gnu
+num_repetitions: 10000
+measurements: []
+error: ''
+...
diff --git a/llvm/test/tools/llvm-exegesis/X86/analysis-skip-unknown-opcode.test b/llvm/test/tools/llvm-exegesis/X86/analysis-skip-unknown-opcode.test
new file mode 100644
index 0000000000000..1e4986ef93161
--- /dev/null
+++ b/llvm/test/tools/llvm-exegesis/X86/analysis-skip-unknown-opcode.test
@@ -0,0 +1,41 @@
+# Verify that in analysis mode an entry referencing an unknown opcode (e.g. a
+# sample that has bitrotted due to instruction renaming) is skipped with a
+# warning instead of aborting, and that the remaining valid entries are still
+# analyzed.
+
+# RUN: llvm-exegesis -mode=analysis -benchmarks-file=%s -analysis-clusters-output-file=- -analysis-clustering-epsilon=0.1 -analysis-inconsistency-epsilon=0.1 -analysis-numpoints=1 2> %t.err | FileCheck -check-prefix=CHECK-CLUSTERS %s
+# RUN: FileCheck -check-prefix=CHECK-WARN %s < %t.err
+
+# CHECK-WARN: warning: skipping benchmark entry: No opcode with name 'NOT_A_REAL_OPCODE'
+# CHECK-WARN: warning: skipped 1 benchmark entries that could not be parsed
+
+# CHECK-CLUSTERS: {{^}}cluster_id,opcode_name,config,sched_class,
+# CHECK-CLUSTERS-NEXT: {{^}}0,
+
+---
+mode: uops
+key:
+ instructions:
+ - 'NOT_A_REAL_OPCODE RAX'
+ config: ''
+ register_initial_values: []
+cpu_name: znver3
+llvm_triple: x86_64-pc-linux-gnu
+num_repetitions: 10000
+measurements: []
+error: ''
+...
+---
+mode: uops
+key:
+ instructions:
+ - 'CMOV16rm AX AX RDI i_0x1 %noreg i_0x0 %noreg i_0x0'
+ config: ''
+ register_initial_values: []
+cpu_name: znver3
+llvm_triple: x86_64-pc-linux-gnu
+num_repetitions: 10000
+measurements:
+ - { key: NumMicroOps, value: 1.0146, per_snippet_value: 14.2044 }
+error: ''
+...
diff --git a/llvm/tools/llvm-exegesis/lib/BenchmarkResult.cpp b/llvm/tools/llvm-exegesis/lib/BenchmarkResult.cpp
index a597f413ee699..1d1a376d3505b 100644
--- a/llvm/tools/llvm-exegesis/lib/BenchmarkResult.cpp
+++ b/llvm/tools/llvm-exegesis/lib/BenchmarkResult.cpp
@@ -19,6 +19,7 @@
#include "llvm/Support/FileOutputBuffer.h"
#include "llvm/Support/FileSystem.h"
#include "llvm/Support/Format.h"
+#include "llvm/Support/WithColor.h"
#include "llvm/Support/raw_ostream.h"
static constexpr char kIntegerPrefix[] = "i_0x";
@@ -63,8 +64,22 @@ struct YamlContext {
std::string &getLastError() { return ErrorStream.str(); }
+ // Returns the recorded error to report to the YAML parser from a
+ // ScalarTraits::input. When ContinueOnError is set, recoverable per-entry
+ // errors (e.g. an unknown opcode in a bitrotted sample) are left recorded for
+ // the caller to inspect but are not reported to the parser, which would
+ // otherwise abort parsing of every remaining document.
+ StringRef getInputError() {
+ return ContinueOnError ? StringRef() : StringRef(getLastError());
+ }
+
raw_string_ostream &getErrorStream() { return ErrorStream; }
+ // When set, ScalarTraits::input swallows recoverable deserialization errors
+ // (see getInputError) so that readYamls can skip the offending entry instead
+ // of aborting the whole file.
+ bool ContinueOnError = false;
+
StringRef getRegName(MCRegister Reg) {
// Special case: Reg may be invalid. We have to deal with it explicitly.
if (!Reg.isValid())
@@ -177,7 +192,7 @@ template <> struct ScalarTraits<MCInst> {
static StringRef input(StringRef Scalar, void *Ctx, MCInst &Value) {
YamlContext &Context = getTypedContext(Ctx);
Context.deserializeMCInst(Scalar, Value);
- return Context.getLastError();
+ return Context.getInputError();
}
// By default strings are quoted only when necessary.
@@ -270,7 +285,7 @@ template <> struct ScalarTraits<exegesis::RegisterValue> {
Context.getErrorStream()
<< "Unknown initial register value: '" << String << "'";
}
- return Context.getLastError();
+ return Context.getInputError();
}
static QuotingType mustQuote(StringRef) { return QuotingType::Single; }
@@ -379,16 +394,32 @@ Expected<std::vector<Benchmark>> Benchmark::readYamls(const LLVMState &State,
MemoryBufferRef Buffer) {
yaml::Input Yin(Buffer);
YamlContext Context(State);
+ // Recoverable per-entry errors (e.g. unknown opcodes) are recorded rather
+ // than reported to the parser, so a single bad entry can be dropped without
+ // aborting the read of the whole file.
+ Context.ContinueOnError = true;
std::vector<Benchmark> Benchmarks;
+ unsigned NumSkippedEntries = 0;
while (Yin.setCurrentDocument()) {
Benchmarks.emplace_back();
yamlize(Yin, Benchmarks.back(), /*unused*/ true, Context);
if (Yin.error())
return errorCodeToError(Yin.error());
- if (!Context.getLastError().empty())
- return make_error<Failure>(Context.getLastError());
+ if (!Context.getLastError().empty()) {
+ // Warn about the unparsable entry (e.g. an unknown opcode from a
+ // bitrotted sample), discard it, and continue so that a single bad entry
+ // doesn't abort the read of the whole file.
+ WithColor::warning() << "skipping benchmark entry: "
+ << Context.getLastError();
+ Context.getLastError().clear();
+ Benchmarks.pop_back();
+ ++NumSkippedEntries;
+ }
Yin.nextDocument();
}
+ if (NumSkippedEntries)
+ WithColor::warning() << "skipped " << NumSkippedEntries
+ << " benchmark entries that could not be parsed\n";
return std::move(Benchmarks);
}
diff --git a/llvm/tools/llvm-exegesis/lib/BenchmarkResult.h b/llvm/tools/llvm-exegesis/lib/BenchmarkResult.h
index 7984c8805cadc..6f3bbdfc7e17f 100644
--- a/llvm/tools/llvm-exegesis/lib/BenchmarkResult.h
+++ b/llvm/tools/llvm-exegesis/lib/BenchmarkResult.h
@@ -136,6 +136,9 @@ struct Benchmark {
static Expected<Benchmark> readYaml(const LLVMState &State,
MemoryBufferRef Buffer);
+ // Deserializes benchmarks from the given Buffer. Entries that fail to parse
+ // (e.g. referencing an unknown opcode from a bitrotted sample) are warned
+ // about and skipped instead of aborting the read of the whole file.
static Expected<std::vector<Benchmark>>
readYamls(const LLVMState &State, MemoryBufferRef Buffer);
More information about the llvm-commits
mailing list