[llvm] 7d80b94 - Add a utility for converting between different types of remarks
Jessica Paquette via llvm-commits
llvm-commits at lists.llvm.org
Mon Sep 12 15:04:56 PDT 2022
Author: Jessica Paquette
Date: 2022-09-12T15:04:19-07:00
New Revision: 7d80b94ca3ab4563c964ebbf3662c0d3033dc91a
URL: https://github.com/llvm/llvm-project/commit/7d80b94ca3ab4563c964ebbf3662c0d3033dc91a
DIFF: https://github.com/llvm/llvm-project/commit/7d80b94ca3ab4563c964ebbf3662c0d3033dc91a.diff
LOG: Add a utility for converting between different types of remarks
This adds llvm-remarkutil. This is intended to be a general tool for doing stuff
with/to remark files.
This patch gives it the following powers:
* `bitstream2yaml` - To convert bitstream remarks to YAML
* `yaml2bitstream` - To convert YAML remarks to bitstream remarks
These are both implemented as subcommands, like
`llvm-remarkutil bitstream2yaml <input_file> -o -`
I ran into an issue where I had some bitstream remarks coming from CI, and I
wanted to be able to do stuff with them (e.g. visualize them). But then I
noticed we didn't have any tooling for doing that, so I decided to write this
thing.
Being able to output YAML as a start seemed like a good idea, since it
would allow people to reuse any tooling they may have written based around YAML
remarks.
Hopefully it can grow into a more featureful remark utility. :)
Currently there are is an outstanding performance issue (see the TODO) with
the bitstream2yaml case. I decided that I'd keep the tool small to start with
and have the yaml2bitstream and bitstream2yaml cases be symmetric.
Also I moved the remarks documentation to its own header because it seems
a little out of place with "basic commands" and "developer tools"; it's
really kind of its own thing.
Differential Revision: https://reviews.llvm.org/D133646
Added:
llvm/docs/CommandGuide/LLVMRemarkUtil.rst
llvm/test/tools/llvm-remarkutil/Inputs/broken-remark
llvm/test/tools/llvm-remarkutil/Inputs/broken-remark.bitstream
llvm/test/tools/llvm-remarkutil/Inputs/empty-file
llvm/test/tools/llvm-remarkutil/Inputs/two-remarks.bitstream
llvm/test/tools/llvm-remarkutil/Inputs/two-remarks.yaml
llvm/test/tools/llvm-remarkutil/broken-bitstream-remark.test
llvm/test/tools/llvm-remarkutil/broken-yaml-remark.test
llvm/test/tools/llvm-remarkutil/convert.test
llvm/test/tools/llvm-remarkutil/empty-file.test
llvm/test/tools/llvm-remarkutil/file-does-not-exist.test
llvm/test/tools/llvm-remarkutil/missing-subcommand.test
llvm/tools/llvm-remarkutil/CMakeLists.txt
llvm/tools/llvm-remarkutil/RemarkUtil.cpp
Modified:
llvm/docs/CommandGuide/index.rst
llvm/test/CMakeLists.txt
llvm/test/lit.cfg.py
Removed:
################################################################################
diff --git a/llvm/docs/CommandGuide/LLVMRemarkUtil.rst b/llvm/docs/CommandGuide/LLVMRemarkUtil.rst
new file mode 100644
index 0000000000000..b068dc318f2c6
--- /dev/null
+++ b/llvm/docs/CommandGuide/LLVMRemarkUtil.rst
@@ -0,0 +1,50 @@
+llvm-remarkutil -
+==============================================================
+
+.. program:: llvm-remarkutil
+
+SYNOPSIS
+--------
+
+:program:`llvm-remarkutil` [*subcommmand*] [*options*]
+
+DESCRIPTION
+-----------
+
+Utility for displaying information from, and converting between
diff erent
+`remark <https://llvm.org/docs/Remarks.html>`_ formats.
+
+Subcommands
+-----------
+
+ * :ref:`bitstream2yaml_subcommand` - Reserialize bitstream remarks to YAML.
+ * :ref:`yaml2bitstream_subcommand` - Reserialize YAML remarks to bitstream.
+
+.. _bitstream2yaml_subcommand:
+
+bitstream2yaml
+~~~~~~
+
+.. program:: llvm-remarkutil bitstream2yaml
+
+USAGE: :program:`llvm-remarkutil` bitstream2yaml <input file> -o <output file>
+
+Summary
+^^^^^^^^^^^
+
+Takes a bitstream remark file as input, and reserializes that file as YAML.
+
+.. _yaml2bitstream_subcommand:
+
+yaml2bitstream
+~~~~~~
+
+.. program:: llvm-remarkutil yaml2bitstream
+
+USAGE: :program:`llvm-remarkutil` yaml2bitstream <input file> -o <output file>
+
+Summary
+^^^^^^^^^^^
+
+Takes a YAML remark file as input, and reserializes that file in the bitstream
+format.
diff --git a/llvm/docs/CommandGuide/index.rst b/llvm/docs/CommandGuide/index.rst
index da9995a5a3e4f..fc87f15c9d588 100644
--- a/llvm/docs/CommandGuide/index.rst
+++ b/llvm/docs/CommandGuide/index.rst
@@ -33,7 +33,6 @@ Basic Commands
llvm-otool
llvm-profdata
llvm-readobj
- llvm-remark-size-
diff
llvm-stress
llvm-symbolizer
opt
@@ -86,3 +85,12 @@ Developer Tools
llvm-pdbutil
llvm-profgen
llvm-tli-checker
+
+Remarks Tools
+~~~~~~~~~~~~~~
+
+.. toctree::
+ :maxdepth: 1
+
+ llvm-remark-size-
diff
+ llvm-remarkutil
diff --git a/llvm/test/CMakeLists.txt b/llvm/test/CMakeLists.txt
index 86ca20ada7b80..5cf783c175842 100644
--- a/llvm/test/CMakeLists.txt
+++ b/llvm/test/CMakeLists.txt
@@ -118,6 +118,7 @@ set(LLVM_TEST_DEPENDS
llvm-readelf
llvm-reduce
llvm-remark-size-
diff
+ llvm-remarkutil
llvm-rtdyld
llvm-sim
llvm-size
diff --git a/llvm/test/lit.cfg.py b/llvm/test/lit.cfg.py
index 75a38b4c5dad5..1622b80356e5d 100644
--- a/llvm/test/lit.cfg.py
+++ b/llvm/test/lit.cfg.py
@@ -171,7 +171,7 @@ def get_asan_rtlib():
'llvm-tblgen', 'llvm-tapi-
diff ', 'llvm-undname', 'llvm-windres',
'llvm-c-test', 'llvm-cxxfilt', 'llvm-xray', 'yaml2obj', 'obj2yaml',
'yaml-bench', 'verify-uselistorder', 'bugpoint', 'llc', 'llvm-symbolizer',
- 'opt', 'sancov', 'sanstats'])
+ 'opt', 'sancov', 'sanstats', 'llvm-remarkutil'])
# The following tools are optional
tools.extend([
diff --git a/llvm/test/tools/llvm-remarkutil/Inputs/broken-remark b/llvm/test/tools/llvm-remarkutil/Inputs/broken-remark
new file mode 100644
index 0000000000000..e3aaf6a440895
--- /dev/null
+++ b/llvm/test/tools/llvm-remarkutil/Inputs/broken-remark
@@ -0,0 +1,15 @@
+--- !Analysis
+Name: StackSize
+Function: func0
+Args:
+ - NumStackBytes: '1'
+ - String: ' stack bytes in function'
+...
+--- !Analysis
+Pass: asm-printer
+Name: InstructionCount
+Function: func0
+Args:
+ - NumInstructions: '1'
+ - String: ' instructions in function'
+...
diff --git a/llvm/test/tools/llvm-remarkutil/Inputs/broken-remark.bitstream b/llvm/test/tools/llvm-remarkutil/Inputs/broken-remark.bitstream
new file mode 100644
index 0000000000000..4b14f8fee2d04
Binary files /dev/null and b/llvm/test/tools/llvm-remarkutil/Inputs/broken-remark.bitstream
diff er
diff --git a/llvm/test/tools/llvm-remarkutil/Inputs/empty-file b/llvm/test/tools/llvm-remarkutil/Inputs/empty-file
new file mode 100644
index 0000000000000..e69de29bb2d1d
diff --git a/llvm/test/tools/llvm-remarkutil/Inputs/two-remarks.bitstream b/llvm/test/tools/llvm-remarkutil/Inputs/two-remarks.bitstream
new file mode 100644
index 0000000000000..2a528436791ae
Binary files /dev/null and b/llvm/test/tools/llvm-remarkutil/Inputs/two-remarks.bitstream
diff er
diff --git a/llvm/test/tools/llvm-remarkutil/Inputs/two-remarks.yaml b/llvm/test/tools/llvm-remarkutil/Inputs/two-remarks.yaml
new file mode 100644
index 0000000000000..ae02fc1102d0b
--- /dev/null
+++ b/llvm/test/tools/llvm-remarkutil/Inputs/two-remarks.yaml
@@ -0,0 +1,16 @@
+--- !Analysis
+Pass: prologepilog
+Name: StackSize
+Function: func0
+Args:
+ - NumStackBytes: '1'
+ - String: ' stack bytes in function'
+...
+--- !Analysis
+Pass: asm-printer
+Name: InstructionCount
+Function: func0
+Args:
+ - NumInstructions: '1'
+ - String: ' instructions in function'
+...
diff --git a/llvm/test/tools/llvm-remarkutil/broken-bitstream-remark.test b/llvm/test/tools/llvm-remarkutil/broken-bitstream-remark.test
new file mode 100644
index 0000000000000..3a81beed03333
--- /dev/null
+++ b/llvm/test/tools/llvm-remarkutil/broken-bitstream-remark.test
@@ -0,0 +1,2 @@
+RUN: not llvm-remarkutil bitstream2yaml %p/Inputs/broken-remark -o - 2>&1 | FileCheck %s
+CHECK: error: Unknown magic number: expecting RMRK, got --- .
diff --git a/llvm/test/tools/llvm-remarkutil/broken-yaml-remark.test b/llvm/test/tools/llvm-remarkutil/broken-yaml-remark.test
new file mode 100644
index 0000000000000..d966725d63ac1
--- /dev/null
+++ b/llvm/test/tools/llvm-remarkutil/broken-yaml-remark.test
@@ -0,0 +1,2 @@
+RUN: not llvm-remarkutil yaml2bitstream %p/Inputs/broken-remark -o - 2>&1 | FileCheck %s
+CHECK: error: Type, Pass, Name or Function missing
diff --git a/llvm/test/tools/llvm-remarkutil/convert.test b/llvm/test/tools/llvm-remarkutil/convert.test
new file mode 100644
index 0000000000000..83023c8ce6a89
--- /dev/null
+++ b/llvm/test/tools/llvm-remarkutil/convert.test
@@ -0,0 +1,20 @@
+RUN: llvm-remarkutil bitstream2yaml %p/Inputs/two-remarks.bitstream -o - | FileCheck %s -strict-whitespace
+RUN: llvm-remarkutil yaml2bitstream %p/Inputs/two-remarks.yaml -o %t
+RUN: llvm-remarkutil bitstream2yaml %t -o - | FileCheck %s -strict-whitespace
+
+; CHECK: --- !Analysis
+; CHECK-NEXT: Pass: prologepilog
+; CHECK-NEXT: Name: StackSize
+; CHECK-NEXT: Function: func0
+; CHECK-NEXT: Args:
+; CHECK-NEXT: - NumStackBytes: '1'
+; CHECK-NEXT: - String: ' stack bytes in function'
+; CHECK-NEXT: ...
+; CHECK-NEXT: --- !Analysis
+; CHECK-NEXT: Pass: asm-printer
+; CHECK-NEXT: Name: InstructionCount
+; CHECK-NEXT: Function: func0
+; CHECK-NEXT: Args:
+; CHECK-NEXT: - NumInstructions: '1'
+; CHECK-NEXT: - String: ' instructions in function'
+; CHECK-NEXT: ...
diff --git a/llvm/test/tools/llvm-remarkutil/empty-file.test b/llvm/test/tools/llvm-remarkutil/empty-file.test
new file mode 100644
index 0000000000000..96bd1871b2410
--- /dev/null
+++ b/llvm/test/tools/llvm-remarkutil/empty-file.test
@@ -0,0 +1,7 @@
+RUN: not llvm-remarkutil yaml2bitstream %p/Inputs/empty-file -o - 2>&1 | FileCheck %s --check-prefix=YAML2BITSTREAM
+RUN: llvm-remarkutil bitstream2yaml %p/Inputs/empty-file -o - 2>&1 | FileCheck %s --allow-empty --check-prefix=BITSTREAM2YAML
+
+; YAML2BITSTREAM: error: document root is not of mapping type.
+
+; An empty bitstream file is valid.
+; BITSTREAM2YAML-NOT: error
diff --git a/llvm/test/tools/llvm-remarkutil/file-does-not-exist.test b/llvm/test/tools/llvm-remarkutil/file-does-not-exist.test
new file mode 100644
index 0000000000000..71ed5ee9453a7
--- /dev/null
+++ b/llvm/test/tools/llvm-remarkutil/file-does-not-exist.test
@@ -0,0 +1,3 @@
+RUN: not llvm-remarkutil bitstream2yaml %p/Inputs/i-do-not-exist -o - 2>&1 | FileCheck %s
+RUN: not llvm-remarkutil yaml2bitstream %p/Inputs/i-do-not-exist -o - 2>&1 | FileCheck %s
+CHECK: error: Cannot open file
diff --git a/llvm/test/tools/llvm-remarkutil/missing-subcommand.test b/llvm/test/tools/llvm-remarkutil/missing-subcommand.test
new file mode 100644
index 0000000000000..9f4e9d3146643
--- /dev/null
+++ b/llvm/test/tools/llvm-remarkutil/missing-subcommand.test
@@ -0,0 +1,2 @@
+RUN: not llvm-remarkutil 2>&1 | FileCheck %s
+CHECK: error: Please specify a subcommand. (See -help for options)
diff --git a/llvm/tools/llvm-remarkutil/CMakeLists.txt b/llvm/tools/llvm-remarkutil/CMakeLists.txt
new file mode 100644
index 0000000000000..2def6b8fc0bdb
--- /dev/null
+++ b/llvm/tools/llvm-remarkutil/CMakeLists.txt
@@ -0,0 +1,5 @@
+set(LLVM_LINK_COMPONENTS Core Demangle Object Remarks Support)
+
+add_llvm_tool(llvm-remarkutil
+ RemarkUtil.cpp
+)
diff --git a/llvm/tools/llvm-remarkutil/RemarkUtil.cpp b/llvm/tools/llvm-remarkutil/RemarkUtil.cpp
new file mode 100644
index 0000000000000..5cb641a0ba014
--- /dev/null
+++ b/llvm/tools/llvm-remarkutil/RemarkUtil.cpp
@@ -0,0 +1,196 @@
+//===--------- llvm-remarkutil/RemarkUtil.cpp -----------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+/// Utility for remark files.
+//===----------------------------------------------------------------------===//
+
+#include "llvm-c/Remarks.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Remarks/Remark.h"
+#include "llvm/Remarks/RemarkFormat.h"
+#include "llvm/Remarks/RemarkParser.h"
+#include "llvm/Remarks/YAMLRemarkSerializer.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/Compiler.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/InitLLVM.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/ToolOutputFile.h"
+#include "llvm/Support/WithColor.h"
+
+using namespace llvm;
+using namespace remarks;
+
+static ExitOnError ExitOnErr;
+static cl::OptionCategory RemarkUtilCategory("llvm-remarkutil options");
+namespace subopts {
+static cl::SubCommand
+ YAML2Bitstream("yaml2bitstream",
+ "Convert YAML remarks to bitstream remarks");
+static cl::SubCommand
+ Bitstream2YAML("bitstream2yaml",
+ "Convert bitstream remarks to YAML remarks");
+} // namespace subopts
+
+// Conversions have the same command line options. AFAIK there is no way to
+// reuse them, so to avoid duplication, let's just stick this in a hideous
+// macro.
+#define CONVERSION_COMMAND_LINE_OPTIONS(SUBOPT) \
+ static cl::opt<std::string> InputFileName( \
+ cl::Positional, cl::cat(RemarkUtilCategory), cl::init("-"), \
+ cl::desc("<input file>"), cl::sub(SUBOPT)); \
+ static cl::opt<std::string> OutputFileName( \
+ "o", cl::init("-"), cl::cat(RemarkUtilCategory), cl::desc("Output"), \
+ cl::value_desc("filename"), cl::sub(SUBOPT));
+namespace yaml2bitstream {
+/// Remark format to parse.
+static constexpr Format InputFormat = Format::YAML;
+/// Remark format to output.
+static constexpr Format OutputFormat = Format::Bitstream;
+CONVERSION_COMMAND_LINE_OPTIONS(subopts::YAML2Bitstream)
+} // namespace yaml2bitstream
+
+namespace bitstream2yaml {
+/// Remark format to parse.
+static constexpr Format InputFormat = Format::Bitstream;
+/// Remark format to output.
+static constexpr Format OutputFormat = Format::YAML;
+CONVERSION_COMMAND_LINE_OPTIONS(subopts::Bitstream2YAML)
+} // namespace bitstream2yaml
+
+/// \returns A MemoryBuffer for the input file on success, and an Error
+/// otherwise.
+static Expected<std::unique_ptr<MemoryBuffer>>
+getInputMemoryBuffer(StringRef InputFileName) {
+ auto MaybeBuf = MemoryBuffer::getFileOrSTDIN(InputFileName);
+ if (auto ErrorCode = MaybeBuf.getError())
+ return createStringError(ErrorCode,
+ Twine("Cannot open file '" + InputFileName +
+ "': " + ErrorCode.message()));
+ return std::move(*MaybeBuf);
+}
+
+/// Parses all remarks in the input file.
+/// \p [out] StrTab - A string table populated for later remark serialization.
+/// \returns A vector of parsed remarks on success, and an Error otherwise.
+static Expected<std::vector<std::unique_ptr<Remark>>>
+tryParseRemarksFromInputFile(StringRef InputFileName, Format InputFormat,
+ StringTable &StrTab) {
+ auto MaybeBuf = getInputMemoryBuffer(InputFileName);
+ if (!MaybeBuf)
+ return MaybeBuf.takeError();
+ auto MaybeParser = createRemarkParser(InputFormat, (*MaybeBuf)->getBuffer());
+ if (!MaybeParser)
+ return MaybeParser.takeError();
+ auto &Parser = **MaybeParser;
+ auto MaybeRemark = Parser.next();
+ // TODO: If we are converting from bitstream to YAML, we don't need to parse
+ // early because the string table is not necessary.
+ std::vector<std::unique_ptr<Remark>> ParsedRemarks;
+ for (; MaybeRemark; MaybeRemark = Parser.next()) {
+ StrTab.internalize(**MaybeRemark);
+ ParsedRemarks.push_back(std::move(*MaybeRemark));
+ }
+ auto E = MaybeRemark.takeError();
+ if (!E.isA<EndOfFileError>())
+ return std::move(E);
+ consumeError(std::move(E));
+ return ParsedRemarks;
+}
+
+/// \returns A ToolOutputFile which can be used for writing remarks on success,
+/// and an Error otherwise.
+static Expected<std::unique_ptr<ToolOutputFile>>
+getOutputFile(StringRef OutputFileName, Format OutputFormat) {
+ if (OutputFileName == "")
+ OutputFileName = "-";
+ auto Flags = OutputFormat == Format::YAML ? sys::fs::OF_TextWithCRLF
+ : sys::fs::OF_None;
+ std::error_code ErrorCode;
+ auto OF = std::make_unique<ToolOutputFile>(OutputFileName, ErrorCode, Flags);
+ if (ErrorCode)
+ return errorCodeToError(ErrorCode);
+ return std::move(OF);
+}
+
+/// Reserialize a list of remarks into the desired output format, and output
+/// to the user-specified output file.
+/// \p ParsedRemarks - A list of remarks.
+/// \p StrTab - The string table for the remarks.
+/// \returns Error::success() on success.
+static Error tryReserializeParsedRemarks(
+ StringRef OutputFileName, Format OutputFormat,
+ const std::vector<std::unique_ptr<Remark>> &ParsedRemarks,
+ StringTable &StrTab) {
+ auto MaybeOF = getOutputFile(OutputFileName, OutputFormat);
+ if (!MaybeOF)
+ return MaybeOF.takeError();
+ auto OF = std::move(*MaybeOF);
+ auto MaybeSerializer = createRemarkSerializer(
+ OutputFormat, SerializerMode::Standalone, OF->os(), std::move(StrTab));
+ if (!MaybeSerializer)
+ return MaybeSerializer.takeError();
+ auto Serializer = std::move(*MaybeSerializer);
+ for (const auto &Remark : ParsedRemarks)
+ Serializer->emit(*Remark);
+ OF->keep();
+ return Error::success();
+}
+
+/// Parses remarks in the input format, and reserializes them in the desired
+/// output format.
+/// \returns Error::success() on success, and an Error otherwise.
+static Error tryReserialize(StringRef InputFileName, StringRef OutputFileName,
+ Format InputFormat, Format OutputFormat) {
+ StringTable StrTab;
+ auto MaybeParsedRemarks =
+ tryParseRemarksFromInputFile(InputFileName, InputFormat, StrTab);
+ if (!MaybeParsedRemarks)
+ return MaybeParsedRemarks.takeError();
+ return tryReserializeParsedRemarks(OutputFileName, OutputFormat,
+ *MaybeParsedRemarks, StrTab);
+}
+
+/// Reserialize bitstream remarks as YAML remarks.
+/// \returns An Error if reserialization fails, or Error::success() on success.
+static Error tryBitstream2YAML() {
+ // Use the namespace to get the correct command line globals.
+ using namespace bitstream2yaml;
+ return tryReserialize(InputFileName, OutputFileName, InputFormat,
+ OutputFormat);
+}
+
+/// Reserialize YAML remarks as bitstream remarks.
+/// \returns An Error if reserialization fails, or Error::success() on success.
+static Error tryYAML2Bitstream() {
+ // Use the namespace to get the correct command line globals.
+ using namespace yaml2bitstream;
+ return tryReserialize(InputFileName, OutputFileName, InputFormat,
+ OutputFormat);
+}
+
+/// Handle user-specified suboptions (e.g. yaml2bitstream, bitstream2yaml).
+/// \returns An Error if the specified suboption fails or if no suboption was
+/// specified. Otherwise, Error::success().
+static Error handleSuboptions() {
+ if (subopts::Bitstream2YAML)
+ return tryBitstream2YAML();
+ if (subopts::YAML2Bitstream)
+ return tryYAML2Bitstream();
+ return make_error<StringError>(
+ "Please specify a subcommand. (See -help for options)",
+ inconvertibleErrorCode());
+}
+
+int main(int argc, const char **argv) {
+ InitLLVM X(argc, argv);
+ cl::HideUnrelatedOptions(RemarkUtilCategory);
+ cl::ParseCommandLineOptions(argc, argv, "Remark file utilities\n");
+ ExitOnErr.setBanner(std::string(argv[0]) + ": error: ");
+ ExitOnErr(handleSuboptions());
+}
More information about the llvm-commits
mailing list